Re: Question on Windows event log ingest and parse

[Simon Elliston Ball]

Hi Ed, 

Sounds like a really nice piece of work to get pushed into the core… how would 
you feel about taking that grok parser and formalising it into the core of 
Metron (happy to help there by the way).

On the actual issue, is sounds like it’s likely to be something to do with 
conversion of the timestamp format to the unixtime used in Metron. We can look 
at that. Did you see any log messages in the storm logs from the topology that 
died? 

Simon


> On 3 May 2017, at 22:34, ed d  wrote:
> 
> Metron version – 0.4.0
> Single node install, bare metal install
> No significant changes to base install besides maintenance mode on 
> elasticsearch mpack and manual configuration.
>  
> I have a Windows 2012 server running AD, AD LDS, DNS, and DHCP. I installed 
> Winlogbeat 5.3.2 64 bit 
> onto the server. It was configured to push logs to the Elasticsearch on my 
> Metron install, and it works great. No issues.
>  
> I modified the Winlogbeat configuration to push logs directly to Kafka as I 
> want to enrich the logs. I followed this guide 
> .
>  
> I can see logs coming into the Kafka topic, so I built a Grok parser to slice 
> and dice. It seems to work fine on Grok Constructor 
>  and Grok Debugger 
> , but when I load it into Metron as a 
> parser, it kills the Storm topology. It seems to be sticking on the 
> timestamp, which is ISO_8601  format 
> (2017-05-03T21:04:33Z).
>  
> My question to the group, before troubleshooting my install, is to see if 
> anyone else has had success ingesting and parsing Windows event logs?
>  
> Does anyone pull Windows log into Kafka, Nifi, or other with the intent to 
> enrich the elements of the log? And if yes, what have you found to be most 
> useful?
>  
> FYI here is my Grok parser for reference:
>  
> timestamp"\:"%{TIMESTAMP_ISO8601:timestamp}","beat"\:\{"hostname"\:%{QUOTEDSTRING:hostname},"name"\:%{QUOTEDSTRING:name},"version"\:%{QUOTEDSTRING:beat_version}\},"computer_name"\:%{QUOTEDSTRING:computer_name},"event_data"\:\{("AuthenticationPackageName"\:%{QUOTEDSTRING:AuthenticationPackageName},?)?("ImpersonationLevel"\:%{QUOTEDSTRING:ImpersonationLevel},?)?("FailureReason"\:%{QUOTEDSTRING:FailureReason},?)?("IpAddress"\:"%{IP:ip_src_addr}",?)?("IpPort"\:%{QUOTEDSTRING:IpPort},?)?("KeyLength"\:%{QUOTEDSTRING:KeyLength},?)?("LmPackageName"\:%{QUOTEDSTRING:LmPackageName},?)?("LogonGuid"\:%{QUOTEDSTRING:LogonGuid},?)?("LogonProcessName"\:%{QUOTEDSTRING:LogonProcessName},?)?("LogonType"\:%{QUOTEDSTRING:LogonType},?)?("PrivilegeList"\:%{QUOTEDSTRING:PrivilegeList},?)?("ProcessId"\:%{QUOTEDSTRING:ProcessId},?)?("ProcessName"\:%{QUOTEDSTRING:ProcessName},?)?("PackageName"\:%{QUOTEDSTRING:PackageName},?)?("Status"\:%{QUOTEDSTRING:Status},?)?("SubStatus"\:%{QUOTEDSTRING:SubStatus},?)?("SubjectDomainName"\:%{QUOTEDSTRING:SubjectDomainName},?)?("SubjectLogonId"\:%{QUOTEDSTRING:SubjectLogonId},?)?("SubjectUserName"\:%{QUOTEDSTRING:SubjectUserName},?)?("SubjectUserSid"\:%{QUOTEDSTRING:SubjectUserSid},?)?("TargetDomainName"\:%{QUOTEDSTRING:TargetDomainName},?)?("TargetLogonId"\:%{QUOTEDSTRING:TargetLogonId},?)?("TargetUserName"\:%{QUOTEDSTRING:TargetUserName},?)?("TargetUserSid"\:%{QUOTEDSTRING:TargetUserSid},?)?("TransmittedServices"\:%{QUOTEDSTRING:TransmittedServices},?)?("Workstation"\:%{QUOTEDSTRING:Workstation},?)?("WorkstationName"\:%{QUOTEDSTRING:WorkstationName},?)?\},"event_id"\:%{NUMBER:event_id},"keywords"\:\[%{QUOTEDSTRING:keywords}\],"level"\:%{QUOTEDSTRING:level},"log_name"\:%{QUOTEDSTRING:log_name},"message"\:%{QUOTEDSTRING:message},"opcode"\:%{QUOTEDSTRING:opcode},"process_id"\:%{NUMBER:process_id},"provider_guid"\:%{QUOTEDSTRING:provider_guid},"record_number"\:%{QUOTEDSTRING:record_number},"source_name"\:%{QUOTEDSTRING:source_name},"task"\:%{QUOTEDSTRING:task},"thread_id"\:%{NUMBER:thread_id},"type"\:%{QUOTEDSTRING:type},?("version"\:%{NUMBER:version},?)?\}

Re: Question on Windows event log ingest and parse

[Simon Elliston Ball]

And just to check… you have the pattern definition you previously sent in 
/patterns/winlogbeat (file) on HDFS.

It looks like the most likely problem from your config is that you have two 
parserConfig elements. I suspect the second is over-riding the first, and hence 
you are losing the grokPath config, if you move the dc2tz element into the 
first parserConfig, you should be good.

As an aside from a quick look at your pattern, it looks like it may be easier 
to use the JSONMapParser for this particular sensor. 

Simon

> On 4 May 2017, at 01:28, ed d  wrote:
> 
> Correction, deploying the Storm topology is this:
> 
> /usr/metron/$METRON_VERSION/bin/start_parser_topology.sh -z `hostname 
> -f`:2181 -k `hostname -f`:6667 -s winlogbeat
> 
> 
> 
> 
> 
> From: Simon Elliston Ball 
> Sent: Wednesday, May 3, 2017 5:59 PM
> To: user@metron.apache.org
> Subject: Re: Question on Windows event log ingest and parse
>  
> Hi Ed, 
> 
> Sounds like a really nice piece of work to get pushed into the core… how 
> would you feel about taking that grok parser and formalising it into the core 
> of Metron (happy to help there by the way).
> 
> On the actual issue, is sounds like it’s likely to be something to do with 
> conversion of the timestamp format to the unixtime used in Metron. We can 
> look at that. Did you see any log messages in the storm logs from the 
> topology that died? 
> 
> Simon
> 
> 
>> On 3 May 2017, at 22:34, ed d > <mailto:ragdel...@hotmail.com>> wrote:
>> 
>> Metron version – 0.4.0
>> Single node install, bare metal install
>> No significant changes to base install besides maintenance mode on 
>> elasticsearch mpack and manual configuration.
>>  
>> I have a Windows 2012 server running AD, AD LDS, DNS, and DHCP. I installed 
>> Winlogbeat <https://www.elastic.co/downloads/beats/winlogbeat>5.3.2 64 bit 
>> onto the server. It was configured to push logs to the Elasticsearch on my 
>> Metron install, and it works great. No issues.
>>  
>> I modified the Winlogbeat configuration to push logs directly to Kafka as I 
>> want to enrich the logs. I followed this guide 
>> <https://www.elastic.co/guide/en/beats/winlogbeat/master/kafka-output.html>.
>>  
>> I can see logs coming into the Kafka topic, so I built a Grok parser to 
>> slice and dice. It seems to work fine on Grok Constructor 
>> <http://grokconstructor.appspot.com/do/match> and Grok Debugger 
>> <https://grokdebug.herokuapp.com/>, but when I load it into Metron as a 
>> parser, it kills the Storm topology. It seems to be sticking on the 
>> timestamp, which is ISO_8601 <https://en.wikipedia.org/wiki/ISO_8601> format 
>> (2017-05-03T21:04:33Z).
>>  
>> My question to the group, before troubleshooting my install, is to see if 
>> anyone else has had success ingesting and parsing Windows event logs?
>>  
>> Does anyone pull Windows log into Kafka, Nifi, or other with the intent to 
>> enrich the elements of the log? And if yes, what have you found to be most 
>> useful?
>>  
>> FYI here is my Grok parser for reference:
>>  
>> timestamp"\:"%{TIMESTAMP_ISO8601:timestamp}","beat"\:\{"hostname"\:%{QUOTEDSTRING:hostname},"name"\:%{QUOTEDSTRING:name},"version"\:%{QUOTEDSTRING:beat_version}\},"computer_name"\:%{QUOTEDSTRING:computer_name},"event_data"\:\{("AuthenticationPackageName"\:%{QUOTEDSTRING:AuthenticationPackageName},?)?("ImpersonationLevel"\:%{QUOTEDSTRING:ImpersonationLevel},?)?("FailureReason"\:%{QUOTEDSTRING:FailureReason},?)?("IpAddress"\:"%{IP:ip_src_addr}",?)?("IpPort"\:%{QUOTEDSTRING:IpPort},?)?("KeyLength"\:%{QUOTEDSTRING:KeyLength},?)?("LmPackageName"\:%{QUOTEDSTRING:LmPackageName},?)?("LogonGuid"\:%{QUOTEDSTRING:LogonGuid},?)?("LogonProcessName"\:%{QUOTEDSTRING:LogonProcessName},?)?("LogonType"\:%{QUOTEDSTRING:LogonType},?)?("PrivilegeList"\:%{QUOTEDSTRING:PrivilegeList},?)?("ProcessId"\:%{QUOTEDSTRING:ProcessId},?)?("ProcessName"\:%{QUOTEDSTRING:ProcessName},?)?("PackageName"\:%{QUOTEDSTRING:PackageName},?)?("Status"\:%{QUOTEDSTRING:Status},?)?("SubStatus"\:%{QUOTEDSTRING:SubStatus},?)?("SubjectDomainName"\:%{QUOTEDSTRING:SubjectDomainName},?)?("SubjectLogonId"\:%{QUOTEDSTRING:SubjectLogonId},?)?("SubjectUserName"\:%{QUOTEDSTRING:SubjectUserName},?)?("SubjectUserSid"\:%{QUOTEDSTRING:SubjectUserSid},?)?("TargetDomainName"\:%{QUOTEDSTRING:TargetDomainName},?)?("TargetLogonId"\:%{QUOTEDSTRIN

Re: Build fails - unable to find https://raw.github.com

[Simon Elliston Ball]

Is your network or proxy blocking github access for any reason? I’ve seen some 
corporate environment block github.

Simon

> On 5 May 2017, at 04:10, Kevin Waterson  wrote:
> 
> Yes, seems to fail at the same point each time.
> 
> On Thu, May 4, 2017 at 9:09 AM, Michael Miklavcic 
> mailto:michael.miklav...@gmail.com>> wrote:
> Hi Kevin, Thanks for trying Metron. This looks like an intermittent network 
> failure of some sort. Are you able to repeatably run this and fail at the 
> place each time?
> 
> On May 3, 2017 3:00 PM, "Kevin Waterson"  > wrote:
> Alll seems to go well until I get to raw.github.. output below
> 
> My build 
> 
> sudo apt-get -y install ansible vagrant virtualbox python maven git
> 
> sudo apt-get -y install openjdk-8-jdk
> 
> javac -version
> 
> export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
> export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/lib
> 
> git clone https://github.com/apache/incubator-metron.git 
> 
> 
> cd incubator-metron
> 
> mvn clean package -DskipTests
> 
> sudo dpkg -i vagrant_1.9.4_x86_64.deb
> 
> vagrant plugin install vagrant-hostmanager
> 
> cd metron-deployment/vagrant/quick-dev-platform
> 
> /run.sh
> 
> 
> 
> [INFO] 
> 
> [INFO] Building metron-writer 0.4.0
> [INFO] 
> 
> Downloading: 
> http://clojars.org/repo/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>  
> 
> Downloading: 
> http://repo.hortonworks.com/content/repositories/releases/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>  
> 
> Downloading: 
> https://raw.github.com/benelog/multiline/master/maven-repository/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>  
> 
> Downloading: 
> https://repo.maven.apache.org/maven2/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>  
> 
> May 04, 2017 6:26:09 AM 
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
> execute
> INFO: I/O exception (java.net.SocketException) caught when processing request 
> to {s}->https://repo.maven.apache.org:443 : 
> Connection timed out (Read failed)
> May 04, 2017 6:26:09 AM 
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
> execute
> INFO: Retrying request to {s}->https://repo.maven.apache.org:443 
> 
> Downloaded: 
> https://repo.maven.apache.org/maven2/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>  
> 
>  (10 KB at 0.0 KB/sec)
> Downloading: 
> http://clojars.org/repo/org/yaml/snakeyaml/1.11/snakeyaml-1.11.pom 
> 
> Downloading: 
> http://repo.hortonworks.com/content/repositories/releases/org/yaml/snakeyaml/1.11/snakeyaml-1.11.pom
>  
> 
> Downloading: 
> https://raw.github.com/benelog/multiline/master/maven-repository/org/yaml/snakeyaml/1.11/snakeyaml-1.11.pom
>  
> 
> May 04, 2017 6:42:17 AM 
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
> execute
> INFO: I/O exception (java.net.SocketException) caught when processing request 
> to {s}->
> 
> https://raw.github.com:443 : Connection timed out 
> (Read failed)
> May 04, 2017 6:42:17 AM 
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
> execute
> INFO: Retrying request to {s}->https://raw.github.com:443 
> 
> 
> 

Re: Build fails - unable to find https://raw.github.com

[Simon Elliston Ball]

The 404 should be expected there. Maven will go through a number of repos until 
it find the one it needs. Your issue here is that it’s failing to connect to 
some repos at all. In this instance it seems like you’re having trouble 
connecting to the apache repo server. 

Simon

> On 5 May 2017, at 13:00, Kevin Waterson  wrote:
> 
> Ahh, there is a 404 on this file
> https://raw.github.com/benelog/multiline/master/maven-repository/org/springframework/plugin/spring-plugin/1.2.0.RELEASE/spring-plugin-1.2.0.RELEASE.pom
>  
> <https://raw.github.com/benelog/multiline/master/maven-repository/org/springframework/plugin/spring-plugin/1.2.0.RELEASE/spring-plugin-1.2.0.RELEASE.pom>
> which is breaking the install.
> 
> Kev
> 
> INFO: I/O exception (java.net.SocketException) caught when processing request 
> to {s}->https://repo.maven.apache.org:443 <https://repo.maven.apache.org/>: 
> Connection timed out (Read failed)
> May 05, 2017 4:26:14 PM 
> org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
> execute
> INFO: Retrying request to {s}->https://repo.maven.apache.org:443 
> <https://repo.maven.apache.org/>
> Downloaded: 
> https://repo.maven.apache.org/maven2/org/springframework/plugin/spring-plugin-core/1.2.0.RELEASE/spring-plugin-core-1.2.0.RELEASE.pom
>  
> <https://repo.maven.apache.org/maven2/org/springframework/plugin/spring-plugin-core/1.2.0.RELEASE/spring-plugin-core-1.2.0.RELEASE.pom>
>  (2 KB at 0.0 KB/sec)
> Downloading: 
> https://raw.github.com/benelog/multiline/master/maven-repository/org/springframework/plugin/spring-plugin/1.2.0.RELEASE/spring-plugin-1.2.0.RELEASE.pom
>  
> <https://raw.github.com/benelog/multiline/master/maven-repository/org/springframework/plugin/spring-plugin/1.2.0.RELEASE/spring-plugin-1.2.0.RELEASE.pom>
> 
> 
> On Fri, May 5, 2017 at 8:21 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> Is your network or proxy blocking github access for any reason? I’ve seen 
> some corporate environment block github.
> 
> Simon
> 
> 
>> On 5 May 2017, at 04:10, Kevin Waterson > <mailto:kevin.water...@gmail.com>> wrote:
>> 
>> Yes, seems to fail at the same point each time.
>> 
>> On Thu, May 4, 2017 at 9:09 AM, Michael Miklavcic 
>> mailto:michael.miklav...@gmail.com>> wrote:
>> Hi Kevin, Thanks for trying Metron. This looks like an intermittent network 
>> failure of some sort. Are you able to repeatably run this and fail at the 
>> place each time?
>> 
>> On May 3, 2017 3:00 PM, "Kevin Waterson" > <mailto:kevin.water...@gmail.com>> wrote:
>> Alll seems to go well until I get to raw.github.. output below
>> 
>> My build 
>> 
>> sudo apt-get -y install ansible vagrant virtualbox python maven git
>> 
>> sudo apt-get -y install openjdk-8-jdk
>> 
>> javac -version
>> 
>> export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
>> export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/lib
>> 
>> git clone https://github.com/apache/incubator-metron.git 
>> <https://github.com/apache/incubator-metron.git>
>> 
>> cd incubator-metron
>> 
>> mvn clean package -DskipTests
>> 
>> sudo dpkg -i vagrant_1.9.4_x86_64.deb
>> 
>> vagrant plugin install vagrant-hostmanager
>> 
>> cd metron-deployment/vagrant/quick-dev-platform
>> 
>> /run.sh
>> 
>> 
>> 
>> [INFO] 
>> 
>> [INFO] Building metron-writer 0.4.0
>> [INFO] 
>> 
>> Downloading: 
>> http://clojars.org/repo/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>>  
>> <http://clojars.org/repo/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom>
>> Downloading: 
>> http://repo.hortonworks.com/content/repositories/releases/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>>  
>> <http://repo.hortonworks.com/content/repositories/releases/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom>
>> Downloading: 
>> https://raw.github.com/benelog/multiline/master/maven-repository/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>>  
>> <https://raw.github.com/benelog/multiline/master/maven-repository/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom>
>> Downloading: 
>> https://repo.maven.apache.org/maven2/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom
>>  
>> <https://repo.maven.apache.org/maven2/org/apache/storm/storm-hdfs/1.0.3/storm-hdfs-1.0.3.pom>
>> M

Re: yaf data ingestion and topology

[Simon Elliston Ball]

YAF is still very much supported, and the process is very similar to the 0.2.0 
approach. There are some sensible roles which help you build and setup YAF 
itself, or you can use an existing YAF install. The approach we use is to pipe 
yaf output into yafscii which in turn pushes CSV to kafka, at which point the 
metron parser is pretty straightforward grok parser using the YAF_DELIMITED 
pattern we still provide in the default install. 

Simon


> On 11 May 2017, at 08:46, moshe jarusalem  wrote:
> 
> Hi All,
> I would like to deploy and test  netflow information from a network 
> interface. I have seen some yaf related information for 0.2.0 release but 
> there is no information about yaf on 0.4.0 release.
> 
> Would you please indicate current status of yaf ingestion and processing? How 
> to deploy them if still metron supports it?
> 
> Regards,

Re: AWS cloudformation script for Metron deployment

[Simon Elliston Ball]

This might be a neater solution than the current ansible build, however, it 
might also be worth considering something more cloud neutral, since more and 
more of the project is moving into ambari, blueprints and cloud break might 
also be an option worth looking at.

Simon 

> On 23 May 2017, at 21:49, Satish Abburi  wrote:
> 
>  
> Want to check with community before we invest to create one? If we don’t have 
> one, can we leverage any existing scripts to create this.
>  
> Thanks,
> Satish
>  

Re: Metron in-memory enrichment

[Simon Elliston Ball]

Surely the caching should make this effectively an in memory lookup. Does the 
stellar enrichment function not use the same clientside caching as the Hbase 
bolt?

Simon 

> On 19 Jun 2017, at 06:21, Casey Stella  wrote:
> 
> In order to do that, the easiest thing to do is to create a stellar function 
> to load and do in-memory lookups.
> 
>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian  wrote:
>> Hi all,
>> 
>> We are using Metron HBase enrichment for a few use cases, but we have 
>> noticed the achievable throughput is not very great. I was wondering whether 
>> there is a way to load the external enrichment data in-memory and use it 
>> with normal Stellar enrichments. In our use cases, the number of rows in the 
>> external enrichments that we are dealing with is less than a 100k and it is 
>> a static list, so it is feasible to load them in-memory and use that for the 
>> enrichment. However, I am not sure how that would be achievable from the 
>> Metron capabilities.
>> 
>> Regards,
>> Ali
> 

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

[Simon Elliston Ball]

Looks like you’ve set the profile to purge (expires) every 30 ms, and your 
period is set to 30 minutes, so the data is being expired long before it has a 
change to write. 

Simon


> On 12 Jul 2017, at 06:17, Krishna Dhanekula  
> wrote:
> 
> I have an problem where profiled data is not storing to Hbase instance:
>  
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>  
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>  
> 
>  
> Attached my profiler.properties and profiler.json files, remote.yaml and 
> worker.log files.
>  
> Once I place the following message into kafka topic 'indexing'
>  
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>  
> It's picked by splitterBolt and passed till HBaseBolt. But data is not 
> getting stored to Hbase.
>  
> Attached metron-artifacts.zip
>  
> Attached storm-ui screenshot.storm-ui-screen.png
>  
> Please suggest what am I missing here.
>  
> Thanks in advance.
>  
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>  
>  
> Regards,
> Balakrishna
> 

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

[Simon Elliston Ball]

In your config for the individual profile though, you expire the content every 
30ms (per the zip file)

Simon



> On 12 Jul 2017, at 12:20, Krishna Dhanekula  
> wrote:
> 
> Thanks Simon for replying.
>  
> This is my ‘profiler.properties’ file content. TTL is 30 minutes and for 
> every 30 seconds I am flushing to Hbase.
>  
> profiler.workers=1
> profiler.executors=0
> profiler.input.topic=indexing
> profiler.period.duration=1
> profiler.period.duration.units=MINUTES
> profiler.ttl=30
> profiler.ttl.units=MINUTES
> profiler.hbase.salt.divisor=1000
> profiler.hbase.table=profiler
> profiler.hbase.column.family=P
> profiler.hbase.batch=1
> profiler.hbase.flush.interval.seconds=30
>  
> # Kafka #
>  
> kafka.zk=10.10.110.184:2181
> kafka.broker=10.10.110.184:6667
> kafka.start=WHERE_I_LEFT_OFF
>  
>  
>  
> Regards,
>  
> Balakrishna
>  
>  
> From: Simon Elliston Ball [mailto:si...@simonellistonball.com] 
> Sent: Wednesday, July 12, 2017 3:28 PM
> To: user@metron.apache.org
> Subject: Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase 
> Instance
>  
> Looks like you’ve set the profile to purge (expires) every 30 ms, and your 
> period is set to 30 minutes, so the data is being expired long before it has 
> a change to write. 
>  
> Simon
>  
>  
> On 12 Jul 2017, at 06:17, Krishna Dhanekula  <mailto:krishna.dhanek...@sstech.us>> wrote:
>  
> I have an problem where profiled data is not storing to Hbase instance:
>  
> Installed metron profiler in ‘HDP-2.5.5.0-157’ with the help of this link:
>  
> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>  
> <https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html>
>  
> Attached my profiler.properties and profiler.json files, remote.yaml and 
> worker.log files.
>  
> Once I place the following message into kafka topic 'indexing'
>  
> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>  
> It's picked by splitterBolt and passed till HBaseBolt. But data is not 
> getting stored to Hbase.
>  
> Attached metron-artifacts.zip
>  
> Attached storm-ui screenshot.storm-ui-screen.png
>  
> Please suggest what am I missing here.
>  
> Thanks in advance.
>  
> Attachments:
> storm-ui-screen.png (17.4 kB)
> metron-artifacts.zip (36.6 kB)
>  
>  
> Regards,
> Balakrishna
> 

Re: metron profiler hbase storage issue

[Simon Elliston Ball]

As I answered earlier today, your profile in this zip file is set to expire 
data after 30 milliseconds, so you will not be writing any data.


> On 12 Jul 2017, at 13:40, Dima Kovalyov  wrote:
> 
> Attached artifacts.
> 
> On 07/12/2017 07:39 PM, Dima Kovalyov wrote:
>> Hello Team,
>> 
>> We have installed metron profiler with the help of this link:
>> https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html
>> 
>> Attached my profiler.properties and profiler.json files, remote.yaml and
>> worker.log files in metron-artifacts.zip.
>> 
>> Once I place the following message into kafka topic 'indexing':
>>> [{"ip_src_addr":"10.0.0.1","protocol":"HTTPS","length":"10","bytes_in":"234"},{"ip_src_addr":"10.0.0.2","protocol":"HTTP","length":"20","bytes_in":"390"},{"ip_src_addr":"10.0.0.3","protocol":"DNS","length":"30","bytes_in":"560"}]
>> It's picked by splitterBolt and passed till HBaseBolt. But data is not
>> getting stored to Hbase.
>> Attached storm-ui screenshot.storm-ui-screen.png
>> 
>> Can you please advise if am I missing something here?
>> Thanks in advance.
>> 
>> p.s. also posted it here:
>> https://community.hortonworks.com/questions/110486/metron-profiler-hbasebolt-not-storing-data-to-hbas.html
>> 
>> - Dima
> 
> 

Re: How to change Elasticsearch indexing policy

[Simon Elliston Ball]

You could change the index data format. One word of caution here though; the 
last time I saw this done it caused huge problems with locking on ingest 
against people running queries on the current day’s data and tended to knock 
recent relevant indexes out of disk cache at the OS level. It might look like 
it will help a bit for ingest initially, but with load on the end user side, 
it’s probably going to kill your disks at any reasonable scale. 

Simon

> On 14 Jul 2017, at 10:31, Ali Nazemian  wrote:
> 
> Hi,
> 
> I am investigating different tuning aspects, and I was wondering how I can 
> change the policy of Elasticsearch indexing. Currently, as a default 
> behaviour, events are stored in separate indices hourly. How can I change 
> this behaviour? Is this a hard-coded design or I can change it through 
> configurations?
> 
> Cheers,
> Ali

Re: Debug enrichment topology

[Simon Elliston Ball]

Topology logs appear in the storm worker logs. The easiest way to get to them 
is through the storm UI, you can also use this when necessary to increase the 
log level of various packages, which can help to find obscure processing bugs. 

Simon


> On 31 Jul 2017, at 15:29, bharath phatak  wrote:
> 
> Hi All,
> 
> Which is the location where topology logs are stored? 
> 
> I am having issue in enrichment topology. The json is not being passed on to 
> "indexing" topic from "enrichments" topic.
> 
> 
> Let me know if any configuration needs to be done in this aspect.
> 
> 
> Thanks,
> Bharath

Re: Threat triage rules using stellar geo enrichment

[Simon Elliston Ball]

A much better way of doing this is to run the geo enrichment as part of the 
regular enrichment process and then just use the output field for the rule. 
Your config already does this, so your rule is in effect running the same 
enrichment twice. Just use enrichments.geo.ip_dst_addr.country != ‘US’ for a 
significantly simpler and more performant rule.

Simon


> On 8 Aug 2017, at 14:47, Anand Subramanian  
> wrote:
> 
> Thank you, Casey. That worked!
> 
> Regards,
> Anand
> 
> From: Casey Stella mailto:ceste...@gmail.com>>
> Reply-To: "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Date: Tuesday, August 8, 2017 at 7:12 PM
> To: "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Subject: Re: Threat triage rules using stellar geo enrichment
> 
> I think you want:
> GEO_GET( ip_dst_addr, ['country']) != 'US'
> 
> 
> On Tue, Aug 8, 2017 at 7:29 AM, Anand Subramanian 
> mailto:asubraman...@hortonworks.com>> wrote:
> Hello All,
> 
> I am trying to write a triage rule where I would like to set the alert score 
> based on Geo enrichment output, as follows. 
> 
> $ cat $METRON_HOME/config/zookeeper/enrichments/snort.json
> {
>   "enrichment" : {
> "fieldMap":
>   {
>   "geo": ["ip_dst_addr", "ip_src_addr"],
>   "host": ["host"]
> }
>   },
>   "threatIntel" : {
> "fieldMap":
>   {
>   "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap":
>   {
>   "ip_src_addr" : ["malicious_ip"],
>   "ip_dst_addr" : ["malicious_ip"]
> },
> "triageConfig" : {
>   "riskLevelRules" : [
> {
>   "name" : "Rule 1",
>   "rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24') 
> )",
>   "score" : 10
> },
> {
>   "name" : "Rule 2",
>   "rule" : "not(GEO_GET(ip_dst_addr, '[country]'), 'US')",
>   "score" : 20
> }
>   ],
>   "aggregator" : "MAX"
> }
>   }
> }
> 
> But I am getting the following error when trying to push the configuration 
> into zookeeper:
> 
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
>   "enrichment" : {
> "fieldMap":
>   {
>   "geo": ["ip_dst_addr", "ip_src_addr"],
>   "host": ["host"]
> }
> 
> at 
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$2(ConfigurationType.java:54)
> at 
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.writeSensorEnrichmentConfigToZookeeper(ConfigurationsUtils.java:123)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.uploadConfigsToZookeeper(ConfigurationsUtils.java:265)
> at 
> org.apache.metron.common.configuration.ConfigurationsUtils.uploadConfigsToZookeeper(ConfigurationsUtils.java:226)
> at 
> org.apache.metron.common.cli.ConfigurationManager.push(ConfigurationManager.java:155)
> at 
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:170)
> at 
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> at 
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: N/A
>  at [Source: {
> 
> }
> ; line: 31, column: 7] (through reference chain: 
> org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig["threatIntel"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatIntelConfig["triageConfig"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig["riskLevelRules"])
> at 
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:262)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:537)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:518)
> at 
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> at 
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> at 
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> at 
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> at 
> org.apache.metron.jackson.

Re: Clearing of data to start over

[Simon Elliston Ball]

Multiple Kafka brokers will help a lot. The wizard allows you too add more by 
using the plus symbol next to Kafka on the master selection screen. After the 
fact you can add more with the add service button on the hosts screen in ambari.

When adding brokers, don't forget to also alter your topics to have more 
partitions to make use of those brokers. Out of the box the default is a pretty 
useless 1. You should have at least as many partitions as you have disk 
spindles for kafka.

For pulling data from remote sites into metron I would suggest something like 
apache NiFi, using NiFi site to site to a NiFi collocated with your metron. 
That would then just write to kafka. So you can think of NiFi as being a bit 
like an agent or a forwarder. 

Good luck!

Simon 

Sent from my iPhone

> On 7 Sep 2017, at 04:01, Frank Horsfall  
> wrote:
> 
> I'm on a role with questions.
> 
> I'm curious to see if I can relieve processing pressure by adding a new vm. 
> 
> Would you know how I would go about it?
> 
> Also
> I would like to pull data from sources instead of have the sources push data 
> to my site. Have you come across this scenario?
> 
> F
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Frank Horsfall 
> Date: 2017-09-06 10:51 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
> 
> Also 
> 
> Laurens you recommended to make 3 Kafka brokers but the install wizard would 
> not let me. 
> 
> As a result my node1 is the only broker currently.  Would this cause a 
> bottleneck?
> 
> If so is there a method to install and configures the 2 additional brokers 
> post initial install?
> 
> kindest regards 
> 
> Frank 
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Frank Horsfall 
> Date: 2017-09-06 10:38 PM (GMT-05:00)
> To: user@metron.apache.org
> Subject: Re: Clearing of data to start over
> 
> Thanks Laurens and Nick.
> 
> I want to let the queues run over night to give us some possible insights 
> into heap sizes etc.
> 
> I currently have 3 vms configured each with 8 cores  500 gigs of data 
> capacity  and 30 gigs of memory.
> 
> Elasticsearch has been configured with 10 gigs xmx.
> 
> I've set storm worker childopts at 7 gigs for now so it takes a while to max 
> out and generate heap errors.
> 
> I deleted approx 6 million events and shut off the data generating apps.
> 
> The idea is to see how much will be processed overnight.
> 
> One thing that has me puzzled is why my bro app isn't emitting events. I 
> double checked my config based on what's recommended but nothing is coming 
> through. A mystery. lol
> 
> 
> Also I kept some notes during the whole process and want to share them if you 
> are interested.  let me know
> 
> Frank
> 
> 
> 
> 
> 
> 
> 
> 
> Sent from my Bell Samsung device over Canada's largest network.
> 
> 
>  Original message 
> From: Laurens Vets 
> Date: 2017-09-06 6:17 PM (GMT-05:00)
> To: user@metron.apache.org
> Cc: Frank Horsfall 
> Subject: Re: Clearing of data to start over
> 
> Hi Frank,
> 
> If you all your queues (Kafka/Storm) are empty, the following should work:
> 
> - Deleting your elasticsearch indices: curl -X DELETE 
> 'http://localhost:9200/snort_index_*', curl -X DELETE 
> 'http://localhost:9200/yaf_index_*', etc...
> 
> - Deleting your Hadoop data:
> 
> Become the hdfs user: sudo su - hdfs
> Show what's been indexed in Hadoop: hdfs dfs -ls 
> /apps/metron/indexing/indexed/ 
> Output should show the following probably:
> /apps/metron/indexing/indexed/error
> /apps/metron/indexing/indexed/snort
> /apps/metron/indexing/indexed/yaf
> ...
> 
> You can remove these with:
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/
> 
> Or the individial files with
> 
> hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME
> 
> 
>> On 2017-09-06 13:59, Frank Horsfall wrote:
>> 
>> Hello all,
>> 
>> I have installed a 3 node system using the bare metal Centos 7 guideline.
>> 
>>  
>> 
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>> 
>>  
>> 
>> It has taken me a while to have all components working properly and I left 
>> the yaf,bro,snort apps running so quite a lot of data has been generated.  
>> Currently, I have almost 18 million events identified in Kibana. 16+ million 
>> are yaf based, and 2+ million are snort  …. 190 events are my new squid 
>> telemetry,  J.   It looks like it still has a while to go before it catches 
>> up to current day.   I recently shutdown the apps.
>> 
>>  
>> 
>>  
>> 
>> My questions are:
>> 
>>  
>> 
>> 1.   Is there a way to wipe all my data and indices clean so that I may 
>> now begin with a fresh dataset?
>> 
>> 2.   Is there a way to config

Re: 192.168.138.158 address in yaf index

[Simon Elliston Ball]

That sounds like an address from the standard example.pcap used to demo metron 
capability. In a real deployment you should not run pcap-replay which is what 
inserts this demo data.

Simon 

> On 21 Sep 2017, at 00:29, Frank Horsfall  
> wrote:
> 
> Morning all,
>  
> I have several logs showing an address of 192.168.138.158 as ip_src_addr and 
> 192.168.138.2 as ip_dst_addr.
>  
> My internal network does not have the 192.168.0.0/24 range which leads me to 
> believe that somewhere there is a test record with the data.
>  
> Would anybody know where I might be able to find it?
>  
> Frank
>  

Re: Metron Installation

[Simon Elliston Ball]

This looks like it’s probably a timeout. From your other posts it sounds like 
the machine you’re using is really not up to running the base platform for 
Metron. I would strongly recommend going for something cloud based. 

I would also consider using the mpack method on an existing ambari, and 
avoiding the ansible method, that will be a little less brittle. 

Simon


> On 25 Sep 2017, at 06:49, Syed Hammad Tahir  wrote:
> 
> Any fix for this?
> 
> 

Re: Unable to add the hosts

[Simon Elliston Ball]

The list says it wants one host per line, you have given it comma separated. 

> On 25 Sep 2017, at 09:31, kotipalli venkatesh 
>  wrote:
> 
> 
> Hi All,
> 
> Please help on the below error, Target host, we added nodes and import the 
> id_rsa file on the main node. and click the ok button but confirm host status 
> is failed.
> 
> please give suggestion on the below error. 
> 
> 
> 
> 
> 
> 
> 
> 

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

the _score field is actually an elastic search matching score field, and is not 
relevant to metron. You should see the scores in the threat:triage:score field. 
However, your rules will only be run if the telemetry has is_alert set true, so 
you should ensure that the enrichment phase sets is_alert: true somewhere for 
alerts you want to go to triage? 

Simon

> On 25 Sep 2017, at 18:46, Laurens Vets  wrote:
> 
> I have the following configuration:
> 
> "threatIntel": {
>  "fieldMap": {},
>"fieldToTypeMap": {},
>"config": {},
>"triageConfig": {
>  "riskLevelRules": [
>{
>  "name": "Rule1",
>  "comment": "Checks whatever 1.",
>  "rule": "test == \"false\"",
>  "score": 20,
>  "reason": null
>},
>{
>  "name": "Rule1",
>  "comment": "Checks whatever 2.",
>  "rule": "test2 == \"False\"",
>  "score": 20,
>  "reason": null
>},
>{
>  "name": "Rule3",
>  "comment": "Checks whatever 2.",
>  "rule": "test3 == \"No\"",
>  "score": 20,
>  "reason": null
>}
>  ],
>  "aggregator": "SUM",
>  "aggregationConfig": {}
>}
> },
> 
> I have no additional configuration in enrichment besides filling a specific 
> with true or false based on a Stellar expression.
> 
> I expected that when events would match my above rules, the _score field 
> would be filled in. That does not seem to be the case.
> 
> Does anyone know what I might be missing?

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

Usually you would have the is_alert set based on more complex rules, and then 
potentially have different rules to determine the importance of the alert, so 
they do tend to serve different purposes. 

For example a triage rule might be set on levels of an indicator after is_alert 
has been triggered by a simple presence of a non-zero result for that 
indicator, e.g. is it 2x std_devs, or 4x std_devs as different rule levels. 
We’re adding the ability to make score a stellar statement which simplifies 
this further by allowing score to be a function, but thresholds are still 
useful to determine the text content of the alert for example. 

Simon

> On 25 Sep 2017, at 19:09, Laurens Vets  wrote:
> 
> Oh, I didn't know I had to set is_alert to True.
> 
> Doesn't that mean that we have to add all rules twice? First to check whether 
> is_alert needs to be set to True. Next to apply the actual scores?
> 
> On 2017-09-25 11:00, Simon Elliston Ball wrote:
>> the _score field is actually an elastic search matching score field,
>> and is not relevant to metron. You should see the scores in the
>> threat:triage:score field. However, your rules will only be run if the
>> telemetry has is_alert set true, so you should ensure that the
>> enrichment phase sets is_alert: true somewhere for alerts you want to
>> go to triage?
>> Simon
>>> On 25 Sep 2017, at 18:46, Laurens Vets  wrote:
>>> I have the following configuration:
>>> "threatIntel": {
>>> "fieldMap": {},
>>>   "fieldToTypeMap": {},
>>>   "config": {},
>>>   "triageConfig": {
>>> "riskLevelRules": [
>>>   {
>>> "name": "Rule1",
>>> "comment": "Checks whatever 1.",
>>> "rule": "test == \"false\"",
>>> "score": 20,
>>> "reason": null
>>>   },
>>>   {
>>> "name": "Rule1",
>>> "comment": "Checks whatever 2.",
>>> "rule": "test2 == \"False\"",
>>> "score": 20,
>>> "reason": null
>>>   },
>>>   {
>>> "name": "Rule3",
>>> "comment": "Checks whatever 2.",
>>> "rule": "test3 == \"No\"",
>>> "score": 20,
>>> "reason": null
>>>   }
>>> ],
>>> "aggregator": "SUM",
>>> "aggregationConfig": {}
>>>   }
>>> },
>>> I have no additional configuration in enrichment besides filling a specific 
>>> with true or false based on a Stellar expression.
>>> I expected that when events would match my above rules, the _score field 
>>> would be filled in. That does not seem to be the case.
>>> Does anyone know what I might be missing?

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

The second statement overwrites the first, but also uses the previous value. 

Technically that is an or. Note this construct is designed to allow multiple 
different trigger conditions to make is_alert true, hence the second one being 
is_alert := is_alert || something_else. 

&& is bitwise and
|| is bitwise or

Simon

> On 25 Sep 2017, at 21:12, Laurens Vets  wrote:
> 
> Thanks! Followup question, the below is_alert 'rules' in the snippet from 
> http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html,
>  are those an AND or OR?
> 
>  "threatIntel": {
>"fieldMap": {
>  "stellar" : {
>"config" : [
>  "geo_distance_distr:= STATS_MERGE( PROFILE_GET( 
> 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 2, 'MINUTES')))",
>  "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
>  "dist_sd := STATS_SD(geo_distance_distr)",
>  "geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
>  "is_alert := exists(is_alert) && is_alert",
>  "is_alert := is_alert || (geo_outlier != null && geo_outlier == 
> true)",
>  "geo_distance_distr := null"
>]
>  }
>},
> 
> For instance, can the 2nd is_alert line overwrite the value assigned in the 
> first is_alert rule?
> 
> On 2017-09-25 11:13, Simon Elliston Ball wrote:
>> Usually you would have the is_alert set based on more complex rules,
>> and then potentially have different rules to determine the importance
>> of the alert, so they do tend to serve different purposes.
>> For example a triage rule might be set on levels of an indicator after
>> is_alert has been triggered by a simple presence of a non-zero result
>> for that indicator, e.g. is it 2x std_devs, or 4x std_devs as
>> different rule levels. We’re adding the ability to make score a
>> stellar statement which simplifies this further by allowing score to
>> be a function, but thresholds are still useful to determine the text
>> content of the alert for example.
>> Simon
>>> On 25 Sep 2017, at 19:09, Laurens Vets  wrote:
>>> Oh, I didn't know I had to set is_alert to True.
>>> Doesn't that mean that we have to add all rules twice? First to check 
>>> whether is_alert needs to be set to True. Next to apply the actual scores?
>>> On 2017-09-25 11:00, Simon Elliston Ball wrote:
>>>> the _score field is actually an elastic search matching score field,
>>>> and is not relevant to metron. You should see the scores in the
>>>> threat:triage:score field. However, your rules will only be run if the
>>>> telemetry has is_alert set true, so you should ensure that the
>>>> enrichment phase sets is_alert: true somewhere for alerts you want to
>>>> go to triage?
>>>> Simon
>>>>> On 25 Sep 2017, at 18:46, Laurens Vets  wrote:
>>>>> I have the following configuration:
>>>>> "threatIntel": {
>>>>> "fieldMap": {},
>>>>>  "fieldToTypeMap": {},
>>>>>  "config": {},
>>>>>  "triageConfig": {
>>>>>"riskLevelRules": [
>>>>>  {
>>>>>"name": "Rule1",
>>>>>"comment": "Checks whatever 1.",
>>>>>"rule": "test == \"false\"",
>>>>>"score": 20,
>>>>>"reason": null
>>>>>  },
>>>>>  {
>>>>>"name": "Rule1",
>>>>>"comment": "Checks whatever 2.",
>>>>>"rule": "test2 == \"False\"",
>>>>>"score": 20,
>>>>>"reason": null
>>>>>  },
>>>>>  {
>>>>>"name": "Rule3",
>>>>>"comment": "Checks whatever 2.",
>>>>>"rule": "test3 == \"No\"",
>>>>>"score": 20,
>>>>>"reason": null
>>>>>  }
>>>>>],
>>>>>"aggregator": "SUM",
>>>>>"aggregationConfig": {}
>>>>>  }
>>>>> },
>>>>> I have no additional configuration in enrichment besides filling a 
>>>>> specific with true or false based on a Stellar expression.
>>>>> I expected that when events would match my above rules, the _score field 
>>>>> would be filled in. That does not seem to be the case.
>>>>> Does anyone know what I might be missing?

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

There should be, though you may need to update your templates in ES if you’ve 
got any custom templates there, and make sure you refresh the fields in 
kibana’s index config. 

Simon


> On 26 Sep 2017, at 17:13, Laurens Vets  wrote:
> 
> After setting is_alert to true, this field is now shown in my event in 
> Kibana. I would expect there also to be a field "threat:triage:level" in 
> those same events (if my rules work?)
> 
> On 2017-09-25 16:46, zeo...@gmail.com wrote:
> 
>> I was quickly reading through this on my mobile device so sorry if I'm off 
>> base here, but it may be because threat.triage.level is changed to 
>> threat:triage:level just before indexing due to the inability to use a 
>> period in keys on older versions of ES.  Not sure exactly what you mean by 
>> you don't get a threat.triage.level field.
>> 
>> Jon
>> 
>> 
>> On Mon, Sep 25, 2017, 19:34 Laurens Vets > <mailto:laur...@daemon.be>> wrote:
>> Next problem:
>> 
>> I'm setting the "is_alert" field to true. It shows up in Kibana, but I
>> don't get a threat.triage.level field which means that either my
>> riskLevelRules rules don't trigger or something else goes wrong.
>> 
>> How and where can I look for additional information on why my rules
>> might not be working? (Metron UI accepts my JSON without issues)
>> 
>> On 2017-09-25 13:39, Laurens Vets wrote:
>> > Thanks!
>> >
>> > On 2017-09-25 13:16, Simon Elliston Ball wrote:
>> >> The second statement overwrites the first, but also uses the previous
>> >> value.
>> >>
>> >> Technically that is an or. Note this construct is designed to allow
>> >> multiple different trigger conditions to make is_alert true, hence the
>> >> second one being is_alert := is_alert || something_else.
>> >>
>> >> && is bitwise and
>> >> || is bitwise or
>> >>
>> >> Simon
>> >>
>> >>> On 25 Sep 2017, at 21:12, Laurens Vets > >>> <mailto:laur...@daemon.be>> wrote:
>> >>>
>> >>> Thanks! Followup question, the below is_alert 'rules' in the snippet
>> >>> from
>> >>> http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html
>> >>>  
>> >>> <http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html>,
>> >>> are those an AND or OR?
>> >>>
>> >>>  "threatIntel": {
>> >>>"fieldMap": {
>> >>>  "stellar" : {
>> >>>"config" : [
>> >>>  "geo_distance_distr:= STATS_MERGE( PROFILE_GET(
>> >>> 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 2,
>> >>> 'MINUTES')))",
>> >>>  "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
>> >>>  "dist_sd := STATS_SD(geo_distance_distr)",
>> >>>  "geo_outlier := ABS(dist_median - geo_distance) >=
>> >>> 5*dist_sd",
>> >>>  "is_alert := exists(is_alert) && is_alert",
>> >>>  "is_alert := is_alert || (geo_outlier != null && geo_outlier
>> >>> == true)",
>> >>>  "geo_distance_distr := null"
>> >>>]
>> >>>  }
>> >>>},
>> >>>
>> >>> For instance, can the 2nd is_alert line overwrite the value assigned
>> >>> in the first is_alert rule?
>> >>>
>> >>> On 2017-09-25 11:13, Simon Elliston Ball wrote:
>> >>>> Usually you would have the is_alert set based on more complex rules,
>> >>>> and then potentially have different rules to determine the
>> >>>> importance
>> >>>> of the alert, so they do tend to serve different purposes.
>> >>>> For example a triage rule might be set on levels of an indicator
>> >>>> after
>> >>>> is_alert has been triggered by a simple presence of a non-zero
>> >>>> result
>> >>>> for that indicator, e.g. is it 2x std_devs, or 4x std_devs as
>> >>>> different rule levels. We're adding the ability to make score a
>> >>>> st

Re: Not seeing any Metron alerts.

[Simon Elliston Ball]

Probably, though there are things (unlikely things) you can do to templates 
that would prevent that. 

> On 26 Sep 2017, at 17:25, Laurens Vets  wrote:
> 
> Why would I need to update my ES template? I should see the field (possibly 
> with the wrong type) anyways in the event after I refreshed the fields in 
> Kibana right?
> 
> On 2017-09-26 09:16, Simon Elliston Ball wrote:
> 
>> There should be, though you may need to update your templates in ES if 
>> you've got any custom templates there, and make sure you refresh the fields 
>> in kibana's index config. 
>>  
>> Simon
>>  
>> 
>>> On 26 Sep 2017, at 17:13, Laurens Vets >> <mailto:laur...@daemon.be>> wrote:
>>> 
>>> After setting is_alert to true, this field is now shown in my event in 
>>> Kibana. I would expect there also to be a field "threat:triage:level" in 
>>> those same events (if my rules work?)
>>> 
>>> On 2017-09-25 16:46, zeo...@gmail.com <mailto:zeo...@gmail.com> wrote:
>>> 
>>> I was quickly reading through this on my mobile device so sorry if I'm off 
>>> base here, but it may be because threat.triage.level is changed to 
>>> threat:triage:level just before indexing due to the inability to use a 
>>> period in keys on older versions of ES.  Not sure exactly what you mean by 
>>> you don't get a threat.triage.level field.
>>> 
>>> Jon
>>> 
>>> 
>>> On Mon, Sep 25, 2017, 19:34 Laurens Vets >> <mailto:laur...@daemon.be>> wrote:
>>> Next problem:
>>> 
>>> I'm setting the "is_alert" field to true. It shows up in Kibana, but I
>>> don't get a threat.triage.level field which means that either my
>>> riskLevelRules rules don't trigger or something else goes wrong.
>>> 
>>> How and where can I look for additional information on why my rules
>>> might not be working? (Metron UI accepts my JSON without issues)
>>> 
>>> On 2017-09-25 13:39, Laurens Vets wrote:
>>> > Thanks!
>>> >
>>> > On 2017-09-25 13:16, Simon Elliston Ball wrote:
>>> >> The second statement overwrites the first, but also uses the previous
>>> >> value.
>>> >>
>>> >> Technically that is an or. Note this construct is designed to allow
>>> >> multiple different trigger conditions to make is_alert true, hence the
>>> >> second one being is_alert := is_alert || something_else.
>>> >>
>>> >> && is bitwise and
>>> >> || is bitwise or
>>> >>
>>> >> Simon
>>> >>
>>> >>> On 25 Sep 2017, at 21:12, Laurens Vets >> >>> <mailto:laur...@daemon.be>> wrote:
>>> >>>
>>> >>> Thanks! Followup question, the below is_alert 'rules' in the snippet
>>> >>> from
>>> >>> http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html
>>> >>>  
>>> >>> <http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html>,
>>> >>> are those an AND or OR?
>>> >>>
>>> >>>  "threatIntel": {
>>> >>>    "fieldMap": {
>>> >>>  "stellar" : {
>>> >>>"config" : [
>>> >>>  "geo_distance_distr:= STATS_MERGE( PROFILE_GET(
>>> >>> 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 2,
>>> >>> 'MINUTES')))",
>>> >>>  "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
>>> >>>  "dist_sd := STATS_SD(geo_distance_distr)",
>>> >>>  "geo_outlier := ABS(dist_median - geo_distance) >=
>>> >>> 5*dist_sd",
>>> >>>  "is_alert := exists(is_alert) && is_alert",
>>> >>>  "is_alert := is_alert || (geo_outlier != null && geo_outlier
>>> >>> == true)",
>>> >>>  "geo_distance_distr := null"
>>> >>>]
>>> >>>  }
>>> >>>    },
>>> >>>
>>> >>> For instance, can the 2nd is_alert line overwrite the value assigned
>>> >>>

Re: Metron Installation error

[Simon Elliston Ball]

Just try a yum install of the package manually.

> On 28 Sep 2017, at 11:29, Syed Hammad Tahir  wrote:
> 
> My internet connection seems to be ok but to remove the doubt, is there any 
> way to install the failed package manually? From where do I get the python 
> script it ran before failure. The script which tries to download the packages
> 
>> On Thu, Sep 28, 2017 at 3:23 PM, Simon Elliston Ball 
>>  wrote:
>> It looks like you do not have access to the internet, or at least your 
>> connection is not good enough to download the packages. 
>> 
>> Verify that you're not getting rpms blocked by a corporate proxy (a common 
>> problem) or something of the sort, or use a clean connection.
>> 
>> Simon 
>> 
>>> On 28 Sep 2017, at 11:17, Syed Hammad Tahir  wrote:
>>> 
>>> OK, I reran everything on my machine (destroyed vagrant and then doing 
>>> vagrant up)
>>> Screenshots are in the order:
>>> 
>>> 
>>> 
>>> 
>>> The Ambari page soon after getting the above error:
>>> 
>>> 
>>> Clicked ops on top left
>>> 
>>> 
>>> 
>>> Clicked "install components on host node 1"
>>> 
>>> 
>>> 
>>> 
>>> Clicked node1:
>>> 
>>> 
>>> 
>>> clicked App timeline server Install
>>> 
>>> 
>>> 
>>> here is the pastebin of output of this error log:
>>> 
>>> https://pastebin.com/eFqHTbxQ
>>> 
>>> Please let me know whats the real issue here. WHy cant it install these 
>>> services.
>>> 
>>> 
>>> 
>>>> On Thu, Sep 28, 2017 at 10:25 AM, Syed Hammad Tahir  
>>>> wrote:
>>>> Here is the ambari-agent.log
>>>> 
>>>>> On Thu, Sep 28, 2017 at 10:22 AM, Syed Hammad Tahir 
>>>>>  wrote:
>>>>> Hello All,
>>>>> 
>>>>> This is what I see
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Now going to last error
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> I think the connection got refused because these components never got 
>>>>> installed. What could be the reason?
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Wed, Sep 27, 2017 at 11:26 PM, Dima Kovalyov 
>>>>>>  wrote:
>>>>>> I agree with Jon, the most effective way to start troubleshooting is to 
>>>>>> look at error in Ambari UI. I have shown in red rectangle in the top 
>>>>>> left corner where you should click on the ATT2.png.
>>>>>> 
>>>>>> There will be list of ambari tasks (ambari_background.png), you can see 
>>>>>> the in my case everything went just fine. What do you see in there on 
>>>>>> your side? And if there are errors, can you drill down to the exact 
>>>>>> message of the error?
>>>>>> Thank you.
>>>>>> 
>>>>>> - Dima
>>>>>> 
>>>>>>> On 09/27/2017 05:41 PM, David Lyle wrote:
>>>>>>> Is there any chance your VM can't reach the internet? 
>>>>>>> 
>>>>>>> Each component failed to install, not just Metron. The Ambari Server 
>>>>>>> log clearly states that, but doesn't give the reason. The reasons 
>>>>>>> should be in the Ambari Agent log and and are definitely accessible 
>>>>>>> from the Ambari page if you click on ops directly to the right of your 
>>>>>>> cluster name. It should show failed operations and give you more 
>>>>>>> information. You can capture the command line Ambari used and run that 
>>>>>>> on node1 if you want to try to recreate the failure.
>>>>>>> 
>>>>>>> -D...
>>>>>>> 
>>>>>>> 
>>>>>>>> On Wed, Sep 27, 2017 at 10:25 AM, Syed Hammad Tahir 
>>>>>>>>  wrote:
>>>>>>>> I re ran everything, still getting this error: 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I ahve also attached the ambari server log file
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
> 

Re: Metron Installation error

[Simon Elliston Ball]

There are a lot of packages involved in installation. If you have a failure 
here, you are likely to fail some of the others too. Often you can retry in 
these scenarios, or go to ambari and in hosts view try reinstalling (on the 
install failed services). The ensile method of install is not the method I 
would recommend personally, but it should be retryable. 

I would honestly look at what’s blocking your package downloads.

Simon

> On 28 Sep 2017, at 11:38, Syed Hammad Tahir  wrote:
> 
> Ok, I guess it failed to install package:  hadoop_2_5_3_0_37-yarn
> If I do it succesfully then should I do vagrant provision again or anything 
> else?
> 
> On Thu, Sep 28, 2017 at 3:32 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> Just try a yum install of the package manually.
> 
> On 28 Sep 2017, at 11:29, Syed Hammad Tahir  <mailto:mscs16...@itu.edu.pk>> wrote:
> 
>> My internet connection seems to be ok but to remove the doubt, is there any 
>> way to install the failed package manually? From where do I get the python 
>> script it ran before failure. The script which tries to download the packages
>> 
>> On Thu, Sep 28, 2017 at 3:23 PM, Simon Elliston Ball 
>> mailto:si...@simonellistonball.com>> wrote:
>> It looks like you do not have access to the internet, or at least your 
>> connection is not good enough to download the packages. 
>> 
>> Verify that you're not getting rpms blocked by a corporate proxy (a common 
>> problem) or something of the sort, or use a clean connection.
>> 
>> Simon 
>> 
>> On 28 Sep 2017, at 11:17, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>>> OK, I reran everything on my machine (destroyed vagrant and then doing 
>>> vagrant up)
>>> Screenshots are in the order:
>>> 
>>> 
>>> 
>>> 
>>> The Ambari page soon after getting the above error:
>>> 
>>> 
>>> Clicked ops on top left
>>> 
>>> 
>>> 
>>> Clicked "install components on host node 1"
>>> 
>>> 
>>> 
>>> 
>>> Clicked node1:
>>> 
>>> 
>>> 
>>> clicked App timeline server Install
>>> 
>>> 
>>> 
>>> here is the pastebin of output of this error log:
>>> 
>>> https://pastebin.com/eFqHTbxQ <https://pastebin.com/eFqHTbxQ>
>>> 
>>> Please let me know whats the real issue here. WHy cant it install these 
>>> services.
>>> 
>>> 
>>> 
>>> On Thu, Sep 28, 2017 at 10:25 AM, Syed Hammad Tahir >> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> Here is the ambari-agent.log
>>> 
>>> On Thu, Sep 28, 2017 at 10:22 AM, Syed Hammad Tahir >> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> Hello All,
>>> 
>>> This is what I see
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Now going to last error
>>> 
>>> 
>>> 
>>> 
>>> 
>>> I think the connection got refused because these components never got 
>>> installed. What could be the reason?
>>> 
>>> 
>>> 
>>> On Wed, Sep 27, 2017 at 11:26 PM, Dima Kovalyov >> <mailto:dima.koval...@sstech.us>> wrote:
>>> I agree with Jon, the most effective way to start troubleshooting is to 
>>> look at error in Ambari UI. I have shown in red rectangle in the top left 
>>> corner where you should click on the ATT2.png.
>>> 
>>> There will be list of ambari tasks (ambari_background.png), you can see the 
>>> in my case everything went just fine. What do you see in there on your 
>>> side? And if there are errors, can you drill down to the exact message of 
>>> the error?
>>> Thank you.
>>> 
>>> - Dima
>>> 
>>> On 09/27/2017 05:41 PM, David Lyle wrote:
>>>> Is there any chance your VM can't reach the internet? 
>>>> 
>>>> Each component failed to install, not just Metron. The Ambari Server log 
>>>> clearly states that, but doesn't give the reason. The reasons should be in 
>>>> the Ambari Agent log and and are definitely accessible from the Ambari 
>>>> page if you click on ops directly to the right of your cluster name. It 
>>>> should show failed operations and give you more information. You can 
>>>> capture the command line Ambari used and run that on node1 if you want to 
>>>> try to recreate the failure.
>>>> 
>>>> -D...
>>>> 
>>>> 
>>>> On Wed, Sep 27, 2017 at 10:25 AM, Syed Hammad Tahir >>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>>> I re ran everything, still getting this error: 
>>>> 
>>>> 
>>>> 
>>>> I ahve also attached the ambari server log file
>>>> 
>>> 
>>> 
>>> 
>>> 
>> 
> 

Re: Metron Installation error

[Simon Elliston Ball]

Nothing. The alerts are a red-herring, your problem is that you have not been 
able to install most of the services. 

You can: try provision again, and be patient OR, if you want to control the 
process… click on hosts, find your one node, and when all the Install Failed 
buttons appear, click them and select re-install. 

> On 28 Sep 2017, at 13:28, Syed Hammad Tahir  wrote:
> 
> 
> 
> What do I need to do here?
> 
> 
> On Thu, Sep 28, 2017 at 5:22 PM, Dima Kovalyov  > wrote:
> You've clicked  on the alerts, not the tasks, which are left to the "25 
> alerts" on you screenshot - "0 ops", that's where we wanted to drill-down.
> 
> - Dima
> 
> On 09/28/2017 08:22 AM, Syed Hammad Tahir wrote:
>> Hello All,
>> 
>> This is what I see
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Now going to last error
>> 
>> 
>> 
>> 
>> 
>> I think the connection got refused because these components never got 
>> installed. What could be the reason?
>> 
>> 
>> 
>> On Wed, Sep 27, 2017 at 11:26 PM, Dima Kovalyov > > wrote:
>> I agree with Jon, the most effective way to start troubleshooting is to look 
>> at error in Ambari UI. I have shown in red rectangle in the top left corner 
>> where you should click on the ATT2.png.
>> 
>> There will be list of ambari tasks (ambari_background.png), you can see the 
>> in my case everything went just fine. What do you see in there on your side? 
>> And if there are errors, can you drill down to the exact message of the 
>> error?
>> Thank you.
>> 
>> - Dima
>> 
>> On 09/27/2017 05:41 PM, David Lyle wrote:
>>> Is there any chance your VM can't reach the internet? 
>>> 
>>> Each component failed to install, not just Metron. The Ambari Server log 
>>> clearly states that, but doesn't give the reason. The reasons should be in 
>>> the Ambari Agent log and and are definitely accessible from the Ambari page 
>>> if you click on ops directly to the right of your cluster name. It should 
>>> show failed operations and give you more information. You can capture the 
>>> command line Ambari used and run that on node1 if you want to try to 
>>> recreate the failure.
>>> 
>>> -D...
>>> 
>>> 
>>> On Wed, Sep 27, 2017 at 10:25 AM, Syed Hammad Tahir >> > wrote:
>>> I re ran everything, still getting this error: 
>>> 
>>> 
>>> 
>>> I ahve also attached the ambari server log file
>>> 
>> 
>> 
> 
> 

Re: Metron Alerts UI, no alerts

[Simon Elliston Ball]

Right now, you can't. I believe we should be taking the lost of index prefixes 
we use in the ui from the index config via the rest api, we can pull the names 
from each sensor index config and use that as the prefix in the ui. That way we 
pickup any new index automatically.

Simon 

> On 28 Sep 2017, at 20:04, Otto Fowler  wrote:
> 
> How would you add a new sensor in?  Like squid if you were doing the tutorial?
> 
> 
>> On September 28, 2017 at 14:52:11, RaghuMitra Kandikonda 
>> (raghumitra@gmail.com) wrote:
>> 
>> Alerts UI shows all the records in the indexes for the following 
>> sensors 'websphere', 'snort', 'asa', 'bro', 'yaf'. It does not show 
>> records under .kibana as they are not the alerts generated by the 
>> system. Usually the index names for the sensors would have a sensor 
>> name prefix followed by timestamp Ex: snort_index_2017.09.28.18 
>> 
>> -Raghu 
>> 
>> On Thu, Sep 28, 2017 at 11:08 PM, Laurens Vets  wrote: 
>> > Hello, 
>> > 
>> > I've got the Alerts UI up and running. However, I do not see any alerts. I 
>> > can see events in Kibana with "is_alert" set to "true" and with a score as 
>> > well, but they do not show up in the Alerts UI. 
>> > 
>> > How and where does the Alerts UI get actual alerts? 

Re: Metron Alerts UI, no alerts

[Simon Elliston Ball]

Don't think we have a JIRA yet... I'm just thinking aloud, but if this works as 
a discuss, we can turn it into a JIRA.

Simon 

> On 28 Sep 2017, at 20:39, Otto Fowler  wrote:
> 
> Simon, is there a jira for this?  I would think this was a known limitation 
> to be addressed later.
> 
> 
>> On September 28, 2017 at 15:29:57, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> Right now, you can't. I believe we should be taking the lost of index 
>> prefixes we use in the ui from the index config via the rest api, we can 
>> pull the names from each sensor index config and use that as the prefix in 
>> the ui. That way we pickup any new index automatically.
>> 
>> Simon 
>> 
>> On 28 Sep 2017, at 20:04, Otto Fowler  wrote:
>> 
>>> How would you add a new sensor in?  Like squid if you were doing the 
>>> tutorial?
>>> 
>>> 
>>>> On September 28, 2017 at 14:52:11, RaghuMitra Kandikonda 
>>>> (raghumitra@gmail.com) wrote:
>>>> 
>>>> Alerts UI shows all the records in the indexes for the following
>>>> sensors 'websphere', 'snort', 'asa', 'bro', 'yaf'. It does not show
>>>> records under .kibana as they are not the alerts generated by the
>>>> system. Usually the index names for the sensors would have a sensor
>>>> name prefix followed by timestamp Ex: snort_index_2017.09.28.18
>>>> 
>>>> -Raghu
>>>> 
>>>> On Thu, Sep 28, 2017 at 11:08 PM, Laurens Vets  wrote:
>>>> > Hello,
>>>> >
>>>> > I've got the Alerts UI up and running. However, I do not see any alerts. 
>>>> > I
>>>> > can see events in Kibana with "is_alert" set to "true" and with a score 
>>>> > as
>>>> > well, but they do not show up in the Alerts UI.
>>>> >
>>>> > How and where does the Alerts UI get actual alerts?

Re: Metron Alerts UI, no alerts

[Simon Elliston Ball]

It would, yes (that’s what I do at the moment) but you need to rebuild the 
alerts ui as stands to make that take effect. I expect we’ll get that fixed 
very shortly.

> On 28 Sep 2017, at 22:10, Laurens Vets  wrote:
> 
> I didn't know that only the default sensors are shown. Thanks!
> 
> Which brings me to my next question :) Would editing "export let INDEXES = 
> ['websphere', 'snort', 'asa', 'bro', 'yaf'];" in 
> metron/metron-interface/metron-alerts/src/app/utils/constants.ts be enough to 
> add support for another sensor right now? Or do other files need to be 
> changed as well to support another sensor?
> 
> 
> On 2017-09-28 11:52, RaghuMitra Kandikonda wrote:
>> Alerts UI shows all the records in the indexes for the following
>> sensors 'websphere', 'snort', 'asa', 'bro', 'yaf'. It does not show
>> records under .kibana as they are not the alerts generated by the
>> system. Usually the index names for the sensors would have a sensor
>> name prefix followed by timestamp Ex: snort_index_2017.09.28.18
>> -Raghu
>> On Thu, Sep 28, 2017 at 11:08 PM, Laurens Vets  wrote:
>>> Hello,
>>> I've got the Alerts UI up and running. However, I do not see any alerts. I
>>> can see events in Kibana with "is_alert" set to "true" and with a score as
>>> well, but they do not show up in the Alerts UI.
>>> How and where does the Alerts UI get actual alerts?

Re: Installation

[Simon Elliston Ball]

host actions -> start all components.

> On 2 Oct 2017, at 07:22, Syed Hammad Tahir  wrote:
> 
> Hi,
> 
> I manually installed every component. Now how do I start them all. I need to 
> use metron. WHat do I do at this step?
> 
> 

Re: Metron Installation

[Simon Elliston Ball]

All of them.

> On 3 Oct 2017, at 11:00, Syed Hammad Tahir  wrote:
> 
> WHat services are necessary to run metron?
> 
> 

[DISCUSS] Dropping support for elastic 2.x

[Simon Elliston Ball]

A number of people are currently working on upgrading the ES support in Metron 
to 5.x (including the clients, and the mpack managed install).

Would anyone have any objections to dropping formal support for 2.x as a result 
of this work? In theory the clients should be backward compatible against older 
data stores, so metron could be upgraded without needing an elastic upgrade. 

In practice, we would need to do pretty extensive testing and I wouldn’t want 
us to have to code around long term support on older clients if no-one in the 
community cares enough about the older ES. Do we think there is a case to be 
made for maintaining long term support for older clients? 

Simon

Re: Initial Testing

[Simon Elliston Ball]

Syed, I would strongly suggest you go through the Squid based tutorial to get 
an idea of how enrichment and indexing works. See: 
https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application 


> On 5 Oct 2017, at 09:13, Syed Hammad Tahir  wrote:
> 
> Thanks for the information. Can I get any tutorial or guide on that 
> enrichment and labelling phase in metron?
> 
> On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik  > wrote:
> Yes, after passing your data from enrichment and labelling phase you can 
> further take it do data modelling phase where you can use python kind of 
> language to apply different modelling techniques on your data.
> 
> Cheers,
> Umesh Kaushik
> 9620023458
> 
> Sent from mobile device, kindly ignore the typographical errors.
> 
> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir"  > wrote:
> Hi,
> 
> Lets say I have dumped snort data. Can I apply some machine learning on it in 
> metron?
> 
> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota  > wrote:
> 1 - It us up to you to install and configure snort however you want. Metron 
> simply consumes the Snort telemetry, but is not opinionated about how you 
> setup your sensors. I would recommend starting with the community rule set: 
> https://www.snort.org/faq/what-are-community-rules 
> 
>  
> 2 - Again, this is outside of scope of Metron. You can view this video to get 
> you started: https://www.youtube.com/watch?v=RUmYojxy3Xw 
> 
>  
> 3 - Metron is not a network mapping tool (although support for graph 
> databases is not too far in the future). Today, the best way to generate a 
> network map (graph) is by using kibana. I would refer you to the following 
> article: https://www.elastic.co/products/x-pack/graph 
> 
>  
> 4 - The snort generated data would be indexed in Elasticsearch and/or stored 
> on HDFS, depending on how you configured the system
>  
> Thanks,
> James
> 
> 
> 04.10.2017, 03:23, "Syed Hammad Tahir"  >:
>> Hi all,
>> 
>> Now that I have installed metron (single node installation on ubuntu 
>> machine), I want to do some initial testing on snort data. I have a few 
>> questions regarding this:
>> 
>> 1- In how many configurations can I use snort with metron (for ex packet 
>> capture in sniffing mode etc)?
>> 
>> 2- How can I change the rules in snort
>> 
>> 3- Can I map the network using metron?
>> 
>> 4- Is snort generated data stored somewhere?
>> 
>> KIndly also give me some tutorial to follow for better understanding.
>> Regards.
>> 
>> 
> 
> 
> --- 
> Thank you,
>  
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
> 
> 
> 

Re: Initial Testing

[Simon Elliston Ball]

Try the ambari files view.

> On 5 Oct 2017, at 09:24, Syed Hammad Tahir  wrote:
> 
> THanks again, also how can I access the snort log via hdfs? Is there any web 
> based hdfs portal or will I have to sneak into the vagrant VM file system to 
> access that?
> 
>> On Thu, Oct 5, 2017 at 1:21 PM, Umesh Kaushik  
>> wrote:
>> I am sorry I will not be able to provide you the exact tutorials. However, I 
>> believe you can find something here:
>> https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture
>> 
>> If not exact answer you will the enough idea to do R&D to achieve your goals.
>> 
>>> On 5 October 2017 at 13:43, Syed Hammad Tahir  wrote:
>>> Thanks for the information. Can I get any tutorial or guide on that 
>>> enrichment and labelling phase in metron?
>>> 
 On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik  
 wrote:
 Yes, after passing your data from enrichment and labelling phase you can 
 further take it do data modelling phase where you can use python kind of 
 language to apply different modelling techniques on your data.
 
 Cheers,
 Umesh Kaushik
 9620023458
 
 Sent from mobile device, kindly ignore the typographical errors.
 
> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir"  wrote:
> Hi,
> 
> Lets say I have dumped snort data. Can I apply some machine learning on 
> it in metron?
> 
>> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota  wrote:
>> 1 - It us up to you to install and configure snort however you want. 
>> Metron simply consumes the Snort telemetry, but is not opinionated about 
>> how you setup your sensors. I would recommend starting with the 
>> community rule set: https://www.snort.org/faq/what-are-community-rules
>>  
>> 2 - Again, this is outside of scope of Metron. You can view this video 
>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw
>>  
>> 3 - Metron is not a network mapping tool (although support for graph 
>> databases is not too far in the future). Today, the best way to generate 
>> a network map (graph) is by using kibana. I would refer you to the 
>> following article: https://www.elastic.co/products/x-pack/graph
>>  
>> 4 - The snort generated data would be indexed in Elasticsearch and/or 
>> stored on HDFS, depending on how you configured the system
>>  
>> Thanks,
>> James
>> 
>> 
>> 04.10.2017, 03:23, "Syed Hammad Tahir" :
>>> Hi all,
>>> 
>>> Now that I have installed metron (single node installation on ubuntu 
>>> machine), I want to do some initial testing on snort data. I have a few 
>>> questions regarding this:
>>> 
>>> 1- In how many configurations can I use snort with metron (for ex 
>>> packet capture in sniffing mode etc)?
>>> 
>>> 2- How can I change the rules in snort
>>> 
>>> 3- Can I map the network using metron?
>>> 
>>> 4- Is snort generated data stored somewhere?
>>> 
>>> KIndly also give me some tutorial to follow for better understanding.
>>> Regards.
>>> 
>>> 
>> 
>> 
>> --- 
>> Thank you,
>>  
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
>> 
> 
>>> 
>> 
>> 
>> 
>> -- 
>> Cheers,
>> Umesh Kaushik
>> (Full Stack Developer- Cyber security analyst: Bhujang Innovations)
>> (9620023458)
> 

Re: Enable geo enrichment

[Simon Elliston Ball]

http://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html
 


Shows you how to configure geo enrichment. 

Simon

> On 5 Oct 2017, at 22:33, Laurens Vets  wrote:
> 
> What's the quickest way to enable geo enrichment on a source ip address in 
> 0.4.1-release? Is there a simple document somewhere with instructions?

Re: Enable geo enrichment

[Simon Elliston Ball]

And incase your install didn’t pick up the latest geo database (or you want to 
update it, the bottom of 
http://metron.apache.org/current-book/metron-platform/metron-data-management/index.html
 
<http://metron.apache.org/current-book/metron-platform/metron-data-management/index.html>
 gives you the relevant info. 


> On 5 Oct 2017, at 22:36, Simon Elliston Ball  
> wrote:
> 
> http://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html
>  
> <http://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html>
> 
> Shows you how to configure geo enrichment. 
> 
> Simon
> 
>> On 5 Oct 2017, at 22:33, Laurens Vets > <mailto:laur...@daemon.be>> wrote:
>> 
>> What's the quickest way to enable geo enrichment on a source ip address in 
>> 0.4.1-release? Is there a simple document somewhere with instructions?
> 

Re: Metron Error in Barematel Installation

[Simon Elliston Ball]

This looks like an error in the frontend build. Sometimes this is transient 
(problems downloading npm packages) so a retry may help. However, we really 
should be looking at pinning the dependency versions, as this can also be 
caused by third-party npm packages being updated in the wild and breaking 
backward compatibility. 

Btw: an easier way to avoid rat errors in the build is to just use 
"-Drat.skip=true" instead of "-Dlicense.skip=true 
-Drat.numUnapprovedLicenses=100"

Simon


> On 16 Oct 2017, at 12:20, Farrukh Naveed Anjum  
> wrote:
> 
> Hi,
> 
> I am experience error during build. Mean while following 0.4.1 Installation. 
> Any help will be appericated
> 
> cd metron
> mvn clean package -DskipTests=true -Dlicense.skip=true 
> -Drat.numUnapprovedLicenses=100  -T 2C -P HDP-2.5.0.0,mpack
> 
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>  
> 
> 
> 
> Log output
> 0 info it worked if it ends with ok
> 1 verbose cli [ '/root/metron/metron-interface/metron-config/node/node',
> 1 verbose cli   
> '/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js',
> 1 verbose cli   'run',
> 1 verbose cli   'build' ]
> 2 info using npm@3.8.9
> 3 info using node@v6.2.0
> 4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
> 5 info lifecycle metron-management-ui@0.4.1~prebuild: 
> metron-management-ui@0.4.1
> 6 silly lifecycle metron-management-ui@0.4.1~prebuild: no script for 
> prebuild, continuing
> 7 info lifecycle metron-management-ui@0.4.1~build: metron-management-ui@0.4.1
> 8 verbose lifecycle metron-management-ui@0.4.1~build: unsafe-perm in 
> lifecycle true
> 9 verbose lifecycle metron-management-ui@0.4.1~build: PATH: 
> /root/metron/metron-interface/metron-config/node/node_modules/npm/bin/node-gyp-bin:/root/metron/metron-interface/metron-config/node_modules/.bin:/root/metron/metron-interface/metron-config/node:/root/metron/metron-interface/metron-config/node:/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:
> 10 verbose lifecycle metron-management-ui@0.4.1~build: CWD: 
> /root/metron/metron-interface/metron-config
> 11 silly lifecycle metron-management-ui@0.4.1~build: Args: [ '-c', 
> './node_modules/angular-cli/bin/ng build -prod' ]
> 12 silly lifecycle metron-management-ui@0.4.1~build: Returned: code: 1  
> signal: null
> 13 info lifecycle metron-management-ui@0.4.1~build: Failed to exec build 
> script
> 14 verbose stack Error: metron-management-ui@0.4.1 build: 
> `./node_modules/angular-cli/bin/ng build -prod`
> 14 verbose stack Exit status 1
> 14 verbose stack at EventEmitter. 
> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/lifecycle.js:245:16)
> 14 verbose stack at emitTwo (events.js:106:13)
> 14 verbose stack at EventEmitter.emit (events.js:191:7)
> 14 verbose stack at ChildProcess. 
> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/spawn.js:24:14)
> 14 verbose stack at emitTwo (events.js:106:13)
> 14 verbose stack at ChildProcess.emit (events.js:191:7)
> 14 verbose stack at maybeClose (internal/child_process.js:850:16)
> 14 verbose stack at Process.ChildProcess._handle.onexit 
> (internal/child_process.js:215:5)
> 15 verbose pkgid metron-management-ui@0.4.1
> 16 verbose cwd /root/metron/metron-interface/metron-config
> 17 error Linux 3.10.0-327.el7.x86_64
> 18 error argv "/root/metron/metron-interface/metron-config/node/node" 
> "/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
>  "run" "build"
> 19 error node v6.2.0
> 20 error npm  v3.8.9
> 21 error code ELIFECYCLE
> 22 error metron-management-ui@0.4.1 build: `./node_modules/angular-cli/bin/ng 
> build -prod`
> 22 error Exit status 1
> 23 error Failed at the metron-management-ui@0.4.1 build script 
> './node_modules/angular-cli/bin/ng build -prod'.
> 23 error Make sure you have the latest version of node.js and npm installed.
> 23 error If you do, this is most likely a problem with the 
> metron-management-ui package,
> 23 error not with npm itself.
> 23 error Tell the author that this fails on your system:
> 23 error ./node_modules/angular-cli/bin/ng build -prod
> 23 error You can get information on how to open an issue for this project 
> with:
> 23 error npm bugs metron-management-ui
> 23 error Or if that isn't available, you can get their info via:
> 23 error npm owner ls metron-management-ui
> 23 error There is likely additional logging output above.
> 24 verbose exit [ 1, true ]
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Metron Error in Barematel Installation

[Simon Elliston Ball]

If you can run again with -X and post the debug output somewhere, we should be 
able to figure out where the dependency that’s failing is. 

Simon

> On 16 Oct 2017, at 12:30, Farrukh Naveed Anjum  
> wrote:
> 
> Used that too but, It seems like as you said some dependency got updated... 
> and its is breaking it.
> 
> On Mon, Oct 16, 2017 at 4:25 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> This looks like an error in the frontend build. Sometimes this is transient 
> (problems downloading npm packages) so a retry may help. However, we really 
> should be looking at pinning the dependency versions, as this can also be 
> caused by third-party npm packages being updated in the wild and breaking 
> backward compatibility. 
> 
> Btw: an easier way to avoid rat errors in the build is to just use 
> "-Drat.skip=true" instead of "-Dlicense.skip=true 
> -Drat.numUnapprovedLicenses=100"
> 
> Simon
> 
> 
>> On 16 Oct 2017, at 12:20, Farrukh Naveed Anjum > <mailto:anjum.farr...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I am experience error during build. Mean while following 0.4.1 Installation. 
>> Any help will be appericated
>> 
>> cd metron
>> mvn clean package -DskipTests=true -Dlicense.skip=true 
>> -Drat.numUnapprovedLicenses=100  -T 2C -P HDP-2.5.0.0,mpack
>> 
>> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST
>>  
>> <https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST>
>> 
>> 
>> Log output
>> 0 info it worked if it ends with ok
>> 1 verbose cli [ '/root/metron/metron-interface/metron-config/node/node',
>> 1 verbose cli   
>> '/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js',
>> 1 verbose cli   'run',
>> 1 verbose cli   'build' ]
>> 2 info using npm@3.8.9
>> 3 info using node@v6.2.0
>> 4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
>> 5 info lifecycle metron-management-ui@0.4.1~prebuild: 
>> metron-management-ui@0.4.1
>> 6 silly lifecycle metron-management-ui@0.4.1~prebuild: no script for 
>> prebuild, continuing
>> 7 info lifecycle metron-management-ui@0.4.1~build: metron-management-ui@0.4.1
>> 8 verbose lifecycle metron-management-ui@0.4.1~build: unsafe-perm in 
>> lifecycle true
>> 9 verbose lifecycle metron-management-ui@0.4.1~build: PATH: 
>> /root/metron/metron-interface/metron-config/node/node_modules/npm/bin/node-gyp-bin:/root/metron/metron-interface/metron-config/node_modules/.bin:/root/metron/metron-interface/metron-config/node:/root/metron/metron-interface/metron-config/node:/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:
>> 10 verbose lifecycle metron-management-ui@0.4.1~build: CWD: 
>> /root/metron/metron-interface/metron-config
>> 11 silly lifecycle metron-management-ui@0.4.1~build: Args: [ '-c', 
>> './node_modules/angular-cli/bin/ng build -prod' ]
>> 12 silly lifecycle metron-management-ui@0.4.1~build: Returned: code: 1  
>> signal: null
>> 13 info lifecycle metron-management-ui@0.4.1~build: Failed to exec build 
>> script
>> 14 verbose stack Error: metron-management-ui@0.4.1 build: 
>> `./node_modules/angular-cli/bin/ng build -prod`
>> 14 verbose stack Exit status 1
>> 14 verbose stack at EventEmitter. 
>> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/lifecycle.js:245:16)
>> 14 verbose stack at emitTwo (events.js:106:13)
>> 14 verbose stack at EventEmitter.emit (events.js:191:7)
>> 14 verbose stack at ChildProcess. 
>> (/root/metron/metron-interface/metron-config/node/node_modules/npm/lib/utils/spawn.js:24:14)
>> 14 verbose stack at emitTwo (events.js:106:13)
>> 14 verbose stack at ChildProcess.emit (events.js:191:7)
>> 14 verbose stack at maybeClose (internal/child_process.js:850:16)
>> 14 verbose stack at Process.ChildProcess._handle.onexit 
>> (internal/child_process.js:215:5)
>> 15 verbose pkgid metron-management-ui@0.4.1
>> 16 verbose cwd /root/metron/metron-interface/metron-config
>> 17 error Linux 3.10.0-327.el7.x86_64
>> 18 error argv "/root/metron/metron-interface/metron-config/node/node" 
>> "/root/metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
>>  "run" "build"
>> 19 error node v6.2.0
>> 20 erro

Re: event correlation on metron

[Simon Elliston Ball]

What you want to do in this setting is just TailFile, the just push to Kafka. 
The grok piece is more efficiently handled in the Metron grok parser.

Push to a kafka topic named for your sensor, then setup a sensor (a parser 
topology to do the grok parsing and any transformation you need). Each sensor 
gets its own parser topology.

Simon 

> On 17 Oct 2017, at 19:00, Youzha  wrote:
> 
> after nifi procces :
> 
> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
> 
> what metron topology that i can use to procces the data in kafka? so it can 
> be enrichment by metron. i’ve check the article about adding new telemetry 
> source with squid, there is a squid topology that will ingest from the squid 
> topic in kafka and then put on enrichment kafka topic. 
> so how about my use case above? is there any topology that i can use?
> 
>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>> So, 
>> There are several options parsing the data and enriching.
>> 
>> 1.  A native parser ( java ), which you have noticed is not there
>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>> 3.  If it is CSV an instance of the CSV parser
>> 4.  If it is JSON an instance of the JSONMap parser
>> 
>> If these cannot be applied to your file then your options are:
>> 
>> 1.  Write or open a jira for a native parser
>> 2. find a way to transform your data to one of the above formats, so you can 
>> use those parsers.  This again is where nifi can help.  Something like:
>> 
>> 
>> [nifi]
>> 
>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>> 
>> where TRANSFORM_TO_JSON is a script processor or something built in 
>> depending on your format.
>> 
>> 
>> 
>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>> 
>>> Hi Lauren thx for your reply,
>>> 
>>> yeah your suggestion absolutely right. i was able to ingest the logs to 
>>> kafka. but how metron can enrich and index all of it? i think there are 
>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron for 
>>> parsers. so, how metron can read the logs telemetry and proccess it so i 
>>> can use it to event correlation
>>> 
 On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
 Hi Youzha,
 
 Either check how the snort logs on the full dev installation are ingested 
 (I believe it's with a script) or check the Apache NiFi project which 
 makes it very easy to read logs from almost any format and ingest them to 
 Metron via Kafka.
 
> On 2017-10-17 08:53, Youzha wrote:
> 
> is it possible to ingest other logs like /var/log/secure for example to 
> be new telemetry on metron? i've seen the metron architecture on the 
> website like picture below. host logs, email, av, etc can be telemetry 
> event buffer on metron. if this possible, could you give me some 
> suggestion how to do it ?
>  
> 
>> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>> If you want to look at failed login attempts for each user over time, 
>> then the Profiler might be a good solution.  Your profile will depend on 
>> the fields available in your telemetry, but it would look something like 
>> this, as an example.
>>  
>> {
>>   "profile": "failed-logins",
>>   "foreach": "user.name",
>>   "onlyif": "source.type == 'activedirectory' and event.type == 
>> 'failed_login'"
>>   "init": { "count": 0 },
>>   "update": { "count" : "count + 1" },
>>   "result": "count"
>> }
>>  
>> You can find an introduction and more information on using the Profiler 
>> below.
>> * 
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler
>> * https://www.slideshare.net/secret/GFBf2RTXBG35PB
>>  
>> Best of luck
>> 
>>> On Tue, Oct 17, 2017 at 4:51 AM, tkg_cangkul  
>>> wrote:
>>> for example,
>>> 
>>> i wanna try to correlate between logs.
>>> how many times user A have login failed and how many times user A have 
>>> login succeed. include detail IP, timestamp etc.
>>> is this possible to do with metron?
>>> 
>>> 
>>> 
>>> 
 On 17/10/17 02:56, James Sirota wrote:
 What specifically are you looking to correlate?  Can you talk a little 
 more about your use case?
 
 16.10.2017, 02:23, "tkg_cangkul" :
> hi,
> 
> anyone could explain me about event correlation using apache metron?
> does metron support event correlation?
> 
> Pls Advice
 ---
 Thank you,
 
 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org
 

Re: event correlation on metron

[Simon Elliston Ball]

Best bet there is to create a new sensor config using the grok parser type. So 
you would for example have a kafka topic called host_dhcp and a sensor called 
host_dhcp with the relevant grok pattern. 

Simon 

> On 17 Oct 2017, at 19:19, Youzha  wrote:
> 
> that’s what i mean.
> what sensor that i need if i want to do this case? 
> especially when i wanna parse some host logs into metron enrichment and 
> indexing
> 
>> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball 
>>  wrote:
>> What you want to do in this setting is just TailFile, the just push to 
>> Kafka. The grok piece is more efficiently handled in the Metron grok parser.
>> 
>> Push to a kafka topic named for your sensor, then setup a sensor (a parser 
>> topology to do the grok parsing and any transformation you need). Each 
>> sensor gets its own parser topology.
>> 
>> Simon 
>> 
>> 
>>> On 17 Oct 2017, at 19:00, Youzha  wrote:
>>> 
>>> after nifi procces :
>>> 
>>> TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
>>> 
>>> what metron topology that i can use to procces the data in kafka? so it can 
>>> be enrichment by metron. i’ve check the article about adding new telemetry 
>>> source with squid, there is a squid topology that will ingest from the 
>>> squid topic in kafka and then put on enrichment kafka topic. 
>>> so how about my use case above? is there any topology that i can use?
>>> 
>>>> On Wed, 18 Oct 2017 at 00.30 Otto Fowler  wrote:
>>>> So, 
>>>> There are several options parsing the data and enriching.
>>>> 
>>>> 1.  A native parser ( java ), which you have noticed is not there
>>>> 2.  An instance of the GROK parser, with GROK rules that parser the input
>>>> 3.  If it is CSV an instance of the CSV parser
>>>> 4.  If it is JSON an instance of the JSONMap parser
>>>> 
>>>> If these cannot be applied to your file then your options are:
>>>> 
>>>> 1.  Write or open a jira for a native parser
>>>> 2. find a way to transform your data to one of the above formats, so you 
>>>> can use those parsers.  This again is where nifi can help.  Something like:
>>>> 
>>>> 
>>>> [nifi]
>>>> 
>>>> TAILFILE -> TRANSFORM_TO_JSON -> PUSH_KAFKA
>>>> 
>>>> where TRANSFORM_TO_JSON is a script processor or something built in 
>>>> depending on your format.
>>>> 
>>>> 
>>>> 
>>>>> On October 17, 2017 at 13:16:05, Youzha (yuza.ras...@gmail.com) wrote:
>>>>> 
>>>>> Hi Lauren thx for your reply,
>>>>> 
>>>>> yeah your suggestion absolutely right. i was able to ingest the logs to 
>>>>> kafka. but how metron can enrich and index all of it? i think there are 
>>>>> only  bro, snort, yaf, snort, pcap, websphere topology storm on metron 
>>>>> for parsers. so, how metron can read the logs telemetry and proccess it 
>>>>> so i can use it to event correlation
>>>>> 
>>>>>> On Tue, 17 Oct 2017 at 23.11 Laurens Vets  wrote:
>>>>>> Hi Youzha,
>>>>>> 
>>>>>> Either check how the snort logs on the full dev installation are 
>>>>>> ingested (I believe it's with a script) or check the Apache NiFi project 
>>>>>> which makes it very easy to read logs from almost any format and ingest 
>>>>>> them to Metron via Kafka.
>>>>>> 
>>>>>>> On 2017-10-17 08:53, Youzha wrote:
>>>>>>> 
>>>>>>> is it possible to ingest other logs like /var/log/secure for example to 
>>>>>>> be new telemetry on metron? i've seen the metron architecture on the 
>>>>>>> website like picture below. host logs, email, av, etc can be telemetry 
>>>>>>> event buffer on metron. if this possible, could you give me some 
>>>>>>> suggestion how to do it ?
>>>>>>>  
>>>>>>> 
>>>>>>>> On Tue, 17 Oct 2017 at 21.00 Nick Allen  wrote:
>>>>>>>> If you want to look at failed login attempts for each user over time, 
>>>>>>>> then the Profiler might be a good solution.  Your profile will depend 
>>>>>>>> on the fields available in your telemetry, but it would look something 
>>>>>>>> like this, as an example.
>

Re: Sizing of components proportional to EPS

[Simon Elliston Ball]

To an extent it very much depends on the use case. I have seen over a million 
EPS on a six node cluster for pcap and basic net flow. If you add a lot of 
complex enrichment and profiling that will obviously increase the load. Tuning 
the components for the workload can also make a significant difference. There 
are some good tips on that in the tuning guide in the source.

It would be great to hear some of the experiences other people on the list have 
had on eps and infrastructure for deployments. If anyone can post specs of a 
deployments, that would be fantastic to see.

Simon 


> On 17 Oct 2017, at 20:14, ed d  wrote:
> 
> Is there a rough guide to match EPS to an architectural sizing guide? I know 
> its very difficult to extrapolate out, but a rough estimate would be nice. 
> This may have already been attempted, and if yes, then please disregard.
> 
> 
> Or can anyone share what they have found to work best?
> 
> 
> For example, 
> 
> POC - 1 machine
> 1 big machine (16 CPU, 128 RAM, 5 Tb HDD)
> 
> 100 EPS - 3 machines
> 1 Nifi (8 CPU, 64 RAM, 5 Tb HDD)
> 1 Hadoop/Metron (8 CPU, 64 RAM, 5 Tb HDD)
> 1 Elasticsearch/Kibana (8 CPU, 64 RAM, 5 Tb HDD)
> 
> 1000 EPS - 8 machines
> 2 Nifi cliustered (8 CPU, 64 RAM, 5 Tb HDD)
> 2 Hadoop (16 CPU, 128 RAM, 20 Tb HDD)
> 1 Metron (16 CPU, 128 RAM, 1 Tb HDD)
> 1 Elasticsearch data (8 CPU, 64 RAM, 20 Tb HDD)
> 1 Elasticsearch master (8 CPU, 64 RAM, 1 Tb HDD)
> 1 Kibana (8 CPU, 64 RAM, 1 Tb HDD)
> 
> 1 EPS - 14 machines
> 4 Nifi clustered (16 CPU, 64 RAM, 5 Tb HDD)
> 2 Hadoop (32 CPU, 128 RAM, 5 Tb HDD)
> 2 Hadoop Data Nodes (32 CPU, 128 RAM, 40 Tb HDD)
> 1 Metron (16 CPU, 128 RAM, 5 Tb HDD)
> 1 Zeppelin (32 CPU, 128 RAM, 5 Tb HDD)
> 2 ES data (32 CPU, 64 RAM, 40 Tb HDD)
> 1 ES master (32 CPU, 64 RAM, 1 Tb HDD)
> 1 Kibana (16 CPU, 64 RAM, 1 Tb HDD)
> 
> 
> 

Re: Snort

[Simon Elliston Ball]

I would recommend just using a text editor if you’re not familiar with sed. To 
solve your sed problem…

sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile 

sed -i  means run the sed command (in this case a find replace) inplace on the 
file, the text following the -i is the name to append to a backup version (ie 
the original file unchanged).

Metron does tend to assume a good knowledge of linux admin, you’ll find we have 
a lot of shell gurus in the community, but if you’re struggling with this, 
maybe a simple text editor would be easier. All you’re trying to do here is 
change a config value. 

Simon

> On 19 Oct 2017, at 11:46, Syed Hammad Tahir  wrote:
> 
> Ran it without -i swtich, gives this:
> 
> 
> 
> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com  
> mailto:zeo...@gmail.com>> wrote:
> The sed command is falling.  It's written for a Mac so it will need an 
> alteration to be portable.  Run it without the '' after -i, from 
> ~/metron-master
> 
> Jon
> 
> 
> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir  > wrote:
> I did what this guide said to install the original sensor:
> https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs
>  
> 
> 
> Still didnt work. How do I install snort into this?
> 
> 
> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir  > wrote:
> Maybe I did something wrong
> 
> 
> 
> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir  > wrote:
> Ok, thankyou. It will install all the sensors (bro, snort etc) ?
> 
> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com  > wrote:
> When you set up full dev if you remove the sensors skip tag it will set up 
> snort for you.  I have a sed one liner in my bro security patch pr to do 
> this, just need to do it before vagrant up.
> 
> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
> metron-deployment/vagrant/full-dev-platform/Vagrantfile
> cd metron-deployment/vagrant/full-dev-platform/
> vagrant up
> 
> Jon
> 
> 
> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir  > wrote:
> I followed this guide exactly:  
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 
> 
> 
> And the did vagrant up in full-development-platform folder. And Snort is not 
> installed because when I type snort -v in vagrant ssh, it returns error of 
> not able to find the snort command.
> 
> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets  > wrote:
> Hi Syed,
> 
> I was under the impression that you installed the full-dev environment? If 
> so, snort should already be installed...
> 
> On 2017-10-18 09:45, Syed Hammad Tahir wrote:
> 
>> It has become a mess. Apparently snort is released for centos 7 whereas 
>> metron one is centos 6.8. Whenever I try to install snort it gives me this:
>>  
>> 
>> 
>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen > > wrote:
>> Just use those as a guide to run the commands  yourself.
>> 
>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir > > wrote:
>> please help me install the snort in metron. I tried doing it the normal way 
>> but i cant install the libraries
>> 
>> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir > > wrote:
>> ok, This is the snort.yml file
>>  
>> 
>>  
>> Do I need to run these commands myself or how do I put these yml files into 
>> play?
>> 
>> On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir > > wrote:
>> I am so noob in all of this. I am using full-dev vm metron install to do my 
>> research. So I have 2 options to install snort: as per my understanding
>>  
>> 1- Install it in a usual way (like that on a regular linux machine) and then 
>> make its kafka topic
>>  
>> 2- Use ansible role to do all of that. Read the content of those yml files 
>> given in main.yml to understand the procedure?
>>  
>> Which one do you suggest?
>>  
>>  
>> 
>> On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen > >wrote:
>> No special commands.  Install and configure Snort however you like and get 
>> those logs into a Kafka topic.  Metron is completely agnostic to how sensor 
>> telemetry lands in Kafka.
>>  
>> We also have an Ansible role that will install Snort along with a simple 
>> mechanism to transport its logs to Kafka.  This is only useful for 
>> development environments; not a production install.  
>>  
>> Using the Ansible role directly may be beyond the knowledge level of some.  
>> I only offer this as a guide that you can use to follow along and manually 
>> install it yourself. 
>>  
>> https:/

Re: multiple pattern grok parser in 1 file

[Simon Elliston Ball]

That is not valid grok. Pattern names should be unique in the grok. 

What you probably mean is something like:

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}
AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})

Simon

> On 23 Oct 2017, at 08:53, tkg_cangkul  wrote:
> 
> FYI,
> 
> i've trying to using Grok parser metron with multiple pattern in single file 
> but it doesn't work. this is my sample grok pattern on 
> /apps/metron/patterns/authlog :
> 
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
> %{USERNAME:username}
> 
> When the sensor started, the second grok pattern doesn't work. Only first 
> pattern works.
> There is an error message like this on storm logs:
> 
> Caused by: java.lang.RuntimeException: Grok statement produced a null message.
> 
> 
> On 23/10/17 10:49, tkg_cangkul wrote:
>> Hi Wasim, 
>> 
>> thx for your reply.
>> So it means i should use logstash parser for metron?
>> Is there any documentation about use logstash parser for metron?
>> I didn't found any documentation about that on metron. 
>> i just find logstash basic parser but there is no documentation about that.
>> 
>> 
>> 
>> On 23/10/17 10:33, Wasim Halani wrote:
>>> Hi Youzha,
>>> 
>>> It should be possible to add multiple patterns in a single config file. For 
>>> reference, you can check out the use of multiple patterns in a repo I 
>>> maintain [1].
>>> You would find the patterns in [2] useful for your use-case.
>>> 
>>> However, do note that there is a cost to every grok failure [3] - so you 
>>> need to ensure that your most common event patterns are at the top of the 
>>> list.
>>> 
>>> As a side-note, if you have any logstash parsers which are not available in 
>>> the repo, please feel to submit a PR to [4] 
>>> 
>>> 
>>> [1] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
>>> [2] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
>>> [3] https://www.elastic.co/blog/do-you-grok-grok
>>> [4] https://bitbucket.org/networkintelligence/logstash-configs/
>>> 
>>> Regards,
>>> ---
>>> Wasim Halani
>>> http://twitter.com/washalsec
>>> http://securitythoughts.wordpress.com
>>> --
>>> To keep silent when you can say something wise and useful is as bad as 
>>> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
>>> 
>>> On Mon, Oct 23, 2017 at 8:08 AM, Youzha  wrote:
>>> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
>>> file?
>>> i’m trying to parsing authlog file in /var/log/secure into metron. the 
>>> problem is there are different structures of logs inside /var/log/secure. 
>>> any suggest for this pls?
>>> 
>>> 
>>> Best Regards,
>>> 
>>> 
>> 
> 

Re: multiple pattern grok parser in 1 file

[Simon Elliston Ball]

My bad, the pattern surpasses names of capture groups.

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}

AUTHLOG (%{AUTHLOG1}|%{AUTHLOG2})

should work… though to be honest, your patterns look a little unusual. You seem 
to have logs with a timestamp in epoch at the front, which is a very weird way 
to setup syslog, so the issue might be that your patterns flat out don’t match 
the logs. 

Simon


> On 23 Oct 2017, at 10:36, tkg_cangkul  wrote:
> 
> Hi Simon,
> 
> I've tried your suggestion but i have an error msg like below :
> 
> 
> 
> On 23/10/17 16:22, Simon Elliston Ball wrote:
>> That is not valid grok. Pattern names should be unique in the grok. 
>> 
>> What you probably mean is something like:
>> 
>> AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>> AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>> %{USERNAME:username}
>> AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})
>> 
>> Simon
>> 
>> 
>>> On 23 Oct 2017, at 08:53, tkg_cangkul 
>>>  wrote:
>>> 
>>> FYI,
>>> 
>>> i've trying to using Grok parser metron with multiple pattern in single 
>>> file but it doesn't work. this is my sample grok pattern on 
>>> /apps/metron/patterns/authlog :
>>> 
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>>> %{USERNAME:username}
>>> 
>>> When the sensor started, the second grok pattern doesn't work. Only first 
>>> pattern works.
>>> There is an error message like this on storm logs:
>>> 
>>> Caused by: java.lang.RuntimeException: Grok statement produced a null 
>>> message.
>>> 
>>> 
>>> On 23/10/17 10:49, tkg_cangkul wrote:
>>> 
>>>> Hi Wasim, 
>>>> 
>>>> thx for your reply.
>>>> So it means i should use logstash parser for metron?
>>>> Is there any documentation about use logstash parser for metron?
>>>> I didn't found any documentation about that on metron. 
>>>> i just find logstash basic parser but there is no documentation about that.
>>>> 
>>>> 
>>>> 
>>>> On 23/10/17 10:33, Wasim Halani wrote:
>>>> 
>>>>> Hi Youzha,
>>>>> 
>>>>> It should be possible to add multiple patterns in a single config file. 
>>>>> For reference, you can check out the use of multiple patterns in a repo I 
>>>>> maintain [1].
>>>>> You would find the patterns in [2] useful for your use-case.
>>>>> 
>>>>> However, do note that there is a cost to every grok failure [3] - so you 
>>>>> need to ensure that your most common event patterns are at the top of the 
>>>>> list.
>>>>> 
>>>>> As a side-note, if you have any logstash parsers which are not available 
>>>>> in the repo, please feel to submit a PR to [4] 
>>>>> 
>>>>> 
>>>>> [1] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
>>>>> 
>>>>> [2] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
>>>>> 
>>>>> [3] 
>>>>> https://www.elastic.co/blog/do-you-grok-grok
>>>>> 
>>>>> [4] 
>>>>> https://bitbucket.org/networkintelligence/logstash-configs/
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> ---
>>>>> Wasim Halani
>>>>> 
>>>>> http://twitter.com/washalsec
>>>>> http://securitythoughts.wordpress.com
>>>>> 
>>>>> --
>>>>> To keep silent when you can say something wise and useful is as bad as 
>>>>> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
>>>>> 
>>>>> On Mon, Oct 23, 2017 at 8:08 AM, Youzha 
>>>>> 
>>>>>  wrote:
>>>>> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
>>>>> file?
>>>>> i’m trying to parsing authlog file in /var/log/secure into metron. the 
>>>>> problem is there are different structures of logs inside /var/log/secure. 
>>>>> any suggest for this pls?
>>>>> 
>>>>> 
>>>>> Best Regards,
>>>>> 
>>>>> 
>>>>> 
> 

Re: ask about profiler rule

[Simon Elliston Ball]

The profiler reads direct from the ingest stream, so sees data before it gets 
to ES. 

The onlyif config you are asking about is a filter condition, so only data 
which matches that expression will be considered by this particular profile. 

The activedirectory example here assumes that you have a sensor setup from 
something like active directory, that has fields called user.name and 
event.type in. It will then count those failures per user.name. 

Simon

> On 24 Oct 2017, at 07:38, tkg_cangkul  wrote:
> 
> Hi,
> 
> anybody can explained to me this rule of profiler config please ?
> 
> {
>   "profile": "failed-logins",
>   "foreach": "user.name ",
>   "onlyif": "source.type == 'activedirectory' and event.type == 
> 'failed_login'"
>   "init": { "count": 0 },
>   "update": { "count" : "count + 1" },
>   "result": "count"
> }
>  
> 
> what is "source.type == 'activedirectory' and event.type == 'failed_login'" 
> means?
> does it means the profiler will read from ES index that have condition 
> source.type == 'activedirectory' . if yes, so i must index to ES first where 
> source type = activedirectory ? 
> 
> I've just read on Nick article here :
> 
> https://www.slideshare.net/NickAllen4/apache-metron-profiler 
> 
> 
> In the other rules config there are  "source.type == 'yaf'" , "source.type == 
> 'bro'". What i know that "source.type == 'yaf'" & "source.type == 'bro'" have 
> indexed by default on metron. how about activedirectory? 
> 
> 
> Best Regards,

Re: SysLog Parser in Metron

[Simon Elliston Ball]

Short answer: grok parsers. 

Longer answer: syslog is more a transport, not just a log format, so it 
encapsulates a wide variety of data sources. Your best bet is probably to use 
NiFi to listen for syslog from a remote host (ListenSyslog) and then route each 
application in the syslog to a different kafka topic. That way you have kafka 
topics for each type of data you care about eg sshd, login, cups... whatever. 
From there it’s easiest to use a grok parser in metron to pull out the fields. 
There are many prebuilt patterns for the common services around on the web.

Simon 

> On 25 Oct 2017, at 05:55, Farrukh Naveed Anjum  
> wrote:
> 
> Hi,
> 
> How can I get syslog in metron any help (pattern / parser). Kindly help ?
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Kibana Error

[Simon Elliston Ball]

Your elastic search instance has died. Try given it more heap size in the 
elastic section on ambari.


> On 25 Oct 2017, at 09:16, Syed Hammad Tahir  wrote:
> 
> When I try to open node1:5000 I see this.
> 
> 
> 
> What could be the problem and its solution?

Re: Kibana Error

[Simon Elliston Ball]

I strongly suggest you spend some time learning about elastic search and some 
of the basic components. This is not a bug, it’s that elastic is down. The 
default heap (use the ambari search in the elastic section) is probably set too 
low. The default is 128m. Change this to more, probably more like 2048m.

Essential background reading for metron is an understanding of elastic search, 
kafka, hadoop (hdfs in particular) and Linux. Our docs will assume you have at 
least some familiarity with those technologies.

Simon 

> On 25 Oct 2017, at 11:40, Syed Hammad Tahir  wrote:
> 
> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
> googled it and found no help. Please help me guys, there are bigger issues at 
> hand and I cant afford to waste much time on this problem :( 
> 
>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>>  wrote:
>> Its a bug reported in metron, 
>> 
>> Look into barematel guide, Turn Red to green Cluster google it.
>> 
>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir"  wrote:
>>> SHould I do it from here? If yes then please guide me how to
>>> 
>>> 
>>> 
>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>>>  wrote:
>>>> Your elastic search instance has died. Try given it more heap size in the 
>>>> elastic section on ambari.
>>>> 
>>>> 
>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir  wrote:
>>>> >
>>>> > When I try to open node1:5000 I see this.
>>>> >
>>>> > 
>>>> >
>>>> > What could be the problem and its solution?
>>>> 
>>> 
> 

Re: Kibana Error

[Simon Elliston Ball]

Did you check the elastic service was running and healthy with the health 
checks. Try a few of the quick links from the elastic section in ambari.

> On 25 Oct 2017, at 13:05, Syed Hammad Tahir  wrote:
> 
> I have increased size to 2048mb. Still seeing it
> 
> 
> 
> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> I strongly suggest you spend some time learning about elastic search and some 
> of the basic components. This is not a bug, it’s that elastic is down. The 
> default heap (use the ambari search in the elastic section) is probably set 
> too low. The default is 128m. Change this to more, probably more like 2048m.
> 
> Essential background reading for metron is an understanding of elastic 
> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
> you have at least some familiarity with those technologies.
> 
> Simon 
> 
> On 25 Oct 2017, at 11:40, Syed Hammad Tahir  <mailto:mscs16...@itu.edu.pk>> wrote:
> 
>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>> googled it and found no help. Please help me guys, there are bigger issues 
>> at hand and I cant afford to waste much time on this problem :( 
>> 
>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>> mailto:anjum.farr...@gmail.com>> wrote:
>> Its a bug reported in metron, 
>> 
>> Look into barematel guide, Turn Red to green Cluster google it.
>> 
>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" > <mailto:mscs16...@itu.edu.pk>> wrote:
>> SHould I do it from here? If yes then please guide me how to
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>> mailto:si...@simonellistonball.com>> wrote:
>> Your elastic search instance has died. Try given it more heap size in the 
>> elastic section on ambari.
>> 
>> 
>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir > > <mailto:mscs16...@itu.edu.pk>> wrote:
>> >
>> > When I try to open node1:5000 I see this.
>> >
>> > 
>> >
>> > What could be the problem and its solution?
>> 
>> 
>> 
> 

Re: Kibana Error

[Simon Elliston Ball]

That just shows running, not health. The problem is that it is not responding. 
I assume you have tried restarting elastic. 

> On 25 Oct 2017, at 13:12, Syed Hammad Tahir  wrote:
> 
> It shows healthy
> 
> 
> But when I click in any quick link it shows this
> 
> 
> 
> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> Did you check the elastic service was running and healthy with the health 
> checks. Try a few of the quick links from the elastic section in ambari.
> 
>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>> I have increased size to 2048mb. Still seeing it
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
>> mailto:si...@simonellistonball.com>> wrote:
>> I strongly suggest you spend some time learning about elastic search and 
>> some of the basic components. This is not a bug, it’s that elastic is down. 
>> The default heap (use the ambari search in the elastic section) is probably 
>> set too low. The default is 128m. Change this to more, probably more like 
>> 2048m.
>> 
>> Essential background reading for metron is an understanding of elastic 
>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
>> you have at least some familiarity with those technologies.
>> 
>> Simon 
>> 
>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>>> googled it and found no help. Please help me guys, there are bigger issues 
>>> at hand and I cant afford to waste much time on this problem :( 
>>> 
>>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>>> mailto:anjum.farr...@gmail.com>> wrote:
>>> Its a bug reported in metron, 
>>> 
>>> Look into barematel guide, Turn Red to green Cluster google it.
>>> 
>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" >> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> SHould I do it from here? If yes then please guide me how to
>>> 
>>> 
>>> 
>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>> mailto:si...@simonellistonball.com>> wrote:
>>> Your elastic search instance has died. Try given it more heap size in the 
>>> elastic section on ambari.
>>> 
>>> 
>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >> > <mailto:mscs16...@itu.edu.pk>> wrote:
>>> >
>>> > When I try to open node1:5000 I see this.
>>> >
>>> > 
>>> >
>>> > What could be the problem and its solution?
>>> 
>>> 
>>> 
>> 
> 
> 

Re: Kibana Error

[Simon Elliston Ball]

Ok, this is an elastic problem which prevents it shutting down. Find the 
elastic processes, kill them, and start it up again.


> On 25 Oct 2017, at 13:15, Syed Hammad Tahir  wrote:
> 
> Just gave the command but its stuck here. I restart it earleir via ambari 
> after changing heapsize. Now doing it via console
> 
> 
> 
> On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> That just shows running, not health. The problem is that it is not 
> responding. I assume you have tried restarting elastic. 
> 
>> On 25 Oct 2017, at 13:12, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote:
>> 
>> It shows healthy
>> 
>> 
>> But when I click in any quick link it shows this
>> 
>> 
>> 
>> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball 
>> mailto:si...@simonellistonball.com>> wrote:
>> Did you check the elastic service was running and healthy with the health 
>> checks. Try a few of the quick links from the elastic section in ambari.
>> 
>>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir >> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> 
>>> I have increased size to 2048mb. Still seeing it
>>> 
>>> 
>>> 
>>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball 
>>> mailto:si...@simonellistonball.com>> wrote:
>>> I strongly suggest you spend some time learning about elastic search and 
>>> some of the basic components. This is not a bug, it’s that elastic is down. 
>>> The default heap (use the ambari search in the elastic section) is probably 
>>> set too low. The default is 128m. Change this to more, probably more like 
>>> 2048m.
>>> 
>>> Essential background reading for metron is an understanding of elastic 
>>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume 
>>> you have at least some familiarity with those technologies.
>>> 
>>> Simon 
>>> 
>>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir >> <mailto:mscs16...@itu.edu.pk>> wrote:
>>> 
>>>> Sorry, I didnt understand. Which baremetal guide should I look into? And I 
>>>> googled it and found no help. Please help me guys, there are bigger issues 
>>>> at hand and I cant afford to waste much time on this problem :( 
>>>> 
>>>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum 
>>>> mailto:anjum.farr...@gmail.com>> wrote:
>>>> Its a bug reported in metron, 
>>>> 
>>>> Look into barematel guide, Turn Red to green Cluster google it.
>>>> 
>>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" >>> <mailto:mscs16...@itu.edu.pk>> wrote:
>>>> SHould I do it from here? If yes then please guide me how to
>>>> 
>>>> 
>>>> 
>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball 
>>>> mailto:si...@simonellistonball.com>> wrote:
>>>> Your elastic search instance has died. Try given it more heap size in the 
>>>> elastic section on ambari.
>>>> 
>>>> 
>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >>> > <mailto:mscs16...@itu.edu.pk>> wrote:
>>>> >
>>>> > When I try to open node1:5000 I see this.
>>>> >
>>>> > 
>>>> >
>>>> > What could be the problem and its solution?
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
> 
> 

Re: analytics exchange platform

[Simon Elliston Ball]

The analytics exchange concept is not really part of Apache Metron, but some 
commercial offerings include it. In terms of Metron itself, are you maybe 
thinking about Model as a Service: 
http://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html
 


Simon

> On 15 Nov 2017, at 00:54, Satish Abburi  wrote:
> 
>  
> Any pointers to this? We are looking to deploy few analytics packages on top 
> of Metron platform.
>  
> Thanks,
> Satish 
>  

Re: Snort enrichment issue

[Simon Elliston Ball]

Did you setup and load the geo enrichment database? 
https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader
 


Also, we can’t really see the error from screenshots, please send log entries. 

Simon

> On 17 Nov 2017, at 07:11, Syed Hammad Tahir  wrote:
> 
> Hi all, I am starting it again. Last one got a bit messy
> 
> Ok, Now I have started everything again from scratch (redeployed single node 
> based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when 
> I execute this command: 
> 
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date 
> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | 
> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
> node1:6667 --topic snort
> 
> (format of ths command was taken from: 
> https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub
>  
> )
> 
> I get this under enrichment storm topology :
> 
> 
> 
> 
> 
> I have come this far, please help me push these dummy preformatted snort logs 
> into kibana dashboard.
> 
> Regards.
> 

Re: ML in Metron

[Simon Elliston Ball]

Use MaaS: 
http://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html
 



> On 21 Nov 2017, at 11:43, Syed Hammad Tahir  wrote:
> 
> HI all,
> 
> I have succesfully pushed real snort logs in to metron, now I need to apply a 
> machine learning or data science algorithm on it. How could I do that? I want 
> to code in python/R and then apply it in metron.
> 
> Regards.

Re: Not able to run metron.

[Simon Elliston Ball]

You shouldn’t need ansible for the full-dev build, but you will need maven, 
docker and an up-to-date nodejs and npm package to do the actual build. I would 
recommend against using the OS provided nodejs and go with the packages from 
nodesource instead. 

The full-dev build is also the best starting point if you’re looking for an 
environment to test extensions or contributions to the platform, though I would 
make sure you give it plenty of RAM (16GB is a good starting point for the VM I 
find, so you may need to adjust the Vagrantfile).

In a ‘real’ environment, the best way to install is through the Mpack method on 
an existing Ambari install. I would not recommend using ansible at all. It is 
extremely sensitive to ansible version number and very brittle as a result 
because ansible apis keep changing from build to build. 

Simon

> On 22 Nov 2017, at 12:21, Pawel Bialasiewicz  wrote:
> 
> Thank you for the answer Zeolla!
> 
> As for now lets focus on the vagrant spin up.
> 
> Correct me if I'm wrong: the vagrant full-dev install should work on any 
> linux distro that has: Vagrant, Virtualbox, Ansible, git, Maven, 
> vagrant-hostmanager. 
> 
> I'm have all of those things installed on a fresh install of Ubuntu 16 LTS. 
> And it still crashes with the output that I included in the Issue 1327.
> 
> Pawel 
> 
> 
> 
> On Wed, Nov 22, 2017 at 1:11 PM, zeo...@gmail.com  > wrote:
> It looks like you have two issues - one related to having java_home unset, 
> and one with an old version of npm.
> 
> I would suggest focusing on the Vagrant spin-up, as it is the easiest to get 
> running.  Are you running this on a CentOS 6 machine, or are you referring to 
> the full-dev VM's CentOS 6 OS?  Also, you have noted that this affects 
> version 0.4.0, but the latest release is 0.4.1, is that accurate?  Have you 
> retried the `vagrant up` after your npm upgrade and a npm` cache clean`?
> 
> Jon
> 
> On Wed, Nov 22, 2017 at 7:01 AM Pawel Bialasiewicz  > wrote:
> Hi,
> 
> I have been trying to run Metron in many ways and all of the deployment 
> methods failed:
> 
> 1) Bare metal –-> [Issue 1320]
> 2) AWS –-> [Issue 1318]
> 3) Vagrant –-> [Issue 1327]
> 
> So currently I'm out of options. All the deployments methods failed.
> 
> Is the all of documentation outdated, or maybe I should use a more specific 
> branch(other then master)?
> 
> Can somebody confirm that it is currently possible to build metron-config? It 
> fails for me in every deployment.
> 
> Any help would be appreciated.  
> -- 
> Jon
> 
> 

Re: Not able to run metron.

[Simon Elliston Ball]

Sorry, you’re right, you do need ansible. Make sure the version is EXACTLY the 
version in the docs. 

Simon

> On 22 Nov 2017, at 13:03, Otto Fowler  wrote:
> 
> You DO need ansible for full_dev deployment.
> You do need Docker installed and running
> 
> 
> 
> On November 22, 2017 at 07:51:47, Pawel Bialasiewicz (pa...@evengx.com 
> ) wrote:
> 
>> Here is the output:
>> 
>> platform-info.sh
>> Metron 0.4.2
>> --
>> * master
>> --
>> commit 8022f2c8c4e9018a15a4f04d0a66f8bc0ea653c3
>> Author: merrimanr mailto:merrim...@gmail.com>>
>> Date:   Tue Nov 21 13:46:35 2017 -0600
>> 
>> METRON-1319 Column Metadata REST service should use default indices on 
>> empty input (merrimanr) closes apache/metron#843
>> --
>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> --
>> ansible 2.0.0.2
>>   config file = /etc/ansible/ansible.cfg
>>   configured module search path = Default w/o overrides
>> --
>> Vagrant 2.0.1
>> --
>> Python 2.7.12
>> --
>> Apache Maven 3.3.9
>> Maven home: /usr/share/maven
>> Java version: 1.8.0_151, vendor: Oracle Corporation
>> Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "linux", version: "4.4.0-87-generic", arch: "amd64", family: "unix"
>> --
>> Docker version 1.13.1, build 092cba3
>> --
>> node
>> v8.9.1
>> --
>> npm
>> 5.5.1
>> --
>> Linux visor 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 
>> x86_64 x86_64 x86_64 GNU/Linux
>> --
>> Total System Memory = 15996.9 MB
>> Processor Model: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
>> Processor Speed: 3427.968 MHz
>> Processor Speed: 3443.437 MHz
>> Processor Speed: 3351.210 MHz
>> Processor Speed: 3438.046 MHz
>> Total Physical Processors: 4
>> Total cores: 16
>> Disk information:
>> /dev/sda1   219G   20G  189G  10% /
>> This CPU appears to support virtualization
>> 
>> 
>> On Wed, Nov 22, 2017 at 1:50 PM, zeo...@gmail.com  
>> mailto:zeo...@gmail.com>> wrote:
>> You will need docker to be installed.  In theory this should work across any 
>> *nix distro, but the instructions provided are most thoroughly tested on 
>> macOS endpoints so I always like to clarify.  I know in the past I had some 
>> bumps with spinning it up on CentOS just due to lack of testing.
>> 
>> Can you run this[1] and report back with the output?  Thanks,
>> 
>> 1:  
>> https://github.com/apache/metron/blob/master/metron-deployment/scripts/platform-info.sh
>>  
>> 
>> 
>> Jon
>> 
>> On Wed, Nov 22, 2017 at 7:21 AM Pawel Bialasiewicz > > wrote:
>> Thank you for the answer Zeolla!
>> 
>> As for now lets focus on the vagrant spin up.
>> 
>> Correct me if I'm wrong: the vagrant full-dev install should work on any 
>> linux distro that has: Vagrant, Virtualbox, Ansible, git, Maven, 
>> vagrant-hostmanager. 
>> 
>> I'm have all of those things installed on a fresh install of Ubuntu 16 LTS. 
>> And it still crashes with the output that I included in the Issue 1327.
>> 
>> Pawel 
>> 
>> 
>> 
>> On Wed, Nov 22, 2017 at 1:11 PM, zeo...@gmail.com  
>> mailto:zeo...@gmail.com>> wrote:
>> It looks like you have two issues - one related to having java_home unset, 
>> and one with an old version of npm.
>> 
>> I would suggest focusing on the Vagrant spin-up, as it is the easiest to get 
>> running.  Are you running this on a CentOS 6 machine, or are you referring 
>> to the full-dev VM's CentOS 6 OS?  Also, you have noted that this affects 
>> version 0.4.0, but the latest release is 0.4.1, is that accurate?  Have you 
>> retried the `vagrant up` after your npm upgrade and a npm` cache clean`?
>> 
>> Jon
>> 
>> On Wed, Nov 22, 2017 at 7:01 AM Pawel Bialasiewicz > > wrote:
>> Hi,
>> 
>> I have been trying to run Metron in many ways and all of the deployment 
>> methods failed:
>> 
>> 1) Bare metal –-> [Issue 1320]
>> 2) AWS –-> [Issue 1318]
>> 3) Vagrant –-> [Issue 1327]
>> 
>> So currently I'm out of options. All the deployments methods failed.
>> 
>> Is the all of documentation outdated, or maybe I should use a more specific 
>> branch(other then master)?
>> 
>> Can somebody confirm that it is currently possible to build metron-config? 
>> It fails for me in every deployment.
>> 
>> Any help would be appreciated.  
>> --
>> Jon
>> 
>> 
>> --
>> Jon
>> 

Re: Not able to run metron.

[Simon Elliston Ball]

Just FYI, that’s a little outdated, Ubuntu builds are now included in the 
hortonworks distribution, but if you want to build from source the advice there 
very much applies. 

> On 22 Nov 2017, at 13:19, Otto Fowler  wrote:
> 
> Btw:  If you have ubuntu clusters, have you seen : 
> https://community.hortonworks.com/articles/88843/manually-installing-apache-metron-on-ubuntu-1404.html
>  
> <https://community.hortonworks.com/articles/88843/manually-installing-apache-metron-on-ubuntu-1404.html>
> 
> On November 22, 2017 at 08:17:41, Otto Fowler (ottobackwa...@gmail.com 
> <mailto:ottobackwa...@gmail.com>) wrote:
> 
>> I build on mac, and have :
>> 
>> --
>> node
>> v6.10.2
>> --
>> npm
>> 3.10.10
>> 
>> for my node versions.
>> 
>> 
>> On November 22, 2017 at 08:05:02, Simon Elliston Ball 
>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>> 
>>> Sorry, you’re right, you do need ansible. Make sure the version is EXACTLY 
>>> the version in the docs. 
>>> 
>>> Simon
>>> 
>>>> On 22 Nov 2017, at 13:03, Otto Fowler >>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>> 
>>>> You DO need ansible for full_dev deployment.
>>>> You do need Docker installed and running
>>>> 
>>>> 
>>>> 
>>>> On November 22, 2017 at 07:51:47, Pawel Bialasiewicz (pa...@evengx.com 
>>>> <mailto:pa...@evengx.com>) wrote:
>>>> 
>>>>> Here is the output:
>>>>> 
>>>>> platform-info.sh
>>>>> Metron 0.4.2
>>>>> --
>>>>> * master
>>>>> --
>>>>> commit 8022f2c8c4e9018a15a4f04d0a66f8bc0ea653c3
>>>>> Author: merrimanr mailto:merrim...@gmail.com>>
>>>>> Date:   Tue Nov 21 13:46:35 2017 -0600
>>>>> 
>>>>> METRON-1319 Column Metadata REST service should use default indices 
>>>>> on empty input (merrimanr) closes apache/metron#843
>>>>> --
>>>>>  metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +-
>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>> --
>>>>> ansible 2.0.0.2
>>>>>   config file = /etc/ansible/ansible.cfg
>>>>>   configured module search path = Default w/o overrides
>>>>> --
>>>>> Vagrant 2.0.1
>>>>> --
>>>>> Python 2.7.12
>>>>> --
>>>>> Apache Maven 3.3.9
>>>>> Maven home: /usr/share/maven
>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation
>>>>> Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>> OS name: "linux", version: "4.4.0-87-generic", arch: "amd64", family: 
>>>>> "unix"
>>>>> --
>>>>> Docker version 1.13.1, build 092cba3
>>>>> --
>>>>> node
>>>>> v8.9.1
>>>>> --
>>>>> npm
>>>>> 5.5.1
>>>>> --
>>>>> Linux visor 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>> --
>>>>> Total System Memory = 15996.9 MB
>>>>> Processor Model: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
>>>>> Processor Speed: 3427.968 MHz
>>>>> Processor Speed: 3443.437 MHz
>>>>> Processor Speed: 3351.210 MHz
>>>>> Processor Speed: 3438.046 MHz
>>>>> Total Physical Processors: 4
>>>>> Total cores: 16
>>>>> Disk information:
>>>>> /dev/sda1   219G   20G  189G  10% /
>>>>> This CPU appears to support virtualization
>>>>> 
>>>>> 
>>>>> On Wed, Nov 22, 2017 at 1:50 PM, zeo...@gmail.com 
>>>>> <mailto:zeo...@gmail.com> mailto:zeo...@gmail.com>> 
>>>>> wrote:
>>>>> You will need docker to be installed.  In theory this should work across 
>>>>> any *nix distro, but the instructions provided are most thoroughly tested 
>>>>> on macOS endpoints so I always like to clarify.  I know in the past I had 
>>>>> some bumps with spinning it up on CentOS just due to lack of testing.
>>>>> 
>>>>> Can you run this[1] and report back with the output?  Thanks,
>

Re: Basic analysis

[Simon Elliston Ball]

Yes. Consider a zeppelin notebook, or kibana dashboard for this. 

If you want to use these values for detection, consider building a profile 
based on the stats objects (see the profiler section of the documentation under 
analytics.

Simon

> On 6 Dec 2017, at 07:42, Syed Hammad Tahir  wrote:
> 
> Hi,
> 
> Can I setup custom visualization to show lets say the peak netrwork usage 
> traffic in a certain time?
> 
> Regards.

Re: Basic analysis

[Simon Elliston Ball]

We do already have a number of example of exactly this, but sure if someone 
feels like adding to those that would be great. 

Simon

> On 6 Dec 2017, at 13:14, Otto Fowler  wrote:
> 
> Maybe a Jira logged for an ‘example’ notebook for this would be appropriate 
> as well?
> 
> 
> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>> 
>> If you want to use these values for detection, consider building a profile 
>> based on the stats objects (see the profiler section of the documentation 
>> under analytics. 
>> 
>> Simon 
>> 
>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir > > <mailto:mscs16...@itu.edu.pk>> wrote: 
>> >  
>> > Hi, 
>> >  
>> > Can I setup custom visualization to show lets say the peak netrwork usage 
>> > traffic in a certain time? 
>> >  
>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

In product… Install Zeppelin Notebooks, and the samples including notebooks at 
https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
 
<https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>

as of course there are similar Kibana dashboards included, which are examples 
of custom visualisation of metron data, there is also the run book for 
visualising squid data in kibana on the docs wiki 
https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard 
<https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>

Should at least get us started. 

Simon

> On 6 Dec 2017, at 14:00, Otto Fowler  wrote:
> 
> Links?
> 
> 
> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> We do already have a number of example of exactly this, but sure if someone 
>> feels like adding to those that would be great. 
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 13:14, Otto Fowler >> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Maybe a Jira logged for an ‘example’ notebook for this would be appropriate 
>>> as well?
>>> 
>>> 
>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>> 
>>>> If you want to use these values for detection, consider building a profile 
>>>> based on the stats objects (see the profiler section of the documentation 
>>>> under analytics. 
>>>> 
>>>> Simon 
>>>> 
>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir >>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>> >  
>>>> > Hi, 
>>>> >  
>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>> > usage traffic in a certain time? 
>>>> >  
>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

No problem, I’ll grant you it’s not in the most intuitive part of the source 
tree to go digging in, but you can also get to the zeppelin bits via the 
actions button on the Metron config section (Install Notebooks)

If anyone has any good ideas (or code!) for sample zeppelin notebooks that 
would be useful, you can add them to a specific instance of the platform via 
the config/zeppelin/metron location and run the action again I believe, and 
this would be a great place for more security people to contribute sample run 
books for example. There are also efforts by commercial support providers I 
believe to add more samples of both dashboards and use cases.

Simon

> On 6 Dec 2017, at 14:12, Otto Fowler  wrote:
> 
> Thanks Simon
> 
> 
> On December 6, 2017 at 09:11:50, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> In product… Install Zeppelin Notebooks, and the samples including notebooks 
>> at 
>> https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
>>  
>> <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>
>> 
>> as of course there are similar Kibana dashboards included, which are 
>> examples of custom visualisation of metron data, there is also the run book 
>> for visualising squid data in kibana on the docs wiki 
>> https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard
>>  
>> <https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>
>> 
>> Should at least get us started. 
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 14:00, Otto Fowler >> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Links?
>>> 
>>> 
>>> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> We do already have a number of example of exactly this, but sure if 
>>>> someone feels like adding to those that would be great. 
>>>> 
>>>> Simon
>>>> 
>>>>> On 6 Dec 2017, at 13:14, Otto Fowler >>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>> 
>>>>> Maybe a Jira logged for an ‘example’ notebook for this would be 
>>>>> appropriate as well?
>>>>> 
>>>>> 
>>>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>>> 
>>>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>>>> 
>>>>>> If you want to use these values for detection, consider building a 
>>>>>> profile based on the stats objects (see the profiler section of the 
>>>>>> documentation under analytics. 
>>>>>> 
>>>>>> Simon 
>>>>>> 
>>>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir >>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>>>> >  
>>>>>> > Hi, 
>>>>>> >  
>>>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>>>> > usage traffic in a certain time? 
>>>>>> >  
>>>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

Agreed… for the users list I would just say use the Install Notebooks action, 
and look at the squid example on the wiki, but since it was you who asked for 
links, Otto, I went a bit dev list ;)

Simon

> On 6 Dec 2017, at 14:33, Otto Fowler  wrote:
> 
> The issue is the requirement for people on the user list to go to the source.
> 
> 
> On December 6, 2017 at 09:16:39, Simon Elliston Ball 
> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
> 
>> No problem, I’ll grant you it’s not in the most intuitive part of the source 
>> tree to go digging in, but you can also get to the zeppelin bits via the 
>> actions button on the Metron config section (Install Notebooks)
>> 
>> If anyone has any good ideas (or code!) for sample zeppelin notebooks that 
>> would be useful, you can add them to a specific instance of the platform via 
>> the config/zeppelin/metron location and run the action again I believe, and 
>> this would be a great place for more security people to contribute sample 
>> run books for example. There are also efforts by commercial support 
>> providers I believe to add more samples of both dashboards and use cases.
>> 
>> Simon
>> 
>>> On 6 Dec 2017, at 14:12, Otto Fowler >> <mailto:ottobackwa...@gmail.com>> wrote:
>>> 
>>> Thanks Simon
>>> 
>>> 
>>> On December 6, 2017 at 09:11:50, Simon Elliston Ball 
>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>> 
>>>> In product… Install Zeppelin Notebooks, and the samples including 
>>>> notebooks at 
>>>> https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
>>>>  
>>>> <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>
>>>> 
>>>> as of course there are similar Kibana dashboards included, which are 
>>>> examples of custom visualisation of metron data, there is also the run 
>>>> book for visualising squid data in kibana on the docs wiki 
>>>> https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard
>>>>  
>>>> <https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>
>>>> 
>>>> Should at least get us started. 
>>>> 
>>>> Simon
>>>> 
>>>>> On 6 Dec 2017, at 14:00, Otto Fowler >>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>> 
>>>>> Links?
>>>>> 
>>>>> 
>>>>> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>>> 
>>>>>> We do already have a number of example of exactly this, but sure if 
>>>>>> someone feels like adding to those that would be great. 
>>>>>> 
>>>>>> Simon
>>>>>> 
>>>>>>> On 6 Dec 2017, at 13:14, Otto Fowler >>>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>>>> 
>>>>>>> Maybe a Jira logged for an ‘example’ notebook for this would be 
>>>>>>> appropriate as well?
>>>>>>> 
>>>>>>> 
>>>>>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>>>>>> 
>>>>>>>> If you want to use these values for detection, consider building a 
>>>>>>>> profile based on the stats objects (see the profiler section of the 
>>>>>>>> documentation under analytics. 
>>>>>>>> 
>>>>>>>> Simon 
>>>>>>>> 
>>>>>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir >>>>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>>>>>> >  
>>>>>>>> > Hi, 
>>>>>>>> >  
>>>>>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>>>>>> > usage traffic in a certain time? 
>>>>>>>> >  
>>>>>>>> > Regards.

Re: Basic analysis

[Simon Elliston Ball]

I suggest reading some of the kibana documentation. There are a lot of 
tutorials available online on kibana. 

Simon

> On 6 Dec 2017, at 15:12, Syed Hammad Tahir  wrote:
> 
> How do I do it using kibana dashboard? What would be the most easy way?
> 
> 
> On Wed, Dec 6, 2017 at 7:35 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> Agreed… for the users list I would just say use the Install Notebooks action, 
> and look at the squid example on the wiki, but since it was you who asked for 
> links, Otto, I went a bit dev list ;)
> 
> Simon
> 
> 
>> On 6 Dec 2017, at 14:33, Otto Fowler > <mailto:ottobackwa...@gmail.com>> wrote:
>> 
>> The issue is the requirement for people on the user list to go to the source.
>> 
>> 
>> On December 6, 2017 at 09:16:39, Simon Elliston Ball 
>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>> 
>>> No problem, I’ll grant you it’s not in the most intuitive part of the 
>>> source tree to go digging in, but you can also get to the zeppelin bits via 
>>> the actions button on the Metron config section (Install Notebooks)
>>> 
>>> If anyone has any good ideas (or code!) for sample zeppelin notebooks that 
>>> would be useful, you can add them to a specific instance of the platform 
>>> via the config/zeppelin/metron location and run the action again I believe, 
>>> and this would be a great place for more security people to contribute 
>>> sample run books for example. There are also efforts by commercial support 
>>> providers I believe to add more samples of both dashboards and use cases.
>>> 
>>> Simon
>>> 
>>>> On 6 Dec 2017, at 14:12, Otto Fowler >>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>> 
>>>> Thanks Simon
>>>> 
>>>> 
>>>> On December 6, 2017 at 09:11:50, Simon Elliston Ball 
>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>> 
>>>>> In product… Install Zeppelin Notebooks, and the samples including 
>>>>> notebooks at 
>>>>> https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron
>>>>>  
>>>>> <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing/src/main/config/zeppelin/metron>
>>>>> 
>>>>> as of course there are similar Kibana dashboards included, which are 
>>>>> examples of custom visualisation of metron data, there is also the run 
>>>>> book for visualising squid data in kibana on the docs wiki 
>>>>> https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard
>>>>>  
>>>>> <https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard>
>>>>> 
>>>>> Should at least get us started. 
>>>>> 
>>>>> Simon
>>>>> 
>>>>>> On 6 Dec 2017, at 14:00, Otto Fowler >>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>>> 
>>>>>> Links?
>>>>>> 
>>>>>> 
>>>>>> On December 6, 2017 at 08:18:23, Simon Elliston Ball 
>>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote:
>>>>>> 
>>>>>>> We do already have a number of example of exactly this, but sure if 
>>>>>>> someone feels like adding to those that would be great. 
>>>>>>> 
>>>>>>> Simon
>>>>>>> 
>>>>>>>> On 6 Dec 2017, at 13:14, Otto Fowler >>>>>>> <mailto:ottobackwa...@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> Maybe a Jira logged for an ‘example’ notebook for this would be 
>>>>>>>> appropriate as well?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On December 6, 2017 at 07:06:30, Simon Elliston Ball 
>>>>>>>> (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Yes. Consider a zeppelin notebook, or kibana dashboard for this.  
>>>>>>>>> 
>>>>>>>>> If you want to use these values for detection, consider building a 
>>>>>>>>> profile based on the stats objects (see the profiler section of the 
>>>>>>>>> documentation under analytics. 
>>>>>>>>> 
>>>>>>>>> Simon 
>>>>>>>>> 
>>>>>>>>> > On 6 Dec 2017, at 07:42, Syed Hammad Tahir >>>>>>>> > <mailto:mscs16...@itu.edu.pk>> wrote: 
>>>>>>>>> >  
>>>>>>>>> > Hi, 
>>>>>>>>> >  
>>>>>>>>> > Can I setup custom visualization to show lets say the peak netrwork 
>>>>>>>>> > usage traffic in a certain time? 
>>>>>>>>> >  
>>>>>>>>> > Regards.
> 
> 

Re: machine learning libraries supported

[Simon Elliston Ball]

I would recommend starting out with something like Spark, but the short answer 
is that anything that will run inside a yarn container, so the answer is most 
ML libraries. 

Using Spark to train models on the historical store is a good bet, and then 
using the trained models with model as a service.

See 
https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
 

 for information on models and some sample boilerplate for deploying your own 
python based models. 

You could as some have suggested use spark streaming, but to be honest, the 
spark ML models are not well suited to streaming use cases, and you would be 
very much breaking the metron flow rather than benefitting from elements like 
MaaS (you’d basically be building a 100% custom side project, which would be 
fine, but you’re missing a lot of the benefits of Metron that way). If you do 
go down that route I would strong recommend having the output of your streaming 
jobs feed back into a Metron sensor. To be honest though, you’re much better 
off training in batch and scoring / inferring via the Model as a Service 
approach. 

Simon


> On 6 Dec 2017, at 07:45, moshe jarusalem  wrote:
> 
> Hi All,
> Would you please suggest some documentation about machine learning libraries 
> can be used in metron architecture? and how ? any examples appretiated.
> 
> regards,
> 

Re: machine learning libraries supported

[Simon Elliston Ball]

Spark’s ML models are primarily batch in their nature. There is talk about 
incorporating things like naive bayes and streaming kmeans to structured 
streaming (which will require some schema work in metron to make sense). These 
are still open issues not seeing a lot of progress in the spark community. 

The most common mistake I’ve seen using spark streaming with ML in the cyber 
world is people thinking that the FP Growth association rules models can be 
online-learnt, because there exists a class of streaming FP Growth models. The 
Spark implementations of FP Growth however, rely on Batch (mathematically!) and 
while technically can be run on the micro-batches Spark streaming provides, are 
not actually meaningful. Just because your model runs and gives you an output, 
doesn’t mean it’s mathematically defensible to do so. 

All that said...

Streaming inference makes some sense in spark, but that’s probably better 
handled through MaaS in Metron, which will generalise to spark and other 
libraries, and absolutely, use the spark models and the ML pipelining to 
perform inference in a spark job run with parallel instances in MaaS. Note that 
the reason for this is primality that Spark is a data parallel engine, where as 
Metron MaaS applies task parallelism, in order to reduce latency. 

To the point of a good example of python / spark / MaaS / Metron, I would 
recommend taking a look at Casey’s blog at 
https://hortonworks.com/blog/model-service-modern-streaming-data-science-apache-metron/
 
<https://hortonworks.com/blog/model-service-modern-streaming-data-science-apache-metron/>
 which is a walk though on score ad python scikit-learn model in MaaS. For the 
spark piece, I’ve seen a number of examples based on these same principals, 
using the spark classes for scoring based on saved models produced by a batch 
trainer. Apologies, I don’t have any readily publishable examples of the whole 
thing, but may work something synthetic up if it would be useful. 

Simon

> On 7 Dec 2017, at 13:09, Martin Andreoni  wrote:
> 
> Hello Simon,
> 
> thanks for the information.
> 
> However, why do u affirm that the streaming models are not well suited?
> 
>> You could as some have suggested use spark streaming, but to be honest, the 
>> spark ML models are not well suited to streaming use cases
> Is there a performance problem or how would you justify that phrase? 
> 
> thanks
> 
> Le 07/12/2017 à 13:55, Simon Elliston Ball a écrit :
>> I would recommend starting out with something like Spark, but the short 
>> answer is that anything that will run inside a yarn container, so the answer 
>> is most ML libraries. 
>> 
>> Using Spark to train models on the historical store is a good bet, and then 
>> using the trained models with model as a service.
>> 
>> See 
>> https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
>>  
>> <https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service>
>>  for information on models and some sample boilerplate for deploying your 
>> own python based models. 
>> 
>> You could as some have suggested use spark streaming, but to be honest, the 
>> spark ML models are not well suited to streaming use cases, and you would be 
>> very much breaking the metron flow rather than benefitting from elements 
>> like MaaS (you’d basically be building a 100% custom side project, which 
>> would be fine, but you’re missing a lot of the benefits of Metron that way). 
>> If you do go down that route I would strong recommend having the output of 
>> your streaming jobs feed back into a Metron sensor. To be honest though, 
>> you’re much better off training in batch and scoring / inferring via the 
>> Model as a Service approach. 
>> 
>> Simon
>> 
>> 
>>> On 6 Dec 2017, at 07:45, moshe jarusalem >> <mailto:tuu...@gmail.com>> wrote:
>>> 
>>> Hi All,
>>> Would you please suggest some documentation about machine learning 
>>> libraries can be used in metron architecture? and how ? any examples 
>>> appretiated.
>>> 
>>> regards,
>>> 
>> 
> 
> -- 
> Martin Andreoni
> PhD. Candidate at GTA/LIP6
> 
> UFRJ/UPMC
> 
> www.gta.ufrj.br/~martin <http://www.gta.ufrj.br/%7Emartin>

Re: metron vs ossec

[Simon Elliston Ball]

In many ways it’s a matter of scale. OSSIM is a kind of lite version of 
AlienVault, and used by them. I’ve seen people move from an OSSIM architecture 
to Metron specifically to get better scaling, things like PCAP capabilities 
etc. but also retain the OSSEC agents to handle endpoint and scanning use 
cases, which they then feed into Metron. In these cases it was mostly about 
scalability and flexibility to extend, as well as manageability of multi-tenant 
environments. 

In functional terms, Metron also emphasises behaviour profiling and machine 
learning, whereas OSSIM is a more traditional rules-centric way of looking at 
security and log monitoring. 

Hope that helps you understand the difference a little better,
Simon

> On 21 Dec 2017, at 12:22, moshe jarusalem  wrote:
> 
> Jon thanks for the information.
> 
> I am indeed trying to learn both of them just wanted to get expert ideas. 
> 
> OSSEC is also supported by OSSIM which is somewhat like metron. I  would like 
> to hear  ideas which may make metron better alternative and or composite 
> usage.
> 
> Regards,
> 
> 
> On Thu, Dec 21, 2017 at 2:39 PM, zeo...@gmail.com  > wrote:
> Yes, I run both in my environment and they are both security products but 
> that's about where the similarities end.  Ossec is a host based solution that 
> monitors local activity with it's tree based rules engine, Metron is a 
> distributed solution that handles large sets of data from many sources and a 
> lot more.  A possible connection between the two may be that ossec 
> logs/alerts could be fed into Metron for enrichment, triage, alerting, and 
> analysis.
> 
> I would recommend either reading the documentation for both of them in more 
> detail, or spinning them both up to get a better handle on the differences.
> 
> Jon
> 
> 
> On Thu, Dec 21, 2017, 00:34 moshe jarusalem  > wrote:
> Hi All, 
> I have come across OSSEC project and find it similar to metron. I am confused 
> a bit. 
> is anyone aware of Ossec and give some comparisons?
> 
> Regards,
> -- 
> Jon
> 
> 

Re: Metron Version

[Simon Elliston Ball]

Are the logs you’re sending with syslog in CEF format? You will note that the 
CEF sensor uses the CEF parser, which means unless your logs are in CEF format, 
they will fail to parse and be dropped into the error index (worth checking the 
error index in kibana via the Metron Error Dashboard. That will likely tell you 
why things aren’t parsing. 

The most likely scenario is that you are sending something non-CEF on the 
syslog feed, in which case you will need something like a Grok parser. I 
suggest reading through the Squid example in the documentation on how to do 
this. 

Simon

> On 4 Jan 2018, at 18:49, Gaurav Bapat  wrote:
> 
> They are syslogs and my topic name is cef, I get one parsed logs out of 1000+ 
> and I want to do analytics using Spark but I cant find a way out.

Re: Metron Rest Kerberos -- Kafka topic ACL

[Simon Elliston Ball]

The ansible roles and playbooks included with Metron install Ambari to handle 
the setup of the Metron and the Hadoop, Kafka etc. components, so yes. 

> On 10 Jan 2018, at 03:18, varsha mordi  wrote:
> 
> Can Ambari UI work with Ansible?
> 
> On Wed, Jan 10, 2018 at 3:46 PM, Mohan Venkateshaiah 
> mailto:mvenkatesha...@hortonworks.com>> 
> wrote:
> Srikanth,
> 
>  
> 
> There is no way you can list all topics to particular user, there is PR for 
> adding REST endpoints to provide required ACL to topic.
> 
>  
> 
> Thanks
> 
> Mohan DV
> 
>  
> 
> From: prakash r mailto:rprakashd...@gmail.com>>
> Reply-To: "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Date: Wednesday, January 10, 2018 at 7:50 AM
> To: "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Subject: Metron Rest Kerberos -- Kafka topic ACL
> 
>  
> 
> Hello,
> 
>  
> 
> We have kerberosed Hadoop Cluster.
> 
>  
> 
> Metron is trying to access all the Kafka topics (ir-respective of Kafka 
> topics which needed for Metron)
> 
>  
> 
> Since it does not have access to all topics, in UI Kafka related infos are 
> not displayed.
> 
>  
> 
> For Ex :
> 
>  
> 
> Kafka has some other topics like (checking123) Metron need authorization for 
> those topic as well.
> 
>  
> 
> 2018-01-10T11:17:39.576 DEBUG 
> [org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor]
>  - Written [{timestamp=Wed Jan 10 11:17:39 AEDT 2018, status=500, 
> error=Internal Server Error, 
> exception=org.apache.kafka.common.errors.TopicAuthorizationException, 
> message=Not authorized to access topics: [checking123], 
> path=/api/v1/kafka/topic/snort}] as "application/json" using 
> [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@ab327c]
> 
>  
> 
>  
> 
> Can Metron Rest restrict access only to those topics which is needed for the 
> same, thanks
> 
> 
> Regards,
> 
> Prakash R
> 
> 
> 
> 
> -- 
> Thanks & Regards,
> Varsha Mordi
> Prodevans Technologies LLP.
> M: +91 9637109734  | L: +91 80 64533365 | www.prodevans.com 
> 
> 

Re: Metron Reference Application (Profiling Your Streams Fails)

[Simon Elliston Ball]

Looks like a docs typo on the wiki: 

What you need is CONFIG_PUT(“PROFILER”, profilerConfig)

Simon

> On 15 Jan 2018, at 10:45, Farrukh Naveed Anjum  
> wrote:
> 
> Can you help on this ?
> 
> On Mon, Jan 15, 2018 at 3:42 PM, Farrukh Naveed Anjum 
>  wrote:
> Any Idea of getting ride of this problem ?
> 
> On Mon, Jan 15, 2018 at 3:38 PM, Farrukh Naveed Anjum 
>  wrote:
> Hi,
> 
> I am trying to setup up Metron Reference Application (SQUID) Example given on 
> main metron website. 
> 
> I am facing a problem during (Profiling Your Streams ) Config_Put Step
> 
> 
> CONFIG_PUT(profilerConfig)
> 
> It causes to display following error
> 
> [!] No enum constant 
> org.apache.metron.common.configuration.ConfigurationType.{
>   "profiles": [
> {
>   "profile": "squid-miss",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid' and action == 'TCP_MISS'",
>   "update": {
> "m": "STATS_ADD(m, 1)"
>   },
>   "result": "m"
> },
> {
>   "profile": "url-length",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid'",
>   "update": {
> "m": "STATS_ADD(m, LENGTH(url))"
>   },
>   "result": "m"
> }
>   ]
> }
> java.lang.IllegalArgumentException: No enum constant 
> org.apache.metron.common.configuration.ConfigurationType.{
>   "profiles": [
> {
>   "profile": "squid-miss",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid' and action == 'TCP_MISS'",
>   "update": {
> "m": "STATS_ADD(m, 1)"
>   },
>   "result": "m"
> },
> {
>   "profile": "url-length",
>   "foreach": "ip_src_addr",
>   "onlyif": "source.type == 'squid'",
>   "update": {
> "m": "STATS_ADD(m, LENGTH(url))"
>   },
>   "result": "m"
> }
>   ]
> }
> at java.lang.Enum.valueOf(Enum.java:238)
> at 
> org.apache.metron.common.configuration.ConfigurationType.valueOf(ConfigurationType.java:31)
> at 
> org.apache.metron.management.ConfigurationFunctions$ConfigPut.apply(ConfigurationFunctions.java:269)
> at 
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:556)
> at 
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160)
> at 
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
> at 
> org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:287)
> at 
> org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:270)
> at 
> org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:409)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 
> 
> How can I solve this problem
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

At present you can certainly create custom stellar functions in Java. I’m 
guessing however that what you’re looking to do is create a kind of function 
that combines a number of stellar functions to avoid repetition, or to ensure 
consistency of certain parameters for example. Is that what you’re looking for? 
Maybe some sort of syntax to create a named stellar function similar to the way 
we create lambdas? 

Simon

> On 17 Jan 2018, at 07:25, Ali Nazemian  wrote:
> 
> Hi all,
> 
> Is there any way that we can define a function that can be used rather than 
> duplicating a logic multiple times?
> 
> Cheers,
> Ali

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 
<https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
 gives good details on how to add a stellar function.

Stellar will pick up an annotated function on its class path, so to add 
function there is no need to rebuild metron module, but you do need your 
modules on the classpath, and, pending 777, to deal with things like class path 
clash in your dependencies. 

Another idea worth discussion on the dev list is probably the notion of 
defining stellar functions in stellar, which would be a much simpler solution 
than custom java functions if you can already express you logic in stellar. 

Simon

> On 17 Jan 2018, at 10:37, Ali Nazemian  wrote:
> 
> Hi Simon,
> 
> Yes, that is exactly what we are looking for. Is there any example regarding 
> adding a Stellar function in Java? Hopefully, we don't need to rebuild the 
> corresponding modules for this?
> 
> Cheers,
> Ali
> 
> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> At present you can certainly create custom stellar functions in Java. I’m 
> guessing however that what you’re looking to do is create a kind of function 
> that combines a number of stellar functions to avoid repetition, or to ensure 
> consistency of certain parameters for example. Is that what you’re looking 
> for? Maybe some sort of syntax to create a named stellar function similar to 
> the way we create lambdas?
> 
> Simon
> 
> > On 17 Jan 2018, at 07:25, Ali Nazemian  > <mailto:alinazem...@gmail.com>> wrote:
> >
> > Hi all,
> >
> > Is there any way that we can define a function that can be used rather than 
> > duplicating a logic multiple times?
> >
> > Cheers,
> > Ali
> 
> 
> 
> 
> -- 
> A.Nazemian

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

Have you looked at the recent TLSH functions in Stellar? We already have that 
for similarity preserving hashes.

Simon

> On 17 Jan 2018, at 12:35, Ali Nazemian  wrote:
> 
> It is a bit complex. We want to create a function that accepts a list of 
> arguments for an asset and generate an asset identifier that can be used as a 
> row_key for the enrichment store. The logic would be complicated, though. We 
> may need to include some sort of similarity aware hash function as a part of 
> this custom function.
> 
>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen  wrote:
>> Ali - Can you describe the logic that you are trying to perform? That would 
>> be useful as a use case to help drive a discussion around creating named 
>> functions in Stellar.
>> 
>> 
>> 
>> 
>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian  wrote:
>>> Thanks, Simon. We have already got a script to deal with classpath 
>>> management for the parsers. We should be able to use it for this extension 
>>> as well.
>>> 
>>> Yeah, I agree. It will be much easier to define functions on the fly and 
>>> use them afterwards. It could be defined as Lambda or custom function. 
>>> 
>>> Regards,
>>> Ali
>>> 
>>> 
>>> 
>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball 
>>>>  wrote:
>>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>>>>  gives good details on how to add a stellar function.
>>>> 
>>>> Stellar will pick up an annotated function on its class path, so to add 
>>>> function there is no need to rebuild metron module, but you do need your 
>>>> modules on the classpath, and, pending 777, to deal with things like class 
>>>> path clash in your dependencies. 
>>>> 
>>>> Another idea worth discussion on the dev list is probably the notion of 
>>>> defining stellar functions in stellar, which would be a much simpler 
>>>> solution than custom java functions if you can already express you logic 
>>>> in stellar. 
>>>> 
>>>> Simon
>>>> 
>>>> 
>>>>> On 17 Jan 2018, at 10:37, Ali Nazemian  wrote:
>>>>> 
>>>>> Hi Simon,
>>>>> 
>>>>> Yes, that is exactly what we are looking for. Is there any example 
>>>>> regarding adding a Stellar function in Java? Hopefully, we don't need to 
>>>>> rebuild the corresponding modules for this?
>>>>> 
>>>>> Cheers,
>>>>> Ali
>>>>> 
>>>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball 
>>>>>>  wrote:
>>>>>> At present you can certainly create custom stellar functions in Java. 
>>>>>> I’m guessing however that what you’re looking to do is create a kind of 
>>>>>> function that combines a number of stellar functions to avoid 
>>>>>> repetition, or to ensure consistency of certain parameters for example. 
>>>>>> Is that what you’re looking for? Maybe some sort of syntax to create a 
>>>>>> named stellar function similar to the way we create lambdas?
>>>>>> 
>>>>>> Simon
>>>>>> 
>>>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian  wrote:
>>>>>> >
>>>>>> > Hi all,
>>>>>> >
>>>>>> > Is there any way that we can define a function that can be used rather 
>>>>>> > than duplicating a logic multiple times?
>>>>>> >
>>>>>> > Cheers,
>>>>>> > Ali
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> A.Nazemian
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> A.Nazemian
> 
> 
> 
> -- 
> A.Nazemian

Re: SysLog using CEF Parser (RSysLogs)

[Simon Elliston Ball]

Are there any errors in the logs for the indexing bolt? I would expect the 
errors are probably at the elastic ingest point, and probably caused by an 
incorrect elastic template for the CEF data. 

Simon

> On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum  
> wrote:
> 
> Yes its Strom Indexing Bolt that is halting it. Any one working on CEF Parser 
> (Can Syslog work with it like RSyslog). We are stuck at that point.
> 
> Please see the above error and suggest
> 
> On Mon, Jan 22, 2018 at 1:10 PM, Gaurav Bapat  > wrote:
> Hi,
> 
> Even I am stuck with the same, and dont know how to solve the issue.
> 
> Looks like this is a parsing error
> 
> On 22 January 2018 at 13:00, Farrukh Naveed Anjum  > wrote:
> Hi,
> 
> I am trying to Ingest syslog using CEF Parser it is not creating any Elastic 
> Search Index based on. 
> 
> Any suggestion how can I achieve it ?
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Some Metron Alerts UI questions

[Simon Elliston Ball]

Hi Laurens, 

A few quick answers inline…

Simon

> On 20 Jan 2018, at 00:37, Laurens Vets  wrote:
> 
> Hi list,
> 
> I have some general Alerts UI questions/comments/remarks, I hope you don't 
> mind :) I'm using the UI that's part of Metron 0.4.2. These apply to my 
> specific use case, so I might be completely wrong in how I use the UI…

Comment and feedback are always welcome!

> 
> - When you're talking about 'alerts', from what I can see in the UI, that's 
> synonymous with just events in elasticsearch right? Wouldn't it make more 
> sense to treat alerts as events where "is_alert" == True?
> 

At present the search does not exclude non-alerts… it’s maybe a little odd to 
call it the alerts view right now, but right now it’s the only way to see 
everything, so this should probably separate out into an ‘everything’ hunting 
focused view and a alerts only view.

The reasons I kinda like the current approach is that it’s good for picking up 
things that have become alerts because they’re in threat intel for example, 
along with things clustered against them by something like the new TLSH 
functions, which makes it easier to combine known alerts with un-detected 
events in a meta alert.

> - It seems that everything I do in the UI is only stored locally? See 
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts. 
> Can this made persistent for multiple people?

Yep. A lot of the preferences, saved searched, column layouts etc, are stored 
in local storage by the browser right now. We need a REST endpoint and to 
figure out how to store them (against user / against a group / global??? 
thoughts?) server side. A lot of the mechanism to do that is in, it’s just not 
quite done done because of those open questions I expect. 

> 
> - How can I change the content "Filters" on the left of the UI?

You wait for https://github.com/apache/metron/pull/853 
 to land. 

> 
> - How do I create a MetaAlert?

You can create a meta-alert from a grouped set of alerts, use the grouping 
buttons at the top and you’ll find a merge alert. Slightly odd process at the 
moment true, but a button to create a meta-alert from all the selected, or all 
the visible alerts on the results page might be a good addition, what do you 
think?

Very quick video of the current method here: https://youtu.be/JkFeNKTOd38

> 
> - What's the plan regarding notifying someone when alerts triggers?

Currently there is no external notification, but the answer here would likely 
be to consume the indexing topic in kafka and integrate to an enterprise alarm 
or monitoring system (alerting and alarms is a massive topic which probably 
deserves its own project beyond metron and I’ve seen people use all sorts of 
things for this, usually some big enterprisey thing mandated by IT).

Re: Indexing Bolt Error

[Simon Elliston Ball]

Yes, configure your indexing. 
https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html
 


Note it’s a warning, not an error, that default values are being used because 
you do not have a specific configuration entry for the snort indexing.

Simon

> On 24 Jan 2018, at 08:31, Farrukh Naveed Anjum  
> wrote:
> 
> Any Idea how to fix this up ?
> 
> On Wed, Jan 24, 2018 at 1:27 PM, Farrukh Naveed Anjum 
> mailto:anjum.farr...@gmail.com>> wrote:
> Hi,
> 
> I am getting this error while starting up squid parsing again. Upon 
> restarting Apache Strom Indexing Bolt is showing up this error any idea how 
> can I fix this ?
> 
> java.lang.Exception: WARNING: Default and (likely) unoptimized writer config 
> used for hdfs writer and sensor snort
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at 
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
>  [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> 
> -- 
> With Regards
> Farrukh Naveed Anjum
> 
> 
> 
> -- 
> With Regards
> Farrukh Naveed Anjum

Re: Metron User Community Meeting Call

[Simon Elliston Ball]

This is going to be a really exciting call. Looking forward to seeing how the 
GCR Canary sings :) 

I’m going to volunteer https://hortonworks.zoom.us/my/simonellistonball as a 
location for the meeting.

I would also support the idea of a quick poll on what people are doing with 
Metron, and maybe if anyone wants to volunteer at the end of the meeting it 
would be great to have an open mic of use cases. 

Talk to you all Wednesday. 

Simon

> On 26 Jan 2018, at 22:10, Seal, Steve  wrote:
> 
> HI all,
>  
> I have several people on my team that are looking forward to hearing about 
> Ahmed’s work. 
>  
> Steve
>  
>  
> From: Daniel Schafer [mailto:daniel.scha...@sstech.us] 
> Sent: Friday, January 26, 2018 5:05 PM
> To: user@metron.apache.org; d...@metron.apache.org
> Subject: Re: Metron User Community Meeting Call
>  
> My team members and me would like to join as well.
> We can provide Zoom Meeting login if necessary.
>  
> Thanks
>  
> Daniel
> 7134806608 
>  
> From: Ahmed Shah  >
> Reply-To: "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Date: Friday, January 26, 2018 at 2:06 PM
> To: "d...@metron.apache.org " 
> mailto:d...@metron.apache.org>>, 
> "user@metron.apache.org " 
> mailto:user@metron.apache.org>>
> Subject: Re: Metron User Community Meeting Call
>  
> Looking forward to presenting!
>  
> Just a thought...
> In advanced should we create a Google Forms to collect survey data on who is 
> using Metron, how they are using it, ext.. and present the results to the 
> group? 
>  
> -Ahmed
> ___
> Ahmed Shah (PMP, M. Eng.)
> Cybersecurity Analyst & Developer 
> GCR - Cybersecurity Operations Center
> Carleton University - cugcr.com 
> 
>  
> 
> From: Andrew Psaltis  >
> Sent: January 26, 2018 1:53 PM
> To: d...@metron.apache.org 
> Subject: Re: Metron User Community Meeting Call
>  
> Count me in. Very interested to hear about Ahmed's journey.
> 
> On Fri, Jan 26, 2018 at 8:58 AM, Kyle Richardson  >
> wrote:
> 
> > Thanks! I'll be there. Excited to hear Ahmed's successes and challenges.
> >
> > -Kyle
> >
> > On Thu, Jan 25, 2018 at 7:44 PM zeo...@gmail.com  
> > mailto:zeo...@gmail.com>> wrote:
> >
> > > Thanks Otto, I'm in to attend at that time/place.
> > >
> > > Jon
> > >
> > > On Thu, Jan 25, 2018, 14:45 Otto Fowler  > > > wrote:
> > >
> > >> I would like to propose a Metron user community meeting. I propose that
> > >> we set the meeting next week, and will throw out Wednesday, January
> > 31st at
> > >> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This
> > meeting
> > >> will be held over a web-ex, the details of which will be included in the
> > >> actual meeting notice.
> > >> Topics
> > >>
> > >> We have a volunteer for a community member presentation:
> > >>
> > >> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
> > >> Cybersecurity Operations Center Carleton University - cugcr.com 
> > >> 
> > >>
> > >> Ahmed would like to talk to the community about
> > >>
> > >>-
> > >>
> > >>Who the GCR group is
> > >>-
> > >>
> > >>How they use Metron 0.4.1
> > >>-
> > >>
> > >>Walk through their dashboards, UI management screen, nifi
> > >>-
> > >>
> > >>Challenges we faced up until now
> > >>
> > >> I would like to thank Ahmed for stepping forward for this meeting.
> > >>
> > >> If you have something you would like to present or talk about please
> > >> reply here! Maybe we can have people ask for “A better explanation of
> > >> feature X” type things?
> > >> Metron User Community Meetings
> > >>
> > >> User Community Meetings are a means for realtime discussion of
> > >> experiences with Apache Metron, or demonstration of how the community is
> > >> using or will be using Apache Metron.
> > >>
> > >> These meetings are geared towards:
> > >>
> > >>-
> > >>
> > >>Demonstrations and knowledge sharing as opposed to technical
> > >>discussion or implementation details from members of the Apache
> > Metron
> > >>Community
> > >>-
> > >>
> > >>Existing Feature demonstrations
> > >>-
> > >>
> > >>Proposed Feature demonstrations
> > >>-
> > >>
> > >>Com

Re: HBase enrichment vs Stellar enrichment for HBase look up

[Simon Elliston Ball]

There shouldn’t be. Both run through the same kind of bolt-side caching, so you 
should be able to use the Stellar version, and in fact that’s the general 
direction the project is heading. We haven’t quite deprecated the plain HBase 
Bolt… but Stellar is definitely the preferred option. 

Simon

> On 2 Feb 2018, at 07:10, Ali Nazemian  wrote:
> 
> Hi All,
> 
> Is there any performance difference between HBase enrichment and Stellar 
> enrichment? We have an HBase enrichment that we need to have a customised key 
> for it. HBase enrichment doesn't give us the full flexibility of using any 
> logic for a Key generation, so I was wondering whether there will be any 
> performance difference if we try to proceed with Stellar enrichment and look 
> up HBase based on our logic or not?
> 
> Regards,
> Ali

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

Shouldn’t be. The one this I would point out though is that you don’t 
necessarily know which supervisor you will be running from, so pulling from 
HDFS would make sense. That said, the performance implications are probably not 
great. A good option here would be to have the config available in the global 
config for example and refer to that, since most instances of stellar apply 
global config to their context. 

Simon


> On 2 Feb 2018, at 07:14, Ali Nazemian  wrote:
> 
> Will be any problem if the Stellar function we want to implement need to load 
> an external config file?
> 
> Cheers,
> Ali
> 
> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian  <mailto:alinazem...@gmail.com>> wrote:
> Thanks, All.
> 
> Yes, Nick. It is highly related to our use case and the way that we are going 
> to enrich events with assets and vulnerability properties. It is not a 
> general case at all.
> 
> Cheers,
> Ali
> 
> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley  <mailto:ma...@apache.org>> wrote:
> Besides the example code Simon mentioned at 
> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>  
> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>  ,
> there is some documentation at 
> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>  
> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>  
> 
> From: Nick Allen mailto:n...@nickallen.org>>
> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
> mailto:user@metron.apache.org>>
> Date: Wednesday, January 17, 2018 at 4:46 AM
> To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
> mailto:user@metron.apache.org>>
> Subject: Re: Define a function that can be used in Stellar
> 
>  
> 
>  
> 
>  
> 
> If something we have already does not fit the bill, I would recommend 
> creating that function in Java.   Since you described it as "a bit complex" 
> and "the logic would be complicated" I don't see any value in defining 
> something like this in Stellar with named functions.
> 
>  
> 
> Best
> 
>  
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> 
> Have you looked at the recent TLSH functions in Stellar? We already have that 
> for similarity preserving hashes.
> 
>  
> 
> Simon
> 
>  
> 
> 
> On 17 Jan 2018, at 12:35, Ali Nazemian  <mailto:alinazem...@gmail.com>> wrote:
> 
> It is a bit complex. We want to create a function that accepts a list of 
> arguments for an asset and generate an asset identifier that can be used as a 
> row_key for the enrichment store. The logic would be complicated, though. We 
> may need to include some sort of similarity aware hash function as a part of 
> this custom function.
> 
>  
> 
> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen  <mailto:n...@nickallen.org>> wrote:
> 
> Ali - Can you describe the logic that you are trying to perform? That would 
> be useful as a use case to help drive a discussion around creating named 
> functions in Stellar.
> 
>  
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian  <mailto:alinazem...@gmail.com>> wrote:
> 
> Thanks, Simon. We have already got a script to deal with classpath management 
> for the parsers. We should be able to use it for this extension as well.
> 
>  
> 
> Yeah, I agree. It will be much easier to define functions on the fly and use 
> them afterwards. It could be defined as Lambda or custom function. 
> 
>  
> 
> Regards,
> 
> Ali
> 
>  
> 
>  
> 
>  
> 
> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball 
> mailto:si...@simonellistonball.com>> wrote:
> 
> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>  
> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>  gives good details on how to add a stellar function.
> 
>  
> 
> Stellar will pick up an annotated function on its class path, so to add 
> function there is no need to rebuild metron module, but you do need your 
> modules on the classpath, and, pending 777, to deal with things like class 
> path clash in your dependencies. 
> 
>  
> 
> Another idea worth discussion on the dev list is probably the notion of 
> defining stellar functions in stellar, which would be a much simpler solution 
> than custom java functions if 

Re: Apache Metron functions implementation

[Simon Elliston Ball]

Hi Helder, 

It is very much possible, and very easy to create your own functions and models 
on top of Metron. 

There are two main ways in which you would do this, depending on the type of 
use case you’re looking at. 

Metron uses a language called Stellar as part of the enrichment stage (and 
elsewhere) to implement a number of algorithms which can then be composed in 
configuration. You can also extend this language to implement your own 
algorithms in the real time stream 
(https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 

 gives a toy example, also checkout some of the source for the more interesting 
stellar functions in 
https://github.com/apache/metron/tree/master/metron-analytics/metron-statistics 
.
 

If your algorithms tend more towards the traditional ML approach, using for 
example Spark, python, or R, then the Model as a Service extension points might 
be more useful. This allows you to run arbitrary micro-service type model 
inference, or scoring, and plug that into he Metron real-time stream 
(https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service
 

 provides more information and a worked example of how you would plug in an 
example python based model).  

I would also suggest taking a look at some of the recent custom use-cases we 
have included in the project to get some starters: 
https://github.com/apache/metron/tree/master/use-cases 
.

I hope that helps, and wish you the best of luck with your project. Also, do 
let the community know what you’re working on, and I’m sure we will be more 
than happy to provide any help and assistance we can. Looking forward to seeing 
what you come up with, and welcome to Metron. 

Simon

> On 2 Feb 2018, at 12:11, Helder Reia  wrote:
> 
> Hello,
> I am a student currently finishing my master degree and for my final work I 
> am proposing to make a security analytics tool. I will want to make it on 
> Apache Metron framework but I have some questions:
> - Is it possible to implement my own functions ? ( I will want to have 
> clustering and classification algorithms )
> - If so, can you give me helpon how to implement those algorithms?
> 
> Thank you for the help !
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

Depends how you write the function class, but most likely, yes. Hence global 
config option. 

Simon

> On 2 Feb 2018, at 13:42, Ali Nazemian  wrote:
> 
> Does it mean every time the function gets called it will load the config, but 
> if I use the global one it will only read it one time and it will be 
> available in memory?
> 
> On 2 Feb. 2018 21:53, "Simon Elliston Ball"  <mailto:si...@simonellistonball.com>> wrote:
> Shouldn’t be. The one this I would point out though is that you don’t 
> necessarily know which supervisor you will be running from, so pulling from 
> HDFS would make sense. That said, the performance implications are probably 
> not great. A good option here would be to have the config available in the 
> global config for example and refer to that, since most instances of stellar 
> apply global config to their context. 
> 
> Simon
> 
> 
>> On 2 Feb 2018, at 07:14, Ali Nazemian > <mailto:alinazem...@gmail.com>> wrote:
>> 
>> Will be any problem if the Stellar function we want to implement need to 
>> load an external config file?
>> 
>> Cheers,
>> Ali
>> 
>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian > <mailto:alinazem...@gmail.com>> wrote:
>> Thanks, All.
>> 
>> Yes, Nick. It is highly related to our use case and the way that we are 
>> going to enrich events with assets and vulnerability properties. It is not a 
>> general case at all.
>> 
>> Cheers,
>> Ali
>> 
>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley > <mailto:ma...@apache.org>> wrote:
>> Besides the example code Simon mentioned at 
>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>>  
>> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>>  ,
>> there is some documentation at 
>> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>>  
>> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>>  
>> 
>> From: Nick Allen mailto:n...@nickallen.org>>
>> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>> mailto:user@metron.apache.org>>
>> Date: Wednesday, January 17, 2018 at 4:46 AM
>> To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>> mailto:user@metron.apache.org>>
>> Subject: Re: Define a function that can be used in Stellar
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> If something we have already does not fit the bill, I would recommend 
>> creating that function in Java.   Since you described it as "a bit complex" 
>> and "the logic would be complicated" I don't see any value in defining 
>> something like this in Stellar with named functions.
>> 
>>  
>> 
>> Best
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball 
>> mailto:si...@simonellistonball.com>> wrote:
>> 
>> Have you looked at the recent TLSH functions in Stellar? We already have 
>> that for similarity preserving hashes.
>> 
>>  
>> 
>> Simon
>> 
>>  
>> 
>> 
>> On 17 Jan 2018, at 12:35, Ali Nazemian > <mailto:alinazem...@gmail.com>> wrote:
>> 
>> It is a bit complex. We want to create a function that accepts a list of 
>> arguments for an asset and generate an asset identifier that can be used as 
>> a row_key for the enrichment store. The logic would be complicated, though. 
>> We may need to include some sort of similarity aware hash function as a part 
>> of this custom function.
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen > <mailto:n...@nickallen.org>> wrote:
>> 
>> Ali - Can you describe the logic that you are trying to perform? That would 
>> be useful as a use case to help drive a discussion around creating named 
>> functions in Stellar.
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian > <mailto:alinazem...@gmail.com>> wrote:
>> 
>> Thanks, Simon. We have already got a script to deal with classpath 
>> management for the parsers. We should be able to use it for this extension 
>> as well.
>> 
>>  
>> 
>> Yeah, I agree. It will be much easier to define functions on the fly and use 
>> them afterwards. It could be de

Re: Define a function that can be used in Stellar

[Simon Elliston Ball]

I forgot we added OBJECT_GET. How does the caching work on that? 

Simn

> On 2 Feb 2018, at 14:33, Nick Allen  wrote:
> 
> There are many functions that use the global configuration.  For example, 
> GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions.  
> There might be a better example, but that is one is staring at me at the 
> moment.
>   
> There is an OBJECT_GET function defined in 
> org.apache.metron.enrichment.stellar.ObjectGet that was purpose-built to 
> retrieve files from HDFS.  If you wanted to retrieve a configuration from 
> HDFS that would be a good example (if you can't just use that functions 
> directly).
> 
> On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian  <mailto:alinazem...@gmail.com>> wrote:
> Is there any Stellar function already been implemented in Metron that has a 
> config file associated with it? I am trying to get an idea of how it works.
> 
> On 3 Feb. 2018 00:44, "Simon Elliston Ball"  <mailto:si...@simonellistonball.com>> wrote:
> Depends how you write the function class, but most likely, yes. Hence global 
> config option. 
> 
> Simon
> 
>> On 2 Feb 2018, at 13:42, Ali Nazemian > <mailto:alinazem...@gmail.com>> wrote:
>> 
>> Does it mean every time the function gets called it will load the config, 
>> but if I use the global one it will only read it one time and it will be 
>> available in memory?
>> 
>> On 2 Feb. 2018 21:53, "Simon Elliston Ball" > <mailto:si...@simonellistonball.com>> wrote:
>> Shouldn’t be. The one this I would point out though is that you don’t 
>> necessarily know which supervisor you will be running from, so pulling from 
>> HDFS would make sense. That said, the performance implications are probably 
>> not great. A good option here would be to have the config available in the 
>> global config for example and refer to that, since most instances of stellar 
>> apply global config to their context. 
>> 
>> Simon
>> 
>> 
>>> On 2 Feb 2018, at 07:14, Ali Nazemian >> <mailto:alinazem...@gmail.com>> wrote:
>>> 
>>> Will be any problem if the Stellar function we want to implement need to 
>>> load an external config file?
>>> 
>>> Cheers,
>>> Ali
>>> 
>>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian >> <mailto:alinazem...@gmail.com>> wrote:
>>> Thanks, All.
>>> 
>>> Yes, Nick. It is highly related to our use case and the way that we are 
>>> going to enrich events with assets and vulnerability properties. It is not 
>>> a general case at all.
>>> 
>>> Cheers,
>>> Ali
>>> 
>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley >> <mailto:ma...@apache.org>> wrote:
>>> Besides the example code Simon mentioned at 
>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
>>>  
>>> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
>>>  ,
>>> there is some documentation at 
>>> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>>>  
>>> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>>>  
>>> 
>>> From: Nick Allen mailto:n...@nickallen.org>>
>>> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>>> mailto:user@metron.apache.org>>
>>> Date: Wednesday, January 17, 2018 at 4:46 AM
>>> To: "user@metron.apache.org <mailto:user@metron.apache.org>" 
>>> mailto:user@metron.apache.org>>
>>> Subject: Re: Define a function that can be used in Stellar
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> If something we have already does not fit the bill, I would recommend 
>>> creating that function in Java.   Since you described it as "a bit complex" 
>>> and "the logic would be complicated" I don't see any value in defining 
>>> something like this in Stellar with named functions.
>>> 
>>>  
>>> 
>>> Best
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball 
>>> mailto:si...@simonellistonball.com>> wrote:
>>> 
>>> Have you looked at the recent TLSH functions in Stellar? We already have 
>>> that f

Re: elasticsearch template question.

[Simon Elliston Ball]

Hi Laurens, 

In Metron all fields tend to get flattened into an un-nested structure of keys 
and values. Some of the keys do represent a flattened tree structure (for 
example our standard enrichment fields). The reason for this is essentially 
ingest speed for nested documents in lucene based indices like Elastic and 
Solr. So, we never used, nor need nested template, and tend to just use the ‘:’ 
separated fields to define the hierarchy.

Is there a particular use case you need the nesting for? 

Simon

> On 7 Feb 2018, at 01:26, Laurens Vets  wrote:
> 
> I hope there's an elasticsearch expert on the mailing list :D
> 
> I have a field called "responseElements:subnets" which can either contain:
> 
> {
>  "subnetIdentifier": "subnet-abcdefgh",
>  "subnetStatus": "Active",
>  "subnetAvailabilityZone": {
>"name": "us-west-2c"
>  }
> },
> {
>  "subnetIdentifier": "subnet-12345678",
>  "subnetStatus": "Active",
>  "subnetAvailabilityZone": {
>"name": "us-west-2b"
>  }
> }
> 
> or:
> 
> subnet-abcdefgh, subnet-12345678, subnet-a1b2c3d4
> 
> Any idea how I can map this in my template?
> 
> For the first case, I got:
> 
> "responseElements:subnets": {
>  "type": "nested",
>  "properties": {
>"subnetIdentifier": { "type": "string"  },
>"subnetStatus": { "type": "string"  },
>"subnetAvailabilityZone": {
>  "type": "nested",
>  "properties": {
>"name": { "type": "string" } } } } }
> 
> But how can I map the 2nd case?

Re: CentOS and Ubuntu

[Simon Elliston Ball]

Not particularly. The centos builds seem to be used by more people on dev, 
probably because they’ve been around for longer, and so are arguably more 
tested. The area where it’s most likely to be relevant is in the install of 
repos for ES and potentially the fastcapa pcap probe (don’t quote me on that 
though, I don’t know if anyone has run that on Ubuntu yet), but other than that 
they’re pretty similar these days. 

Simon

Sent from my iPhone

> On 7 Feb 2018, at 13:00, Helder Reia  wrote:
> 
> Hey everyone!
> I am new to Apache Metron and I don't know much about this! Are there any 
> differences on using CentOS or Ubuntu ? I am used to work with Ubuntu but I 
> can look for CentOS if it is easier to use / has advantages !
> 
> Thank you for your help!
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Stellar post-parsing transformation conditional statement

[Simon Elliston Ball]

You either want a MAP_GET in your IF or a match statement in there I expect. 
See the match statement at 
https://github.com/apache/metron/blob/master/metron-stellar/stellar-common/README.md
 under core functions (it’s relatively new)

Simon 

Sent from my iPhone

> On 9 Feb 2018, at 03:55, Ali Nazemian  wrote:
> 
> Hi All,
> 
> I was wondering how we can address if statement in the config section to have 
> a different mapping in certain conditions. The following syntax is not 
> acceptable.
> 
> {
>   "parserClassName": "org.apache.metron.parsers.asa.BasicAsaParser",
>   "filterClassName": null,
>   "sensorTopic": "test-asa",
>   "writerClassName": null,
>   "errorWriterClassName": null,
>   "invalidWriterClassName": null,
>   "parserConfig": {},
>   "fieldTransformations": [
>   {
>   "input": [],
>   "output": [
>   "x",
>   "y",
>   "z",
>   ],
>   "transformation": "STELLAR",
>   "config": {
> IF "ip_src_port" == 39296 THEN
>   "x": "something",
>   "y": "something else"
> 
> ELSE
>   "y": "something",
>   "z": "something else"
>   }
>   }
>   ]
> }
> 
> Regards,
> Ali

Re: Error when trying to install Apache Metron CentOS7

[Simon Elliston Ball]

To be honest, rather than messing about with grub for this, I would follow the 
alternative route outlines in the wiki page.

To be even more honest, I wouldn’t use that method from the wiki and would 
probably go with something like the full dev VM platform if you’re looking to 
do development work. If you’re looking for production scale, a proper mpack 
based install on an existing Ambari managed cluster (see elsewhere for docs on 
how to set one of these up with a hadoop distribution) is probably better. 

Simon

> On 14 Feb 2018, at 13:07, Helder Reia  wrote:
> 
> Hi all,
> I am following the installation guide for installing Metron 0.4.1 with HDP 
> 2.5 on CentOS 7 and I am having trouble after I change this line and reboot:
> 
> # Change the line:
> GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv =cl/root 
> rd.lvm.lv =cl/swap rhgb quiet"
> # To:
> GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv =cl/root 
> rd.lvm.lv =cl/swap rhgb quiet transparent_hugepage=never"
> # Afterwards, run:
> grub2-mkconfig -o /boot/grub2/grub.cfg
> 
> Basically after I reboot, the OS won't start again, gives a fatal error
> 
> I am installing it on a VM with CentOS7, I don't know if that is the problem, 
> but I wanted to install it here first before I install on some nodes.
> 
> 
> Thank you all for the help,
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Best Metron version for development

[Simon Elliston Ball]

The full dev platform may be the easiest to test things like that on. It can be 
a little brittle if you’re running it in limited RAM, but it also has things 
like the sensor-stubs, which provides an easy means to fake up some input 
traffic. That may be useful for your development and testing. 

Simon

> On 15 Feb 2018, at 18:58, Helder Reia  wrote:
> 
> I want to implement some algorithms, in order to perform cluster and 
> classification over some data, that is why I asked if the better version is 
> the latest one or one of the older ones
> 
> Thank you,
> 
> 2018-02-15 18:53 GMT+00:00 Laurens Vets  >:
> I'm not sure I understand the question completely, but my guess would be the 
> latest release, i.e. 0.4.2?
> 
> On 2018-02-15 10:19, Helder Reia wrote:
> 
>> Hi, I am trying to build a intrusion detection system and I was thinking on 
>> using Apache Metron, but I have a question: which is the best version to 
>> development? I ask this because I will need to implement some 
>> cluster-classify algorithms. Also which guide should I take to install 
>> Apache Metron?
>>  
>> Thank you for your help
>> 
>>  
>> 
>>Sem vírus. www.avast.com 
>> 
> 
> 
> 
> -- 
> Helder Reia
> ALF-AL TM
> 
> 

Re: Metron Indexing Service Failing Shortly After Start

[Simon Elliston Ball]

Anything appearing on the indexing topic in kafka? 

Anything in the logs for the indexing topology in Storm UI? 

Master isn’t always the best place to start, might be worth sticking to a 
released build to kick the tyres. 

Simon

> On 27 Feb 2018, at 17:38, David McGinnis  wrote:
> 
> All,
> 
> I have a kerberized HDP 2.6.3 cluster which I have installed Metron on 
> through an MPack. I am using code straight from the master branch of the 
> github as of last week. 
> 
> When I start Metron, all of the components correct start up. Shortly after, 
> however, the Indexing Service stops working. I have not been able to find any 
> logs that seem to capture any issues, and both indexing storm topologies are 
> still active and running after the failure. Those topologies only report a 
> warning about one of my parsers using the default HDFS writer config, which 
> appears to be normal. 
> 
> Not only is this making my dashboard red when it shouldn't be, but also it 
> seems as if data doesn't go through the pipeline until the entire Metron 
> system is restarted. I am not confident this is related, but I suspect it is. 
> 
> Where should I go to find logs about this service, or does anyone have any 
> ideas why this would be happening?
> 
> -- 
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
>  M: (513) 439-0082
> LinkedIn  | Google+ 
>  | Twitter 
> 
> -
> This message (including any attachments) contains confidential information 
> intended for a specific individual and purpose, and is protected by law. If 
> you are not the intended recipient, you should delete this message. Any 
> disclosure, copying, or distribution of this message, or the taking of any 
> action based on it, is strictly prohibited.

Re: Alerts Not Being Generated?

[Simon Elliston Ball]

Hi David,

One quick thing just in case, is_alert, not is_alarm. 

That said that should not affect what’s in the alerts ui. You should see data 
from your geo source as well (whatever you called it). It is possible there may 
be a problem with your elastic template. You might be interested in 
https://github.com/simonellistonball/metron-field-demos/blob/master/geo/es.json 

 which is based on the use case. Note that there is a field in there:  { alert: 
{ type: nested } } this is necessary for the Alerts UI and specifically the 
meta alerts capability. 

Note that you may also need to reload your alerts ui, and possibly restart the 
REST service to pickup new index types in the alerts ui, there may be issues 
with caching.

Simon


> On 1 Mar 2018, at 15:46, David McGinnis  wrote:
> 
> All,
> 
> I am following the instructions located here for creating a parser which 
> detects user logins distant from their recent logins, and raising alarms: 
> https://github.com/apache/metron/tree/master/use-cases/geographic_login_outliers
>  
> .
>  I have been able to successfully see the data show up in Kibana, including 
> the is_alarm field, which shows true when distant logins are reported, and 
> null or empty otherwise (I believe this is the correct behavior?).
> 
> The issue I'm having is that none of these distant logins are reported in the 
> Alarms UI. I have made the condition the same as the one I'm using for 
> is_alarm, and also used conditions that should always be true, but the only 
> alarms that show up are alarms from some sample Bro data that I can pass 
> through the system and see alerts for. 
> 
> Any ideas for how I can get alarms to show up correctly in the UI, or where 
> else I can check? I am not very familiar with the process of going from 
> enrichments to alerts UI at this point.
> 
> Thanks!
> 
> -- 
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
>  M: (513) 439-0082
> LinkedIn  | Google+ 
>  | Twitter 
> 
> -
> This message (including any attachments) contains confidential information 
> intended for a specific individual and purpose, and is protected by law. If 
> you are not the intended recipient, you should delete this message. Any 
> disclosure, copying, or distribution of this message, or the taking of any 
> action based on it, is strictly prohibited.

Re: How to delete the original message field once the message parsed?

[Simon Elliston Ball]

Very sorry... posted on the wrong thread...

The original string serves purposes well beyond debugging. Many users will
need to be able to prove provenance to the raw logs in order to prove or
prosecute an attack from an internal threat, or provide evidence to law
enforcement or an external threat. As such, the original string is
important.

It also provides a valuable source for the free text search where parsing
has not extracted all the necessary tokens for a hunt use case, so it can
be a valuable field to have in Elastic or Solr for text rather than keyword
indexing.

That said, it may make sense to remove a heavy weight processing and
storage field like this from the lucene store. We have been talking for a
while about filtering some of the data out of the realtime index, and
preserving full copies in the batch index, which could meet the forensic
use cases above, and would make it a matter of user choice. That would
probably be configured through indexing config to filter fields.

Simon


On 25 June 2018 at 23:49, Michel Sumbul  wrote:

> Hi James,
>
> Will it not be interesting, to have an option to remove that field just
> before indexing? This save storage space/Cost in HDFS and ES?
> For example, during development/debugging you keep that field and when
> everything is ready for prod, you check a box to remove that field before
> indexing?
>
> Michel
>
> 2018-06-25 23:37 GMT+01:00 James Sirota :
>
>> Hi Michael, the original_string is there for a reason. It's an immutable
>> field that preserves the original message. While enrichments are added,
>> various parts of the message are parsed out, changed, filtered out,
>> ocncantenated, etc., you can always recover the original message from the
>> original string.
>>
>> Thanks,
>> James
>>
>>
>> 25.06.2018, 15:18, "Michel Sumbul" :
>>
>> Hello,
>>
>> Is there a way to avoid to keep the field "original message", once the
>> message have been parsed?
>> The objectif is to reduce the size of the message to store in HDFS, ES
>> and the traffic between storm/kafka.
>> Currently, we have all the fields + the original message which means that
>> we are going to used 2 time more space to store an information.
>>
>> Thanks for the help,
>> Michel
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>


-- 
--
simon elliston ball
@sireb

Re: How to delete the original message field once the message parsed?

[Simon Elliston Ball]

Agreed. I think of the hdfs batch store as the throw away nothing store, and 
the lucene real-time store as more of an index or cache which does not have to 
be quite so complete, where we could definitely optimise down some of the 
fields.

Simon

> On 26 Jun 2018, at 04:57, Otto Fowler  wrote:
> 
> Also, theoretically, ‘not throwing anything away’ allows future 
> processing/reprocessing of data to gain new insights.  It is not uncommon 
> from the SEIM’s that I’ve seen to store the raw log information for the 
> reasons Simon states for example.
> 
> 
> So all these things that Simon and James have mentioned are true, and are the 
> why from a capabilities perspective.
> 
> That doesn’t invalidate your very practical point Michel however, and it is 
> important to understand field issues as people put Metron into use.  If these 
> features are not being used, or don’t exist yet (replay) can someone not tune
> them down for their scenario with some understanding of the tradeoffs?
> 
> I don’t think there is currently a way to do this, but it is worth having a 
> discussion on the issue.
> 
> 
>> On June 25, 2018 at 20:04:16, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> Very sorry... posted on the wrong thread...
>> 
>> The original string serves purposes well beyond debugging. Many users will 
>> need to be able to prove provenance to the raw logs in order to prove or 
>> prosecute an attack from an internal threat, or provide evidence to law 
>> enforcement or an external threat. As such, the original string is 
>> important. 
>> 
>> It also provides a valuable source for the free text search where parsing 
>> has not extracted all the necessary tokens for a hunt use case, so it can be 
>> a valuable field to have in Elastic or Solr for text rather than keyword 
>> indexing.
>> 
>> That said, it may make sense to remove a heavy weight processing and storage 
>> field like this from the lucene store. We have been talking for a while 
>> about filtering some of the data out of the realtime index, and preserving 
>> full copies in the batch index, which could meet the forensic use cases 
>> above, and would make it a matter of user choice. That would probably be 
>> configured through indexing config to filter fields.
>> 
>> Simon
>> 
>> 
>>> On 25 June 2018 at 23:49, Michel Sumbul  wrote:
>>> Hi James,
>>> 
>>> Will it not be interesting, to have an option to remove that field just 
>>> before indexing? This save storage space/Cost in HDFS and ES?
>>> For example, during development/debugging you keep that field and when 
>>> everything is ready for prod, you check a box to remove that field before 
>>> indexing?
>>> 
>>> Michel
>>> 
>>> 2018-06-25 23:37 GMT+01:00 James Sirota :
>>>> Hi Michael, the original_string is there for a reason. It's an immutable 
>>>> field that preserves the original message. While enrichments are added, 
>>>> various parts of the message are parsed out, changed, filtered out, 
>>>> ocncantenated, etc., you can always recover the original message from the 
>>>> original string.
>>>>  
>>>> Thanks,
>>>> James
>>>> 
>>>> 
>>>> 25.06.2018, 15:18, "Michel Sumbul" :
>>>>> Hello,
>>>>> 
>>>>> Is there a way to avoid to keep the field "original message", once the 
>>>>> message have been parsed?
>>>>> The objectif is to reduce the size of the message to store in HDFS, ES 
>>>>> and the traffic between storm/kafka. 
>>>>> Currently, we have all the fields + the original message which means that 
>>>>> we are going to used 2 time more space to store an information.
>>>>> 
>>>>> Thanks for the help,
>>>>> Michel
>>>> 
>>>> 
>>>> --- 
>>>> Thank you,
>>>>  
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> --
>> simon elliston ball
>> @sireb

Re: CEF Parser not Indexing data via Nifi (SysLogs)

[Simon Elliston Ball]

What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF 
parsing. 

Just use NiFi to do the listen syslog (no need to parse in NiFi) then SplitText 
to get one line of CEF per kafka message (if your syslog is batching, this may 
not be necessary. Set up a sensor in Metron using the CEF parser and you should 
be fine. 

Simon 


> On 20 Jul 2018, at 09:39, Srikanth Nagarajan  wrote:
> 
> Hi Farrukh,
> 
> You can try using the Grok Parser and search for regular expression pattern 
> for your log.  You can customize the regex to meet your needs.   
> 
> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
> 
> Look at Step-5 on how to create a regex for grok parser. Grok parser also 
> allows to validate the fields.
> 
> Good luck !
> 
> Thanks
> Srikanth
> 
>> On July 20, 2018 at 4:23 AM Farrukh Naveed Anjum  
>> wrote: 
>> 
>> Hi,
>> 
>> I am trying to index the Syslog using CEF Parser with Nifi.
>> 
>> It does not give any error though, transport data to kafa without indexing 
>> it. It keepg giving FAILED in Spout.
>> 
>> I believe indexing Syslog are most basic usecase for all. But metron fails 
>> to do it with each in standard format.
>> 
>> I tried bro for it. But even it keeps giving PARSER Error.
>> 
>> Any help ? Fast will be apperciated.
>> 
>> 
>> 
>> 
>> -- 
>> With Regards 
>> Farrukh Naveed Anjum
> 
> __
> 
> Srikanth Nagarajan 
> Principal
> 
> Gandiva Networks Inc
> 
> 732.690.1884 Mobile
> 
> s...@gandivanetworks.com
> 
> www.gandivanetworks.com
> 
> Please consider the environment before printing this. NOTICE: The information 
> contained in this e-mail message is intended for addressee(s) only. If you 
> have received this message in error please notify the sender.

Re: Metron Not Reading From Kafka?

[Simon Elliston Ball]

It might be worth looking in the error and invalid topics, if you have any
validation, or your parser is not producing proper timestamps (that's what
I usually forget to check!) you may be getting messages routed to the error
index. Are you indexing topologies picking any of this up? Is there
anything in the ES error index?

The other thing worth doing is upgrading 0.4.3 is a very very old version
and a lot of changes have come in since then.

Simon

On 17 August 2018 at 16:07, David McGinnis 
wrote:

> All,
>
> We have a Metron 0.4.3 installation on an HDP cluster which has a sensor
> set up to read from a Kafka topic and write the data out to Elasticsearch.
> Data is being inserted into the Kafka topic, and we can read that through
> Kafka console consumer, but the system is not reporting any data coming
> through. The Storm spout says no data has been processed, and the index
> hasn't even been created in Elastic, despite running for nearly a month
> now.
>
> We've searched the worker logs for Storm, and the only error that comes up
> is a (we think) unrelated error about not being able to find the jmxmetrics
> JAR file. Metron reports that the topic is found, and does not tell us that
> the topic is not emitting, so we suspect it sees the data in there.
>
> Do you all have any ideas on where we can look to determine the cause of
> this issue, or things to try?
>
> Thanks!
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
> 
> -
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>



-- 
--
simon elliston ball
@sireb

Re: Metron Not Reading From Kafka?

[Simon Elliston Ball]

That's a pretty out dated version, and sounds like you may have an old
version of HDP if you have the jmx error.

>From the info here, there are a lot of things it could potentially be. I
would suggest working through the stages piece by piece, and checking that
your parsers are producing valid messages (correct timestamp?) and check
what is going through the error and invalid topics.

Also worth turning up the logging in storm for the parser topologies for a
bit (via storm ui) on org.apache.metron.

Simon

On Fri, 17 Aug 2018 at 16:07, David McGinnis 
wrote:

> All,
>
> We have a Metron 0.4.3 installation on an HDP cluster which has a sensor
> set up to read from a Kafka topic and write the data out to Elasticsearch.
> Data is being inserted into the Kafka topic, and we can read that through
> Kafka console consumer, but the system is not reporting any data coming
> through. The Storm spout says no data has been processed, and the index
> hasn't even been created in Elastic, despite running for nearly a month
> now.
>
> We've searched the worker logs for Storm, and the only error that comes up
> is a (we think) unrelated error about not being able to find the jmxmetrics
> JAR file. Metron reports that the topic is found, and does not tell us that
> the topic is not emitting, so we suspect it sees the data in there.
>
> Do you all have any ideas on where we can look to determine the cause of
> this issue, or things to try?
>
> Thanks!
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
>
> -
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>


-- 
--
simon elliston ball
@sireb

Re: Requesting access to slack

[Simon Elliston Ball]

Hi Tarik,

You’re quite right, ES costs can get very high, which is why most Metron users 
store a short term amount of data in ES and use the HDFS store for longer term 
data access.

Most I know of keep 1 to 3 months in elastic, and use HDFS to store data for, 
in some cases, years. This is usually done by deleting older ES indexes via 
curator.

Some people also limit the fields stored in ES through templates, which is 
something we’ve talked about making even more efficient with field level 
filtering in Metron. That helps keep the hot layer storage costs (ES) down. 

Simon

> On 12 Sep 2018, at 20:08, Tarik Courdy  wrote:
> 
> Hello - 
> 
> I was wondering if I could receive an invite to the apache metron slack 
> organization?
> 
> I also wanted to ask a question about supported data stores.  One of the 
> supported data stores is elasticsearch.  Does this mean that all data is 
> stored directly in elasticsearch or is some other approach taken?  If all of 
> the data is stored in ES, I imagine that scaling costs could get out of hand 
> as the size of the data continues to grow.  Is this a valid concern with 
> metron or no?
> 
> Thank you for your time.
> 
> -Tarik

Re: https access to Metron Alert UI

[Simon Elliston Ball]

Metron doesn’t fully support this yet out of the box, but you can hack it up by 
changing the templates for the spring yaml config. 

More commonly, put it behind a reverse proxy for the ssl. There was talk about 
integrating that with Knox for ssl proxying, but that’s on pause now. 

Simon 

Sent from my iPhone

> On 30 Sep 2018, at 02:18, Charles Lo  wrote:
> 
> Hello,
> Is https access to Metron Alert UI (via port 4201) supported?
> If so, how do I configure it?
> Thank you.
>  

Re: Syslog parser issue

[Simon Elliston Ball]

Field transformations allow you to remove fields. That’s probably what you’re 
looking for.

Simon

> On 30 Oct 2018, at 19:03, Muhammed Irshad  wrote:
> 
> Thanks Otto. BasicISEParser worked well. Could you please elaborate more on 
> structured data ? Is it something in header or message field in syslog 
> message in my example ? Just to under stand the working of syslogparser 
> library in detail to extend in future. 
> Also can I filter fields when using BasicISEParser ? I know we can filter 
> message with stellar but can we filter fields ? Like index only interested 
> fields ? 
> 
>> On Tue, Oct 30, 2018 at 11:29 PM Otto Fowler  wrote:
>> Per the spec which this is written to, if you don’t have structured data, 
>> you need to have a ‘-‘ marker.  So this is not valid 5424.  That is from a 
>> cursory look.
>> Metron has a dedicated ISE parser, have you tried that?
>> 
>> If you would like to have the parser have a setting to optionally accept 
>> missing structured data, you can open an issue @ 
>> https://github.com/palindromicity/simple-syslog-5424/issues
>> If/when resolved there, a jira to pick up the change in metron can be logged.
>> 
>> 
>> 
>>> On October 30, 2018 at 13:38:39, Muhammed Irshad (irshadkt@gmail.com) 
>>> wrote:
>>> 
>>> I am trying to test existing Syslog5424Parser with the logs from my 
>>> cisco:ise log data. I am getting the below error message under 
>>> MessageParserResult. Is the below format supported by existing syslog 
>>> parser ? Or can I configure it to support this format ?
>>> 
>>> Message sample :
>>> <182>1 2018-10-05T08:46:06+00:00 lxapp1492-admin.in.mycompany.com  
>>> CISE_Profiler 0038547765 1 0 2018-10-05 18:46:06.972 +10:00 0538115228 
>>> 80002 INFO  Profiler: Profiler EndPoint profiling event occurred, 
>>> ConfigVersionId=267, OperatingSystem=FreeBSD 10.0-CURRENT (accuracy 92%), 
>>> EndpointCertainityMetric=160, EndpointIPAddress=192.168.88.55, 
>>> EndpointMacAddress=F8:0D:60:FF:86:E5, EndpointMatchedPolicy=Canon-Printer,
>>> 
>>> Error message :
>>> com.github.palindromicity.syslog.dsl.ParseException: Syntax error @ 1:93 no 
>>> viable alternative at input '1'
>>> 
>>> --
>>> Muhammed Irshad K T
>>> Senior Software Engineer
>>> +919447946359
>>> irshadkt@gmail.com
>>> Skype : muhammed.irshad.k.t
> 
> 
> -- 
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t

Re: Issue with BasicIseParser

[Simon Elliston Ball]

Sounds like a perfect opportunity to contribute a fix, or a test case for
the broken log types. I would suggest raising a JIRA, and even a PR.

Simon

On Thu, 1 Nov 2018 at 14:35, Muhammed Irshad  wrote:

> Hi ,
>
> Seems string escaping is not handled in built in ISE parser. I am getting
> wired output for some of the log from cisco ise collected via splunk. The
> same issue is there for the test logs as well. PFA input string and output
> json. Same issue is there for the unit test case messages as well.
>
> --
> Muhammed Irshad K T
> Senior Software Engineer
> +919447946359
> irshadkt@gmail.com
> Skype : muhammed.irshad.k.t
>


-- 
--
simon elliston ball
@sireb

Re: Running MAAS in batch

[Simon Elliston Ball]

MaaS is designed to wrap model inference (scoring) an event at a time, via
a REST api. As such, running it batch doesn't make a lot of sense, since
each message would be processed individually. Most of the models you're
likely to run in MaaS however, are also likely to be easily batchable, and
are probable better wrapped up in a batch engine like Spark to take
advantage of more efficient "mass" scoring.

Simon

On Fri, 16 Nov 2018 at 15:18, deepak kumar  wrote:

> Hi All
> Right now MAAS supports running the model against real time events being
> streamed into metron platform.
> Is there any way to run the models deployed in MAAS on the batch events /
> data that have been indexed into hdfs ?
> If anyone have tried this batch model , please share some insights.
> Thanks
> Deepak.
>
>

-- 
--
simon elliston ball
@sireb

Re: Running MAAS in batch

[Simon Elliston Ball]

You model is really just a function that you wrap in a REST service in
order to deploy in MaaS. In the case of something like spark, you would
just wrap it in a udf instead of wrapping it in a REST service, at that
point, applying it in batch is just a case of a simple dataframe query.

On Fri, 16 Nov 2018 at 15:51, deepak kumar  wrote:

> Simon,
> Can you elaborate more on this:
> '
>
> *wrapped up in a batch engine like Spark to takeadvantage of more
> efficient "mass" scoring.*
> '
> How the mass model wrapped in spark  can take advantage of mass scoring?
>
> Thanks
> Deepak
>
> On Fri, Nov 16, 2018 at 9:15 PM Otto Fowler 
> wrote:
>
>> That may be the best MAAS explanation I’ve seen Simon.
>>
>>
>> On November 16, 2018 at 10:28:57, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> MaaS is designed to wrap model inference (scoring) an event at a time,
>> via a REST api. As such, running it batch doesn't make a lot of sense,
>> since each message would be processed individually. Most of the models
>> you're likely to run in MaaS however, are also likely to be easily
>> batchable, and are probable better wrapped up in a batch engine like Spark
>> to take advantage of more efficient "mass" scoring.
>>
>> Simon
>>
>> On Fri, 16 Nov 2018 at 15:18, deepak kumar  wrote:
>>
>>> Hi All
>>> Right now MAAS supports running the model against real time events being
>>> streamed into metron platform.
>>> Is there any way to run the models deployed in MAAS on the batch events
>>> / data that have been indexed into hdfs ?
>>> If anyone have tried this batch model , please share some insights.
>>> Thanks
>>> Deepak.
>>>
>>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>>

-- 
--
simon elliston ball
@sireb

Re: Error deploying Metron 0.3.1 single Node

[Simon Elliston Ball]

Are you looking to install a dev build? If not and you just want to use the 
system, you may be better off with a pre-built distribution. 

Simon 

> On 30 Nov 2018, at 12:48, Babak Abbaschian  wrote:
> 
> It’s two weeks that I’m trying to install metron 0.6.1, but I end up with an 
> error with mpm failing to install some dependencies. 
> From the other side everything in Metron’s documentation is too old, CentOs 
> 6, Ubuntu 14, Ansible 2.0.0.2 etc. And at the same time we need NodeJs 
> 9(accompanied with NPM 5) with NPM 6(accompanies NodeJs 10).
> So I thought instead of this amount of patchwork try the one with all same 
> age components, but it failed as well. :(
> 
> 
> Sent from Yahoo Mail for iPhone
> 
> On Thursday, November 29, 2018, 10:09 PM, Laurens Vets  
> wrote:
> 
> I would suggest to try with a newer version (0.6.0), 0.3.1 is very old.
> 
>> On 2018-11-29 6:20 p.m., Babak Abbaschian wrote:
> 
> Followed this link: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
> With the following info:
> 
> Metron 0.3.1
> --
> * (detached from origin/Metron_0.3.1)
> --
> commit 7abd7e8a231c6cbe9ee4ab23a5df1e97344f5212
> Author: justinleet 
> Date:   Thu Feb 23 10:40:14 2017 -0500
> 
> METRON-734 Builds failing because of MaxMind DB transitive dependency 
> (justi
> --
> --
> ansible 2.0.0.2
>   config file = /etc/ansible/ansible.cfg
>   configured module search path = Default w/o overrides
> --
> Vagrant 2.2.0
> --
> Python 2.7.12
> --
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-10T11:41:47-05:00)
> Maven home: /usr/local/apache-maven/apache-maven-3.3.9
> Java version: 1.8.0_191, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-139-generic", arch: "amd64", family: "unix"
> --
> Linux upctv 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 
> 2018 x86_64 x86_64 x86_64 GNU/Linux
> --
> Total System Memory = 15968.6 MB
> Processor Model: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
> Processor Speed: 3899.902 MHz
> Total Physical Processors: 4
> Total cores: 16
> Disk information:
> /dev/sda3   269G   33G  223G  13% /
> /dev/sda1   659G  453G  206G  69% /media/ubuntu/DropBox
> **
> 
> And got this error: 
> 
> TASK [kibana : Install the Metron Dashboard] 
> ***
> fatal: [node1]: FAILED! => {"changed": true, "cmd": "elasticdump 
> --output=http://node1:9200/.kibana --input=/tmp/kibana-index.json", "delta": 
> "0:00:00.069235", "end": "2018-11-30 01:39:46.092021", "failed": true, "rc": 
> 1, "start": "2018-11-30 01:39:46.022786", "stderr": 
> "/usr/lib/node_modules/elasticdump/elasticdump.js:3\nconst {EventEmitter} = 
> require('events')\n  ^\n\nSyntaxError: Unexpected token {\nat 
> exports.runInThisContext (vm.js:53:16)\nat Module._compile 
> (module.js:373:25)\nat Object.Module._extensions..js (module.js:416:10)\n 
>at Module.load (module.js:343:32)\nat Function.Module._load 
> (module.js:300:12)\nat Module.require (module.js:353:17)\nat require 
> (internal/module.js:12:17)\nat Object. 
> (/usr/lib/node_modules/elasticdump/bin/elasticdump:6:19)\nat 
> Module._compile (module.js:409:26)\nat Object.Module._extensions..js 
> (module.js:416:10)", "stdout": "", "stdout_lines": [], "warnings": []}
> 
> 
> 
> 
> 
> 

Re: Raw Message Strategy "Envelope"

[Simon Elliston Ball]

The envelope strategy controls how the parser views it's incoming data. In
other words, if the incoming data is json, should it treat one field as is
it were the original message, or treat the whole json as original.

To your example, the syslog parser would produce JSON, probably with a
bunch of syslog header fields and a field called something like "message".
(Note that these syslog wrappers may be useful or significant so you
probably don't want to just throw them away. This means:

A: (syslog wrapped) -> JSON (with message fields) -> Parser (P) for /This
is the .*message/
B: (raw) -> Parser (P) for /This is the .*message/

So you will need a syslog kafka topic, a syslog_parsed (json) kafka topic,
and a raw_message topic, along with two copies of P, one with envelope, one
without.

A better answer would be
A: (syslog wrapped) -> JSON (with message fields) -
B: (raw) ->  NoOp parser (e.g. Grok GREEDYDATA:message) which wraps into
metron JSON (with message fields)

now both those outputs can go into the same logical input topic for the
common envelope strategy parser:
Parser (P) for /This is the .*message/

It may seem counter intuitive to wrap the raw with an extra parser, but
this means you will end up with one Parser P set to anchor things like
enrichment and indexing config off further down the line, instead of two.

Simon




On Mon, 3 Dec 2018 at 18:32, Stefan Kupstaitis-Dunkler 
wrote:

> Hi,
>
> just out of interest: what is/should be the expected behaviour of the raw
> message strategy "ENVELOPE"?
>
>
>- Should a parser with this strategy only accept message that were
>already pre-processed by another parser?
>- Or should parser like this accept both? Direct ingests as well as
>ingests that are chained from a previous parser?
>
>
> Imagine you have 2 different log sources. One adds a syslog header the
> other doesn't.
>
> Example message from source 1: "<86>Dec 3 18:25:10 my.hostname.com This
> is the message"
> Example message from source 2: "This is the other message".
>
> Assumption is, that both "This is the message" and "This is the other
> message" can be parsed using the same pattern.
>
> Would I/Should I need to use 3 Kafka topics  (1 for the syslog parser, 1
> for the chained parser and another identical for the direct ingestion) or 2
> Kafka topics (1 for the syslog parser, 1 for both, the enveloped/chained
> source and the "default" source).
>
> Appreciate your thoughts and comments.
>
> Best,
> Stefan
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> https://www.meetup.com/Hadoop-User-Group-Vienna/
> https://twitter.com/StefanDunkler
>


-- 
--
simon elliston ball
@sireb