date:20171023

Re: multiple pattern grok parser in 1 file

2017-10-23 Thread Simon Elliston Ball

My bad, the pattern surpasses names of capture groups.

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}

AUTHLOG (%{AUTHLOG1}|%{AUTHLOG2})

should work… though to be honest, your patterns look a little unusual. You seem 
to have logs with a timestamp in epoch at the front, which is a very weird way 
to setup syslog, so the issue might be that your patterns flat out don’t match 
the logs. 

Simon


> On 23 Oct 2017, at 10:36, tkg_cangkul  wrote:
> 
> Hi Simon,
> 
> I've tried your suggestion but i have an error msg like below :
> 
> 
> 
> On 23/10/17 16:22, Simon Elliston Ball wrote:
>> That is not valid grok. Pattern names should be unique in the grok. 
>> 
>> What you probably mean is something like:
>> 
>> AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>> AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>> %{USERNAME:username}
>> AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})
>> 
>> Simon
>> 
>> 
>>> On 23 Oct 2017, at 08:53, tkg_cangkul 
>>>  wrote:
>>> 
>>> FYI,
>>> 
>>> i've trying to using Grok parser metron with multiple pattern in single 
>>> file but it doesn't work. this is my sample grok pattern on 
>>> /apps/metron/patterns/authlog :
>>> 
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
>>> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
>>> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
>>> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
>>> %{USERNAME:username}
>>> 
>>> When the sensor started, the second grok pattern doesn't work. Only first 
>>> pattern works.
>>> There is an error message like this on storm logs:
>>> 
>>> Caused by: java.lang.RuntimeException: Grok statement produced a null 
>>> message.
>>> 
>>> 
>>> On 23/10/17 10:49, tkg_cangkul wrote:
>>> 
 Hi Wasim, 
 
 thx for your reply.
 So it means i should use logstash parser for metron?
 Is there any documentation about use logstash parser for metron?
 I didn't found any documentation about that on metron. 
 i just find logstash basic parser but there is no documentation about that.
 
 
 
 On 23/10/17 10:33, Wasim Halani wrote:
 
> Hi Youzha,
> 
> It should be possible to add multiple patterns in a single config file. 
> For reference, you can check out the use of multiple patterns in a repo I 
> maintain [1].
> You would find the patterns in [2] useful for your use-case.
> 
> However, do note that there is a cost to every grok failure [3] - so you 
> need to ensure that your most common event patterns are at the top of the 
> list.
> 
> As a side-note, if you have any logstash parsers which are not available 
> in the repo, please feel to submit a PR to [4] 
> 
> 
> [1] 
> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
> 
> [2] 
> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
> 
> [3] 
> https://www.elastic.co/blog/do-you-grok-grok
> 
> [4] 
> https://bitbucket.org/networkintelligence/logstash-configs/
> 
> 
> Regards,
> ---
> Wasim Halani
> 
> http://twitter.com/washalsec
> http://securitythoughts.wordpress.com
> 
> --
> To keep silent when you can say something wise and useful is as bad as 
> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
> 
> On Mon, Oct 23, 2017 at 8:08 AM, Youzha 
> 
>  wrote:
> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
> file?
> i’m trying to parsing authlog file in /var/log/secure into metron. the 
> problem is there are different structures of logs inside /var/log/secure. 
> any suggest for this pls?
> 
> 
> Best Regards,
> 
> 
> 
>

Re: Snort Installation

2017-10-23 Thread Syed Hammad Tahir

yes nut I am a bit confused here. Let me ask them as well then.

On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com  wrote:

> Hi Syed,
>
> Just to clarify, this a snort issue you are having?  If so I suggest
> looking at their documentation (https://snort.org/documents) or reaching
> out to their community (https://snort.org/community), as they have more
> expertise in this area.
>
> Jon
>
> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
> wrote:
>
>> Hi guys,
>>
>> I tried to add another network interface in order to bridge it to LAN. I
>> tried to do it on virtualbox vm settings and when i did vagrant up after
>> that, there was no bridged interface. Can anyone help me on this?
>>
>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir > > wrote:
>>
>>> Ok, thankyou. I will let you know once  I make snort sniff the traffic
>>> in the given configuration, might be helpful for others. I will then try to
>>> do that kafka topic and will ask if any help is needed.
>>>
>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>>>
 Hi Syed,

 See inline.

 On 2017-10-20 00:32, Syed Hammad Tahir wrote:

> I have installed the snort manually. Now I need help with :
>
> 1- Capturing the data of my lan and dumping it via snort :Snort cant
> see the traffic outside vagrant vm, how do I make it see that traffic?
>

 To be honest, configuring Snort to work on your LAN is out of scope of
 the project. Have a look at the documentation at https://www.snort.org/
 .
 You will probably have to add a 2nd network interface bridged to your
 LAN in promiscuous mode. Additionally, I think most of us expect some basic
 Linux & network administration knowledge when using Metron.

 2- Making a kafka topic to push those saved logs in metron for
> preprocessing
>

 Have a look at the Metron documentation at https://metron.apache.org/
 current-book/index.html. Adding a new sensor in the Metron UI will
 create the Kafka iirc.

 3- Applying a basic Machine learning algorithm on the captured data.
>

 I can't help you with this :)

>>>
>>>
>> --
>
> Jon
>

Re: Snort Installation

2017-10-23 Thread zeo...@gmail.com

Hi Syed,

Just to clarify, this a snort issue you are having?  If so I suggest
looking at their documentation (https://snort.org/documents) or reaching
out to their community (https://snort.org/community), as they have more
expertise in this area.

Jon

On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir  wrote:

> Hi guys,
>
> I tried to add another network interface in order to bridge it to LAN. I
> tried to do it on virtualbox vm settings and when i did vagrant up after
> that, there was no bridged interface. Can anyone help me on this?
>
> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir 
> wrote:
>
>> Ok, thankyou. I will let you know once  I make snort sniff the traffic in
>> the given configuration, might be helpful for others. I will then try to do
>> that kafka topic and will ask if any help is needed.
>>
>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>>
>>> Hi Syed,
>>>
>>> See inline.
>>>
>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>>
 I have installed the snort manually. Now I need help with :

 1- Capturing the data of my lan and dumping it via snort :Snort cant
 see the traffic outside vagrant vm, how do I make it see that traffic?

>>>
>>> To be honest, configuring Snort to work on your LAN is out of scope of
>>> the project. Have a look at the documentation at https://www.snort.org/.
>>> You will probably have to add a 2nd network interface bridged to your
>>> LAN in promiscuous mode. Additionally, I think most of us expect some basic
>>> Linux & network administration knowledge when using Metron.
>>>
>>> 2- Making a kafka topic to push those saved logs in metron for
 preprocessing

>>>
>>> Have a look at the Metron documentation at
>>> https://metron.apache.org/current-book/index.html. Adding a new sensor
>>> in the Metron UI will create the Kafka iirc.
>>>
>>> 3- Applying a basic Machine learning algorithm on the captured data.

>>>
>>> I can't help you with this :)
>>>
>>
>>
> --

Jon

Re: multiple pattern grok parser in 1 file

2017-10-23 Thread Simon Elliston Ball

That is not valid grok. Pattern names should be unique in the grok. 

What you probably mean is something like:

AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG2 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}
AUTHLOG (?:%{AUTHLOG1}|%{AUTHLOG2})

Simon

> On 23 Oct 2017, at 08:53, tkg_cangkul  wrote:
> 
> FYI,
> 
> i've trying to using Grok parser metron with multiple pattern in single file 
> but it doesn't work. this is my sample grok pattern on 
> /apps/metron/patterns/authlog :
> 
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
> %{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
> AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
> %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
> %{USERNAME:username}
> 
> When the sensor started, the second grok pattern doesn't work. Only first 
> pattern works.
> There is an error message like this on storm logs:
> 
> Caused by: java.lang.RuntimeException: Grok statement produced a null message.
> 
> 
> On 23/10/17 10:49, tkg_cangkul wrote:
>> Hi Wasim, 
>> 
>> thx for your reply.
>> So it means i should use logstash parser for metron?
>> Is there any documentation about use logstash parser for metron?
>> I didn't found any documentation about that on metron. 
>> i just find logstash basic parser but there is no documentation about that.
>> 
>> 
>> 
>> On 23/10/17 10:33, Wasim Halani wrote:
>>> Hi Youzha,
>>> 
>>> It should be possible to add multiple patterns in a single config file. For 
>>> reference, you can check out the use of multiple patterns in a repo I 
>>> maintain [1].
>>> You would find the patterns in [2] useful for your use-case.
>>> 
>>> However, do note that there is a cost to every grok failure [3] - so you 
>>> need to ensure that your most common event patterns are at the top of the 
>>> list.
>>> 
>>> As a side-note, if you have any logstash parsers which are not available in 
>>> the repo, please feel to submit a PR to [4] 
>>> 
>>> 
>>> [1] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
>>> [2] 
>>> https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf
>>> [3] https://www.elastic.co/blog/do-you-grok-grok
>>> [4] https://bitbucket.org/networkintelligence/logstash-configs/
>>> 
>>> Regards,
>>> ---
>>> Wasim Halani
>>> http://twitter.com/washalsec
>>> http://securitythoughts.wordpress.com
>>> --
>>> To keep silent when you can say something wise and useful is as bad as 
>>> keeping on propagating foolish and unwise thoughts. -- Imam Ali (p.b.u.h.)
>>> 
>>> On Mon, Oct 23, 2017 at 8:08 AM, Youzha  wrote:
>>> Hi, is that possible to using multiple pattern grok parser ini 1 pattern 
>>> file?
>>> i’m trying to parsing authlog file in /var/log/secure into metron. the 
>>> problem is there are different structures of logs inside /var/log/secure. 
>>> any suggest for this pls?
>>> 
>>> 
>>> Best Regards,
>>> 
>>> 
>> 
>

Re: multiple pattern grok parser in 1 file

2017-10-23 Thread tkg_cangkul


FYI,

i've trying to using Grok parser metron with multiple pattern in single 
file but it doesn't work. this is my sample grok pattern on 
/apps/metron/patterns/authlog :


AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for 
%{USERNAME:username} from %{IP:ip} %{GREEDYDATA}
AUTHLOG %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} 
%{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} closed for user 
%{USERNAME:username}


When the sensor started, the second grok pattern doesn't work. Only 
first pattern works.

There is an error message like this on storm logs:

Caused by: java.lang.RuntimeException: Grok statement produced a null message.



On 23/10/17 10:49, tkg_cangkul wrote:

Hi Wasim,

thx for your reply.
So it means i should use logstash parser for metron?
Is there any documentation about use logstash parser for metron?
I didn't found any documentation about that on metron.
i just find logstash basic parser but there is no documentation about 
that.




On 23/10/17 10:33, Wasim Halani wrote:

Hi Youzha,

It should be possible to add multiple patterns in a single config 
file. For reference, you can check out the use of multiple patterns 
in a repo I maintain [1].

You would find the patterns in [2] useful for your use-case.

However, do note that there is a cost to every grok failure [3] - so 
you need to ensure that your most common event patterns are at the 
top of the list.


As a side-note, if you have any logstash parsers which are not 
available in the repo, please feel to submit a PR to [4]



[1] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/cisco-asa.conf
[2] 
https://bitbucket.org/networkintelligence/logstash-configs/raw/aae3d61bb6c53beb0678536e2e9b33d7996e2960/linux-system.conf

[3] https://www.elastic.co/blog/do-you-grok-grok
[4] https://bitbucket.org/networkintelligence/logstash-configs/

Regards,
---
Wasim Halani
http://twitter.com/washalsec
http://securitythoughts.wordpress.com
--
To keep silent when you can say something wise and useful is as bad 
as keeping on propagating foolish and unwise thoughts. -- Imam Ali 
(p.b.u.h.)


On Mon, Oct 23, 2017 at 8:08 AM, Youzha > wrote:


Hi, is that possible to using multiple pattern grok parser ini 1
pattern file?
i’m trying to parsing authlog file in /var/log/secure into
metron. the problem is there are different structures of logs
inside /var/log/secure. any suggest for this pls?


Best Regards,

Re: Snort Installation

2017-10-23 Thread Syed Hammad Tahir

Hi guys,

I tried to add another network interface in order to bridge it to LAN. I
tried to do it on virtualbox vm settings and when i did vagrant up after
that, there was no bridged interface. Can anyone help me on this?

On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir 
wrote:

> Ok, thankyou. I will let you know once  I make snort sniff the traffic in
> the given configuration, might be helpful for others. I will then try to do
> that kafka topic and will ask if any help is needed.
>
> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>
>> Hi Syed,
>>
>> See inline.
>>
>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>
>>> I have installed the snort manually. Now I need help with :
>>>
>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see
>>> the traffic outside vagrant vm, how do I make it see that traffic?
>>>
>>
>> To be honest, configuring Snort to work on your LAN is out of scope of
>> the project. Have a look at the documentation at https://www.snort.org/.
>> You will probably have to add a 2nd network interface bridged to your LAN
>> in promiscuous mode. Additionally, I think most of us expect some basic
>> Linux & network administration knowledge when using Metron.
>>
>> 2- Making a kafka topic to push those saved logs in metron for
>>> preprocessing
>>>
>>
>> Have a look at the Metron documentation at https://metron.apache.org/curr
>> ent-book/index.html. Adding a new sensor in the Metron UI will create
>> the Kafka iirc.
>>
>> 3- Applying a basic Machine learning algorithm on the captured data.
>>>
>>
>> I can't help you with this :)
>>
>
>

Re: multiple pattern grok parser in 1 file

Re: Snort Installation

Re: Snort Installation

Re: multiple pattern grok parser in 1 file

Re: multiple pattern grok parser in 1 file

Re: Snort Installation

6 matches

Site Navigation

Mail list logo

Footer information