Re: Snort

[James Sirota]

Yes, you can use Snort. Metron can consume Snort telemetries out of the box. You have to setup Snort on your own and push the output into a kafka topic (most likely using NiFi). From there on you can use the output of Snort in Metron.10.10.2017, 00:48, "Syed Hammad Tahir" :Hi,Can I use snort in packet capture mode with metron? By default it works in IDS mode only.Regards.
--- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org

Re: Snort

[Syed Hammad Tahir]

You mean that I must start snort from terminal by doing snort -v and then
push it to kafka topic? I need to start snort in packet capture mode.

On Tue, Oct 10, 2017 at 9:52 PM, James Sirota  wrote:

> Yes, you can use Snort. Metron can consume Snort telemetries out of the
> box. You have to setup Snort on your own and push the output into a kafka
> topic (most likely using NiFi). From there on you can use the output of
> Snort in Metron.
>
>
> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>
> Hi,
>
> Can I use snort in packet capture mode with metron? By default it works in
> IDS mode only.
>
> Regards.
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: Snort

[James Sirota]

What I mean is that you should install snort, load the appropriate Snort rules for your use case, set Snort to log to a directory, and send traffic to the network interface where Snort is listening. That will produce Snort log files. Then you can push the contents of Snort logs either to Kafka using NiFi (preferred) or using Kafka utilities such as command line producer. This should be pushed to a Kafka topic called Snort where each message is a log line of the Snort file. Does that make sense? Thanks,James11.10.2017, 23:08, "Syed Hammad Tahir" :You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode.On Tue, Oct 10, 2017 at 9:52 PM, James Sirota  wrote:Yes, you can use Snort. Metron can consume Snort telemetries out of the box. You have to setup Snort on your own and push the output into a kafka topic (most likely using NiFi). From there on you can use the output of Snort in Metron.10.10.2017, 00:48, "Syed Hammad Tahir" :Hi,Can I use snort in packet capture mode with metron? By default it works in IDS mode only.Regards.
--- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org

--- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org

Re: Snort

[Syed Hammad Tahir]

Hi,

Thanks for the support. Can it be performed both on dumped log and real
time data?
Regards.

On Tue, Oct 17, 2017 at 1:02 AM, James Sirota  wrote:

> What I mean is that you should install snort, load the appropriate Snort
> rules for your use case, set Snort to log to a directory, and send traffic
> to the network interface where Snort is listening. That will produce Snort
> log files. Then you can push the contents of Snort logs either to Kafka
> using NiFi (preferred) or using Kafka utilities such as command line
> producer. This should be pushed to a Kafka topic called Snort where each
> message is a log line of the Snort file. Does that make sense?
>
> Thanks,
> James
>
>
> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>
> You mean that I must start snort from terminal by doing snort -v and then
> push it to kafka topic? I need to start snort in packet capture mode.
>
> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota  wrote:
>
> Yes, you can use Snort. Metron can consume Snort telemetries out of the
> box. You have to setup Snort on your own and push the output into a kafka
> topic (most likely using NiFi). From there on you can use the output of
> Snort in Metron.
>
>
> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>
> Hi,
>
> Can I use snort in packet capture mode with metron? By default it works in
> IDS mode only.
>
> Regards.
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: Snort

[Syed Hammad Tahir]

And I am sorry about one confusion but isnt snort builtin into the metron
framework? If so then cant we access that snort and do the tasks you
mentioned earlier?

On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir 
wrote:

> Hi,
>
> Thanks for the support. Can it be performed both on dumped log and real
> time data?
> Regards.
>
> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota  wrote:
>
>> What I mean is that you should install snort, load the appropriate Snort
>> rules for your use case, set Snort to log to a directory, and send traffic
>> to the network interface where Snort is listening. That will produce Snort
>> log files. Then you can push the contents of Snort logs either to Kafka
>> using NiFi (preferred) or using Kafka utilities such as command line
>> producer. This should be pushed to a Kafka topic called Snort where each
>> message is a log line of the Snort file. Does that make sense?
>>
>> Thanks,
>> James
>>
>>
>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>
>> You mean that I must start snort from terminal by doing snort -v and then
>> push it to kafka topic? I need to start snort in packet capture mode.
>>
>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota  wrote:
>>
>> Yes, you can use Snort. Metron can consume Snort telemetries out of the
>> box. You have to setup Snort on your own and push the output into a kafka
>> topic (most likely using NiFi). From there on you can use the output of
>> Snort in Metron.
>>
>>
>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>
>> Hi,
>>
>> Can I use snort in packet capture mode with metron? By default it works
>> in IDS mode only.
>>
>> Regards.
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>

Re: Snort

[Nick Allen]

>From Metron's perspective, Snort is just another sensor.  Snort is
installed, managed and executed completely independent of Metron itself. As
with any sensor, you are responsible for getting the telemetry produced by
Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
wonderful things with it. :)


On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
wrote:

> And I am sorry about one confusion but isnt snort builtin into the metron
> framework? If so then cant we access that snort and do the tasks you
> mentioned earlier?
>
> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir 
> wrote:
>
>> Hi,
>>
>> Thanks for the support. Can it be performed both on dumped log and real
>> time data?
>> Regards.
>>
>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota  wrote:
>>
>>> What I mean is that you should install snort, load the appropriate Snort
>>> rules for your use case, set Snort to log to a directory, and send traffic
>>> to the network interface where Snort is listening. That will produce Snort
>>> log files. Then you can push the contents of Snort logs either to Kafka
>>> using NiFi (preferred) or using Kafka utilities such as command line
>>> producer. This should be pushed to a Kafka topic called Snort where each
>>> message is a log line of the Snort file. Does that make sense?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>>
>>> You mean that I must start snort from terminal by doing snort -v and
>>> then push it to kafka topic? I need to start snort in packet capture mode.
>>>
>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>>> wrote:
>>>
>>> Yes, you can use Snort. Metron can consume Snort telemetries out of the
>>> box. You have to setup Snort on your own and push the output into a kafka
>>> topic (most likely using NiFi). From there on you can use the output of
>>> Snort in Metron.
>>>
>>>
>>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>>
>>> Hi,
>>>
>>> Can I use snort in packet capture mode with metron? By default it works
>>> in IDS mode only.
>>>
>>> Regards.
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>
>

Re: Snort

[Syed Hammad Tahir]

yes,, but when i do snort -v in vagrant ssh console it says snort isnt
installed where as it can be seen working in metron. Due to that reason I
am confused because James Sirota said to install snort.

On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:

> From Metron's perspective, Snort is just another sensor.  Snort is
> installed, managed and executed completely independent of Metron itself. As
> with any sensor, you are responsible for getting the telemetry produced by
> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
> wonderful things with it. :)
>
>
> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
> wrote:
>
>> And I am sorry about one confusion but isnt snort builtin into the metron
>> framework? If so then cant we access that snort and do the tasks you
>> mentioned earlier?
>>
>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir > > wrote:
>>
>>> Hi,
>>>
>>> Thanks for the support. Can it be performed both on dumped log and real
>>> time data?
>>> Regards.
>>>
>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>>> wrote:
>>>
 What I mean is that you should install snort, load the appropriate
 Snort rules for your use case, set Snort to log to a directory, and send
 traffic to the network interface where Snort is listening. That will
 produce Snort log files. Then you can push the contents of Snort logs
 either to Kafka using NiFi (preferred) or using Kafka utilities such as
 command line producer. This should be pushed to a Kafka topic called Snort
 where each message is a log line of the Snort file. Does that make sense?

 Thanks,
 James


 11.10.2017, 23:08, "Syed Hammad Tahir" :

 You mean that I must start snort from terminal by doing snort -v and
 then push it to kafka topic? I need to start snort in packet capture mode.

 On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
 wrote:

 Yes, you can use Snort. Metron can consume Snort telemetries out of the
 box. You have to setup Snort on your own and push the output into a kafka
 topic (most likely using NiFi). From there on you can use the output of
 Snort in Metron.


 10.10.2017, 00:48, "Syed Hammad Tahir" :

 Hi,

 Can I use snort in packet capture mode with metron? By default it works
 in IDS mode only.

 Regards.



 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org




 ---
 Thank you,

 James Sirota
 PMC- Apache Metron
 jsirota AT apache DOT org


>>>
>>
>

Re: Snort

[Nick Allen]

In the Full Dev environment, Snort is not installed.  We install "Sensor
Stubs" which is just a mechanism that continually replays canned telemetry
logs repetitively to mimic real sensors.  We have to do this because of
resource constraints when running all of Metron on a single VM.  See the
following for more information.

https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs



On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir 
wrote:

> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
> installed where as it can be seen working in metron. Due to that reason I
> am confused because James Sirota said to install snort.
>
> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>
>> From Metron's perspective, Snort is just another sensor.  Snort is
>> installed, managed and executed completely independent of Metron itself. As
>> with any sensor, you are responsible for getting the telemetry produced by
>> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
>> wonderful things with it. :)
>>
>>
>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir 
>> wrote:
>>
>>> And I am sorry about one confusion but isnt snort builtin into the
>>> metron framework? If so then cant we access that snort and do the tasks you
>>> mentioned earlier?
>>>
>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 Hi,

 Thanks for the support. Can it be performed both on dumped log and real
 time data?
 Regards.

 On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
 wrote:

> What I mean is that you should install snort, load the appropriate
> Snort rules for your use case, set Snort to log to a directory, and send
> traffic to the network interface where Snort is listening. That will
> produce Snort log files. Then you can push the contents of Snort logs
> either to Kafka using NiFi (preferred) or using Kafka utilities such as
> command line producer. This should be pushed to a Kafka topic called Snort
> where each message is a log line of the Snort file. Does that make sense?
>
> Thanks,
> James
>
>
> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>
> You mean that I must start snort from terminal by doing snort -v and
> then push it to kafka topic? I need to start snort in packet capture mode.
>
> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
> wrote:
>
> Yes, you can use Snort. Metron can consume Snort telemetries out of
> the box. You have to setup Snort on your own and push the output into a
> kafka topic (most likely using NiFi). From there on you can use the output
> of Snort in Metron.
>
>
> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>
> Hi,
>
> Can I use snort in packet capture mode with metron? By default it
> works in IDS mode only.
>
> Regards.
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

>>>
>>
>

Re: Snort

[Syed Hammad Tahir]

Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
way snort is usually install on a linux distro or do I need to run some
special commands again?

On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:

> In the Full Dev environment, Snort is not installed.  We install "Sensor
> Stubs" which is just a mechanism that continually replays canned telemetry
> logs repetitively to mimic real sensors.  We have to do this because of
> resource constraints when running all of Metron on a single VM.  See the
> following for more information.
>
> https://github.com/apache/metron/tree/master/metron-
> deployment/roles/sensor-stubs
>
>
>
> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir 
> wrote:
>
>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
>> installed where as it can be seen working in metron. Due to that reason I
>> am confused because James Sirota said to install snort.
>>
>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>>
>>> From Metron's perspective, Snort is just another sensor.  Snort is
>>> installed, managed and executed completely independent of Metron itself. As
>>> with any sensor, you are responsible for getting the telemetry produced by
>>> Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
>>> wonderful things with it. :)
>>>
>>>
>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir >> > wrote:
>>>
 And I am sorry about one confusion but isnt snort builtin into the
 metron framework? If so then cant we access that snort and do the tasks you
 mentioned earlier?

 On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Hi,
>
> Thanks for the support. Can it be performed both on dumped log and
> real time data?
> Regards.
>
> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
> wrote:
>
>> What I mean is that you should install snort, load the appropriate
>> Snort rules for your use case, set Snort to log to a directory, and send
>> traffic to the network interface where Snort is listening. That will
>> produce Snort log files. Then you can push the contents of Snort logs
>> either to Kafka using NiFi (preferred) or using Kafka utilities such as
>> command line producer. This should be pushed to a Kafka topic called 
>> Snort
>> where each message is a log line of the Snort file. Does that make sense?
>>
>> Thanks,
>> James
>>
>>
>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>
>> You mean that I must start snort from terminal by doing snort -v and
>> then push it to kafka topic? I need to start snort in packet capture 
>> mode.
>>
>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>> wrote:
>>
>> Yes, you can use Snort. Metron can consume Snort telemetries out of
>> the box. You have to setup Snort on your own and push the output into a
>> kafka topic (most likely using NiFi). From there on you can use the 
>> output
>> of Snort in Metron.
>>
>>
>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>
>> Hi,
>>
>> Can I use snort in packet capture mode with metron? By default it
>> works in IDS mode only.
>>
>> Regards.
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>>
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>

>>>
>>
>

Re: Snort

[Nick Allen]

No special commands.  Install and configure Snort however you like and get
those logs into a Kafka topic.  Metron is completely agnostic to how sensor
telemetry lands in Kafka.

We also have an Ansible role that will install Snort along with a simple
mechanism to transport its logs to Kafka.  This is only useful for
development environments; not a production install.

Using the Ansible role directly may be beyond the knowledge level of some.
I only offer this as a guide that you can use to follow along and manually
install it yourself.

https://github.com/apache/metron/blob/master/metron-deployment/roles/snort/tasks/main.yml


If you are not familiar with how Ansible roles are defined, just start at
the main.yml, then follow through each of the other files as they are
included.  It is pretty readable once you get use to the layout.

On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir 
wrote:

> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
> way snort is usually install on a linux distro or do I need to run some
> special commands again?
>
> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:
>
>> In the Full Dev environment, Snort is not installed.  We install "Sensor
>> Stubs" which is just a mechanism that continually replays canned telemetry
>> logs repetitively to mimic real sensors.  We have to do this because of
>> resource constraints when running all of Metron on a single VM.  See the
>> following for more information.
>>
>> https://github.com/apache/metron/tree/master/metron-deployme
>> nt/roles/sensor-stubs
>>
>>
>>
>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir > > wrote:
>>
>>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt
>>> installed where as it can be seen working in metron. Due to that reason I
>>> am confused because James Sirota said to install snort.
>>>
>>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:
>>>
 From Metron's perspective, Snort is just another sensor.  Snort is
 installed, managed and executed completely independent of Metron itself. As
 with any sensor, you are responsible for getting the telemetry produced by
 Snort into Kafka.  Metron can then consume that telemetry from Kafka and do
 wonderful things with it. :)


 On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> And I am sorry about one confusion but isnt snort builtin into the
> metron framework? If so then cant we access that snort and do the tasks 
> you
> mentioned earlier?
>
> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Hi,
>>
>> Thanks for the support. Can it be performed both on dumped log and
>> real time data?
>> Regards.
>>
>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>> wrote:
>>
>>> What I mean is that you should install snort, load the appropriate
>>> Snort rules for your use case, set Snort to log to a directory, and send
>>> traffic to the network interface where Snort is listening. That will
>>> produce Snort log files. Then you can push the contents of Snort logs
>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as
>>> command line producer. This should be pushed to a Kafka topic called 
>>> Snort
>>> where each message is a log line of the Snort file. Does that make 
>>> sense?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> 11.10.2017, 23:08, "Syed Hammad Tahir" :
>>>
>>> You mean that I must start snort from terminal by doing snort -v and
>>> then push it to kafka topic? I need to start snort in packet capture 
>>> mode.
>>>
>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
>>> wrote:
>>>
>>> Yes, you can use Snort. Metron can consume Snort telemetries out of
>>> the box. You have to setup Snort on your own and push the output into a
>>> kafka topic (most likely using NiFi). From there on you can use the 
>>> output
>>> of Snort in Metron.
>>>
>>>
>>> 10.10.2017, 00:48, "Syed Hammad Tahir" :
>>>
>>> Hi,
>>>
>>> Can I use snort in packet capture mode with metron? By default it
>>> works in IDS mode only.
>>>
>>> Regards.
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>>
>>>
>>> ---
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>>
>>
>

>>>
>>
>

Re: Snort

[Syed Hammad Tahir]

I am so noob in all of this. I am using full-dev vm metron install to do my
research. So I have 2 options to install snort: as per my understanding

1- Install it in a usual way (like that on a regular linux machine) and
then make its kafka topic

2- Use ansible role to do all of that. Read the content of those yml files
given in main.yml to understand the procedure?

Which one do you suggest?



On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen  wrote:

> No special commands.  Install and configure Snort however you like and get
> those logs into a Kafka topic.  Metron is completely agnostic to how sensor
> telemetry lands in Kafka.
>
> We also have an Ansible role that will install Snort along with a simple
> mechanism to transport its logs to Kafka.  This is only useful for
> development environments; not a production install.
>
> Using the Ansible role directly may be beyond the knowledge level of
> some.  I only offer this as a guide that you can use to follow along and
> manually install it yourself.
>
> https://github.com/apache/metron/blob/master/metron-
> deployment/roles/snort/tasks/main.yml
>
>
> If you are not familiar with how Ansible roles are defined, just start at
> the main.yml, then follow through each of the other files as they are
> included.  It is pretty readable once you get use to the layout.
>
> On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir 
> wrote:
>
>> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
>> way snort is usually install on a linux distro or do I need to run some
>> special commands again?
>>
>> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen  wrote:
>>
>>> In the Full Dev environment, Snort is not installed.  We install "Sensor
>>> Stubs" which is just a mechanism that continually replays canned telemetry
>>> logs repetitively to mimic real sensors.  We have to do this because of
>>> resource constraints when running all of Metron on a single VM.  See the
>>> following for more information.
>>>
>>> https://github.com/apache/metron/tree/master/metron-deployme
>>> nt/roles/sensor-stubs
>>>
>>>
>>>
>>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 yes,, but when i do snort -v in vagrant ssh console it says snort isnt
 installed where as it can be seen working in metron. Due to that reason I
 am confused because James Sirota said to install snort.

 On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen  wrote:

> From Metron's perspective, Snort is just another sensor.  Snort is
> installed, managed and executed completely independent of Metron itself. 
> As
> with any sensor, you are responsible for getting the telemetry produced by
> Snort into Kafka.  Metron can then consume that telemetry from Kafka and 
> do
> wonderful things with it. :)
>
>
> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> And I am sorry about one confusion but isnt snort builtin into the
>> metron framework? If so then cant we access that snort and do the tasks 
>> you
>> mentioned earlier?
>>
>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the support. Can it be performed both on dumped log and
>>> real time data?
>>> Regards.
>>>
>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota 
>>> wrote:
>>>
 What I mean is that you should install snort, load the appropriate
 Snort rules for your use case, set Snort to log to a directory, and 
 send
 traffic to the network interface where Snort is listening. That will
 produce Snort log files. Then you can push the contents of Snort logs
 either to Kafka using NiFi (preferred) or using Kafka utilities such as
 command line producer. This should be pushed to a Kafka topic called 
 Snort
 where each message is a log line of the Snort file. Does that make 
 sense?

 Thanks,
 James


 11.10.2017, 23:08, "Syed Hammad Tahir" :

 You mean that I must start snort from terminal by doing snort -v
 and then push it to kafka topic? I need to start snort in packet 
 capture
 mode.

 On Tue, Oct 10, 2017 at 9:52 PM, James Sirota 
 wrote:

 Yes, you can use Snort. Metron can consume Snort telemetries out of
 the box. You have to setup Snort on your own and push the output into a
 kafka topic (most likely using NiFi). From there on you can use the 
 output
 of Snort in Metron.


 10.10.2017, 00:48, "Syed Hammad Tahir" :

 Hi,

 Can I use snort in packet capture mode with metron? By default it
 works in IDS mode only.

 Regards.



>>

Re: Snort

[Simon Elliston Ball]

I would recommend just using a text editor if you’re not familiar with sed. To 
solve your sed problem…

sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile 

sed -i  means run the sed command (in this case a find replace) inplace on the 
file, the text following the -i is the name to append to a backup version (ie 
the original file unchanged).

Metron does tend to assume a good knowledge of linux admin, you’ll find we have 
a lot of shell gurus in the community, but if you’re struggling with this, 
maybe a simple text editor would be easier. All you’re trying to do here is 
change a config value. 

Simon

> On 19 Oct 2017, at 11:46, Syed Hammad Tahir  wrote:
> 
> Ran it without -i swtich, gives this:
> 
> 
> 
> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com  
> mailto:zeo...@gmail.com>> wrote:
> The sed command is falling.  It's written for a Mac so it will need an 
> alteration to be portable.  Run it without the '' after -i, from 
> ~/metron-master
> 
> Jon
> 
> 
> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir  > wrote:
> I did what this guide said to install the original sensor:
> https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs
>  
> 
> 
> Still didnt work. How do I install snort into this?
> 
> 
> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir  > wrote:
> Maybe I did something wrong
> 
> 
> 
> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir  > wrote:
> Ok, thankyou. It will install all the sensors (bro, snort etc) ?
> 
> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com  > wrote:
> When you set up full dev if you remove the sensors skip tag it will set up 
> snort for you.  I have a sed one liner in my bro security patch pr to do 
> this, just need to do it before vagrant up.
> 
> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
> metron-deployment/vagrant/full-dev-platform/Vagrantfile
> cd metron-deployment/vagrant/full-dev-platform/
> vagrant up
> 
> Jon
> 
> 
> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir  > wrote:
> I followed this guide exactly:  
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 
> 
> 
> And the did vagrant up in full-development-platform folder. And Snort is not 
> installed because when I type snort -v in vagrant ssh, it returns error of 
> not able to find the snort command.
> 
> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets  > wrote:
> Hi Syed,
> 
> I was under the impression that you installed the full-dev environment? If 
> so, snort should already be installed...
> 
> On 2017-10-18 09:45, Syed Hammad Tahir wrote:
> 
>> It has become a mess. Apparently snort is released for centos 7 whereas 
>> metron one is centos 6.8. Whenever I try to install snort it gives me this:
>>  
>> 
>> 
>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen > > wrote:
>> Just use those as a guide to run the commands  yourself.
>> 
>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir > > wrote:
>> please help me install the snort in metron. I tried doing it the normal way 
>> but i cant install the libraries
>> 
>> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir > > wrote:
>> ok, This is the snort.yml file
>>  
>> 
>>  
>> Do I need to run these commands myself or how do I put these yml files into 
>> play?
>> 
>> On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir > > wrote:
>> I am so noob in all of this. I am using full-dev vm metron install to do my 
>> research. So I have 2 options to install snort: as per my understanding
>>  
>> 1- Install it in a usual way (like that on a regular linux machine) and then 
>> make its kafka topic
>>  
>> 2- Use ansible role to do all of that. Read the content of those yml files 
>> given in main.yml to understand the procedure?
>>  
>> Which one do you suggest?
>>  
>>  
>> 
>> On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen > >wrote:
>> No special commands.  Install and configure Snort however you like and get 
>> those logs into a Kafka topic.  Metron is completely agnostic to how sensor 
>> telemetry lands in Kafka.
>>  
>> We also have an Ansible role that will install Snort along with a simple 
>> mechanism to transport its logs to Kafka.  This is only useful for 
>> development environments; not a production install.  
>>  
>> Using the Ansible role directly may be beyond the knowledge level of some.  
>> I only offer this as a guide that you can use to follow along and manually 
>> install it yourself. 
>>  
>> https:/

Re: Snort

[Syed Hammad Tahir]

Should I edit the vagrant file using text editor and what exactly should I
edit there?

On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> I would recommend just using a text editor if you’re not familiar with
> sed. To solve your sed problem…
>
> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
> metron-deployment/vagrant/full-dev-platform/Vagrantfile
>
> sed -i  means run the sed command (in this case a find replace) inplace on
> the file, the text following the -i is the name to append to a backup
> version (ie the original file unchanged).
>
> Metron does tend to assume a good knowledge of linux admin, you’ll find we
> have a lot of shell gurus in the community, but if you’re struggling with
> this, maybe a simple text editor would be easier. All you’re trying to do
> here is change a config value.
>
> Simon
>
> On 19 Oct 2017, at 11:46, Syed Hammad Tahir  wrote:
>
> Ran it without -i swtich, gives this:
>
> 
>
> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com 
> wrote:
>
>> The sed command is falling.  It's written for a Mac so it will need an
>> alteration to be portable.  Run it without the '' after -i, from
>> ~/metron-master
>>
>> Jon
>>
>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir 
>> wrote:
>>
>>> I did what this guide said to install the original sensor:
>>> https://github.com/apache/metron/tree/master/metron-deployme
>>> nt/roles/sensor-stubs
>>>
>>> Still didnt work. How do I install snort into this?
>>>
>>>
>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 Maybe I did something wrong

 

 On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Ok, thankyou. It will install all the sensors (bro, snort etc) ?
>
> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com 
> wrote:
>
>> When you set up full dev if you remove the sensors skip tag it will
>> set up snort for you.  I have a sed one liner in my bro security patch pr
>> to do this, just need to do it before vagrant up.
>>
>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
>> metron-deployment/vagrant/full-dev-platform/Vagrantfile
>> cd metron-deployment/vagrant/full-dev-platform/
>> vagrant up
>>
>> Jon
>>
>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir 
>> wrote:
>>
>>> I followed this guide exactly:  https://cwiki.apache
>>> .org/confluence/pages/viewpage.action?pageId=68718548
>>>
>>> And the did vagrant up in full-development-platform folder. And
>>> Snort is not installed because when I type snort -v in vagrant ssh, it
>>> returns error of not able to find the snort command.
>>>
>>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets 
>>> wrote:
>>>
 Hi Syed,

 I was under the impression that you installed the full-dev
 environment? If so, snort should already be installed...

 On 2017-10-18 09:45, Syed Hammad Tahir wrote:

 It has become a mess. Apparently snort is released for centos 7
 whereas metron one is centos 6.8. Whenever I try to install snort it 
 gives
 me this:

 

 On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen 
 wrote:

> Just use those as a guide to run the commands  yourself.
>
> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> please help me install the snort in metron. I tried doing it the
>> normal way but i cant install the libraries
>>
>> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> ok, This is the snort.yml file
>>>
>>> 
>>>
>>> Do I need to run these commands myself or how do I put these yml
>>> files into play?
>>>
>>> On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 I am so noob in all of this. I am using full-dev vm metron
 install to do my research. So I have 2 options to install snort: 
 as per my
 understanding

 1- Install it in a usual way (like that on a regular linux
 machine) and then make its kafka topic

 2- Use ansible role to do all of that. Read the content of
 those yml files given in main.yml to understand the procedure?

 Which one do you suggest?



 On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen >>> >wrote:

> No special commands.  Install and configure Snort however you
> like and get those logs into a Kafka topic.  Metron is completely 
>>>

Re: Snort

[zeo...@gmail.com]

In the Vagrantfile for full-dev, edit the line that starts with
ansibleSkipTags (this line
)
to be exactly the following:

ansibleSkipTags='quick_dev'

Jon

On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir 
wrote:

> Should I edit the vagrant file using text editor and what exactly should I
> edit there?
>
> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> I would recommend just using a text editor if you’re not familiar with
>> sed. To solve your sed problem…
>>
>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
>> metron-deployment/vagrant/full-dev-platform/Vagrantfile
>>
>> sed -i  means run the sed command (in this case a find replace) inplace
>> on the file, the text following the -i is the name to append to a backup
>> version (ie the original file unchanged).
>>
>> Metron does tend to assume a good knowledge of linux admin, you’ll find
>> we have a lot of shell gurus in the community, but if you’re struggling
>> with this, maybe a simple text editor would be easier. All you’re trying to
>> do here is change a config value.
>>
>> Simon
>>
>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir  wrote:
>>
>> Ran it without -i swtich, gives this:
>>
>> 
>>
>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com 
>> wrote:
>>
>>> The sed command is falling.  It's written for a Mac so it will need an
>>> alteration to be portable.  Run it without the '' after -i, from
>>> ~/metron-master
>>>
>>> Jon
>>>
>>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir 
>>> wrote:
>>>
 I did what this guide said to install the original sensor:

 https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs

 Still didnt work. How do I install snort into this?


 On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Maybe I did something wrong
>
> 
>
> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Ok, thankyou. It will install all the sensors (bro, snort etc) ?
>>
>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com 
>>  wrote:
>>
>>> When you set up full dev if you remove the sensors skip tag it will
>>> set up snort for you.  I have a sed one liner in my bro security patch 
>>> pr
>>> to do this, just need to do it before vagrant up.
>>>
>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile
>>> cd metron-deployment/vagrant/full-dev-platform/
>>> vagrant up
>>>
>>> Jon
>>>
>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir 
>>> wrote:
>>>
 I followed this guide exactly:
 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548

 And the did vagrant up in full-development-platform folder. And
 Snort is not installed because when I type snort -v in vagrant ssh, it
 returns error of not able to find the snort command.

 On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets 
 wrote:

> Hi Syed,
>
> I was under the impression that you installed the full-dev
> environment? If so, snort should already be installed...
>
> On 2017-10-18 09:45, Syed Hammad Tahir wrote:
>
> It has become a mess. Apparently snort is released for centos 7
> whereas metron one is centos 6.8. Whenever I try to install snort it 
> gives
> me this:
>
> 
>
> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen 
> wrote:
>
>> Just use those as a guide to run the commands  yourself.
>>
>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> please help me install the snort in metron. I tried doing it the
>>> normal way but i cant install the libraries
>>>
>>> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 ok, This is the snort.yml file

 

 Do I need to run these commands myself or how do I put these
 yml files into play?

 On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> I am so noob in all of this. I am using full-dev vm metron
> install to do my research. So I have 2 options to install snort: 
> as per my
> understanding
>
> 1- Install it in a usual way (like that on a regular linux
> machine) and then make its kafka topic
>
> 2- Use ansible ro

Re: Snort

[Syed Hammad Tahir]

would I need to vagrant destroy and then vagrant up again after this or
will vagrant halt and vagrant up will do the job?

On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com  wrote:

> In the Vagrantfile for full-dev, edit the line that starts with
> ansibleSkipTags (this line
> )
> to be exactly the following:
>
> ansibleSkipTags='quick_dev'
>
> Jon
>
> On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir 
> wrote:
>
>> Should I edit the vagrant file using text editor and what exactly should
>> I edit there?
>>
>> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>
>>> I would recommend just using a text editor if you’re not familiar with
>>> sed. To solve your sed problem…
>>>
>>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile
>>>
>>> sed -i  means run the sed command (in this case a find replace) inplace
>>> on the file, the text following the -i is the name to append to a backup
>>> version (ie the original file unchanged).
>>>
>>> Metron does tend to assume a good knowledge of linux admin, you’ll find
>>> we have a lot of shell gurus in the community, but if you’re struggling
>>> with this, maybe a simple text editor would be easier. All you’re trying to
>>> do here is change a config value.
>>>
>>> Simon
>>>
>>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir 
>>> wrote:
>>>
>>> Ran it without -i swtich, gives this:
>>>
>>> 
>>>
>>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com 
>>> wrote:
>>>
 The sed command is falling.  It's written for a Mac so it will need an
 alteration to be portable.  Run it without the '' after -i, from
 ~/metron-master

 Jon

 On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir 
 wrote:

> I did what this guide said to install the original sensor:
> https://github.com/apache/metron/tree/master/metron-
> deployment/roles/sensor-stubs
>
> Still didnt work. How do I install snort into this?
>
>
> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Maybe I did something wrong
>>
>> 
>>
>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ?
>>>
>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >> > wrote:
>>>
 When you set up full dev if you remove the sensors skip tag it will
 set up snort for you.  I have a sed one liner in my bro security patch 
 pr
 to do this, just need to do it before vagrant up.

 sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
 metron-deployment/vagrant/full-dev-platform/Vagrantfile
 cd metron-deployment/vagrant/full-dev-platform/
 vagrant up

 Jon

 On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir 
 wrote:

> I followed this guide exactly:  https://cwiki.
> apache.org/confluence/pages/viewpage.action?pageId=68718548
>
> And the did vagrant up in full-development-platform folder. And
> Snort is not installed because when I type snort -v in vagrant ssh, it
> returns error of not able to find the snort command.
>
> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets 
>  wrote:
>
>> Hi Syed,
>>
>> I was under the impression that you installed the full-dev
>> environment? If so, snort should already be installed...
>>
>> On 2017-10-18 09:45, Syed Hammad Tahir wrote:
>>
>> It has become a mess. Apparently snort is released for centos 7
>> whereas metron one is centos 6.8. Whenever I try to install snort it 
>> gives
>> me this:
>>
>> 
>>
>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen 
>> wrote:
>>
>>> Just use those as a guide to run the commands  yourself.
>>>
>>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 please help me install the snort in metron. I tried doing it
 the normal way but i cant install the libraries

 On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> ok, This is the snort.yml file
>
> 
>
> Do I need to run these commands myself or how do I put these
> yml files into play?
>
> On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> I am so noob in all of this. I am using 

Re: Snort

[Syed Hammad Tahir]

I did all of that and then did vagrant up again. Snort is still not
installed. Will I have to vagrant destroy and then vagrant up again in
order for it to work?

On Thu, Oct 19, 2017 at 8:58 PM, Syed Hammad Tahir 
wrote:

> would I need to vagrant destroy and then vagrant up again after this or
> will vagrant halt and vagrant up will do the job?
>
> On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com 
> wrote:
>
>> In the Vagrantfile for full-dev, edit the line that starts with
>> ansibleSkipTags (this line
>> )
>> to be exactly the following:
>>
>> ansibleSkipTags='quick_dev'
>>
>> Jon
>>
>> On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir 
>> wrote:
>>
>>> Should I edit the vagrant file using text editor and what exactly should
>>> I edit there?
>>>
>>> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball <
>>> si...@simonellistonball.com> wrote:
>>>
 I would recommend just using a text editor if you’re not familiar with
 sed. To solve your sed problem…

 sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
 metron-deployment/vagrant/full-dev-platform/Vagrantfile

 sed -i  means run the sed command (in this case a find replace) inplace
 on the file, the text following the -i is the name to append to a backup
 version (ie the original file unchanged).

 Metron does tend to assume a good knowledge of linux admin, you’ll find
 we have a lot of shell gurus in the community, but if you’re struggling
 with this, maybe a simple text editor would be easier. All you’re trying to
 do here is change a config value.

 Simon

 On 19 Oct 2017, at 11:46, Syed Hammad Tahir 
 wrote:

 Ran it without -i swtich, gives this:

 

 On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com 
 wrote:

> The sed command is falling.  It's written for a Mac so it will need an
> alteration to be portable.  Run it without the '' after -i, from
> ~/metron-master
>
> Jon
>
> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir 
> wrote:
>
>> I did what this guide said to install the original sensor:
>> https://github.com/apache/metron/tree/master/metron-deployme
>> nt/roles/sensor-stubs
>>
>> Still didnt work. How do I install snort into this?
>>
>>
>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Maybe I did something wrong
>>>
>>> 
>>>
>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 Ok, thankyou. It will install all the sensors (bro, snort etc) ?

 On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>> .com> wrote:

> When you set up full dev if you remove the sensors skip tag it
> will set up snort for you.  I have a sed one liner in my bro security 
> patch
> pr to do this, just need to do it before vagrant up.
>
> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
> metron-deployment/vagrant/full-dev-platform/Vagrantfile
> cd metron-deployment/vagrant/full-dev-platform/
> vagrant up
>
> Jon
>
> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> I followed this guide exactly:  https://cwiki.apache
>> .org/confluence/pages/viewpage.action?pageId=68718548
>>
>> And the did vagrant up in full-development-platform folder. And
>> Snort is not installed because when I type snort -v in vagrant ssh, 
>> it
>> returns error of not able to find the snort command.
>>
>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets > > wrote:
>>
>>> Hi Syed,
>>>
>>> I was under the impression that you installed the full-dev
>>> environment? If so, snort should already be installed...
>>>
>>> On 2017-10-18 09:45, Syed Hammad Tahir wrote:
>>>
>>> It has become a mess. Apparently snort is released for centos 7
>>> whereas metron one is centos 6.8. Whenever I try to install snort 
>>> it gives
>>> me this:
>>>
>>> 
>>>
>>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen 
>>>  wrote:
>>>
 Just use those as a guide to run the commands  yourself.

 On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> please help me install the snort in metron. I tried doing it
> the normal way but i cant install the libraries
>
> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>

Re: Snort Installation

[Syed Hammad Tahir]

Help guys !!!

On Fri, Oct 20, 2017 at 12:32 PM, Syed Hammad Tahir 
wrote:

> I have installed the snort manually. Now I need help with :
>
> 1- Capturing the data of my lan and dumping it via snort :Snort cant see
> the traffic outside vagrant vm, how do I make it see that traffic?
>
> 2- Making a kafka topic to push those saved logs in metron for
> preprocessing
>
> 3- Applying a basic Machine learning algorithm on the captured data.
>
> Regards.
>

Re: Snort Installation

[Laurens Vets]

Hi Syed,

See inline.

On 2017-10-20 00:32, Syed Hammad Tahir wrote:

I have installed the snort manually. Now I need help with :

1- Capturing the data of my lan and dumping it via snort :Snort cant 
see the traffic outside vagrant vm, how do I make it see that traffic?


To be honest, configuring Snort to work on your LAN is out of scope of 
the project. Have a look at the documentation at https://www.snort.org/.
You will probably have to add a 2nd network interface bridged to your 
LAN in promiscuous mode. Additionally, I think most of us expect some 
basic Linux & network administration knowledge when using Metron.


2- Making a kafka topic to push those saved logs in metron for 
preprocessing


Have a look at the Metron documentation at 
https://metron.apache.org/current-book/index.html. Adding a new sensor 
in the Metron UI will create the Kafka iirc.



3- Applying a basic Machine learning algorithm on the captured data.


I can't help you with this :)

Re: Snort Installation

[Syed Hammad Tahir]

Ok, thankyou. I will let you know once  I make snort sniff the traffic in
the given configuration, might be helpful for others. I will then try to do
that kafka topic and will ask if any help is needed.

On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:

> Hi Syed,
>
> See inline.
>
> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>
>> I have installed the snort manually. Now I need help with :
>>
>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see
>> the traffic outside vagrant vm, how do I make it see that traffic?
>>
>
> To be honest, configuring Snort to work on your LAN is out of scope of the
> project. Have a look at the documentation at https://www.snort.org/.
> You will probably have to add a 2nd network interface bridged to your LAN
> in promiscuous mode. Additionally, I think most of us expect some basic
> Linux & network administration knowledge when using Metron.
>
> 2- Making a kafka topic to push those saved logs in metron for
>> preprocessing
>>
>
> Have a look at the Metron documentation at https://metron.apache.org/curr
> ent-book/index.html. Adding a new sensor in the Metron UI will create the
> Kafka iirc.
>
> 3- Applying a basic Machine learning algorithm on the captured data.
>>
>
> I can't help you with this :)
>

Re: Snort Installation

[Syed Hammad Tahir]

Hi guys,

I tried to add another network interface in order to bridge it to LAN. I
tried to do it on virtualbox vm settings and when i did vagrant up after
that, there was no bridged interface. Can anyone help me on this?

On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir 
wrote:

> Ok, thankyou. I will let you know once  I make snort sniff the traffic in
> the given configuration, might be helpful for others. I will then try to do
> that kafka topic and will ask if any help is needed.
>
> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>
>> Hi Syed,
>>
>> See inline.
>>
>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>
>>> I have installed the snort manually. Now I need help with :
>>>
>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see
>>> the traffic outside vagrant vm, how do I make it see that traffic?
>>>
>>
>> To be honest, configuring Snort to work on your LAN is out of scope of
>> the project. Have a look at the documentation at https://www.snort.org/.
>> You will probably have to add a 2nd network interface bridged to your LAN
>> in promiscuous mode. Additionally, I think most of us expect some basic
>> Linux & network administration knowledge when using Metron.
>>
>> 2- Making a kafka topic to push those saved logs in metron for
>>> preprocessing
>>>
>>
>> Have a look at the Metron documentation at https://metron.apache.org/curr
>> ent-book/index.html. Adding a new sensor in the Metron UI will create
>> the Kafka iirc.
>>
>> 3- Applying a basic Machine learning algorithm on the captured data.
>>>
>>
>> I can't help you with this :)
>>
>
>

Re: Snort Installation

[zeo...@gmail.com]

Hi Syed,

Just to clarify, this a snort issue you are having?  If so I suggest
looking at their documentation (https://snort.org/documents) or reaching
out to their community (https://snort.org/community), as they have more
expertise in this area.

Jon

On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir  wrote:

> Hi guys,
>
> I tried to add another network interface in order to bridge it to LAN. I
> tried to do it on virtualbox vm settings and when i did vagrant up after
> that, there was no bridged interface. Can anyone help me on this?
>
> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir 
> wrote:
>
>> Ok, thankyou. I will let you know once  I make snort sniff the traffic in
>> the given configuration, might be helpful for others. I will then try to do
>> that kafka topic and will ask if any help is needed.
>>
>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>>
>>> Hi Syed,
>>>
>>> See inline.
>>>
>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>>
 I have installed the snort manually. Now I need help with :

 1- Capturing the data of my lan and dumping it via snort :Snort cant
 see the traffic outside vagrant vm, how do I make it see that traffic?

>>>
>>> To be honest, configuring Snort to work on your LAN is out of scope of
>>> the project. Have a look at the documentation at https://www.snort.org/.
>>> You will probably have to add a 2nd network interface bridged to your
>>> LAN in promiscuous mode. Additionally, I think most of us expect some basic
>>> Linux & network administration knowledge when using Metron.
>>>
>>> 2- Making a kafka topic to push those saved logs in metron for
 preprocessing

>>>
>>> Have a look at the Metron documentation at
>>> https://metron.apache.org/current-book/index.html. Adding a new sensor
>>> in the Metron UI will create the Kafka iirc.
>>>
>>> 3- Applying a basic Machine learning algorithm on the captured data.

>>>
>>> I can't help you with this :)
>>>
>>
>>
> --

Jon

Re: Snort Installation

[Syed Hammad Tahir]

yes nut I am a bit confused here. Let me ask them as well then.

On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com  wrote:

> Hi Syed,
>
> Just to clarify, this a snort issue you are having?  If so I suggest
> looking at their documentation (https://snort.org/documents) or reaching
> out to their community (https://snort.org/community), as they have more
> expertise in this area.
>
> Jon
>
> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
> wrote:
>
>> Hi guys,
>>
>> I tried to add another network interface in order to bridge it to LAN. I
>> tried to do it on virtualbox vm settings and when i did vagrant up after
>> that, there was no bridged interface. Can anyone help me on this?
>>
>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir > > wrote:
>>
>>> Ok, thankyou. I will let you know once  I make snort sniff the traffic
>>> in the given configuration, might be helpful for others. I will then try to
>>> do that kafka topic and will ask if any help is needed.
>>>
>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets  wrote:
>>>
 Hi Syed,

 See inline.

 On 2017-10-20 00:32, Syed Hammad Tahir wrote:

> I have installed the snort manually. Now I need help with :
>
> 1- Capturing the data of my lan and dumping it via snort :Snort cant
> see the traffic outside vagrant vm, how do I make it see that traffic?
>

 To be honest, configuring Snort to work on your LAN is out of scope of
 the project. Have a look at the documentation at https://www.snort.org/
 .
 You will probably have to add a 2nd network interface bridged to your
 LAN in promiscuous mode. Additionally, I think most of us expect some basic
 Linux & network administration knowledge when using Metron.

 2- Making a kafka topic to push those saved logs in metron for
> preprocessing
>

 Have a look at the Metron documentation at https://metron.apache.org/
 current-book/index.html. Adding a new sensor in the Metron UI will
 create the Kafka iirc.

 3- Applying a basic Machine learning algorithm on the captured data.
>

 I can't help you with this :)

>>>
>>>
>> --
>
> Jon
>

Re: Snort Installation

[Syed Hammad Tahir]

Ok, I have fixed everything on my own. Now that I have snort logs saved in
a file, I need to get them to metron. Can anyone help me on that?

On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir 
wrote:

> yes nut I am a bit confused here. Let me ask them as well then.
>
> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com 
> wrote:
>
>> Hi Syed,
>>
>> Just to clarify, this a snort issue you are having?  If so I suggest
>> looking at their documentation (https://snort.org/documents) or reaching
>> out to their community (https://snort.org/community), as they have more
>> expertise in this area.
>>
>> Jon
>>
>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
>> wrote:
>>
>>> Hi guys,
>>>
>>> I tried to add another network interface in order to bridge it to LAN. I
>>> tried to do it on virtualbox vm settings and when i did vagrant up after
>>> that, there was no bridged interface. Can anyone help me on this?
>>>
>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 Ok, thankyou. I will let you know once  I make snort sniff the traffic
 in the given configuration, might be helpful for others. I will then try to
 do that kafka topic and will ask if any help is needed.

 On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets 
 wrote:

> Hi Syed,
>
> See inline.
>
> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>
>> I have installed the snort manually. Now I need help with :
>>
>> 1- Capturing the data of my lan and dumping it via snort :Snort cant
>> see the traffic outside vagrant vm, how do I make it see that traffic?
>>
>
> To be honest, configuring Snort to work on your LAN is out of scope of
> the project. Have a look at the documentation at
> https://www.snort.org/.
> You will probably have to add a 2nd network interface bridged to your
> LAN in promiscuous mode. Additionally, I think most of us expect some 
> basic
> Linux & network administration knowledge when using Metron.
>
> 2- Making a kafka topic to push those saved logs in metron for
>> preprocessing
>>
>
> Have a look at the Metron documentation at
> https://metron.apache.org/current-book/index.html. Adding a new
> sensor in the Metron UI will create the Kafka iirc.
>
> 3- Applying a basic Machine learning algorithm on the captured data.
>>
>
> I can't help you with this :)
>


>>> --
>>
>> Jon
>>
>
>

Re: Snort Installation

[Nick Allen]

Take a look at `kafka-console-producer.sh`, which is installed as part of
Kafka.

On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir 
wrote:

> Ok, I have fixed everything on my own. Now that I have snort logs saved in
> a file, I need to get them to metron. Can anyone help me on that?
>
> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir 
> wrote:
>
>> yes nut I am a bit confused here. Let me ask them as well then.
>>
>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com 
>> wrote:
>>
>>> Hi Syed,
>>>
>>> Just to clarify, this a snort issue you are having?  If so I suggest
>>> looking at their documentation (https://snort.org/documents) or
>>> reaching out to their community (https://snort.org/community), as they
>>> have more expertise in this area.
>>>
>>> Jon
>>>
>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
>>> wrote:
>>>
 Hi guys,

 I tried to add another network interface in order to bridge it to LAN.
 I tried to do it on virtualbox vm settings and when i did vagrant up after
 that, there was no bridged interface. Can anyone help me on this?

 On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Ok, thankyou. I will let you know once  I make snort sniff the traffic
> in the given configuration, might be helpful for others. I will then try 
> to
> do that kafka topic and will ask if any help is needed.
>
> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets 
> wrote:
>
>> Hi Syed,
>>
>> See inline.
>>
>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>
>>> I have installed the snort manually. Now I need help with :
>>>
>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant
>>> see the traffic outside vagrant vm, how do I make it see that traffic?
>>>
>>
>> To be honest, configuring Snort to work on your LAN is out of scope
>> of the project. Have a look at the documentation at
>> https://www.snort.org/.
>> You will probably have to add a 2nd network interface bridged to your
>> LAN in promiscuous mode. Additionally, I think most of us expect some 
>> basic
>> Linux & network administration knowledge when using Metron.
>>
>> 2- Making a kafka topic to push those saved logs in metron for
>>> preprocessing
>>>
>>
>> Have a look at the Metron documentation at
>> https://metron.apache.org/current-book/index.html. Adding a new
>> sensor in the Metron UI will create the Kafka iirc.
>>
>> 3- Applying a basic Machine learning algorithm on the captured data.
>>>
>>
>> I can't help you with this :)
>>
>
>
 --
>>>
>>> Jon
>>>
>>
>>
>

Re: Snort Installation

[Syed Hammad Tahir]

Where do I find this file kafka-console-producer.sh?

On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen  wrote:

> Take a look at `kafka-console-producer.sh`, which is installed as part of
> Kafka.
>
> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir 
> wrote:
>
>> Ok, I have fixed everything on my own. Now that I have snort logs saved
>> in a file, I need to get them to metron. Can anyone help me on that?
>>
>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir 
>> wrote:
>>
>>> yes nut I am a bit confused here. Let me ask them as well then.
>>>
>>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com 
>>> wrote:
>>>
 Hi Syed,

 Just to clarify, this a snort issue you are having?  If so I suggest
 looking at their documentation (https://snort.org/documents) or
 reaching out to their community (https://snort.org/community), as they
 have more expertise in this area.

 Jon

 On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
 wrote:

> Hi guys,
>
> I tried to add another network interface in order to bridge it to LAN.
> I tried to do it on virtualbox vm settings and when i did vagrant up after
> that, there was no bridged interface. Can anyone help me on this?
>
> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Ok, thankyou. I will let you know once  I make snort sniff the
>> traffic in the given configuration, might be helpful for others. I will
>> then try to do that kafka topic and will ask if any help is needed.
>>
>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets 
>> wrote:
>>
>>> Hi Syed,
>>>
>>> See inline.
>>>
>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote:
>>>
 I have installed the snort manually. Now I need help with :

 1- Capturing the data of my lan and dumping it via snort :Snort
 cant see the traffic outside vagrant vm, how do I make it see that 
 traffic?

>>>
>>> To be honest, configuring Snort to work on your LAN is out of scope
>>> of the project. Have a look at the documentation at
>>> https://www.snort.org/.
>>> You will probably have to add a 2nd network interface bridged to
>>> your LAN in promiscuous mode. Additionally, I think most of us expect 
>>> some
>>> basic Linux & network administration knowledge when using Metron.
>>>
>>> 2- Making a kafka topic to push those saved logs in metron for
 preprocessing

>>>
>>> Have a look at the Metron documentation at
>>> https://metron.apache.org/current-book/index.html. Adding a new
>>> sensor in the Metron UI will create the Kafka iirc.
>>>
>>> 3- Applying a basic Machine learning algorithm on the captured data.

>>>
>>> I can't help you with this :)
>>>
>>
>>
> --

 Jon

>>>
>>>
>>
>

Re: Snort Installation

[Syed Hammad Tahir]

All I did was install snort separately on vagrant ssh console. The ran it
to collect logs. Now I need to bring those logs to metron.

On Wed, Oct 25, 2017 at 9:50 AM, Farrukh Naveed Anjum <
anjum.farr...@gmail.com> wrote:

> Hi Syed Hammed,
>
> Can you share the steps how did you connected snort with external source ?
> (Metron Snort ?)
>
> On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen  wrote:
>
>> Take a look at `kafka-console-producer.sh`, which is installed as part of
>> Kafka.
>>
>> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir 
>> wrote:
>>
>>> Ok, I have fixed everything on my own. Now that I have snort logs saved
>>> in a file, I need to get them to metron. Can anyone help me on that?
>>>
>>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> > wrote:
>>>
 yes nut I am a bit confused here. Let me ask them as well then.

 On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com 
 wrote:

> Hi Syed,
>
> Just to clarify, this a snort issue you are having?  If so I suggest
> looking at their documentation (https://snort.org/documents) or
> reaching out to their community (https://snort.org/community), as
> they have more expertise in this area.
>
> Jon
>
> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
> wrote:
>
>> Hi guys,
>>
>> I tried to add another network interface in order to bridge it to
>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up
>> after that, there was no bridged interface. Can anyone help me on this?
>>
>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Ok, thankyou. I will let you know once  I make snort sniff the
>>> traffic in the given configuration, might be helpful for others. I will
>>> then try to do that kafka topic and will ask if any help is needed.
>>>
>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets 
>>> wrote:
>>>
 Hi Syed,

 See inline.

 On 2017-10-20 00:32, Syed Hammad Tahir wrote:

> I have installed the snort manually. Now I need help with :
>
> 1- Capturing the data of my lan and dumping it via snort :Snort
> cant see the traffic outside vagrant vm, how do I make it see that 
> traffic?
>

 To be honest, configuring Snort to work on your LAN is out of scope
 of the project. Have a look at the documentation at
 https://www.snort.org/.
 You will probably have to add a 2nd network interface bridged to
 your LAN in promiscuous mode. Additionally, I think most of us expect 
 some
 basic Linux & network administration knowledge when using Metron.

 2- Making a kafka topic to push those saved logs in metron for
> preprocessing
>

 Have a look at the Metron documentation at
 https://metron.apache.org/current-book/index.html. Adding a new
 sensor in the Metron UI will create the Kafka iirc.

 3- Applying a basic Machine learning algorithm on the captured data.
>

 I can't help you with this :)

>>>
>>>
>> --
>
> Jon
>


>>>
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>

Re: Snort Installation

[Nick Allen]

A quick google search will answer your question.



On Wed, Oct 25, 2017 at 1:30 AM Syed Hammad Tahir 
wrote:

> Where do I find this file kafka-console-producer.sh?
>
> On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen  wrote:
>
>> Take a look at `kafka-console-producer.sh`, which is installed as part of
>> Kafka.
>>
>> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir 
>> wrote:
>>
>>> Ok, I have fixed everything on my own. Now that I have snort logs saved
>>> in a file, I need to get them to metron. Can anyone help me on that?
>>>
>>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> > wrote:
>>>
 yes nut I am a bit confused here. Let me ask them as well then.

 On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com 
 wrote:

> Hi Syed,
>
> Just to clarify, this a snort issue you are having?  If so I suggest
> looking at their documentation (https://snort.org/documents) or
> reaching out to their community (https://snort.org/community), as
> they have more expertise in this area.
>
> Jon
>
> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir 
> wrote:
>
>> Hi guys,
>>
>> I tried to add another network interface in order to bridge it to
>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up
>> after that, there was no bridged interface. Can anyone help me on this?
>>
>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Ok, thankyou. I will let you know once  I make snort sniff the
>>> traffic in the given configuration, might be helpful for others. I will
>>> then try to do that kafka topic and will ask if any help is needed.
>>>
>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets 
>>> wrote:
>>>
 Hi Syed,

 See inline.

 On 2017-10-20 00:32, Syed Hammad Tahir wrote:

> I have installed the snort manually. Now I need help with :
>
> 1- Capturing the data of my lan and dumping it via snort :Snort
> cant see the traffic outside vagrant vm, how do I make it see that 
> traffic?
>

 To be honest, configuring Snort to work on your LAN is out of scope
 of the project. Have a look at the documentation at
 https://www.snort.org/.
 You will probably have to add a 2nd network interface bridged to
 your LAN in promiscuous mode. Additionally, I think most of us expect 
 some
 basic Linux & network administration knowledge when using Metron.

 2- Making a kafka topic to push those saved logs in metron for
> preprocessing
>

 Have a look at the Metron documentation at
 https://metron.apache.org/current-book/index.html. Adding a new
 sensor in the Metron UI will create the Kafka iirc.

 3- Applying a basic Machine learning algorithm on the captured data.
>

 I can't help you with this :)

>>>
>>>
>> --
>
> Jon
>


>>>
>>
>

Re: Snort Logs

[zeo...@gmail.com]

If you have text snort logs you can use Apache nifi or the Kafka producer
script as described in step 4 here[1] to push them to Metron's snort
topic.  You may also want to look at this [2].

1: https://kafka.apache.org/quickstart
2:
https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script

Jon

On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir  wrote:

> Hello everyone,
>
> I have run snort independently on vagrant ssh and dumped the logs in
> tcpdump format. Now I want to bring them to metron to play with them a bit.
> Some of you already replied me with some solutions but thats lost in the
> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please
> give me an easy to understand this solution for this problem.
>
> Regards.
>
-- 

Jon

Re: Snort Logs

[Syed Hammad Tahir]

snort logs are in tcp dump format. I may have to convert them.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

How to give file name or path in this command?

On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com  wrote:

> If you have text snort logs you can use Apache nifi or the Kafka producer
> script as described in step 4 here[1] to push them to Metron's snort
> topic.  You may also want to look at this [2].
>
> 1: https://kafka.apache.org/quickstart
> 2: https://stackoverflow.com/questions/38701179/kafka-
> console-producer-and-bash-script
>
> Jon
>
> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
> wrote:
>
>> Hello everyone,
>>
>> I have run snort independently on vagrant ssh and dumped the logs in
>> tcpdump format. Now I want to bring them to metron to play with them a bit.
>> Some of you already replied me with some solutions but thats lost in the
>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please
>> give me an easy to understand this solution for this problem.
>>
>> Regards.
>>
> --
>
> Jon
>

Re: Snort Logs

[zeo...@gmail.com]

On the 25th I said:

 It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from
memory) on node1, assuming you are running full dev.

 Jon


Jon

On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir 
wrote:

> snort logs are in tcp dump format. I may have to convert them.
>
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>
> How to give file name or path in this command?
>
> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
> wrote:
>
>> If you have text snort logs you can use Apache nifi or the Kafka producer
>> script as described in step 4 here[1] to push them to Metron's snort
>> topic.  You may also want to look at this [2].
>>
>> 1: https://kafka.apache.org/quickstart
>> 2:
>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script
>>
>> Jon
>>
>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I have run snort independently on vagrant ssh and dumped the logs in
>>> tcpdump format. Now I want to bring them to metron to play with them a bit.
>>> Some of you already replied me with some solutions but thats lost in the
>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please
>>> give me an easy to understand this solution for this problem.
>>>
>>> Regards.
>>>
>> --
>>
>> Jon
>>
>
> --

Jon

Re: Snort Logs

[Syed Hammad Tahir]

I have found the kafka-console-producer.sh but I need to know how to make
it read snort.log (tcp dump format) file. May be I am missing something in
the plain sight but it would be awsome if you tell me that.

Regards.

On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com  wrote:

> On the 25th I said:
>
>  It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from
> memory) on node1, assuming you are running full dev.
>
>  Jon
>
>
> Jon
>
> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir 
> wrote:
>
>> snort logs are in tcp dump format. I may have to convert them.
>>
>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>>
>> How to give file name or path in this command?
>>
>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
>> wrote:
>>
>>> If you have text snort logs you can use Apache nifi or the Kafka
>>> producer script as described in step 4 here[1] to push them to Metron's
>>> snort topic.  You may also want to look at this [2].
>>>
>>> 1: https://kafka.apache.org/quickstart
>>> 2: https://stackoverflow.com/questions/38701179/kafka-
>>> console-producer-and-bash-script
>>>
>>> Jon
>>>
>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
>>> wrote:
>>>
 Hello everyone,

 I have run snort independently on vagrant ssh and dumped the logs in
 tcpdump format. Now I want to bring them to metron to play with them a bit.
 Some of you already replied me with some solutions but thats lost in the
 inbox somewhere and engulfed by the elasticsearhc issue that I had. Please
 give me an easy to understand this solution for this problem.

 Regards.

>>> --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>

Re: Snort Logs

[zeo...@gmail.com]

You need text logs. Here's an example of some properly formatted logs -
https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out

Jon

On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir  wrote:

> I have found the kafka-console-producer.sh but I need to know how to make
> it read snort.log (tcp dump format) file. May be I am missing something in
> the plain sight but it would be awsome if you tell me that.
>
> Regards.
>
> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com 
> wrote:
>
>> On the 25th I said:
>>
>>  It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from
>> memory) on node1, assuming you are running full dev.
>>
>>  Jon
>>
>>
>> Jon
>>
>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir 
>> wrote:
>>
>>> snort logs are in tcp dump format. I may have to convert them.
>>>
>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>>>
>>> How to give file name or path in this command?
>>>
>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
>>> wrote:
>>>
 If you have text snort logs you can use Apache nifi or the Kafka
 producer script as described in step 4 here[1] to push them to Metron's
 snort topic.  You may also want to look at this [2].

 1: https://kafka.apache.org/quickstart
 2:
 https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script

 Jon

 On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
 wrote:

> Hello everyone,
>
> I have run snort independently on vagrant ssh and dumped the logs in
> tcpdump format. Now I want to bring them to metron to play with them a 
> bit.
> Some of you already replied me with some solutions but thats lost in the
> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please
> give me an easy to understand this solution for this problem.
>
> Regards.
>
 --

 Jon

>>>
>>> --
>>
>> Jon
>>
>
> --

Jon

Re: Snort Logs

[Syed Hammad Tahir]

Yes, I have converted them to text but those logs are simply captured
packet headers over the local network. Now I just push them via that kafka
producer command under topic name of snort and they will be visible in
metron?

On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com  wrote:

> You need text logs. Here's an example of some properly formatted logs -
> https://raw.githubusercontent.com/apache/metron/master/metron-
> deployment/roles/sensor-stubs/files/snort.out
>
> Jon
>
> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir 
> wrote:
>
>> I have found the kafka-console-producer.sh but I need to know how to
>> make it read snort.log (tcp dump format) file. May be I am missing
>> something in the plain sight but it would be awsome if you tell me that.
>>
>> Regards.
>>
>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com 
>> wrote:
>>
>>> On the 25th I said:
>>>
>>>  It should be in /usr/hdp/current/kafka-broker/bin/ or similar
>>> (from memory) on node1, assuming you are running full dev.
>>>
>>>  Jon
>>>
>>>
>>> Jon
>>>
>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir 
>>> wrote:
>>>
 snort logs are in tcp dump format. I may have to convert them.

 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

 How to give file name or path in this command?

 On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
 wrote:

> If you have text snort logs you can use Apache nifi or the Kafka
> producer script as described in step 4 here[1] to push them to Metron's
> snort topic.  You may also want to look at this [2].
>
> 1: https://kafka.apache.org/quickstart
> 2: https://stackoverflow.com/questions/38701179/kafka-
> console-producer-and-bash-script
>
> Jon
>
> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
> wrote:
>
>> Hello everyone,
>>
>> I have run snort independently on vagrant ssh and dumped the logs in
>> tcpdump format. Now I want to bring them to metron to play with them a 
>> bit.
>> Some of you already replied me with some solutions but thats lost in the
>> inbox somewhere and engulfed by the elasticsearhc issue that I had. 
>> Please
>> give me an easy to understand this solution for this problem.
>>
>> Regards.
>>
> --
>
> Jon
>

 --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>

Re: Snort Logs

[zeo...@gmail.com]

They need to meet the format of the logs I sent earlier.  Look into the
snort output options - may require you rerun snort, depending on your
situation

Jon

On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir  wrote:

> Yes, I have converted them to text but those logs are simply captured
> packet headers over the local network. Now I just push them via that kafka
> producer command under topic name of snort and they will be visible in
> metron?
>
> On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com 
> wrote:
>
>> You need text logs. Here's an example of some properly formatted logs -
>> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out
>>
>> Jon
>>
>> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir 
>> wrote:
>>
>>> I have found the kafka-console-producer.sh but I need to know how to
>>> make it read snort.log (tcp dump format) file. May be I am missing
>>> something in the plain sight but it would be awsome if you tell me that.
>>>
>>> Regards.
>>>
>>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com 
>>> wrote:
>>>
 On the 25th I said:

  It should be in /usr/hdp/current/kafka-broker/bin/ or similar
 (from memory) on node1, assuming you are running full dev.

  Jon


 Jon

 On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir 
 wrote:

> snort logs are in tcp dump format. I may have to convert them.
>
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>
> How to give file name or path in this command?
>
> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
> wrote:
>
>> If you have text snort logs you can use Apache nifi or the Kafka
>> producer script as described in step 4 here[1] to push them to Metron's
>> snort topic.  You may also want to look at this [2].
>>
>> 1: https://kafka.apache.org/quickstart
>> 2:
>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script
>>
>> Jon
>>
>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I have run snort independently on vagrant ssh and dumped the logs in
>>> tcpdump format. Now I want to bring them to metron to play with them a 
>>> bit.
>>> Some of you already replied me with some solutions but thats lost in the
>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. 
>>> Please
>>> give me an easy to understand this solution for this problem.
>>>
>>> Regards.
>>>
>> --
>>
>> Jon
>>
>
> --

 Jon

>>>
>>> --
>>
>> Jon
>>
>
> --

Jon

Re: Snort Logs

[Syed Hammad Tahir]

I sent a random message to that kafka topic and got this

[image: Inline image 1]

I guess this is because I am not following the format of message I should
send? Like those snort logs you showed.

On Mon, Oct 30, 2017 at 5:24 PM, zeo...@gmail.com  wrote:

> They need to meet the format of the logs I sent earlier.  Look into the
> snort output options - may require you rerun snort, depending on your
> situation
>
> Jon
>
> On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir 
> wrote:
>
>> Yes, I have converted them to text but those logs are simply captured
>> packet headers over the local network. Now I just push them via that kafka
>> producer command under topic name of snort and they will be visible in
>> metron?
>>
>> On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com 
>> wrote:
>>
>>> You need text logs. Here's an example of some properly formatted logs -
>>> https://raw.githubusercontent.com/apache/metron/master/metron-
>>> deployment/roles/sensor-stubs/files/snort.out
>>>
>>> Jon
>>>
>>> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir 
>>> wrote:
>>>
 I have found the kafka-console-producer.sh but I need to know how to
 make it read snort.log (tcp dump format) file. May be I am missing
 something in the plain sight but it would be awsome if you tell me that.

 Regards.

 On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com 
 wrote:

> On the 25th I said:
>
>  It should be in /usr/hdp/current/kafka-broker/bin/ or similar
> (from memory) on node1, assuming you are running full dev.
>
>  Jon
>
>
> Jon
>
> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> snort logs are in tcp dump format. I may have to convert them.
>>
>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
>> test
>>
>> How to give file name or path in this command?
>>
>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com 
>> wrote:
>>
>>> If you have text snort logs you can use Apache nifi or the Kafka
>>> producer script as described in step 4 here[1] to push them to Metron's
>>> snort topic.  You may also want to look at this [2].
>>>
>>> 1: https://kafka.apache.org/quickstart
>>> 2: https://stackoverflow.com/questions/38701179/kafka-
>>> console-producer-and-bash-script
>>>
>>> Jon
>>>
>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir 
>>> wrote:
>>>
 Hello everyone,

 I have run snort independently on vagrant ssh and dumped the logs
 in tcpdump format. Now I want to bring them to metron to play with 
 them a
 bit. Some of you already replied me with some solutions but thats lost 
 in
 the inbox somewhere and engulfed by the elasticsearhc issue that I had.
 Please give me an easy to understand this solution for this problem.

 Regards.

>>> --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>

 --
>>>
>>> Jon
>>>
>>
>> --
>
> Jon
>

Re: Snort Logs

[Syed Hammad Tahir]

And how do I install elasticsearch head on the vagrant VM?

Re: Snort Logs

[Otto Fowler]

You can install it into the chrome web browser from the play store.



On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk)
wrote:

And how do I install elasticsearch head on the vagrant VM?

Re: Snort Logs

[zeo...@gmail.com]

It looks like your ES cluster has a health of Red, so there's your
problem.  I would go look in /var/log/elasticsearch/ at some logs.

Jon

On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir 
wrote:

>
> -- Forwarded message --
> From: Syed Hammad Tahir 
> Date: Fri, Nov 3, 2017 at 5:07 PM
> Subject: Re: Snort Logs
> To: Otto Fowler 
>
>
> NVM, I have installed the elastic search head. Now where do I go in this
> to find out why I cant see the snort logs in kibana dashboard, pushed to
> snort topic via kafka producer?
>
> [image: Inline image 1]
>
> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler 
> wrote:
>
>> You can install it into the chrome web browser from the play store.
>>
>>
>>
>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk)
>> wrote:
>>
>> And how do I install elasticsearch head on the vagrant VM?
>>
>>
> --

Jon

Re: Snort Logs

[Syed Hammad Tahir]

hi, I am back at work. lets see if i can find something in logs

On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com  wrote:

> It looks like your ES cluster has a health of Red, so there's your
> problem.  I would go look in /var/log/elasticsearch/ at some logs.
>
> Jon
>
> On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir 
> wrote:
>
>>
>> -- Forwarded message --
>> From: Syed Hammad Tahir 
>> Date: Fri, Nov 3, 2017 at 5:07 PM
>> Subject: Re: Snort Logs
>> To: Otto Fowler 
>>
>>
>> NVM, I have installed the elastic search head. Now where do I go in this
>> to find out why I cant see the snort logs in kibana dashboard, pushed to
>> snort topic via kafka producer?
>>
>> [image: Inline image 1]
>>
>> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler 
>> wrote:
>>
>>> You can install it into the chrome web browser from the play store.
>>>
>>>
>>>
>>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk)
>>> wrote:
>>>
>>> And how do I install elasticsearch head on the vagrant VM?
>>>
>>>
>> --
>
> Jon
>

Re: Snort Logs

[Syed Hammad Tahir]

gt;>>>>>>>>>
>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the logs
>>>>>>>>>>>>> I am trying to push
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data
>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the same 
>>>>>>>>>>>>> thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir <
>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>> here: https://raw.githubusercontent.com/apache/metron/master/
>>>>>>>>>>>>>> metron-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines
>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This was the mechanism I am using to push the logs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler <
>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What I mean is this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort
>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think 
>>>>>>>>>>>>>>> of as live.
>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see 
>>>>>>>>>>>>>>> this error.
>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for things
>>>>>>>>>>>>>>> that are automatically pushed to kafka as you do when you 
>>>>>>>>>>>>>>> manual push them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On November 7, 2017 at 08:51:41, Syed Hammad Tahir (
>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Yes, If the messages cannot be parsed then that would be a
>>>>>>>>>>>>>>> problem.  If you see this error with your ‘live’ messages as 
>>>>>>>>>>>>>>> well then that
>>>>>>>>>>>>>>> could be it.
>>>>>>>>>>>>>>> I wonder if the issue is with the date format?"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If by 'live' messages you mean the time I push them into
>>>>>>>>>>>>>>> kafka topic then no, I dont see 

Re: Snort Logs

[Syed Hammad Tahir]

t;>>>>>>
>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler (
>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column
>>>>>>>>>>>>>> is for a piece of data that is failing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir (
>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the logs
>>>>>>>>>>>>>> I am trying to push
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data
>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the 
>>>>>>>>>>>>>> same thing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir <
>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>>> here: https://raw.githubusercontent.
>>>>>>>>>>>>>>> com/apache/metron/master/metron-deployment/roles/sensor-
>>>>>>>>>>>>>>> stubs/files/snort.out
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines
>>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This was the mechanism I am using to push the logs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler <
>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What I mean is this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort
>>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think 
>>>>>>>>>>>>>>>> of as live.
>>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see 
>>>>>>>>>>>>>>>> this error.
>>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for
>>>>>>>>>>>>>>>> things that are automatically pushed to kafka as you do when 
>>>>>>>>>>>>>>>> you manual
>&g

Re: Snort Logs

[Syed Hammad Tahir]

>>>>>> private transient DateTimeFormatter dateTimeFormatter;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If your records are in dd/MM/yy-  format, then you may see
>>>>>>>>>>>>>>> this error I believe.
>>>>>>>>>>>>>>> Can you verify the timestamp field’s format?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If this is the case, then you will need to modify the
>>>>>>>>>>>>>>> default log timestamp format for snort in the short term.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler (
>>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column
>>>>>>>>>>>>>>> is for a piece of data that is failing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir (
>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the
>>>>>>>>>>>>>>> logs I am trying to push
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data
>>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the 
>>>>>>>>>>>>>>> same thing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir <
>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>>>> here: https://raw.githubusercontent.
>>>>>>>>>>>>>>>> com/apache/metron/master/metro
>>>>>>>>>>>>>>>> n-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines
>>>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This was the mechanism I am using to push the logs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler <
>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>&

Re: Snort Logs

[zeo...@gmail.com]

 problems
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Nov 8, 2017, 06:59 Otto Fowler <
>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The snort parser is coded to support dates in this format:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private static String defaultDateFormat = 
>>>>>>>>>>>>>>>> "MM/dd/yy-HH:mm:ss.SS";
>>>>>>>>>>>>>>>> private transient DateTimeFormatter dateTimeFormatter;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If your records are in dd/MM/yy-  format, then you may see
>>>>>>>>>>>>>>>> this error I believe.
>>>>>>>>>>>>>>>> Can you verify the timestamp field’s format?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If this is the case, then you will need to modify the
>>>>>>>>>>>>>>>> default log timestamp format for snort in the short term.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler (
>>>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column
>>>>>>>>>>>>>>>> is for a piece of data that is failing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir (
>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the
>>>>>>>>>>>>>>>> logs I am trying to push
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data
>>>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the 
>>>>>>>>>>>>>>>> same thing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [image: Inline i

Re: Snort Logs

[Syed Hammad Tahir]

hing my logs to kafka topic
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>>>>>> here: https://raw.githubusercontent.
>>>>>>>>>>>>>>>>>> com/apache/metron/master/metron-deployment/roles/
>>>>>>>>>>>>>>>>>> sensor-stubs/files/snort.out
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not getting any error here. I can also see these
>>>>>>>>>>>>>>>>>> lines pushed out via kafka consumer under topic of snort.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This was the mechanism I am using to push the logs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler <
>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What I mean is this:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort
>>>>>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I 
>>>>>>>>>>>>>>>>>>> think of as live.
>>>>>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you 
>>>>>>>>>>>>>>>>>>> see this error.
>>>>>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for
>>>>>>>>>>>>>>>>>>> things that are automatically pushed to kafka as you do 
>>>>>>>>>>>>>>>>>>> when you manual
>>>>>>>>>>>>>>>>>>> push them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On November 7, 2017 at 08:51:41, Syed Hammad Tahir (
>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> "Yes, If the messages cannot be parsed then that would
>>>>>>>>>>>>>>>>>>> be a problem.  If you see this error with your ‘live’ 
>>>>>>>>>>>>>>>>>>> messages as well then
>>>>>>>>>>>>>>>>>>> that could be it.
>>>>>>>>>>>>>>>>>>> I wonder if the issue is with the date format?"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If by 'live' messages you mean the time I push them into
>>>>>>>>>>>>>>>>>>> kafka topic then no, I dont see any error at that time. If 
>>>>>>>>>>>>>>>>>>> 'live' means
>>>>>>>>>>>>>>>>>>> something else here then please tell me what could it be.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 5:07 PM, Otto Fowler <
>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes, If the messages cannot be parsed then that would
>>>>>>>>>>>>>>>>>>>> be a problem.  If you see this error with your ‘live’ 
>>>>>>>>>>>>>>>>>>>> messages as well then
>>>>>>>>>>>>>>>>>>>> that could be it.
>>>>>>>>>>>>>>>>>>>> I wonder if the issue is with the date format?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You need to confirm that you see these same errors with
>>>>>>>>>>>>>>>>>>>> the live data or not.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Remember, the flow is like this
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> snort -> ??? -> Kafka -> Storm Parser Topology -> kafka
>>>>>>>>>>>>>>>>>>>> -> Storm Enrichment Topology -> Kafka -> Storm Indexing 
>>>>>>>>>>>>>>>>>>>> Topology -> HDFS |
>>>>>>>>>>>>>>>>>>>> ElasticSearch
>>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> Kibana <-> Elastic Search
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Any point in this chain could fail and result in Kibana
>>>>>>>>>>>>>>>>>>>> not seeing things.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On November 7, 2017 at 01:57:19, Syed Hammad Tahir (
>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> could this be related to why I am unable to see logs in
>>>>>>>>>>>>>>>>>>>> kibana dashboard?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am copying a few lines from here
>>>>>>>>>>>>>>>>>>>> https://raw.githubusercontent.com/apache/metron/master/
>>>>>>>>>>>>>>>>>>>> metron-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> and then pushing them to snort kafka topic.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> THis is some error I am seeing in stormUI parser bolt
>>>>>>>>>>>>>>>>>>>> in snort section:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 11:49 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I guess I have hit a dead end. I am not able to get
>>>>>>>>>>>>>>>>>>>>> the snort logs in kibana dashboard. Any help will be 
>>>>>>>>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 1:24 PM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I guess this (metron.log) in /var/log/elasticsearch/
>>>>>>>>>>>>>>>>>>>>>> is also relevant
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 11:46 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cluster health by index shows this:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> looks like some shard is unassigned and that is
>>>>>>>>>>>>>>>>>>>>>>> related to snort. Could it be the logs I was pushing to 
>>>>>>>>>>>>>>>>>>>>>>> kafka topic earlier?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:47 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This is what I see here. What should I be looking
>>>>>>>>>>>>>>>>>>>>>>>> at here?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:33 AM, Syed Hammad Tahir
>>>>>>>>>>>>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> hi, I am back at work. lets see if i can find
>>>>>>>>>>>>>>>>>>>>>>>>> something in logs
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com <
>>>>>>>>>>>>>>>>>>>>>>>>> zeo...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> It looks like your ES cluster has a health of
>>>>>>>>>>>>>>>>>>>>>>>>>> Red, so there's your problem.  I would go look in 
>>>>>>>>>>>>>>>>>>>>>>>>>> /var/log/elasticsearch/
>>>>>>>>>>>>>>>>>>>>>>>>>> at some logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir
>>>>>>>>>>>>>>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -- Forwarded message --
>>>>>>>>>>>>>>>>>>>>>>>>>>> From: Syed Hammad Tahir 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: Fri, Nov 3, 2017 at 5:07 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Snort Logs
>>>>>>>>>>>>>>>>>>>>>>>>>>> To: Otto Fowler 
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> NVM, I have installed the elastic search head.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Now where do I go in this to find out why I cant 
>>>>>>>>>>>>>>>>>>>>>>>>>>> see the snort logs in
>>>>>>>>>>>>>>>>>>>>>>>>>>> kibana dashboard, pushed to snort topic via kafka 
>>>>>>>>>>>>>>>>>>>>>>>>>>> producer?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can install it into the chrome web browser
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the play store.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On November 3, 2017 at 07:47:47, Syed Hammad
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tahir (mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> And how do I install elasticsearch head on the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> vagrant VM?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>
>>>>
>>>
>> --
>
> Jon
>

Re: Snort Logs

[Otto Fowler]

I guess I am wrong.
But from looking at the output, it looks like this is error topic stuff
that is failing doesn’t it?



On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com) wrote:

Isn't sending indexing errors to the indexing topic intentional?  I may
need to refresh myself on the below conversation, but I recall it coming up
in conversation on the mailing lists in the past.

https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33
https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E

Jon

On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler  wrote:

> OK.
>
> I think your sending errors to your indexing topic instead of the error
> topic.
> I think you posted your config before, but I don’t remember off the top of
> my head
> where the error topic is configured.
>
> If the error topic is the same as the indexing topic, and you ‘have
> errors’  I think you may see this.
>
>
>
> On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk)
> wrote:
>
> Here we go. This is what I see when I do kafka client on indexing topic.
>
> [image: Inline image 1]
>
> On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir 
> wrote:
>
>> ok, I will try it again and report results
>>
>> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler 
>> wrote:
>>
>>> You have to be seeing data in the indexing topic, you have errors in the
>>> indexing topology that reads from it.
>>>
>>>
>>>
>>> On November 13, 2017 at 13:42:14, Syed Hammad Tahir (
>>> mscs16...@itu.edu.pk) wrote:
>>>
>>> So you are saying:
>>>
>>> * when you do the kafka client on the enrichment topic things are in json
>>> * when you do the kafka client on the indexing topic they are csv
>>>
>>> 1- Yes, kafka client on enrichment shows json
>>>
>>> 2- No, I dont see anything in kafka client on indexing topic
>>>
>>> On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler 
>>> wrote:
>>>
 So you are saying:

 * when you do the kafka client on the enrichment topic things are in
 json
 * when you do the kafka client on the indexing topic they are csv

 ???



 On November 13, 2017 at 12:28:51, Syed Hammad Tahir (
 mscs16...@itu.edu.pk) wrote:

 From one of your earlier messages, This is what I have figured out so
 far.

 [image: Inline image 1]

 The issue is inducated by red marked portion of the flow.

 On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Which .java file is causing the issue in this hdfsindexbolt. I mean
> which one should I look at because there are so many listed here.
>
> [image: Inline image 1]
>
> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it 
>> correctly since I am getting error in the indexing bolt not in the 
>> parser one.
>>
>>
>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> org.apache.metron.parsers.snort.BasicSnortParser does this parse the 
>>> basic message and then convert it in JSON?
>>>
>>>
>>> On Mon, Nov 13, 2017 at 9:00 PM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 No, I am not seeing it under indexing topic as JSON. I can only see
 JSON objects of stub sensor logs but not from those pushed by me via 
 kafka
 producer.

 On Mon, Nov 13, 2017 at 5:17 PM, zeo...@gmail.com >>> > wrote:

> Please use kafka-console-consumer.sh (same folder as the producer
> script) and pull from the indexing topic.  Are you seeing it in JSON 
> there?
>
> Jon
>
> On Mon, Nov 13, 2017 at 7:03 AM Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Kindly give me the mechanism implemented in metron through which
>> a line such as this
>>
>> 01/11/17-20:49:18.107168 ,1,999158,0,"'snort test 
>> alert'",TCP,192.168.66.1,49581,192.168.66.121,22,0A:00:27:00:00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC,0x56900BB6,,0x1000,64,10,23403,76,77824
>>
>> is converted into a json object. Maybe I am missing something here 
>> is the formatting.
>>
>>
>>
>>
>> On Mon, Nov 13, 2017 at 3:21 PM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Restarted snort, still giving me error for indexing topologies
>>> even though I havent even pushed out any data to snort topic yet. I 
>>> have
>>> not run the kafka-producer command but its still giving error for
>>> something.
>>>
>>> [image: Inline image 1]
>>>
>>>

Re: Snort Logs

[Otto Fowler]

OK,
So the JSONFromPosition MessageGetStrategy is choking the csv…

I don’t know if there is a position index or it is 0, but whatever it is,
it is getting csv not json message.




On November 13, 2017 at 15:49:57, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I guess I am wrong.
But from looking at the output, it looks like this is error topic stuff
that is failing doesn’t it?



On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com) wrote:

Isn't sending indexing errors to the indexing topic intentional?  I may
need to refresh myself on the below conversation, but I recall it coming up
in conversation on the mailing lists in the past.

https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33
https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E

Jon

On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler  wrote:

> OK.
>
> I think your sending errors to your indexing topic instead of the error
> topic.
> I think you posted your config before, but I don’t remember off the top of
> my head
> where the error topic is configured.
>
> If the error topic is the same as the indexing topic, and you ‘have
> errors’  I think you may see this.
>
>
>
> On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk)
> wrote:
>
> Here we go. This is what I see when I do kafka client on indexing topic.
>
> [image: Inline image 1]
>
> On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir 
> wrote:
>
>> ok, I will try it again and report results
>>
>> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler 
>> wrote:
>>
>>> You have to be seeing data in the indexing topic, you have errors in the
>>> indexing topology that reads from it.
>>>
>>>
>>>
>>> On November 13, 2017 at 13:42:14, Syed Hammad Tahir (
>>> mscs16...@itu.edu.pk) wrote:
>>>
>>> So you are saying:
>>>
>>> * when you do the kafka client on the enrichment topic things are in json
>>> * when you do the kafka client on the indexing topic they are csv
>>>
>>> 1- Yes, kafka client on enrichment shows json
>>>
>>> 2- No, I dont see anything in kafka client on indexing topic
>>>
>>> On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler 
>>> wrote:
>>>
 So you are saying:

 * when you do the kafka client on the enrichment topic things are in
 json
 * when you do the kafka client on the indexing topic they are csv

 ???



 On November 13, 2017 at 12:28:51, Syed Hammad Tahir (
 mscs16...@itu.edu.pk) wrote:

 From one of your earlier messages, This is what I have figured out so
 far.

 [image: Inline image 1]

 The issue is inducated by red marked portion of the flow.

 On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> Which .java file is causing the issue in this hdfsindexbolt. I mean
> which one should I look at because there are so many listed here.
>
> [image: Inline image 1]
>
> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it 
>> correctly since I am getting error in the indexing bolt not in the 
>> parser one.
>>
>>
>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> org.apache.metron.parsers.snort.BasicSnortParser does this parse the 
>>> basic message and then convert it in JSON?
>>>
>>>
>>> On Mon, Nov 13, 2017 at 9:00 PM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 No, I am not seeing it under indexing topic as JSON. I can only see
 JSON objects of stub sensor logs but not from those pushed by me via 
 kafka
 producer.

 On Mon, Nov 13, 2017 at 5:17 PM, zeo...@gmail.com >>> > wrote:

> Please use kafka-console-consumer.sh (same folder as the producer
> script) and pull from the indexing topic.  Are you seeing it in JSON 
> there?
>
> Jon
>
> On Mon, Nov 13, 2017 at 7:03 AM Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Kindly give me the mechanism implemented in metron through which
>> a line such as this
>>
>> 01/11/17-20:49:18.107168 ,1,999158,0,"'snort test 
>> alert'",TCP,192.168.66.1,49581,192.168.66.121,22,0A:00:27:00:00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC,0x56900BB6,,0x1000,64,10,23403,76,77824
>>
>> is converted into a json object. Maybe I am missing something here 
>> is the formatting.
>>
>>
>>
>>
>> On Mon, Nov 13, 2017 at 3:21 PM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Restarted snort, still giving me error for indexing topologies

Re: Snort Logs

[zeo...@gmail.com]

Yeah, I think you're right.  I'm not familiar with that code as much
though, I've never had to touch it.

Given all of the lessons learned up to this point, maybe spinning full-dev
up from scratch again makes sense?  Typically I am able to spin up full-dev
pretty hands-off without hitting these types of issues, as long as
my machine has enough resources to give.

Jon

On Mon, Nov 13, 2017 at 3:49 PM Otto Fowler  wrote:

> I guess I am wrong.
> But from looking at the output, it looks like this is error topic stuff
> that is failing doesn’t it?
>
>
>
> On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Isn't sending indexing errors to the indexing topic intentional?  I may
> need to refresh myself on the below conversation, but I recall it coming up
> in conversation on the mailing lists in the past.
>
>
> https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33
>
> https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E
>
> Jon
>
> On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler 
> wrote:
>
>> OK.
>>
>> I think your sending errors to your indexing topic instead of the error
>> topic.
>> I think you posted your config before, but I don’t remember off the top
>> of my head
>> where the error topic is configured.
>>
>> If the error topic is the same as the indexing topic, and you ‘have
>> errors’  I think you may see this.
>>
>>
>>
>> On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk)
>> wrote:
>>
>> Here we go. This is what I see when I do kafka client on indexing topic.
>>
>> [image: Inline image 1]
>>
>> On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir > > wrote:
>>
>>> ok, I will try it again and report results
>>>
>>> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler 
>>> wrote:
>>>
 You have to be seeing data in the indexing topic, you have errors in
 the indexing topology that reads from it.



 On November 13, 2017 at 13:42:14, Syed Hammad Tahir (
 mscs16...@itu.edu.pk) wrote:

 So you are saying:

 * when you do the kafka client on the enrichment topic things are in
 json
 * when you do the kafka client on the indexing topic they are csv

 1- Yes, kafka client on enrichment shows json

 2- No, I dont see anything in kafka client on indexing topic

 On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler 
 wrote:

> So you are saying:
>
> * when you do the kafka client on the enrichment topic things are in
> json
> * when you do the kafka client on the indexing topic they are csv
>
> ???
>
>
>
> On November 13, 2017 at 12:28:51, Syed Hammad Tahir (
> mscs16...@itu.edu.pk) wrote:
>
> From one of your earlier messages, This is what I have figured out so
> far.
>
> [image: Inline image 1]
>
> The issue is inducated by red marked portion of the flow.
>
> On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir <
> mscs16...@itu.edu.pk> wrote:
>
>> Which .java file is causing the issue in this hdfsindexbolt. I mean
>> which one should I look at because there are so many listed here.
>>
>> [image: Inline image 1]
>>
>> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it 
>>> correctly since I am getting error in the indexing bolt not in the 
>>> parser one.
>>>
>>>
>>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir <
>>> mscs16...@itu.edu.pk> wrote:
>>>
 org.apache.metron.parsers.snort.BasicSnortParser does this parse the 
 basic message and then convert it in JSON?


 On Mon, Nov 13, 2017 at 9:00 PM, Syed Hammad Tahir <
 mscs16...@itu.edu.pk> wrote:

> No, I am not seeing it under indexing topic as JSON. I can only
> see JSON objects of stub sensor logs but not from those pushed by me 
> via
> kafka producer.
>
> On Mon, Nov 13, 2017 at 5:17 PM, zeo...@gmail.com <
> zeo...@gmail.com> wrote:
>
>> Please use kafka-console-consumer.sh (same folder as the producer
>> script) and pull from the indexing topic.  Are you seeing it in JSON 
>> there?
>>
>> Jon
>>
>> On Mon, Nov 13, 2017 at 7:03 AM Syed Hammad Tahir <
>> mscs16...@itu.edu.pk> wrote:
>>
>>> Kindly give me the mechanism implemented in metron through which
>>> a line such as this
>>>
>>> 01/11/17-20:49:18.107168 ,1,999158,0,"'snort test 
>>> alert'",TCP,192.168.66.1,49581,192.168.66.121,22,0A:00:27:00:00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC,0x56900BB6,,0x1000,64,10,23403,76,77824
>>>
>>

Re: Snort enrichment issue

[Simon Elliston Ball]

Did you setup and load the geo enrichment database? 
https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader
 


Also, we can’t really see the error from screenshots, please send log entries. 

Simon

> On 17 Nov 2017, at 07:11, Syed Hammad Tahir  wrote:
> 
> Hi all, I am starting it again. Last one got a bit messy
> 
> Ok, Now I have started everything again from scratch (redeployed single node 
> based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when 
> I execute this command: 
> 
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date 
> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | 
> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
> node1:6667 --topic snort
> 
> (format of ths command was taken from: 
> https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub
>  
> )
> 
> I get this under enrichment storm topology :
> 
> 
> 
> 
> 
> I have come this far, please help me push these dummy preformatted snort logs 
> into kibana dashboard.
> 
> Regards.
> 

Re: Snort enrichment issue

[Syed Hammad Tahir]

Here you go, the error part of the log is in the attachment.

On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Did you setup and load the geo enrichment database? https://metron.
> apache.org/current-book/metron-platform/metron-data-management/index.html#
> GeoLite2_Loader
>
> Also, we can’t really see the error from screenshots, please send log
> entries.
>
> Simon
>
> On 17 Nov 2017, at 07:11, Syed Hammad Tahir  wrote:
>
> Hi all, I am starting it again. Last one got a bit messy
>
> Ok, Now I have started everything again from scratch (redeployed single
> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
> now when I execute this command:
>
> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | 
> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
> --broker-list node1:6667 --topic snort
>
> (format of ths command was taken from: https://github.com/apach
> e/metron/blob/master/metron-deployment/roles/sensor-stubs/
> templates/start-snort-stub)
>
> I get this under enrichment storm topology :
>
> 
>
> 
>
> I have come this far, please help me push these dummy preformatted snort
> logs into kibana dashboard.
>
> Regards.
>
>
>


Enrichment
Description: Binary data

Re: Snort enrichment issue

[Syed Hammad Tahir]

ANd I dint load anything. It was supposed to be loaded during installation?
My installation is ambari based single node VM install on ubuntu host.

On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir 
wrote:

> Here you go, the error part of the log is in the attachment.
>
> On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Did you setup and load the geo enrichment database? https://metron.apach
>> e.org/current-book/metron-platform/metron-data-managemen
>> t/index.html#GeoLite2_Loader
>>
>> Also, we can’t really see the error from screenshots, please send log
>> entries.
>>
>> Simon
>>
>> On 17 Nov 2017, at 07:11, Syed Hammad Tahir  wrote:
>>
>> Hi all, I am starting it again. Last one got a bit messy
>>
>> Ok, Now I have started everything again from scratch (redeployed single
>> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and
>> now when I execute this command:
>>
>> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date
>> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | 
>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>> --broker-list node1:6667 --topic snort
>>
>> (format of ths command was taken from: https://github.com/apach
>> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te
>> mplates/start-snort-stub)
>>
>> I get this under enrichment storm topology :
>>
>> 
>>
>> 
>>
>> I have come this far, please help me push these dummy preformatted snort
>> logs into kibana dashboard.
>>
>> Regards.
>>
>>
>>
>

Re: Snort logs flow issue

[Michael Miklavcic]

How did you validate the logs are making it to the indexing topology?

On Fri, Apr 5, 2019 at 8:12 AM Hema malini  wrote:

>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
> sent the sample snort logs copied from metron git repo to snort kafka
> topic.We did the same for bro topic.Logs are getting parsed and reached
> indexing topology . Elastic search indices are not getting created though
> we gave elastic search template install from ambari. So manually created
> the elastic search index using template available in metron repo. Though
> elastic search index is present , data from indexing toplogy neither
> reached elastic search nor hdfs path .There are no errors in storm toplogy
> logs.We could see the sample log in Metron management ui. How we can send
> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
> with no data. Elasticsearch health is yellow and we are able to insert data
> via rest call. Any documentation on sending the smaple snort logs to metron
> alerts ui will be helpful . Any configuration from metron management ui is
> required to pass it to alerts –ui
>
>
>
>
>
> Thanks and Regards
>
> Hema
>
>
>
>
>
>
>

Re: Snort logs flow issue

[Hema malini]

We verified it in Storm ui and in Storm topology logs

On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic 
wrote:

> How did you validate the logs are making it to the indexing topology?
>
> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
> wrote:
>
>>
>> Hi,
>>
>>
>>
>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>> sent the sample snort logs copied from metron git repo to snort kafka
>> topic.We did the same for bro topic.Logs are getting parsed and reached
>> indexing topology . Elastic search indices are not getting created though
>> we gave elastic search template install from ambari. So manually created
>> the elastic search index using template available in metron repo. Though
>> elastic search index is present , data from indexing toplogy neither
>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>> logs.We could see the sample log in Metron management ui. How we can send
>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>> with no data. Elasticsearch health is yellow and we are able to insert data
>> via rest call. Any documentation on sending the smaple snort logs to metron
>> alerts ui will be helpful . Any configuration from metron management ui is
>> required to pass it to alerts –ui
>>
>>
>>
>>
>>
>> Thanks and Regards
>>
>> Hema
>>
>>
>>
>>
>>
>>
>>
>

Re: Snort logs flow issue

[Michael Miklavcic]

Do you get 10 records output to the CLI when you run the following?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10


On Fri, Apr 5, 2019 at 11:38 AM Hema malini  wrote:

> We verified it in Storm ui and in Storm topology logs
>
> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> How did you validate the logs are making it to the indexing topology?
>>
>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
>> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>> sent the sample snort logs copied from metron git repo to snort kafka
>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>> indexing topology . Elastic search indices are not getting created though
>>> we gave elastic search template install from ambari. So manually created
>>> the elastic search index using template available in metron repo.
>>> Though elastic search index is present , data from indexing toplogy neither
>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>> logs.We could see the sample log in Metron management ui. How we can send
>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>> with no data. Elasticsearch health is yellow and we are able to insert data
>>> via rest call. Any documentation on sending the smaple snort logs to metron
>>> alerts ui will be helpful . Any configuration from metron management ui is
>>> required to pass it to alerts –ui
>>>
>>>
>>>
>>>
>>>
>>> Thanks and Regards
>>>
>>> Hema
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: Snort logs flow issue

[Hema malini]

Yes I am getting messages

On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic 
wrote:

> Do you get 10 records output to the CLI when you run the following?
>
> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>
>
> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
> wrote:
>
>> We verified it in Storm ui and in Storm topology logs
>>
>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> How did you validate the logs are making it to the indexing topology?
>>>
>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
>>> wrote:
>>>

 Hi,



 We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
 sent the sample snort logs copied from metron git repo to snort kafka
 topic.We did the same for bro topic.Logs are getting parsed and reached
 indexing topology . Elastic search indices are not getting created though
 we gave elastic search template install from ambari. So manually created
 the elastic search index using template available in metron repo.
 Though elastic search index is present , data from indexing toplogy neither
 reached elastic search nor hdfs path .There are no errors in storm toplogy
 logs.We could see the sample log in Metron management ui. How we can send
 the logs to alerts ui and kibana dashboard. In kibana dashboard we could
 see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
 with no data. Elasticsearch health is yellow and we are able to insert data
 via rest call. Any documentation on sending the smaple snort logs to metron
 alerts ui will be helpful . Any configuration from metron management ui is
 required to pass it to alerts –ui





 Thanks and Regards

 Hema







>>>

Re: Snort logs flow issue

[Hema malini]

Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}


On Fri, Apr 5, 2019, 11:43 PM Hema malini  wrote:

> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
 How did you validate the logs are making it to the indexing topology?

 On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
 wrote:

>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
> sent the sample snort logs copied from metron git repo to snort kafka
> topic.We did the same for bro topic.Logs are getting parsed and reached
> indexing topology . Elastic search indices are not getting created though
> we gave elastic search template install from ambari. So manually created
> the elastic search index using template available in metron repo.
> Though elastic search index is present , data from indexing toplogy 
> neither
> reached elastic search nor hdfs path .There are no errors in storm toplogy
> logs.We could see the sample log in Metron management ui. How we can send
> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
> with no data. Elasticsearch health is yellow and we are able to insert 
> data
> via rest call. Any documentation on sending the smaple snort logs to 
> metron
> alerts ui will be helpful . Any configuration from metron management ui is
> required to pass it to alerts –ui
>
>
>
>
>
> Thanks and Regards
>
> Hema
>
>
>
>
>
>
>

Re: Snort logs flow issue

[Hema malini]

Are we missing any configuration? Initially elastic search was down. We
figured out the issue and fixed it .Now elastic search is up . We restarted
metron indexing but still those indices not created. So we created it
manually.Do we have to change any parser configuration . How logs will flow
into metron alerts dashboard and kibana dashboard..what is the required
congratulation

On Fri, Apr 5, 2019, 11:52 PM Hema malini  wrote:

> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini  wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
>>> wrote:
>>>
 We verified it in Storm ui and in Storm topology logs

 On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
 michael.miklav...@gmail.com> wrote:

> How did you validate the logs are making it to the indexing topology?
>
> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
> wrote:
>
>>
>> Hi,
>>
>>
>>
>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>> we sent the sample snort logs copied from metron git repo to snort kafka
>> topic.We did the same for bro topic.Logs are getting parsed and reached
>> indexing topology . Elastic search indices are not getting created though
>> we gave elastic search template install from ambari. So manually created
>> the elastic search index using template available in metron repo.
>> Though elastic search index is present , data from indexing toplogy 
>> neither
>> reached elastic search nor hdfs path .There are no errors in storm 
>> toplogy
>> logs.We could see the sample log in Metron management ui. How we can send
>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>> with no data. Elasticsearch health is yellow and we are able to insert 
>> data
>> via rest call. Any documentation on sending the smaple snort logs to 
>> metron
>> alerts ui will be helpful . Any configuration from metron management ui 
>> is
>> required to pass it to alerts –ui
>>
>>
>>
>>
>>
>> Thanks and Regards
>>
>> Hema
>>
>>
>>
>>
>>
>>
>>
>

Re: Snort logs flow issue

[Hema malini]

Sorry for the typo. Can you please help with the required configuration.

On Sat, Apr 6, 2019, 5:39 PM Hema malini  wrote:

> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini  wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>> 0x1F5","parallelenricher.splitter.begin.ts":"
>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini 
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
 Do you get 10 records output to the CLI when you run the following?

 /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
 $ZOOKEEPER --topic indexing --from-beginning --max-messages 10


 On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
 wrote:

> We verified it in Storm ui and in Storm topology logs
>
> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> How did you validate the logs are making it to the indexing topology?
>>
>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
>> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>> indexing topology . Elastic search indices are not getting created 
>>> though
>>> we gave elastic search template install from ambari. So manually created
>>> the elastic search index using template available in metron repo.
>>> Though elastic search index is present , data from indexing toplogy 
>>> neither
>>> reached elastic search nor hdfs path .There are no errors in storm 
>>> toplogy
>>> logs.We could see the sample log in Metron management ui. How we can 
>>> send
>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>> with no data. Elasticsearch health is yellow and we are able to insert 
>>> data
>>> via rest call. Any documentation on sending the smaple snort logs to 
>>> metron
>>> alerts ui will be helpful . Any configuration from metron management ui 
>>> is
>>> required to pass it to alerts –ui
>>>
>>>
>>>
>>>
>>>
>>> Thanks and Regards
>>>
>>> Hema
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: Snort logs flow issue

[Hema malini]

After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently
there is no data in alerts UI. How.to configure the logs as alerts

On Sat, Apr 6, 2019, 9:21 PM Hema malini  wrote:

> Sorry for the typo. Can you please help with the required configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini  wrote:
>
>> Are we missing any configuration? Initially elastic search was down. We
>> figured out the issue and fixed it .Now elastic search is up . We restarted
>> metron indexing but still those indices not created. So we created it
>> manually.Do we have to change any parser configuration . How logs will flow
>> into metron alerts dashboard and kibana dashboard..what is the required
>> congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini 
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini 
>>> wrote:
>>>
 Yes I am getting messages

 On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
 michael.miklav...@gmail.com> wrote:

> Do you get 10 records output to the CLI when you run the following?
>
> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>
>
> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
> wrote:
>
>> We verified it in Storm ui and in Storm topology logs
>>
>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> How did you validate the logs are making it to the indexing topology?
>>>
>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
>>> wrote:
>>>

 Hi,



 We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
 we sent the sample snort logs copied from metron git repo to snort 
 kafka
 topic.We did the same for bro topic.Logs are getting parsed and reached
 indexing topology . Elastic search indices are not getting created 
 though
 we gave elastic search template install from ambari. So manually 
 created
 the elastic search index using template available in metron repo.
 Though elastic search index is present , data from indexing toplogy 
 neither
 reached elastic search nor hdfs path .There are no errors in storm 
 toplogy
 logs.We could see the sample log in Metron management ui. How we can 
 send
 the logs to alerts ui and kibana dashboard. In kibana dashboard we 
 could
 see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created 
 but
 with no data. Elasticsearch health is yellow and we are able to insert 
 data
 via rest call. Any documentation on sending the smaple snort logs to 
 metron
 alerts ui will be helpful . Any configuration from metron management 
 ui is
 required to pass it to alerts –ui





 Thanks and Regards

 Hema







>>>

Re: Snort logs flow issue

[Michael Miklavcic]

If you see them in the dashboard you should be able to see them in the
alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
default behavior is that the UI doesn't initiate a search at login, it's up
to the user to click search.

On Mon, Apr 8, 2019, 6:38 AM Hema malini  wrote:

> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini  wrote:
>
>> Sorry for the typo. Can you please help with the required configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini  wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down. We
>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>> metron indexing but still those indices not created. So we created it
>>> manually.Do we have to change any parser configuration . How logs will flow
>>> into metron alerts dashboard and kibana dashboard..what is the required
>>> congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini 
>>> wrote:
>>>
 Sample messages flown in indexing topic
 {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
 "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
 tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
 hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
 geoadapter.begin.ts":"1554384503452","tcpwindow":"
 0x1F5","parallelenricher.splitter.begin.ts":"
 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
 original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
 alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
 0x836687BD,,0x1F5,64,0,62040,52,53248","
 parallelenricher.enrich.end.ts":"1554384505342","threat.
 triage.rules.0.reason":null,"tos":"0","adapter.
 hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
 62040","ip_src_addr":"192.168.66.121","timestamp":
 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
 ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
 1554384505264","ttl":"64","source.type":"snort","adapter.
 geoadapter.end.ts":"1554384503453","ethlen":"0x42"
 ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
 1554384505264","ip_src_port":"8080","tcpflags":"***A","
 guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
 999158","sig_generator":"1"}


 On Fri, Apr 5, 2019, 11:43 PM Hema malini 
 wrote:

> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 
>> 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
 How did you validate the logs are making it to the indexing
 topology?

 On Fri, Apr 5, 2019 at 8:12 AM Hema malini 
 wrote:

>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
> Nifi we sent the sample snort logs copied from metron git repo to 
> snort
> kafka topic.We did the same for bro topic.Logs are getting parsed and
> reached indexing topology . Elastic search indices are not getting 
> created
> though we gave elastic search template install from ambari. So 
> manually
> created the elastic search index using template available in
> metron repo. Though elastic search index is present , data from 
> indexing
> toplogy neither reached elastic search nor hdfs path .There are no 
> errors
> in storm toplogy logs.We could see the sample log in Metron 
> management ui.
> How we can send the logs to alerts ui and kibana dashboard. In kibana
> dashboard we could see two dashboards -
> Metron-Dashboard,Metron-Error-Dashboard created but with no data.
> Elasticsearch health is yellow and we are able to insert data via rest
> call. Any documentation on sending the smaple snort logs to metron 
> alerts
> ui will be helpful . Any configuration from metron management ui is
> required to p

Re: Snort logs flow issue

[Hema malini]

Hi Michael,

Thanks for your reply. I couldn't find any errors in metron alerts UI log .
I clicked the search and changed the date range too. Still no records. Do
we have to run metron rest in dev profile?

On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic 
wrote:

> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini  wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini  wrote:
>>
>>> Sorry for the typo. Can you please help with the required configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini 
>>> wrote:
>>>
 Are we missing any configuration? Initially elastic search was down. We
 figured out the issue and fixed it .Now elastic search is up . We restarted
 metron indexing but still those indices not created. So we created it
 manually.Do we have to change any parser configuration . How logs will flow
 into metron alerts dashboard and kibana dashboard..what is the required
 congratulation

 On Fri, Apr 5, 2019, 11:52 PM Hema malini 
 wrote:

> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
> 52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort
> test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248","
> parallelenricher.enrich.end.ts":"1554384505342","threat.
> triage.rules.0.reason":null,"tos":"0","adapter.
> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini 
> wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 
>>> 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini 
>>> wrote:
>>>
 We verified it in Storm ui and in Storm topology logs

 On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
 michael.miklav...@gmail.com> wrote:

> How did you validate the logs are making it to the indexing
> topology?
>
> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
> nhemamalin...@gmail.com> wrote:
>
>>
>> Hi,
>>
>>
>>
>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>> Nifi we sent the sample snort logs copied from metron git repo to 
>> snort
>> kafka topic.We did the same for bro topic.Logs are getting parsed and
>> reached indexing topology . Elastic search indices are not getting 
>> created
>> though we gave elastic search template install from ambari. So 
>> manually
>> created the elastic search index using template available in
>> metron repo. Though elastic search index is present , data from 
>> indexing
>> toplogy neither reached elastic search nor hdfs path .There are no 
>> errors
>> in storm toplogy logs.We could see the sample log in Metron 
>> management ui.
>> How we can send the logs to alerts ui and kibana dashboard. In kibana

Re: Snort logs flow issue

[Hema malini]

Hi Michael,

Sorry just noticed the error in metron rest logs - Table 'user settings'
was not found. Do we have to create that hbase table . Where to find the
hbase tables created. I could see only two namespace in hbase - default and
hbase. No tables created in that. Do I have to run metron rest in dev
profile.

Thanks & Regards
Hema

On Tue, Apr 9, 2019, 12:44 PM Hema malini  wrote:

> Hi Michael,
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> If you see them in the dashboard you should be able to see them in the
>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>> default behavior is that the UI doesn't initiate a search at login, it's up
>> to the user to click search.
>>
>> On Mon, Apr 8, 2019, 6:38 AM Hema malini  wrote:
>>
>>> After recreating the index, now we are able to visualize the data in
>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>
>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini 
>>> wrote:
>>>
 Sorry for the typo. Can you please help with the required
 configuration.

 On Sat, Apr 6, 2019, 5:39 PM Hema malini 
 wrote:

> Are we missing any configuration? Initially elastic search was down.
> We figured out the issue and fixed it .Now elastic search is up . We
> restarted metron indexing but still those indices not created. So we
> created it manually.Do we have to change any parser configuration . How
> logs will flow into metron alerts dashboard and kibana dashboard..what is
> the required congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini 
> wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>> 52","adapter.hostfromjsonlistadapter.end.
>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>> splitter.begin.ts":"1554384505264","threat.triage.
>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>> triage.rules.0.reason":null,"tos":"0","adapter.
>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>> 1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini 
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
 Do you get 10 records output to the CLI when you run the following?

 /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
 --zookeeper $ZOOKEEPER --topic indexing --from-beginning 
 --max-messages 10


 On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
 nhemamalin...@gmail.com> wrote:

> We verified it in Storm ui and in Storm topology logs
>
> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> How did you validate the logs are making it to the indexing
>> topology?
>>
>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini <
>> nhemamalin...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using
>>> Nifi we sent the sample snort logs copied from metron git repo to 
>>> snort
>>> kafka topic.We did the same for bro topic.Logs are getting parsed 
>>> and
>>> reached indexing topology . Elastic search indice

RE: Snort logs flow issue

[stephane.davy]

Hello Hema,

Unless I’m wrong, this must be setup in MySQL, the database you use for Metron 
REST.


From: Hema malini [mailto:nhemamalin...@gmail.com]
Sent: Tuesday, April 09, 2019 09:42
To: user@metron.apache.org
Subject: Re: Snort logs flow issue

Hi Michael,

Sorry just noticed the error in metron rest logs - Table 'user settings' was 
not found. Do we have to create that hbase table . Where to find the hbase 
tables created. I could see only two namespace in hbase - default and hbase. No 
tables created in that. Do I have to run metron rest in dev profile.

Thanks & Regards
Hema

On Tue, Apr 9, 2019, 12:44 PM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
Hi Michael,

Thanks for your reply. I couldn't find any errors in metron alerts UI log . I 
clicked the search and changed the date range too. Still no records. Do we have 
to run metron rest in dev profile?

On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic 
mailto:michael.miklav...@gmail.com>> wrote:
If you see them in the dashboard you should be able to see them in the alerts 
UI. Any errors in either the alerts UI or REST logs? Also, the new default 
behavior is that the UI doesn't initiate a search at login, it's up to the user 
to click search.

On Mon, Apr 8, 2019, 6:38 AM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
After recreating the index, now we are able to visualize the data in kibana 
metron dashboard. How we can pass alerts to metron alerts UI. Currently there 
is no data in alerts UI. How.to configure the logs as alerts

On Sat, Apr 6, 2019, 9:21 PM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
Sorry for the typo. Can you please help with the required configuration.

On Sat, Apr 6, 2019, 5:39 PM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
Are we missing any configuration? Initially elastic search was down. We figured 
out the issue and fixed it .Now elastic search is up . We restarted metron 
indexing but still those indices not created. So we created it manually.Do we 
have to change any parser configuration . How logs will flow into metron alerts 
dashboard and kibana dashboard..what is the required congratulation

On Fri, Apr 5, 2019, 11:52 PM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
Sample messages flown in indexing topic
{"msg":"'snort test 
alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
 ,1,999158,0,\"'snort test 
alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name<http://threat.triage.rules.0.name/>":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}


On Fri, Apr 5, 2019, 11:43 PM Hema malini 
mailto:nhemamalin...@gmail.com>> wrote:
Yes I am getting messages

On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic 
mailto:michael.miklav...@gmail.com>> wrote:
Do you get 10 records output to the CLI when you run the following?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper 
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10


On Fri, 

Re: Snort logs flow issue

[Hema malini]

Hi Stephanie,

Issue got resolved by creating that table in hbase.

Thanks and regards,
Hema

On Tue, Apr 9, 2019, 1:31 PM  wrote:

> Hello Hema,
>
>
>
> Unless I’m wrong, this must be setup in MySQL, the database you use for
> Metron REST.
>
>
>
>
>
> *From:* Hema malini [mailto:nhemamalin...@gmail.com]
> *Sent:* Tuesday, April 09, 2019 09:42
> *To:* user@metron.apache.org
> *Subject:* Re: Snort logs flow issue
>
>
>
> Hi Michael,
>
>
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
>
>
> Thanks & Regards
>
> Hema
>
>
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini  wrote:
>
> Hi Michael,
>
>
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
>
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
>
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini  wrote:
>
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
>
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini  wrote:
>
> Sorry for the typo. Can you please help with the required configuration.
>
>
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini  wrote:
>
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
>
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini  wrote:
>
> Sample messages flown in indexing topic
>
> {"msg":"'snort test
> alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>
>
>
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini  wrote:
>
> Yes I

Re: Snort logs flow issue

[Hema malini]

Hi Michael,

Issue got resolved after I manually created user settings table in hbase.
There are no contents in that table though it is working. Where are the
records getting stored then for alerts UI. Where can I find the records in
HDFS.

Thanks and regards,
Hema

On Tue, Apr 9, 2019, 1:12 PM Hema malini  wrote:

> Hi Michael,
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
> Thanks & Regards
> Hema
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini  wrote:
>
>> Hi Michael,
>>
>> Thanks for your reply. I couldn't find any errors in metron alerts UI log
>> . I clicked the search and changed the date range too. Still no records. Do
>> we have to run metron rest in dev profile?
>>
>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> If you see them in the dashboard you should be able to see them in the
>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>> to the user to click search.
>>>
>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini 
>>> wrote:
>>>
 After recreating the index, now we are able to visualize the data in
 kibana metron dashboard. How we can pass alerts to metron alerts UI.
 Currently there is no data in alerts UI. How.to configure the logs as 
 alerts

 On Sat, Apr 6, 2019, 9:21 PM Hema malini 
 wrote:

> Sorry for the typo. Can you please help with the required
> configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini 
> wrote:
>
>> Are we missing any configuration? Initially elastic search was down.
>> We figured out the issue and fixed it .Now elastic search is up . We
>> restarted metron indexing but still those indices not created. So we
>> created it manually.Do we have to change any parser configuration . How
>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>> the required congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini 
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>> 52","adapter.hostfromjsonlistadapter.end.
>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>> splitter.begin.ts":"1554384505264","threat.triage.
>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
>>> 52,53248","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":
>>> "true","parallelenricher.enrich.begin.ts":"
>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>> 999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini 
>>> wrote:
>>>
 Yes I am getting messages

 On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
 michael.miklav...@gmail.com> wrote:

> Do you get 10 records output to the CLI when you run the following?
>
> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
> --zookeeper $ZOOKEEPER --topic indexing --from-beginning 
> --max-messages 10
>
>
> On Fri, Apr 5, 2019 at 11:38 AM Hema malini <
> nhemamalin...@gmail.com> wrote:
>
>> We verified it in Storm ui and in Storm topology logs
>>
>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> How did you validate the logs are making it to the indexing
>>> topology?
>>>
>>> On Fri, Apr 5, 2019 at 8:12 AM Hema mal

Re: Snort logs flow issue

[Michael Miklavcic]

That table should have been created by default as part of the Ambari
installation of Metron via our MPack -
https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
You shouldn't have to worry about this at all as an end user, but here is
where it happens if you're curious-
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230

I believe that table will be empty by default. @Ryan Merriman, do you
happen to know if we currently save user UI data there?

On Tue, Apr 9, 2019 at 2:14 AM Hema malini  wrote:

> Hi Michael,
>
> Issue got resolved after I manually created user settings table in hbase.
> There are no contents in that table though it is working. Where are the
> records getting stored then for alerts UI. Where can I find the records in
> HDFS.
>
> Thanks and regards,
> Hema
>
> On Tue, Apr 9, 2019, 1:12 PM Hema malini  wrote:
>
>> Hi Michael,
>>
>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>> was not found. Do we have to create that hbase table . Where to find the
>> hbase tables created. I could see only two namespace in hbase - default and
>> hbase. No tables created in that. Do I have to run metron rest in dev
>> profile.
>>
>> Thanks & Regards
>> Hema
>>
>> On Tue, Apr 9, 2019, 12:44 PM Hema malini 
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>> log . I clicked the search and changed the date range too. Still no
>>> records. Do we have to run metron rest in dev profile?
>>>
>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
 If you see them in the dashboard you should be able to see them in the
 alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
 default behavior is that the UI doesn't initiate a search at login, it's up
 to the user to click search.

 On Mon, Apr 8, 2019, 6:38 AM Hema malini 
 wrote:

> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as 
> alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini 
> wrote:
>
>> Sorry for the typo. Can you please help with the required
>> configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini 
>> wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down.
>>> We figured out the issue and fixed it .Now elastic search is up . We
>>> restarted metron indexing but still those indices not created. So we
>>> created it manually.Do we have to change any parser configuration . How
>>> logs will flow into metron alerts dashboard and kibana dashboard..what 
>>> is
>>> the required congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini 
>>> wrote:
>>>
 Sample messages flown in indexing topic
 {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
 "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
 tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
 52","adapter.hostfromjsonlistadapter.end.
 ts":"1554384503452","adapter.geoadapter.begin.ts":"
 1554384503452","tcpwindow":"0x1F5","parallelenricher.
 splitter.begin.ts":"1554384505264","threat.triage.
 rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
 ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
 ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
 52,53248","parallelenricher.enrich.end.
 ts":"1554384505342","threat.triage.rules.0.reason":null,"
 tos":"0","adapter.hostfromjsonlistadapter.begin.
 ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
 threat.triage.rules.0.name":null,"is_alert":
 "true","parallelenricher.enrich.begin.ts":"
 1554384505264","ttl":"64","source.type":"snort","adapter.
 geoadapter.end.ts":"1554384503453","ethlen":"0x42"
 ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
 1554384505264","ip_src_port":"8080","tcpflags":"***A","
 guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
 999158","sig_generator":"1"}


 On Fri, Apr 5, 2019, 11:43 PM Hema malini 
 wrote:

> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael M

Re: Snort logs flow issue

[Hema malini]

Thanks a lot Michael for your help. Will explore further.

On Wed, Apr 10, 2019, 3:37 AM Michael Miklavcic 
wrote:

> That table should have been created by default as part of the Ambari
> installation of Metron via our MPack -
> https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
> You shouldn't have to worry about this at all as an end user, but here is
> where it happens if you're curious-
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
>
> I believe that table will be empty by default. @Ryan Merriman, do you
> happen to know if we currently save user UI data there?
>
> On Tue, Apr 9, 2019 at 2:14 AM Hema malini 
> wrote:
>
>> Hi Michael,
>>
>> Issue got resolved after I manually created user settings table in hbase.
>> There are no contents in that table though it is working. Where are the
>> records getting stored then for alerts UI. Where can I find the records in
>> HDFS.
>>
>> Thanks and regards,
>> Hema
>>
>> On Tue, Apr 9, 2019, 1:12 PM Hema malini  wrote:
>>
>>> Hi Michael,
>>>
>>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>>> was not found. Do we have to create that hbase table . Where to find the
>>> hbase tables created. I could see only two namespace in hbase - default and
>>> hbase. No tables created in that. Do I have to run metron rest in dev
>>> profile.
>>>
>>> Thanks & Regards
>>> Hema
>>>
>>> On Tue, Apr 9, 2019, 12:44 PM Hema malini 
>>> wrote:
>>>
 Hi Michael,

 Thanks for your reply. I couldn't find any errors in metron alerts UI
 log . I clicked the search and changed the date range too. Still no
 records. Do we have to run metron rest in dev profile?

 On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
 michael.miklav...@gmail.com> wrote:

> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's 
> up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini 
> wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as 
>> alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini 
>> wrote:
>>
>>> Sorry for the typo. Can you please help with the required
>>> configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini 
>>> wrote:
>>>
 Are we missing any configuration? Initially elastic search was
 down. We figured out the issue and fixed it .Now elastic search is up 
 . We
 restarted metron indexing but still those indices not created. So we
 created it manually.Do we have to change any parser configuration . How
 logs will flow into metron alerts dashboard and kibana dashboard..what 
 is
 the required congratulation

 On Fri, Apr 5, 2019, 11:52 PM Hema malini 
 wrote:

> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
> 52","adapter.hostfromjsonlistadapter.end.
> ts":"1554384503452","adapter.geoadapter.begin.ts":"
> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
> splitter.begin.ts":"1554384505264","threat.triage.
> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
> 52,53248","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.
> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name":null,"is_alert":
> "true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c6