Snort
Hi, Can I use snort in packet capture mode with metron? By default it works in IDS mode only. Regards.
Re: Snort
Yes, you can use Snort. Metron can consume Snort telemetries out of the box. You have to setup Snort on your own and push the output into a kafka topic (most likely using NiFi). From there on you can use the output of Snort in Metron.10.10.2017, 00:48, "Syed Hammad Tahir" :Hi,Can I use snort in packet capture mode with metron? By default it works in IDS mode only.Regards. --- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org
Re: Snort
You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode. On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: > Yes, you can use Snort. Metron can consume Snort telemetries out of the > box. You have to setup Snort on your own and push the output into a kafka > topic (most likely using NiFi). From there on you can use the output of > Snort in Metron. > > > 10.10.2017, 00:48, "Syed Hammad Tahir" : > > Hi, > > Can I use snort in packet capture mode with metron? By default it works in > IDS mode only. > > Regards. > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Re: Snort
What I mean is that you should install snort, load the appropriate Snort rules for your use case, set Snort to log to a directory, and send traffic to the network interface where Snort is listening. That will produce Snort log files. Then you can push the contents of Snort logs either to Kafka using NiFi (preferred) or using Kafka utilities such as command line producer. This should be pushed to a Kafka topic called Snort where each message is a log line of the Snort file. Does that make sense? Thanks,James11.10.2017, 23:08, "Syed Hammad Tahir" :You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode.On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> wrote:Yes, you can use Snort. Metron can consume Snort telemetries out of the box. You have to setup Snort on your own and push the output into a kafka topic (most likely using NiFi). From there on you can use the output of Snort in Metron.10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>:Hi,Can I use snort in packet capture mode with metron? By default it works in IDS mode only.Regards. --- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org --- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org
Re: Snort
Hi, Thanks for the support. Can it be performed both on dumped log and real time data? Regards. On Tue, Oct 17, 2017 at 1:02 AM, James Sirota wrote: > What I mean is that you should install snort, load the appropriate Snort > rules for your use case, set Snort to log to a directory, and send traffic > to the network interface where Snort is listening. That will produce Snort > log files. Then you can push the contents of Snort logs either to Kafka > using NiFi (preferred) or using Kafka utilities such as command line > producer. This should be pushed to a Kafka topic called Snort where each > message is a log line of the Snort file. Does that make sense? > > Thanks, > James > > > 11.10.2017, 23:08, "Syed Hammad Tahir" : > > You mean that I must start snort from terminal by doing snort -v and then > push it to kafka topic? I need to start snort in packet capture mode. > > On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: > > Yes, you can use Snort. Metron can consume Snort telemetries out of the > box. You have to setup Snort on your own and push the output into a kafka > topic (most likely using NiFi). From there on you can use the output of > Snort in Metron. > > > 10.10.2017, 00:48, "Syed Hammad Tahir" : > > Hi, > > Can I use snort in packet capture mode with metron? By default it works in > IDS mode only. > > Regards. > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Re: Snort
And I am sorry about one confusion but isnt snort builtin into the metron framework? If so then cant we access that snort and do the tasks you mentioned earlier? On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir wrote: > Hi, > > Thanks for the support. Can it be performed both on dumped log and real > time data? > Regards. > > On Tue, Oct 17, 2017 at 1:02 AM, James Sirota wrote: > >> What I mean is that you should install snort, load the appropriate Snort >> rules for your use case, set Snort to log to a directory, and send traffic >> to the network interface where Snort is listening. That will produce Snort >> log files. Then you can push the contents of Snort logs either to Kafka >> using NiFi (preferred) or using Kafka utilities such as command line >> producer. This should be pushed to a Kafka topic called Snort where each >> message is a log line of the Snort file. Does that make sense? >> >> Thanks, >> James >> >> >> 11.10.2017, 23:08, "Syed Hammad Tahir" : >> >> You mean that I must start snort from terminal by doing snort -v and then >> push it to kafka topic? I need to start snort in packet capture mode. >> >> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: >> >> Yes, you can use Snort. Metron can consume Snort telemetries out of the >> box. You have to setup Snort on your own and push the output into a kafka >> topic (most likely using NiFi). From there on you can use the output of >> Snort in Metron. >> >> >> 10.10.2017, 00:48, "Syed Hammad Tahir" : >> >> Hi, >> >> Can I use snort in packet capture mode with metron? By default it works >> in IDS mode only. >> >> Regards. >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >
Re: Snort
>From Metron's perspective, Snort is just another sensor. Snort is installed, managed and executed completely independent of Metron itself. As with any sensor, you are responsible for getting the telemetry produced by Snort into Kafka. Metron can then consume that telemetry from Kafka and do wonderful things with it. :) On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir wrote: > And I am sorry about one confusion but isnt snort builtin into the metron > framework? If so then cant we access that snort and do the tasks you > mentioned earlier? > > On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir > wrote: > >> Hi, >> >> Thanks for the support. Can it be performed both on dumped log and real >> time data? >> Regards. >> >> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota wrote: >> >>> What I mean is that you should install snort, load the appropriate Snort >>> rules for your use case, set Snort to log to a directory, and send traffic >>> to the network interface where Snort is listening. That will produce Snort >>> log files. Then you can push the contents of Snort logs either to Kafka >>> using NiFi (preferred) or using Kafka utilities such as command line >>> producer. This should be pushed to a Kafka topic called Snort where each >>> message is a log line of the Snort file. Does that make sense? >>> >>> Thanks, >>> James >>> >>> >>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>> >>> You mean that I must start snort from terminal by doing snort -v and >>> then push it to kafka topic? I need to start snort in packet capture mode. >>> >>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>> wrote: >>> >>> Yes, you can use Snort. Metron can consume Snort telemetries out of the >>> box. You have to setup Snort on your own and push the output into a kafka >>> topic (most likely using NiFi). From there on you can use the output of >>> Snort in Metron. >>> >>> >>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>> >>> Hi, >>> >>> Can I use snort in packet capture mode with metron? By default it works >>> in IDS mode only. >>> >>> Regards. >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> >> >
Re: Snort
yes,, but when i do snort -v in vagrant ssh console it says snort isnt installed where as it can be seen working in metron. Due to that reason I am confused because James Sirota said to install snort. On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: > From Metron's perspective, Snort is just another sensor. Snort is > installed, managed and executed completely independent of Metron itself. As > with any sensor, you are responsible for getting the telemetry produced by > Snort into Kafka. Metron can then consume that telemetry from Kafka and do > wonderful things with it. :) > > > On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir > wrote: > >> And I am sorry about one confusion but isnt snort builtin into the metron >> framework? If so then cant we access that snort and do the tasks you >> mentioned earlier? >> >> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir > > wrote: >> >>> Hi, >>> >>> Thanks for the support. Can it be performed both on dumped log and real >>> time data? >>> Regards. >>> >>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>> wrote: >>> >>>> What I mean is that you should install snort, load the appropriate >>>> Snort rules for your use case, set Snort to log to a directory, and send >>>> traffic to the network interface where Snort is listening. That will >>>> produce Snort log files. Then you can push the contents of Snort logs >>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>> command line producer. This should be pushed to a Kafka topic called Snort >>>> where each message is a log line of the Snort file. Does that make sense? >>>> >>>> Thanks, >>>> James >>>> >>>> >>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>> >>>> You mean that I must start snort from terminal by doing snort -v and >>>> then push it to kafka topic? I need to start snort in packet capture mode. >>>> >>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>> wrote: >>>> >>>> Yes, you can use Snort. Metron can consume Snort telemetries out of the >>>> box. You have to setup Snort on your own and push the output into a kafka >>>> topic (most likely using NiFi). From there on you can use the output of >>>> Snort in Metron. >>>> >>>> >>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>> >>>> Hi, >>>> >>>> Can I use snort in packet capture mode with metron? By default it works >>>> in IDS mode only. >>>> >>>> Regards. >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>> >> >
Re: Snort
In the Full Dev environment, Snort is not installed. We install "Sensor Stubs" which is just a mechanism that continually replays canned telemetry logs repetitively to mimic real sensors. We have to do this because of resource constraints when running all of Metron on a single VM. See the following for more information. https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir wrote: > yes,, but when i do snort -v in vagrant ssh console it says snort isnt > installed where as it can be seen working in metron. Due to that reason I > am confused because James Sirota said to install snort. > > On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: > >> From Metron's perspective, Snort is just another sensor. Snort is >> installed, managed and executed completely independent of Metron itself. As >> with any sensor, you are responsible for getting the telemetry produced by >> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >> wonderful things with it. :) >> >> >> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir >> wrote: >> >>> And I am sorry about one confusion but isnt snort builtin into the >>> metron framework? If so then cant we access that snort and do the tasks you >>> mentioned earlier? >>> >>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Hi, >>>> >>>> Thanks for the support. Can it be performed both on dumped log and real >>>> time data? >>>> Regards. >>>> >>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>> wrote: >>>> >>>>> What I mean is that you should install snort, load the appropriate >>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>> traffic to the network interface where Snort is listening. That will >>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>> command line producer. This should be pushed to a Kafka topic called Snort >>>>> where each message is a log line of the Snort file. Does that make sense? >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>>> >>>>> You mean that I must start snort from terminal by doing snort -v and >>>>> then push it to kafka topic? I need to start snort in packet capture mode. >>>>> >>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>>> wrote: >>>>> >>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>> the box. You have to setup Snort on your own and push the output into a >>>>> kafka topic (most likely using NiFi). From there on you can use the output >>>>> of Snort in Metron. >>>>> >>>>> >>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>>> >>>>> Hi, >>>>> >>>>> Can I use snort in packet capture mode with metron? By default it >>>>> works in IDS mode only. >>>>> >>>>> Regards. >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PMC- Apache Metron >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PMC- Apache Metron >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>> >>> >> >
Re: Snort
Ok, Now I get it. Now should I install snort in vagrant ssh in the normal way snort is usually install on a linux distro or do I need to run some special commands again? On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote: > In the Full Dev environment, Snort is not installed. We install "Sensor > Stubs" which is just a mechanism that continually replays canned telemetry > logs repetitively to mimic real sensors. We have to do this because of > resource constraints when running all of Metron on a single VM. See the > following for more information. > > https://github.com/apache/metron/tree/master/metron- > deployment/roles/sensor-stubs > > > > On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir > wrote: > >> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >> installed where as it can be seen working in metron. Due to that reason I >> am confused because James Sirota said to install snort. >> >> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: >> >>> From Metron's perspective, Snort is just another sensor. Snort is >>> installed, managed and executed completely independent of Metron itself. As >>> with any sensor, you are responsible for getting the telemetry produced by >>> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >>> wonderful things with it. :) >>> >>> >>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir >> > wrote: >>> >>>> And I am sorry about one confusion but isnt snort builtin into the >>>> metron framework? If so then cant we access that snort and do the tasks you >>>> mentioned earlier? >>>> >>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Hi, >>>>> >>>>> Thanks for the support. Can it be performed both on dumped log and >>>>> real time data? >>>>> Regards. >>>>> >>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>>> wrote: >>>>> >>>>>> What I mean is that you should install snort, load the appropriate >>>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>>> traffic to the network interface where Snort is listening. That will >>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>> command line producer. This should be pushed to a Kafka topic called >>>>>> Snort >>>>>> where each message is a log line of the Snort file. Does that make sense? >>>>>> >>>>>> Thanks, >>>>>> James >>>>>> >>>>>> >>>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>>>> >>>>>> You mean that I must start snort from terminal by doing snort -v and >>>>>> then push it to kafka topic? I need to start snort in packet capture >>>>>> mode. >>>>>> >>>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>>>> wrote: >>>>>> >>>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>>> the box. You have to setup Snort on your own and push the output into a >>>>>> kafka topic (most likely using NiFi). From there on you can use the >>>>>> output >>>>>> of Snort in Metron. >>>>>> >>>>>> >>>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>>>> >>>>>> Hi, >>>>>> >>>>>> Can I use snort in packet capture mode with metron? By default it >>>>>> works in IDS mode only. >>>>>> >>>>>> Regards. >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Snort
No special commands. Install and configure Snort however you like and get those logs into a Kafka topic. Metron is completely agnostic to how sensor telemetry lands in Kafka. We also have an Ansible role that will install Snort along with a simple mechanism to transport its logs to Kafka. This is only useful for development environments; not a production install. Using the Ansible role directly may be beyond the knowledge level of some. I only offer this as a guide that you can use to follow along and manually install it yourself. https://github.com/apache/metron/blob/master/metron-deployment/roles/snort/tasks/main.yml If you are not familiar with how Ansible roles are defined, just start at the main.yml, then follow through each of the other files as they are included. It is pretty readable once you get use to the layout. On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir wrote: > Ok, Now I get it. Now should I install snort in vagrant ssh in the normal > way snort is usually install on a linux distro or do I need to run some > special commands again? > > On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote: > >> In the Full Dev environment, Snort is not installed. We install "Sensor >> Stubs" which is just a mechanism that continually replays canned telemetry >> logs repetitively to mimic real sensors. We have to do this because of >> resource constraints when running all of Metron on a single VM. See the >> following for more information. >> >> https://github.com/apache/metron/tree/master/metron-deployme >> nt/roles/sensor-stubs >> >> >> >> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir > > wrote: >> >>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >>> installed where as it can be seen working in metron. Due to that reason I >>> am confused because James Sirota said to install snort. >>> >>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: >>> >>>> From Metron's perspective, Snort is just another sensor. Snort is >>>> installed, managed and executed completely independent of Metron itself. As >>>> with any sensor, you are responsible for getting the telemetry produced by >>>> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >>>> wonderful things with it. :) >>>> >>>> >>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> And I am sorry about one confusion but isnt snort builtin into the >>>>> metron framework? If so then cant we access that snort and do the tasks >>>>> you >>>>> mentioned earlier? >>>>> >>>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks for the support. Can it be performed both on dumped log and >>>>>> real time data? >>>>>> Regards. >>>>>> >>>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>>>> wrote: >>>>>> >>>>>>> What I mean is that you should install snort, load the appropriate >>>>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>>>> traffic to the network interface where Snort is listening. That will >>>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>>> command line producer. This should be pushed to a Kafka topic called >>>>>>> Snort >>>>>>> where each message is a log line of the Snort file. Does that make >>>>>>> sense? >>>>>>> >>>>>>> Thanks, >>>>>>> James >>>>>>> >>>>>>> >>>>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>>>>> >>>>>>> You mean that I must start snort from terminal by doing snort -v and >>>>>>> then push it to kafka topic? I need to start snort in packet capture >>>>>>> mode. >>>>>>> >>>>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>>>>> wrote: >>>>>>> >>>>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>>>> the box. You have to setup Snort on your own and push the output into a >>>>>>> kafka topic (most likely using NiFi). From there on you can use the >>>>>>> output >>>>>>> of Snort in Metron. >>>>>>> >>>>>>> >>>>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Can I use snort in packet capture mode with metron? By default it >>>>>>> works in IDS mode only. >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> Thank you, >>>>>>> >>>>>>> James Sirota >>>>>>> PMC- Apache Metron >>>>>>> jsirota AT apache DOT org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> Thank you, >>>>>>> >>>>>>> James Sirota >>>>>>> PMC- Apache Metron >>>>>>> jsirota AT apache DOT org >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Snort
I am so noob in all of this. I am using full-dev vm metron install to do my research. So I have 2 options to install snort: as per my understanding 1- Install it in a usual way (like that on a regular linux machine) and then make its kafka topic 2- Use ansible role to do all of that. Read the content of those yml files given in main.yml to understand the procedure? Which one do you suggest? On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen wrote: > No special commands. Install and configure Snort however you like and get > those logs into a Kafka topic. Metron is completely agnostic to how sensor > telemetry lands in Kafka. > > We also have an Ansible role that will install Snort along with a simple > mechanism to transport its logs to Kafka. This is only useful for > development environments; not a production install. > > Using the Ansible role directly may be beyond the knowledge level of > some. I only offer this as a guide that you can use to follow along and > manually install it yourself. > > https://github.com/apache/metron/blob/master/metron- > deployment/roles/snort/tasks/main.yml > > > If you are not familiar with how Ansible roles are defined, just start at > the main.yml, then follow through each of the other files as they are > included. It is pretty readable once you get use to the layout. > > On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir > wrote: > >> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal >> way snort is usually install on a linux distro or do I need to run some >> special commands again? >> >> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote: >> >>> In the Full Dev environment, Snort is not installed. We install "Sensor >>> Stubs" which is just a mechanism that continually replays canned telemetry >>> logs repetitively to mimic real sensors. We have to do this because of >>> resource constraints when running all of Metron on a single VM. See the >>> following for more information. >>> >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> >>> >>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >>>> installed where as it can be seen working in metron. Due to that reason I >>>> am confused because James Sirota said to install snort. >>>> >>>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: >>>> >>>>> From Metron's perspective, Snort is just another sensor. Snort is >>>>> installed, managed and executed completely independent of Metron itself. >>>>> As >>>>> with any sensor, you are responsible for getting the telemetry produced by >>>>> Snort into Kafka. Metron can then consume that telemetry from Kafka and >>>>> do >>>>> wonderful things with it. :) >>>>> >>>>> >>>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> And I am sorry about one confusion but isnt snort builtin into the >>>>>> metron framework? If so then cant we access that snort and do the tasks >>>>>> you >>>>>> mentioned earlier? >>>>>> >>>>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for the support. Can it be performed both on dumped log and >>>>>>> real time data? >>>>>>> Regards. >>>>>>> >>>>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>>>>> wrote: >>>>>>> >>>>>>>> What I mean is that you should install snort, load the appropriate >>>>>>>> Snort rules for your use case, set Snort to log to a directory, and >>>>>>>> send >>>>>>>> traffic to the network interface where Snort is listening. That will >>>>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>>>> command line producer. This should be pushed to a Kafka topic c
Re: Snort
I would recommend just using a text editor if you’re not familiar with sed. To solve your sed problem… sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile sed -i means run the sed command (in this case a find replace) inplace on the file, the text following the -i is the name to append to a backup version (ie the original file unchanged). Metron does tend to assume a good knowledge of linux admin, you’ll find we have a lot of shell gurus in the community, but if you’re struggling with this, maybe a simple text editor would be easier. All you’re trying to do here is change a config value. Simon > On 19 Oct 2017, at 11:46, Syed Hammad Tahir wrote: > > Ran it without -i swtich, gives this: > > > > On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com <mailto:zeo...@gmail.com> > mailto:zeo...@gmail.com>> wrote: > The sed command is falling. It's written for a Mac so it will need an > alteration to be portable. Run it without the '' after -i, from > ~/metron-master > > Jon > > > On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir <mailto:mscs16...@itu.edu.pk>> wrote: > I did what this guide said to install the original sensor: > https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs > > <https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs> > > Still didnt work. How do I install snort into this? > > > On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir <mailto:mscs16...@itu.edu.pk>> wrote: > Maybe I did something wrong > > > > On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir <mailto:mscs16...@itu.edu.pk>> wrote: > Ok, thankyou. It will install all the sensors (bro, snort etc) ? > > On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com <mailto:zeo...@gmail.com>> wrote: > When you set up full dev if you remove the sensors skip tag it will set up > snort for you. I have a sed one liner in my bro security patch pr to do > this, just need to do it before vagrant up. > > sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" > metron-deployment/vagrant/full-dev-platform/Vagrantfile > cd metron-deployment/vagrant/full-dev-platform/ > vagrant up > > Jon > > > On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir <mailto:mscs16...@itu.edu.pk>> wrote: > I followed this guide exactly: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548> > > And the did vagrant up in full-development-platform folder. And Snort is not > installed because when I type snort -v in vagrant ssh, it returns error of > not able to find the snort command. > > On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets <mailto:laur...@daemon.be>> wrote: > Hi Syed, > > I was under the impression that you installed the full-dev environment? If > so, snort should already be installed... > > On 2017-10-18 09:45, Syed Hammad Tahir wrote: > >> It has become a mess. Apparently snort is released for centos 7 whereas >> metron one is centos 6.8. Whenever I try to install snort it gives me this: >> >> >> >> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen > <mailto:n...@nickallen.org>> wrote: >> Just use those as a guide to run the commands yourself. >> >> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote: >> please help me install the snort in metron. I tried doing it the normal way >> but i cant install the libraries >> >> On Wed, Oct 18, 2017 at 10:35 AM, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote: >> ok, This is the snort.yml file >> >> >> >> Do I need to run these commands myself or how do I put these yml files into >> play? >> >> On Tue, Oct 17, 2017 at 9:44 PM, Syed Hammad Tahir > <mailto:mscs16...@itu.edu.pk>> wrote: >> I am so noob in all of this. I am using full-dev vm metron install to do my >> research. So I have 2 options to install snort: as per my understanding >> >> 1- Install it in a usual way (like that on a regular linux machine) and then >> make its kafka topic >> >> 2- Use ansible role to do all of that. Read the content of those yml files >> given in main.yml to understand the procedure? >> >> Which one do you suggest? >> >> >> >> On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen > <mailto:n...@nickallen.org>>wrote: &
Re: Snort
Should I edit the vagrant file using text editor and what exactly should I edit there? On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > I would recommend just using a text editor if you’re not familiar with > sed. To solve your sed problem… > > sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" > metron-deployment/vagrant/full-dev-platform/Vagrantfile > > sed -i means run the sed command (in this case a find replace) inplace on > the file, the text following the -i is the name to append to a backup > version (ie the original file unchanged). > > Metron does tend to assume a good knowledge of linux admin, you’ll find we > have a lot of shell gurus in the community, but if you’re struggling with > this, maybe a simple text editor would be easier. All you’re trying to do > here is change a config value. > > Simon > > On 19 Oct 2017, at 11:46, Syed Hammad Tahir wrote: > > Ran it without -i swtich, gives this: > > > > On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com > wrote: > >> The sed command is falling. It's written for a Mac so it will need an >> alteration to be portable. Run it without the '' after -i, from >> ~/metron-master >> >> Jon >> >> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >> wrote: >> >>> I did what this guide said to install the original sensor: >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> Still didnt work. How do I install snort into this? >>> >>> >>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Maybe I did something wrong >>>> >>>> >>>> >>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>> >>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>> wrote: >>>>> >>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>> set up snort for you. I have a sed one liner in my bro security patch pr >>>>>> to do this, just need to do it before vagrant up. >>>>>> >>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>> vagrant up >>>>>> >>>>>> Jon >>>>>> >>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir >>>>>> wrote: >>>>>> >>>>>>> I followed this guide exactly: https://cwiki.apache >>>>>>> .org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>> >>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>> returns error of not able to find the snort command. >>>>>>> >>>>>>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> I was under the impression that you installed the full-dev >>>>>>>> environment? If so, snort should already be installed... >>>>>>>> >>>>>>>> On 2017-10-18 09:45, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>> It has become a mess. Apparently snort is released for centos 7 >>>>>>>> whereas metron one is centos 6.8. Whenever I try to install snort it >>>>>>>> gives >>>>>>>> me this: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just use those as a guide to run the commands yourself. >>>>>>>>> >>>>>>>>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir < >>>>>>>>> msc
Re: Snort
In the Vagrantfile for full-dev, edit the line that starts with ansibleSkipTags (this line <https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20>) to be exactly the following: ansibleSkipTags='quick_dev' Jon On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir wrote: > Should I edit the vagrant file using text editor and what exactly should I > edit there? > > On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> I would recommend just using a text editor if you’re not familiar with >> sed. To solve your sed problem… >> >> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >> metron-deployment/vagrant/full-dev-platform/Vagrantfile >> >> sed -i means run the sed command (in this case a find replace) inplace >> on the file, the text following the -i is the name to append to a backup >> version (ie the original file unchanged). >> >> Metron does tend to assume a good knowledge of linux admin, you’ll find >> we have a lot of shell gurus in the community, but if you’re struggling >> with this, maybe a simple text editor would be easier. All you’re trying to >> do here is change a config value. >> >> Simon >> >> On 19 Oct 2017, at 11:46, Syed Hammad Tahir wrote: >> >> Ran it without -i swtich, gives this: >> >> >> >> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com >> wrote: >> >>> The sed command is falling. It's written for a Mac so it will need an >>> alteration to be portable. Run it without the '' after -i, from >>> ~/metron-master >>> >>> Jon >>> >>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >>> wrote: >>> >>>> I did what this guide said to install the original sensor: >>>> >>>> https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs >>>> >>>> Still didnt work. How do I install snort into this? >>>> >>>> >>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Maybe I did something wrong >>>>> >>>>> >>>>> >>>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>>> >>>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>>> set up snort for you. I have a sed one liner in my bro security patch >>>>>>> pr >>>>>>> to do this, just need to do it before vagrant up. >>>>>>> >>>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>>> vagrant up >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir >>>>>>> wrote: >>>>>>> >>>>>>>> I followed this guide exactly: >>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>>> >>>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>>> returns error of not able to find the snort command. >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Syed, >>>>>>>>> >>>>>>>>> I was under the impression that you installed the full-dev >>>>>>>>> environment? If so, snort should already be installed... >>>>>>>>> >>>>>>>>> On 2017-10-18 09:45, Syed Hammad Tahir wrote: >>>>>>>>> >>>>>>>>> It has become a mess.
Re: Snort
would I need to vagrant destroy and then vagrant up again after this or will vagrant halt and vagrant up will do the job? On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com wrote: > In the Vagrantfile for full-dev, edit the line that starts with > ansibleSkipTags (this line > <https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20>) > to be exactly the following: > > ansibleSkipTags='quick_dev' > > Jon > > On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir > wrote: > >> Should I edit the vagrant file using text editor and what exactly should >> I edit there? >> >> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> I would recommend just using a text editor if you’re not familiar with >>> sed. To solve your sed problem… >>> >>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>> >>> sed -i means run the sed command (in this case a find replace) inplace >>> on the file, the text following the -i is the name to append to a backup >>> version (ie the original file unchanged). >>> >>> Metron does tend to assume a good knowledge of linux admin, you’ll find >>> we have a lot of shell gurus in the community, but if you’re struggling >>> with this, maybe a simple text editor would be easier. All you’re trying to >>> do here is change a config value. >>> >>> Simon >>> >>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir >>> wrote: >>> >>> Ran it without -i swtich, gives this: >>> >>> >>> >>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com >>> wrote: >>> >>>> The sed command is falling. It's written for a Mac so it will need an >>>> alteration to be portable. Run it without the '' after -i, from >>>> ~/metron-master >>>> >>>> Jon >>>> >>>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >>>> wrote: >>>> >>>>> I did what this guide said to install the original sensor: >>>>> https://github.com/apache/metron/tree/master/metron- >>>>> deployment/roles/sensor-stubs >>>>> >>>>> Still didnt work. How do I install snort into this? >>>>> >>>>> >>>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Maybe I did something wrong >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>>>> >>>>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>>> > wrote: >>>>>>> >>>>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>>>> set up snort for you. I have a sed one liner in my bro security patch >>>>>>>> pr >>>>>>>> to do this, just need to do it before vagrant up. >>>>>>>> >>>>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>>>> vagrant up >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I followed this guide exactly: https://cwiki. >>>>>>>>> apache.org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>>>> >>>>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>>>> returns error of not able to find the snort command. >>>>>>>>> >>>>>>>>> On Wed, Oct 18,
Re: Snort
I did all of that and then did vagrant up again. Snort is still not installed. Will I have to vagrant destroy and then vagrant up again in order for it to work? On Thu, Oct 19, 2017 at 8:58 PM, Syed Hammad Tahir wrote: > would I need to vagrant destroy and then vagrant up again after this or > will vagrant halt and vagrant up will do the job? > > On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com > wrote: > >> In the Vagrantfile for full-dev, edit the line that starts with >> ansibleSkipTags (this line >> <https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20>) >> to be exactly the following: >> >> ansibleSkipTags='quick_dev' >> >> Jon >> >> On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir >> wrote: >> >>> Should I edit the vagrant file using text editor and what exactly should >>> I edit there? >>> >>> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> I would recommend just using a text editor if you’re not familiar with >>>> sed. To solve your sed problem… >>>> >>>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>> >>>> sed -i means run the sed command (in this case a find replace) inplace >>>> on the file, the text following the -i is the name to append to a backup >>>> version (ie the original file unchanged). >>>> >>>> Metron does tend to assume a good knowledge of linux admin, you’ll find >>>> we have a lot of shell gurus in the community, but if you’re struggling >>>> with this, maybe a simple text editor would be easier. All you’re trying to >>>> do here is change a config value. >>>> >>>> Simon >>>> >>>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir >>>> wrote: >>>> >>>> Ran it without -i swtich, gives this: >>>> >>>> >>>> >>>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> The sed command is falling. It's written for a Mac so it will need an >>>>> alteration to be portable. Run it without the '' after -i, from >>>>> ~/metron-master >>>>> >>>>> Jon >>>>> >>>>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> I did what this guide said to install the original sensor: >>>>>> https://github.com/apache/metron/tree/master/metron-deployme >>>>>> nt/roles/sensor-stubs >>>>>> >>>>>> Still didnt work. How do I install snort into this? >>>>>> >>>>>> >>>>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Maybe I did something wrong >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>>>>> >>>>>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>>>> .com> wrote: >>>>>>>> >>>>>>>>> When you set up full dev if you remove the sensors skip tag it >>>>>>>>> will set up snort for you. I have a sed one liner in my bro security >>>>>>>>> patch >>>>>>>>> pr to do this, just need to do it before vagrant up. >>>>>>>>> >>>>>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>>>>> vagrant up >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk&g
Snort Installation
I have installed the snort manually. Now I need help with : 1- Capturing the data of my lan and dumping it via snort :Snort cant see the traffic outside vagrant vm, how do I make it see that traffic? 2- Making a kafka topic to push those saved logs in metron for preprocessing 3- Applying a basic Machine learning algorithm on the captured data. Regards.
Snort Logs
Hello everyone, I have run snort independently on vagrant ssh and dumped the logs in tcpdump format. Now I want to bring them to metron to play with them a bit. Some of you already replied me with some solutions but thats lost in the inbox somewhere and engulfed by the elasticsearhc issue that I had. Please give me an easy to understand this solution for this problem. Regards.
Re: Snort Installation
Help guys !!! On Fri, Oct 20, 2017 at 12:32 PM, Syed Hammad Tahir wrote: > I have installed the snort manually. Now I need help with : > > 1- Capturing the data of my lan and dumping it via snort :Snort cant see > the traffic outside vagrant vm, how do I make it see that traffic? > > 2- Making a kafka topic to push those saved logs in metron for > preprocessing > > 3- Applying a basic Machine learning algorithm on the captured data. > > Regards. >
Re: Snort Installation
Hi Syed, See inline. On 2017-10-20 00:32, Syed Hammad Tahir wrote: I have installed the snort manually. Now I need help with : 1- Capturing the data of my lan and dumping it via snort :Snort cant see the traffic outside vagrant vm, how do I make it see that traffic? To be honest, configuring Snort to work on your LAN is out of scope of the project. Have a look at the documentation at https://www.snort.org/. You will probably have to add a 2nd network interface bridged to your LAN in promiscuous mode. Additionally, I think most of us expect some basic Linux & network administration knowledge when using Metron. 2- Making a kafka topic to push those saved logs in metron for preprocessing Have a look at the Metron documentation at https://metron.apache.org/current-book/index.html. Adding a new sensor in the Metron UI will create the Kafka iirc. 3- Applying a basic Machine learning algorithm on the captured data. I can't help you with this :)
Re: Snort Installation
Ok, thankyou. I will let you know once I make snort sniff the traffic in the given configuration, might be helpful for others. I will then try to do that kafka topic and will ask if any help is needed. On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: > Hi Syed, > > See inline. > > On 2017-10-20 00:32, Syed Hammad Tahir wrote: > >> I have installed the snort manually. Now I need help with : >> >> 1- Capturing the data of my lan and dumping it via snort :Snort cant see >> the traffic outside vagrant vm, how do I make it see that traffic? >> > > To be honest, configuring Snort to work on your LAN is out of scope of the > project. Have a look at the documentation at https://www.snort.org/. > You will probably have to add a 2nd network interface bridged to your LAN > in promiscuous mode. Additionally, I think most of us expect some basic > Linux & network administration knowledge when using Metron. > > 2- Making a kafka topic to push those saved logs in metron for >> preprocessing >> > > Have a look at the Metron documentation at https://metron.apache.org/curr > ent-book/index.html. Adding a new sensor in the Metron UI will create the > Kafka iirc. > > 3- Applying a basic Machine learning algorithm on the captured data. >> > > I can't help you with this :) >
Re: Snort Installation
Hi guys, I tried to add another network interface in order to bridge it to LAN. I tried to do it on virtualbox vm settings and when i did vagrant up after that, there was no bridged interface. Can anyone help me on this? On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir wrote: > Ok, thankyou. I will let you know once I make snort sniff the traffic in > the given configuration, might be helpful for others. I will then try to do > that kafka topic and will ask if any help is needed. > > On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: > >> Hi Syed, >> >> See inline. >> >> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >> >>> I have installed the snort manually. Now I need help with : >>> >>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see >>> the traffic outside vagrant vm, how do I make it see that traffic? >>> >> >> To be honest, configuring Snort to work on your LAN is out of scope of >> the project. Have a look at the documentation at https://www.snort.org/. >> You will probably have to add a 2nd network interface bridged to your LAN >> in promiscuous mode. Additionally, I think most of us expect some basic >> Linux & network administration knowledge when using Metron. >> >> 2- Making a kafka topic to push those saved logs in metron for >>> preprocessing >>> >> >> Have a look at the Metron documentation at https://metron.apache.org/curr >> ent-book/index.html. Adding a new sensor in the Metron UI will create >> the Kafka iirc. >> >> 3- Applying a basic Machine learning algorithm on the captured data. >>> >> >> I can't help you with this :) >> > >
Re: Snort Installation
Hi Syed, Just to clarify, this a snort issue you are having? If so I suggest looking at their documentation (https://snort.org/documents) or reaching out to their community (https://snort.org/community), as they have more expertise in this area. Jon On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir wrote: > Hi guys, > > I tried to add another network interface in order to bridge it to LAN. I > tried to do it on virtualbox vm settings and when i did vagrant up after > that, there was no bridged interface. Can anyone help me on this? > > On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir > wrote: > >> Ok, thankyou. I will let you know once I make snort sniff the traffic in >> the given configuration, might be helpful for others. I will then try to do >> that kafka topic and will ask if any help is needed. >> >> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: >> >>> Hi Syed, >>> >>> See inline. >>> >>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>> >>>> I have installed the snort manually. Now I need help with : >>>> >>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>> >>> >>> To be honest, configuring Snort to work on your LAN is out of scope of >>> the project. Have a look at the documentation at https://www.snort.org/. >>> You will probably have to add a 2nd network interface bridged to your >>> LAN in promiscuous mode. Additionally, I think most of us expect some basic >>> Linux & network administration knowledge when using Metron. >>> >>> 2- Making a kafka topic to push those saved logs in metron for >>>> preprocessing >>>> >>> >>> Have a look at the Metron documentation at >>> https://metron.apache.org/current-book/index.html. Adding a new sensor >>> in the Metron UI will create the Kafka iirc. >>> >>> 3- Applying a basic Machine learning algorithm on the captured data. >>>> >>> >>> I can't help you with this :) >>> >> >> > -- Jon
Re: Snort Installation
yes nut I am a bit confused here. Let me ask them as well then. On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com wrote: > Hi Syed, > > Just to clarify, this a snort issue you are having? If so I suggest > looking at their documentation (https://snort.org/documents) or reaching > out to their community (https://snort.org/community), as they have more > expertise in this area. > > Jon > > On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir > wrote: > >> Hi guys, >> >> I tried to add another network interface in order to bridge it to LAN. I >> tried to do it on virtualbox vm settings and when i did vagrant up after >> that, there was no bridged interface. Can anyone help me on this? >> >> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir > > wrote: >> >>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>> in the given configuration, might be helpful for others. I will then try to >>> do that kafka topic and will ask if any help is needed. >>> >>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: >>> >>>> Hi Syed, >>>> >>>> See inline. >>>> >>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>> >>>>> I have installed the snort manually. Now I need help with : >>>>> >>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>> >>>> >>>> To be honest, configuring Snort to work on your LAN is out of scope of >>>> the project. Have a look at the documentation at https://www.snort.org/ >>>> . >>>> You will probably have to add a 2nd network interface bridged to your >>>> LAN in promiscuous mode. Additionally, I think most of us expect some basic >>>> Linux & network administration knowledge when using Metron. >>>> >>>> 2- Making a kafka topic to push those saved logs in metron for >>>>> preprocessing >>>>> >>>> >>>> Have a look at the Metron documentation at https://metron.apache.org/ >>>> current-book/index.html. Adding a new sensor in the Metron UI will >>>> create the Kafka iirc. >>>> >>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>> >>>> >>>> I can't help you with this :) >>>> >>> >>> >> -- > > Jon >
Re: Snort Installation
Ok, I have fixed everything on my own. Now that I have snort logs saved in a file, I need to get them to metron. Can anyone help me on that? On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir wrote: > yes nut I am a bit confused here. Let me ask them as well then. > > On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com > wrote: > >> Hi Syed, >> >> Just to clarify, this a snort issue you are having? If so I suggest >> looking at their documentation (https://snort.org/documents) or reaching >> out to their community (https://snort.org/community), as they have more >> expertise in this area. >> >> Jon >> >> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >> wrote: >> >>> Hi guys, >>> >>> I tried to add another network interface in order to bridge it to LAN. I >>> tried to do it on virtualbox vm settings and when i did vagrant up after >>> that, there was no bridged interface. Can anyone help me on this? >>> >>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>>> in the given configuration, might be helpful for others. I will then try to >>>> do that kafka topic and will ask if any help is needed. >>>> >>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> See inline. >>>>> >>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>> >>>>>> I have installed the snort manually. Now I need help with : >>>>>> >>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>>> >>>>> >>>>> To be honest, configuring Snort to work on your LAN is out of scope of >>>>> the project. Have a look at the documentation at >>>>> https://www.snort.org/. >>>>> You will probably have to add a 2nd network interface bridged to your >>>>> LAN in promiscuous mode. Additionally, I think most of us expect some >>>>> basic >>>>> Linux & network administration knowledge when using Metron. >>>>> >>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>> preprocessing >>>>>> >>>>> >>>>> Have a look at the Metron documentation at >>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>> sensor in the Metron UI will create the Kafka iirc. >>>>> >>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>> >>>>> >>>>> I can't help you with this :) >>>>> >>>> >>>> >>> -- >> >> Jon >> > >
Re: Snort Installation
Take a look at `kafka-console-producer.sh`, which is installed as part of Kafka. On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir wrote: > Ok, I have fixed everything on my own. Now that I have snort logs saved in > a file, I need to get them to metron. Can anyone help me on that? > > On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir > wrote: > >> yes nut I am a bit confused here. Let me ask them as well then. >> >> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >> wrote: >> >>> Hi Syed, >>> >>> Just to clarify, this a snort issue you are having? If so I suggest >>> looking at their documentation (https://snort.org/documents) or >>> reaching out to their community (https://snort.org/community), as they >>> have more expertise in this area. >>> >>> Jon >>> >>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>> wrote: >>> >>>> Hi guys, >>>> >>>> I tried to add another network interface in order to bridge it to LAN. >>>> I tried to do it on virtualbox vm settings and when i did vagrant up after >>>> that, there was no bridged interface. Can anyone help me on this? >>>> >>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>>>> in the given configuration, might be helpful for others. I will then try >>>>> to >>>>> do that kafka topic and will ask if any help is needed. >>>>> >>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>> wrote: >>>>> >>>>>> Hi Syed, >>>>>> >>>>>> See inline. >>>>>> >>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>> >>>>>>> I have installed the snort manually. Now I need help with : >>>>>>> >>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>>>> >>>>>> >>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>> of the project. Have a look at the documentation at >>>>>> https://www.snort.org/. >>>>>> You will probably have to add a 2nd network interface bridged to your >>>>>> LAN in promiscuous mode. Additionally, I think most of us expect some >>>>>> basic >>>>>> Linux & network administration knowledge when using Metron. >>>>>> >>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>> preprocessing >>>>>>> >>>>>> >>>>>> Have a look at the Metron documentation at >>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>> >>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>> >>>>>> >>>>>> I can't help you with this :) >>>>>> >>>>> >>>>> >>>> -- >>> >>> Jon >>> >> >> >
Re: Snort Installation
Where do I find this file kafka-console-producer.sh? On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen wrote: > Take a look at `kafka-console-producer.sh`, which is installed as part of > Kafka. > > On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir > wrote: > >> Ok, I have fixed everything on my own. Now that I have snort logs saved >> in a file, I need to get them to metron. Can anyone help me on that? >> >> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> wrote: >> >>> yes nut I am a bit confused here. Let me ask them as well then. >>> >>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >>> wrote: >>> >>>> Hi Syed, >>>> >>>> Just to clarify, this a snort issue you are having? If so I suggest >>>> looking at their documentation (https://snort.org/documents) or >>>> reaching out to their community (https://snort.org/community), as they >>>> have more expertise in this area. >>>> >>>> Jon >>>> >>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>>> wrote: >>>> >>>>> Hi guys, >>>>> >>>>> I tried to add another network interface in order to bridge it to LAN. >>>>> I tried to do it on virtualbox vm settings and when i did vagrant up after >>>>> that, there was no bridged interface. Can anyone help me on this? >>>>> >>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>> >>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>>> wrote: >>>>>> >>>>>>> Hi Syed, >>>>>>> >>>>>>> See inline. >>>>>>> >>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>> >>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>> >>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>> traffic? >>>>>>>> >>>>>>> >>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>> of the project. Have a look at the documentation at >>>>>>> https://www.snort.org/. >>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>> some >>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>> >>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>> preprocessing >>>>>>>> >>>>>>> >>>>>>> Have a look at the Metron documentation at >>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>> >>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>> >>>>>>> >>>>>>> I can't help you with this :) >>>>>>> >>>>>> >>>>>> >>>>> -- >>>> >>>> Jon >>>> >>> >>> >> >
Re: Snort Installation
All I did was install snort separately on vagrant ssh console. The ran it to collect logs. Now I need to bring those logs to metron. On Wed, Oct 25, 2017 at 9:50 AM, Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > Hi Syed Hammed, > > Can you share the steps how did you connected snort with external source ? > (Metron Snort ?) > > On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen wrote: > >> Take a look at `kafka-console-producer.sh`, which is installed as part of >> Kafka. >> >> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir >> wrote: >> >>> Ok, I have fixed everything on my own. Now that I have snort logs saved >>> in a file, I need to get them to metron. Can anyone help me on that? >>> >>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> > wrote: >>> >>>> yes nut I am a bit confused here. Let me ask them as well then. >>>> >>>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> Just to clarify, this a snort issue you are having? If so I suggest >>>>> looking at their documentation (https://snort.org/documents) or >>>>> reaching out to their community (https://snort.org/community), as >>>>> they have more expertise in this area. >>>>> >>>>> Jon >>>>> >>>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I tried to add another network interface in order to bridge it to >>>>>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up >>>>>> after that, there was no bridged interface. Can anyone help me on this? >>>>>> >>>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>>> >>>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> See inline. >>>>>>>> >>>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>>> >>>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>>> traffic? >>>>>>>>> >>>>>>>> >>>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>>> of the project. Have a look at the documentation at >>>>>>>> https://www.snort.org/. >>>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>>> some >>>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>>> >>>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>>> preprocessing >>>>>>>>> >>>>>>>> >>>>>>>> Have a look at the Metron documentation at >>>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>>> >>>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>>> >>>>>>>> >>>>>>>> I can't help you with this :) >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> >>> >> > > > -- > With Regards > Farrukh Naveed Anjum >
Re: Snort Installation
A quick google search will answer your question. On Wed, Oct 25, 2017 at 1:30 AM Syed Hammad Tahir wrote: > Where do I find this file kafka-console-producer.sh? > > On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen wrote: > >> Take a look at `kafka-console-producer.sh`, which is installed as part of >> Kafka. >> >> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir >> wrote: >> >>> Ok, I have fixed everything on my own. Now that I have snort logs saved >>> in a file, I need to get them to metron. Can anyone help me on that? >>> >>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> > wrote: >>> >>>> yes nut I am a bit confused here. Let me ask them as well then. >>>> >>>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> Just to clarify, this a snort issue you are having? If so I suggest >>>>> looking at their documentation (https://snort.org/documents) or >>>>> reaching out to their community (https://snort.org/community), as >>>>> they have more expertise in this area. >>>>> >>>>> Jon >>>>> >>>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I tried to add another network interface in order to bridge it to >>>>>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up >>>>>> after that, there was no bridged interface. Can anyone help me on this? >>>>>> >>>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>>> >>>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> See inline. >>>>>>>> >>>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>>> >>>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>>> traffic? >>>>>>>>> >>>>>>>> >>>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>>> of the project. Have a look at the documentation at >>>>>>>> https://www.snort.org/. >>>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>>> some >>>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>>> >>>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>>> preprocessing >>>>>>>>> >>>>>>>> >>>>>>>> Have a look at the Metron documentation at >>>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>>> >>>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>>> >>>>>>>> >>>>>>>> I can't help you with this :) >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> >>> >> >
Re: Snort Logs
If you have text snort logs you can use Apache nifi or the Kafka producer script as described in step 4 here[1] to push them to Metron's snort topic. You may also want to look at this [2]. 1: https://kafka.apache.org/quickstart 2: https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script Jon On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir wrote: > Hello everyone, > > I have run snort independently on vagrant ssh and dumped the logs in > tcpdump format. Now I want to bring them to metron to play with them a bit. > Some of you already replied me with some solutions but thats lost in the > inbox somewhere and engulfed by the elasticsearhc issue that I had. Please > give me an easy to understand this solution for this problem. > > Regards. > -- Jon
Re: Snort Logs
snort logs are in tcp dump format. I may have to convert them. bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test How to give file name or path in this command? On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com wrote: > If you have text snort logs you can use Apache nifi or the Kafka producer > script as described in step 4 here[1] to push them to Metron's snort > topic. You may also want to look at this [2]. > > 1: https://kafka.apache.org/quickstart > 2: https://stackoverflow.com/questions/38701179/kafka- > console-producer-and-bash-script > > Jon > > On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir > wrote: > >> Hello everyone, >> >> I have run snort independently on vagrant ssh and dumped the logs in >> tcpdump format. Now I want to bring them to metron to play with them a bit. >> Some of you already replied me with some solutions but thats lost in the >> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >> give me an easy to understand this solution for this problem. >> >> Regards. >> > -- > > Jon >
Re: Snort Logs
On the 25th I said: It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from memory) on node1, assuming you are running full dev. Jon Jon On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir wrote: > snort logs are in tcp dump format. I may have to convert them. > > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > > How to give file name or path in this command? > > On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com > wrote: > >> If you have text snort logs you can use Apache nifi or the Kafka producer >> script as described in step 4 here[1] to push them to Metron's snort >> topic. You may also want to look at this [2]. >> >> 1: https://kafka.apache.org/quickstart >> 2: >> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >> >> Jon >> >> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >> wrote: >> >>> Hello everyone, >>> >>> I have run snort independently on vagrant ssh and dumped the logs in >>> tcpdump format. Now I want to bring them to metron to play with them a bit. >>> Some of you already replied me with some solutions but thats lost in the >>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>> give me an easy to understand this solution for this problem. >>> >>> Regards. >>> >> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
I have found the kafka-console-producer.sh but I need to know how to make it read snort.log (tcp dump format) file. May be I am missing something in the plain sight but it would be awsome if you tell me that. Regards. On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com wrote: > On the 25th I said: > > It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from > memory) on node1, assuming you are running full dev. > > Jon > > > Jon > > On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir > wrote: > >> snort logs are in tcp dump format. I may have to convert them. >> >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >> >> How to give file name or path in this command? >> >> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >> wrote: >> >>> If you have text snort logs you can use Apache nifi or the Kafka >>> producer script as described in step 4 here[1] to push them to Metron's >>> snort topic. You may also want to look at this [2]. >>> >>> 1: https://kafka.apache.org/quickstart >>> 2: https://stackoverflow.com/questions/38701179/kafka- >>> console-producer-and-bash-script >>> >>> Jon >>> >>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>> wrote: >>> >>>> Hello everyone, >>>> >>>> I have run snort independently on vagrant ssh and dumped the logs in >>>> tcpdump format. Now I want to bring them to metron to play with them a bit. >>>> Some of you already replied me with some solutions but thats lost in the >>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>> give me an easy to understand this solution for this problem. >>>> >>>> Regards. >>>> >>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
You need text logs. Here's an example of some properly formatted logs - https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out Jon On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir wrote: > I have found the kafka-console-producer.sh but I need to know how to make > it read snort.log (tcp dump format) file. May be I am missing something in > the plain sight but it would be awsome if you tell me that. > > Regards. > > On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com > wrote: > >> On the 25th I said: >> >> It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from >> memory) on node1, assuming you are running full dev. >> >> Jon >> >> >> Jon >> >> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >> wrote: >> >>> snort logs are in tcp dump format. I may have to convert them. >>> >>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>> >>> How to give file name or path in this command? >>> >>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>> wrote: >>> >>>> If you have text snort logs you can use Apache nifi or the Kafka >>>> producer script as described in step 4 here[1] to push them to Metron's >>>> snort topic. You may also want to look at this [2]. >>>> >>>> 1: https://kafka.apache.org/quickstart >>>> 2: >>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>> bit. >>>>> Some of you already replied me with some solutions but thats lost in the >>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>>> give me an easy to understand this solution for this problem. >>>>> >>>>> Regards. >>>>> >>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
Yes, I have converted them to text but those logs are simply captured packet headers over the local network. Now I just push them via that kafka producer command under topic name of snort and they will be visible in metron? On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com wrote: > You need text logs. Here's an example of some properly formatted logs - > https://raw.githubusercontent.com/apache/metron/master/metron- > deployment/roles/sensor-stubs/files/snort.out > > Jon > > On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir > wrote: > >> I have found the kafka-console-producer.sh but I need to know how to >> make it read snort.log (tcp dump format) file. May be I am missing >> something in the plain sight but it would be awsome if you tell me that. >> >> Regards. >> >> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >> wrote: >> >>> On the 25th I said: >>> >>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>> (from memory) on node1, assuming you are running full dev. >>> >>> Jon >>> >>> >>> Jon >>> >>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >>> wrote: >>> >>>> snort logs are in tcp dump format. I may have to convert them. >>>> >>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>> >>>> How to give file name or path in this command? >>>> >>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>> snort topic. You may also want to look at this [2]. >>>>> >>>>> 1: https://kafka.apache.org/quickstart >>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>> console-producer-and-bash-script >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>> bit. >>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>> Please >>>>>> give me an easy to understand this solution for this problem. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
They need to meet the format of the logs I sent earlier. Look into the snort output options - may require you rerun snort, depending on your situation Jon On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir wrote: > Yes, I have converted them to text but those logs are simply captured > packet headers over the local network. Now I just push them via that kafka > producer command under topic name of snort and they will be visible in > metron? > > On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com > wrote: > >> You need text logs. Here's an example of some properly formatted logs - >> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out >> >> Jon >> >> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir >> wrote: >> >>> I have found the kafka-console-producer.sh but I need to know how to >>> make it read snort.log (tcp dump format) file. May be I am missing >>> something in the plain sight but it would be awsome if you tell me that. >>> >>> Regards. >>> >>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >>> wrote: >>> >>>> On the 25th I said: >>>> >>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>> (from memory) on node1, assuming you are running full dev. >>>> >>>> Jon >>>> >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >>>> wrote: >>>> >>>>> snort logs are in tcp dump format. I may have to convert them. >>>>> >>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>>> >>>>> How to give file name or path in this command? >>>>> >>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>>> wrote: >>>>> >>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>> snort topic. You may also want to look at this [2]. >>>>>> >>>>>> 1: https://kafka.apache.org/quickstart >>>>>> 2: >>>>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>>>> >>>>>> Jon >>>>>> >>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>>> wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>>> bit. >>>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>> Please >>>>>>> give me an easy to understand this solution for this problem. >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>> -- >>>>>> >>>>>> Jon >>>>>> >>>>> >>>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
I sent a random message to that kafka topic and got this [image: Inline image 1] I guess this is because I am not following the format of message I should send? Like those snort logs you showed. On Mon, Oct 30, 2017 at 5:24 PM, zeo...@gmail.com wrote: > They need to meet the format of the logs I sent earlier. Look into the > snort output options - may require you rerun snort, depending on your > situation > > Jon > > On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir > wrote: > >> Yes, I have converted them to text but those logs are simply captured >> packet headers over the local network. Now I just push them via that kafka >> producer command under topic name of snort and they will be visible in >> metron? >> >> On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com >> wrote: >> >>> You need text logs. Here's an example of some properly formatted logs - >>> https://raw.githubusercontent.com/apache/metron/master/metron- >>> deployment/roles/sensor-stubs/files/snort.out >>> >>> Jon >>> >>> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir >>> wrote: >>> >>>> I have found the kafka-console-producer.sh but I need to know how to >>>> make it read snort.log (tcp dump format) file. May be I am missing >>>> something in the plain sight but it would be awsome if you tell me that. >>>> >>>> Regards. >>>> >>>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> On the 25th I said: >>>>> >>>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>>> (from memory) on node1, assuming you are running full dev. >>>>> >>>>> Jon >>>>> >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> snort logs are in tcp dump format. I may have to convert them. >>>>>> >>>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic >>>>>> test >>>>>> >>>>>> How to give file name or path in this command? >>>>>> >>>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>>> snort topic. You may also want to look at this [2]. >>>>>>> >>>>>>> 1: https://kafka.apache.org/quickstart >>>>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>>>> console-producer-and-bash-script >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>>>> wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> I have run snort independently on vagrant ssh and dumped the logs >>>>>>>> in tcpdump format. Now I want to bring them to metron to play with >>>>>>>> them a >>>>>>>> bit. Some of you already replied me with some solutions but thats lost >>>>>>>> in >>>>>>>> the inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>>> Please give me an easy to understand this solution for this problem. >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jon >>>>>>> >>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
And how do I install elasticsearch head on the vagrant VM?
Re: Snort Logs
You can install it into the chrome web browser from the play store. On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) wrote: And how do I install elasticsearch head on the vagrant VM?
Fwd: Snort Logs
-- Forwarded message -- From: Syed Hammad Tahir Date: Fri, Nov 3, 2017 at 5:07 PM Subject: Re: Snort Logs To: Otto Fowler NVM, I have installed the elastic search head. Now where do I go in this to find out why I cant see the snort logs in kibana dashboard, pushed to snort topic via kafka producer? [image: Inline image 1] On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler wrote: > You can install it into the chrome web browser from the play store. > > > > On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) > wrote: > > And how do I install elasticsearch head on the vagrant VM? > >
Re: Snort Logs
It looks like your ES cluster has a health of Red, so there's your problem. I would go look in /var/log/elasticsearch/ at some logs. Jon On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir wrote: > > -- Forwarded message -- > From: Syed Hammad Tahir > Date: Fri, Nov 3, 2017 at 5:07 PM > Subject: Re: Snort Logs > To: Otto Fowler > > > NVM, I have installed the elastic search head. Now where do I go in this > to find out why I cant see the snort logs in kibana dashboard, pushed to > snort topic via kafka producer? > > [image: Inline image 1] > > On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler > wrote: > >> You can install it into the chrome web browser from the play store. >> >> >> >> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >> wrote: >> >> And how do I install elasticsearch head on the vagrant VM? >> >> > -- Jon
Re: Snort Logs
hi, I am back at work. lets see if i can find something in logs On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com wrote: > It looks like your ES cluster has a health of Red, so there's your > problem. I would go look in /var/log/elasticsearch/ at some logs. > > Jon > > On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir > wrote: > >> >> -- Forwarded message -- >> From: Syed Hammad Tahir >> Date: Fri, Nov 3, 2017 at 5:07 PM >> Subject: Re: Snort Logs >> To: Otto Fowler >> >> >> NVM, I have installed the elastic search head. Now where do I go in this >> to find out why I cant see the snort logs in kibana dashboard, pushed to >> snort topic via kafka producer? >> >> [image: Inline image 1] >> >> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler >> wrote: >> >>> You can install it into the chrome web browser from the play store. >>> >>> >>> >>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >>> wrote: >>> >>> And how do I install elasticsearch head on the vagrant VM? >>> >>> >> -- > > Jon >
Re: Snort Logs
I have attached the output of this dump /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com wrote: > What is the output of: > > /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP > > ? > > Jon > > On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir > wrote: > >> This is the script/command i used >> >> sudo cat snort.out | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir >> wrote: >> >>> sudo cat snort.out | >>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>> --broker-list node1:6667 --topic snort >>> >>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>> wrote: >>> >>>> What topic? what are the parameters you are calling the script with? >>>> >>>> >>>> >>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> The metron installation I have (single node based vm install) comes >>>> with sensor stubs. I assume that everything has already been done for those >>>> stub sensors to push the canned data. I am doing the similar thing, >>>> directly pushing the preformatted canned data to kafka topic. I can see the >>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>> push the same logs myself, those errors pop that I have shown earlier. >>>> >>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>> wrote: >>>> >>>>> How did you start the snort parser topology and what's the parser >>>>> config (in zookeeper)? >>>>> >>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> This is what I am doing >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> >>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>> wrote: >>>>>> >>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>> parser or from some other source? It looks like there are some records >>>>>>> in >>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>> topic, >>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>> then >>>>>>> the enrichments topology enrich that map and emits the enriched map to >>>>>>> the >>>>>>> indexing topic. >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>> full stack trace >>>>>>>> >>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 1] >>>>>>>> >>>>>>>> from indexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 2] >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>> >>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>> Also, are you saying that you are no longer seeing the parser >>>>>>>>> topology error? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella (ceste...@gmail.com) >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> If you click on the port (6704) there in those errors, what's the >>>>>>>>> full stacktrace (that starts with the suggesti
Re: Snort Logs
Any solution to these issues guys? On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir wrote: > I have attached the output of this dump > > /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP > > > > On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com > wrote: > >> What is the output of: >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> ? >> >> Jon >> >> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >> wrote: >> >>> This is the script/command i used >>> >>> sudo cat snort.out | >>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>> --broker-list node1:6667 --topic snort >>> >>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir >> > wrote: >>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>>> wrote: >>>> >>>>> What topic? what are the parameters you are calling the script with? >>>>> >>>>> >>>>> >>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>> mscs16...@itu.edu.pk) wrote: >>>>> >>>>> The metron installation I have (single node based vm install) comes >>>>> with sensor stubs. I assume that everything has already been done for >>>>> those >>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>> the >>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>> >>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>> wrote: >>>>> >>>>>> How did you start the snort parser topology and what's the parser >>>>>> config (in zookeeper)? >>>>>> >>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> This is what I am doing >>>>>>> >>>>>>> sudo cat snort.out | >>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>> --broker-list node1:6667 --topic snort >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>> wrote: >>>>>>> >>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>> parser or from some other source? It looks like there are some >>>>>>>> records in >>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>> topic, >>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>> then >>>>>>>> the enrichments topology enrich that map and emits the enriched map to >>>>>>>> the >>>>>>>> indexing topic. >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>> >>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>> full stack trace >>>>>>>>> >>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>> >>>>>>>>> [image: Inline image 1] >>>>>>>>> >>>>>>>>> from indexingbolt in indexing topology >>>>>>>>> >>>>>>>>> [image: Inline image 2] >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>> Also, are you saying that you are no longer seeing the parser >&
Re: Snort Logs
hi, This problem still persists guys . On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir wrote: > Any solution to these issues guys? > > On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir > wrote: > >> I have attached the output of this dump >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> >> >> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com >> wrote: >> >>> What is the output of: >>> >>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>> >>> ? >>> >>> Jon >>> >>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >>> wrote: >>> >>>> This is the script/command i used >>>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> sudo cat snort.out | >>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>> --broker-list node1:6667 --topic snort >>>>> >>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>>>> wrote: >>>>> >>>>>> What topic? what are the parameters you are calling the script with? >>>>>> >>>>>> >>>>>> >>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>> mscs16...@itu.edu.pk) wrote: >>>>>> >>>>>> The metron installation I have (single node based vm install) comes >>>>>> with sensor stubs. I assume that everything has already been done for >>>>>> those >>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>>> the >>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>>> wrote: >>>>>> >>>>>>> How did you start the snort parser topology and what's the parser >>>>>>> config (in zookeeper)? >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> This is what I am doing >>>>>>>> >>>>>>>> sudo cat snort.out | >>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>> records in >>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>> topic, >>>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>>> then >>>>>>>>> the enrichments topology enrich that map and emits the enriched map >>>>>>>>> to the >>>>>>>>> indexing topic. >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>>> full stack trace >>>>>>>>>> >>>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>>> >>>>>>>>>> [image: Inline image 1] >>>>>>>>>> >>>>>>>>>> from indexingbolt in indexing topology >>>>
Re: Snort Logs
Can you restart storm and give it another shot? Jon On Mon, Nov 13, 2017, 00:30 Syed Hammad Tahir wrote: > hi, This problem still persists guys . > > On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir > wrote: > >> Any solution to these issues guys? >> >> On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir >> wrote: >> >>> I have attached the output of this dump >>> >>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>> >>> >>> >>> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com >>> wrote: >>> >>>> What is the output of: >>>> >>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>> >>>> ? >>>> >>>> Jon >>>> >>>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >>>> wrote: >>>> >>>>> This is the script/command i used >>>>> >>>>> sudo cat snort.out | >>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>> --broker-list node1:6667 --topic snort >>>>> >>>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>>>> > wrote: >>>>>> >>>>>>> What topic? what are the parameters you are calling the script with? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>> >>>>>>> The metron installation I have (single node based vm install) comes >>>>>>> with sensor stubs. I assume that everything has already been done for >>>>>>> those >>>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>>>> the >>>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>>>> wrote: >>>>>>> >>>>>>>> How did you start the snort parser topology and what's the parser >>>>>>>> config (in zookeeper)? >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>> >>>>>>>>> This is what I am doing >>>>>>>>> >>>>>>>>> sudo cat snort.out | >>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>>> --broker-list >>>>>>>>> node1:6667 --topic snort >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>>> records in >>>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>>> topic, >>>>>>>>>> it should be a JSON map. The parser topology emits that JSON map >>>>>>>>>> and then >>>>>>>>>> the enrichments topology enrich that map and emits the enriched map >>>>>>>>>> to the >>>>>>>>>> indexing topic. >>>>>>>>>> >>>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>&g
Re: Snort Logs
ok, Doing it. On Mon, Nov 13, 2017 at 3:07 PM, zeo...@gmail.com wrote: > Can you restart storm and give it another shot? > > Jon > > On Mon, Nov 13, 2017, 00:30 Syed Hammad Tahir > wrote: > >> hi, This problem still persists guys . >> >> On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir >> wrote: >> >>> Any solution to these issues guys? >>> >>> On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir >>> wrote: >>> >>>> I have attached the output of this dump >>>> >>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>> >>>> >>>> >>>> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com >>>> wrote: >>>> >>>>> What is the output of: >>>>> >>>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>>> >>>>> ? >>>>> >>>>> Jon >>>>> >>>>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> This is the script/command i used >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> sudo cat snort.out | >>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>> --broker-list node1:6667 --topic snort >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler < >>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>> >>>>>>>> What topic? what are the parameters you are calling the script >>>>>>>> with? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>> >>>>>>>> The metron installation I have (single node based vm install) comes >>>>>>>> with sensor stubs. I assume that everything has already been done for >>>>>>>> those >>>>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>>>> directly pushing the preformatted canned data to kafka topic. I can >>>>>>>> see the >>>>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>>>>> wrote: >>>>>>>> >>>>>>>>> How did you start the snort parser topology and what's the parser >>>>>>>>> config (in zookeeper)? >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> This is what I am doing >>>>>>>>>> >>>>>>>>>> sudo cat snort.out | >>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>>>> records in >>>>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>>>> topic, >>>>>>>>>>> it should be a JSON map. The parser topology emits that JSON map >>>>>>>>&
Re: Snort Logs
I guess I am wrong. But from looking at the output, it looks like this is error topic stuff that is failing doesn’t it? On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com) wrote: Isn't sending indexing errors to the indexing topic intentional? I may need to refresh myself on the below conversation, but I recall it coming up in conversation on the mailing lists in the past. https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33 https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E Jon On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler wrote: > OK. > > I think your sending errors to your indexing topic instead of the error > topic. > I think you posted your config before, but I don’t remember off the top of > my head > where the error topic is configured. > > If the error topic is the same as the indexing topic, and you ‘have > errors’ I think you may see this. > > > > On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk) > wrote: > > Here we go. This is what I see when I do kafka client on indexing topic. > > [image: Inline image 1] > > On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir > wrote: > >> ok, I will try it again and report results >> >> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler >> wrote: >> >>> You have to be seeing data in the indexing topic, you have errors in the >>> indexing topology that reads from it. >>> >>> >>> >>> On November 13, 2017 at 13:42:14, Syed Hammad Tahir ( >>> mscs16...@itu.edu.pk) wrote: >>> >>> So you are saying: >>> >>> * when you do the kafka client on the enrichment topic things are in json >>> * when you do the kafka client on the indexing topic they are csv >>> >>> 1- Yes, kafka client on enrichment shows json >>> >>> 2- No, I dont see anything in kafka client on indexing topic >>> >>> On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler >>> wrote: >>> >>>> So you are saying: >>>> >>>> * when you do the kafka client on the enrichment topic things are in >>>> json >>>> * when you do the kafka client on the indexing topic they are csv >>>> >>>> ??? >>>> >>>> >>>> >>>> On November 13, 2017 at 12:28:51, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> From one of your earlier messages, This is what I have figured out so >>>> far. >>>> >>>> [image: Inline image 1] >>>> >>>> The issue is inducated by red marked portion of the flow. >>>> >>>> On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Which .java file is causing the issue in this hdfsindexbolt. I mean >>>>> which one should I look at because there are so many listed here. >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it >>>>>> correctly since I am getting error in the indexing bolt not in the >>>>>> parser one. >>>>>> >>>>>> >>>>>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> org.apache.metron.parsers.snort.BasicSnortParser does this parse the >>>>>>> basic message and then convert it in JSON? >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 13, 2017 at 9:00 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> No, I am not seeing it under indexing topic as JSON. I can only see >>>>>>>> JSON objects of stub sensor logs but not from those pushed by me via >>>>>>>> kafka >>>>>>>> producer. >>>>>>>> >>>>>>>> On Mon, Nov 13, 2017 at 5:17 PM, zeo...@gmail.com >>>>>>> > wrote: >>>>>>>> >>>>>>>>> Please use kafka-console-consumer.sh (
Re: Snort Logs
OK, So the JSONFromPosition MessageGetStrategy is choking the csv… I don’t know if there is a position index or it is 0, but whatever it is, it is getting csv not json message. On November 13, 2017 at 15:49:57, Otto Fowler (ottobackwa...@gmail.com) wrote: I guess I am wrong. But from looking at the output, it looks like this is error topic stuff that is failing doesn’t it? On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com) wrote: Isn't sending indexing errors to the indexing topic intentional? I may need to refresh myself on the below conversation, but I recall it coming up in conversation on the mailing lists in the past. https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33 https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E Jon On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler wrote: > OK. > > I think your sending errors to your indexing topic instead of the error > topic. > I think you posted your config before, but I don’t remember off the top of > my head > where the error topic is configured. > > If the error topic is the same as the indexing topic, and you ‘have > errors’ I think you may see this. > > > > On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk) > wrote: > > Here we go. This is what I see when I do kafka client on indexing topic. > > [image: Inline image 1] > > On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir > wrote: > >> ok, I will try it again and report results >> >> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler >> wrote: >> >>> You have to be seeing data in the indexing topic, you have errors in the >>> indexing topology that reads from it. >>> >>> >>> >>> On November 13, 2017 at 13:42:14, Syed Hammad Tahir ( >>> mscs16...@itu.edu.pk) wrote: >>> >>> So you are saying: >>> >>> * when you do the kafka client on the enrichment topic things are in json >>> * when you do the kafka client on the indexing topic they are csv >>> >>> 1- Yes, kafka client on enrichment shows json >>> >>> 2- No, I dont see anything in kafka client on indexing topic >>> >>> On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler >>> wrote: >>> >>>> So you are saying: >>>> >>>> * when you do the kafka client on the enrichment topic things are in >>>> json >>>> * when you do the kafka client on the indexing topic they are csv >>>> >>>> ??? >>>> >>>> >>>> >>>> On November 13, 2017 at 12:28:51, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> From one of your earlier messages, This is what I have figured out so >>>> far. >>>> >>>> [image: Inline image 1] >>>> >>>> The issue is inducated by red marked portion of the flow. >>>> >>>> On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Which .java file is causing the issue in this hdfsindexbolt. I mean >>>>> which one should I look at because there are so many listed here. >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it >>>>>> correctly since I am getting error in the indexing bolt not in the >>>>>> parser one. >>>>>> >>>>>> >>>>>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> org.apache.metron.parsers.snort.BasicSnortParser does this parse the >>>>>>> basic message and then convert it in JSON? >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 13, 2017 at 9:00 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> No, I am not seeing it under indexing topic as JSON. I can only see >>>>>>>> JSON objects of stub sensor logs but not from those pushed by me via >>>>>>>> kafka >>>>>>>> producer. >>>>>
Re: Snort Logs
Yeah, I think you're right. I'm not familiar with that code as much though, I've never had to touch it. Given all of the lessons learned up to this point, maybe spinning full-dev up from scratch again makes sense? Typically I am able to spin up full-dev pretty hands-off without hitting these types of issues, as long as my machine has enough resources to give. Jon On Mon, Nov 13, 2017 at 3:49 PM Otto Fowler wrote: > I guess I am wrong. > But from looking at the output, it looks like this is error topic stuff > that is failing doesn’t it? > > > > On November 13, 2017 at 15:06:20, zeo...@gmail.com (zeo...@gmail.com) > wrote: > > Isn't sending indexing errors to the indexing topic intentional? I may > need to refresh myself on the below conversation, but I recall it coming up > in conversation on the mailing lists in the past. > > > https://github.com/apache/metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties#L33 > > https://lists.apache.org/thread.html/01e4ed416bda8d1057f09f7717809d2802ae1de3035dc42f001d7bbe@%3Cdev.metron.apache.org%3E > > Jon > > On Mon, Nov 13, 2017 at 2:59 PM Otto Fowler > wrote: > >> OK. >> >> I think your sending errors to your indexing topic instead of the error >> topic. >> I think you posted your config before, but I don’t remember off the top >> of my head >> where the error topic is configured. >> >> If the error topic is the same as the indexing topic, and you ‘have >> errors’ I think you may see this. >> >> >> >> On November 13, 2017 at 14:39:44, Syed Hammad Tahir (mscs16...@itu.edu.pk) >> wrote: >> >> Here we go. This is what I see when I do kafka client on indexing topic. >> >> [image: Inline image 1] >> >> On Tue, Nov 14, 2017 at 12:03 AM, Syed Hammad Tahir > > wrote: >> >>> ok, I will try it again and report results >>> >>> On Tue, Nov 14, 2017 at 12:00 AM, Otto Fowler >>> wrote: >>> >>>> You have to be seeing data in the indexing topic, you have errors in >>>> the indexing topology that reads from it. >>>> >>>> >>>> >>>> On November 13, 2017 at 13:42:14, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> So you are saying: >>>> >>>> * when you do the kafka client on the enrichment topic things are in >>>> json >>>> * when you do the kafka client on the indexing topic they are csv >>>> >>>> 1- Yes, kafka client on enrichment shows json >>>> >>>> 2- No, I dont see anything in kafka client on indexing topic >>>> >>>> On Mon, Nov 13, 2017 at 11:26 PM, Otto Fowler >>>> wrote: >>>> >>>>> So you are saying: >>>>> >>>>> * when you do the kafka client on the enrichment topic things are in >>>>> json >>>>> * when you do the kafka client on the indexing topic they are csv >>>>> >>>>> ??? >>>>> >>>>> >>>>> >>>>> On November 13, 2017 at 12:28:51, Syed Hammad Tahir ( >>>>> mscs16...@itu.edu.pk) wrote: >>>>> >>>>> From one of your earlier messages, This is what I have figured out so >>>>> far. >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> The issue is inducated by red marked portion of the flow. >>>>> >>>>> On Mon, Nov 13, 2017 at 10:14 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Which .java file is causing the issue in this hdfsindexbolt. I mean >>>>>> which one should I look at because there are so many listed here. >>>>>> >>>>>> [image: Inline image 1] >>>>>> >>>>>> On Mon, Nov 13, 2017 at 9:25 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> org.apache.metron.parsers.snort.BasicSnortParser This one is parsing it >>>>>>> correctly since I am getting error in the indexing bolt not in the >>>>>>> parser one. >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 13, 2017 at 9:17 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> org.apach
Re: Snort enrichment issue
Did you setup and load the geo enrichment database? https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader <https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader> Also, we can’t really see the error from screenshots, please send log entries. Simon > On 17 Nov 2017, at 07:11, Syed Hammad Tahir wrote: > > Hi all, I am starting it again. Last one got a bit messy > > Ok, Now I have started everything again from scratch (redeployed single node > based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when > I execute this command: > > shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date > +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | > /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list > node1:6667 --topic snort > > (format of ths command was taken from: > https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub > > <https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub>) > > I get this under enrichment storm topology : > > > > > > I have come this far, please help me push these dummy preformatted snort logs > into kibana dashboard. > > Regards. >
Re: Snort enrichment issue
Here you go, the error part of the log is in the attachment. On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Did you setup and load the geo enrichment database? https://metron. > apache.org/current-book/metron-platform/metron-data-management/index.html# > GeoLite2_Loader > > Also, we can’t really see the error from screenshots, please send log > entries. > > Simon > > On 17 Nov 2017, at 07:11, Syed Hammad Tahir wrote: > > Hi all, I am starting it again. Last one got a bit messy > > Ok, Now I have started everything again from scratch (redeployed single > node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and > now when I execute this command: > > shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date > +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | > /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh > --broker-list node1:6667 --topic snort > > (format of ths command was taken from: https://github.com/apach > e/metron/blob/master/metron-deployment/roles/sensor-stubs/ > templates/start-snort-stub) > > I get this under enrichment storm topology : > > > > > > I have come this far, please help me push these dummy preformatted snort > logs into kibana dashboard. > > Regards. > > > Enrichment Description: Binary data
Re: Snort enrichment issue
ANd I dint load anything. It was supposed to be loaded during installation? My installation is ambari based single node VM install on ubuntu host. On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir wrote: > Here you go, the error part of the log is in the attachment. > > On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Did you setup and load the geo enrichment database? https://metron.apach >> e.org/current-book/metron-platform/metron-data-managemen >> t/index.html#GeoLite2_Loader >> >> Also, we can’t really see the error from screenshots, please send log >> entries. >> >> Simon >> >> On 17 Nov 2017, at 07:11, Syed Hammad Tahir wrote: >> >> Hi all, I am starting it again. Last one got a bit messy >> >> Ok, Now I have started everything again from scratch (redeployed single >> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and >> now when I execute this command: >> >> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date >> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> (format of ths command was taken from: https://github.com/apach >> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te >> mplates/start-snort-stub) >> >> I get this under enrichment storm topology : >> >> >> >> >> >> I have come this far, please help me push these dummy preformatted snort >> logs into kibana dashboard. >> >> Regards. >> >> >> >
Snort logs flow issue
Hi, We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui Thanks and Regards Hema
Suricata support for snort parsers
Hello, In the info discovery phase, and I'm just curious if anyone has tried/had any problems with leveraging existing snort parsers for suricata logs. Cheers, Ian 'z0r0' Abreu
Re: Snort logs flow issue
How did you validate the logs are making it to the indexing topology? On Fri, Apr 5, 2019 at 8:12 AM Hema malini wrote: > > Hi, > > > > We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we > sent the sample snort logs copied from metron git repo to snort kafka > topic.We did the same for bro topic.Logs are getting parsed and reached > indexing topology . Elastic search indices are not getting created though > we gave elastic search template install from ambari. So manually created > the elastic search index using template available in metron repo. Though > elastic search index is present , data from indexing toplogy neither > reached elastic search nor hdfs path .There are no errors in storm toplogy > logs.We could see the sample log in Metron management ui. How we can send > the logs to alerts ui and kibana dashboard. In kibana dashboard we could > see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but > with no data. Elasticsearch health is yellow and we are able to insert data > via rest call. Any documentation on sending the smaple snort logs to metron > alerts ui will be helpful . Any configuration from metron management ui is > required to pass it to alerts –ui > > > > > > Thanks and Regards > > Hema > > > > > > >
Re: Snort logs flow issue
We verified it in Storm ui and in Storm topology logs On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic wrote: > How did you validate the logs are making it to the indexing topology? > > On Fri, Apr 5, 2019 at 8:12 AM Hema malini > wrote: > >> >> Hi, >> >> >> >> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >> sent the sample snort logs copied from metron git repo to snort kafka >> topic.We did the same for bro topic.Logs are getting parsed and reached >> indexing topology . Elastic search indices are not getting created though >> we gave elastic search template install from ambari. So manually created >> the elastic search index using template available in metron repo. Though >> elastic search index is present , data from indexing toplogy neither >> reached elastic search nor hdfs path .There are no errors in storm toplogy >> logs.We could see the sample log in Metron management ui. How we can send >> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >> with no data. Elasticsearch health is yellow and we are able to insert data >> via rest call. Any documentation on sending the smaple snort logs to metron >> alerts ui will be helpful . Any configuration from metron management ui is >> required to pass it to alerts –ui >> >> >> >> >> >> Thanks and Regards >> >> Hema >> >> >> >> >> >> >> >
Re: Snort logs flow issue
Do you get 10 records output to the CLI when you run the following? /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 On Fri, Apr 5, 2019 at 11:38 AM Hema malini wrote: > We verified it in Storm ui and in Storm topology logs > > On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> How did you validate the logs are making it to the indexing topology? >> >> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >> wrote: >> >>> >>> Hi, >>> >>> >>> >>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>> sent the sample snort logs copied from metron git repo to snort kafka >>> topic.We did the same for bro topic.Logs are getting parsed and reached >>> indexing topology . Elastic search indices are not getting created though >>> we gave elastic search template install from ambari. So manually created >>> the elastic search index using template available in metron repo. >>> Though elastic search index is present , data from indexing toplogy neither >>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>> logs.We could see the sample log in Metron management ui. How we can send >>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>> with no data. Elasticsearch health is yellow and we are able to insert data >>> via rest call. Any documentation on sending the smaple snort logs to metron >>> alerts ui will be helpful . Any configuration from metron management ui is >>> required to pass it to alerts –ui >>> >>> >>> >>> >>> >>> Thanks and Regards >>> >>> Hema >>> >>> >>> >>> >>> >>> >>> >>
Re: Snort logs flow issue
Yes I am getting messages On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic wrote: > Do you get 10 records output to the CLI when you run the following? > > /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper > $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 > > > On Fri, Apr 5, 2019 at 11:38 AM Hema malini > wrote: > >> We verified it in Storm ui and in Storm topology logs >> >> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> How did you validate the logs are making it to the indexing topology? >>> >>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>>> sent the sample snort logs copied from metron git repo to snort kafka >>>> topic.We did the same for bro topic.Logs are getting parsed and reached >>>> indexing topology . Elastic search indices are not getting created though >>>> we gave elastic search template install from ambari. So manually created >>>> the elastic search index using template available in metron repo. >>>> Though elastic search index is present , data from indexing toplogy neither >>>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>>> logs.We could see the sample log in Metron management ui. How we can send >>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>>> with no data. Elasticsearch health is yellow and we are able to insert data >>>> via rest call. Any documentation on sending the smaple snort logs to metron >>>> alerts ui will be helpful . Any configuration from metron management ui is >>>> required to pass it to alerts –ui >>>> >>>> >>>> >>>> >>>> >>>> Thanks and Regards >>>> >>>> Hema >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>
Re: Snort logs flow issue
Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> How did you validate the logs are making it to the indexing topology?
>>>>
>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>>>> sent the sample snort logs copied from metron git repo to snort kafka
>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>> indexing topology . Elastic search indices are not getting created though
>>>>> we gave elastic search template install from ambari. So manually created
>>>>> the elastic search index using template available in metron repo.
>>>>> Though elastic search index is present , data from indexing toplogy
>>>>> neither
>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>> data
>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>> metron
>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>> required to pass it to alerts –ui
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards
>>>>>
>>>>> Hema
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
Re: Snort logs flow issue
Are we missing any configuration? Initially elastic search was down. We
figured out the issue and fixed it .Now elastic search is up . We restarted
metron indexing but still those indices not created. So we created it
manually.Do we have to change any parser configuration . How logs will flow
into metron alerts dashboard and kibana dashboard..what is the required
congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>> wrote:
>>>
>>>> We verified it in Storm ui and in Storm topology logs
>>>>
>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>
>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>> the elastic search index using template available in metron repo.
>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>> neither
>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>> toplogy
>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>> data
>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>> metron
>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>> is
>>>>>> required to pass it to alerts –ui
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Hema
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
Re: Snort logs flow issue
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>> 0x1F5","parallelenricher.splitter.begin.ts":"
>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A****,0x8DF34F4B,
>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> Do you get 10 records output to the CLI when you run the following?
>>>>
>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>
>>>>
>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>> wrote:
>>>>
>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>
>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>> though
>>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>>> the elastic search index using template available in metron repo.
>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>> neither
>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>> toplogy
>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>> send
>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>> data
>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>> metron
>>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>>> is
>>>>>>> required to pass it to alerts –ui
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Hema
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
Re: Snort logs flow issue
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently
there is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
> Sorry for the typo. Can you please help with the required configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
>> Are we missing any configuration? Initially elastic search was down. We
>> figured out the issue and fixed it .Now elastic search is up . We restarted
>> metron indexing but still those indices not created. So we created it
>> manually.Do we have to change any parser configuration . How logs will flow
>> into metron alerts dashboard and kibana dashboard..what is the required
>> congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>> wrote:
>>>
>>>> Yes I am getting messages
>>>>
>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>
>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>>> we sent the sample snort logs copied from metron git repo to snort
>>>>>>>> kafka
>>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>>> though
>>>>>>>> we gave elastic search template install from ambari. So manually
>>>>>>>> created
>>>>>>>> the elastic search index using template available in metron repo.
>>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>>> neither
>>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>>> toplogy
>>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>>> send
>>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we
>>>>>>>> could
>>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created
>>>>>>>> but
>>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>>> data
>>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>>> metron
>>>>>>>> alerts ui will be helpful . Any configuration from metron management
>>>>>>>> ui is
>>>>>>>> required to pass it to alerts –ui
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Hema
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
Re: Snort logs flow issue
If you see them in the dashboard you should be able to see them in the
alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
default behavior is that the UI doesn't initiate a search at login, it's up
to the user to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
>> Sorry for the typo. Can you please help with the required configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down. We
>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>> metron indexing but still those indices not created. So we created it
>>> manually.Do we have to change any parser configuration . How logs will flow
>>> into metron alerts dashboard and kibana dashboard..what is the required
>>> congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>> wrote:
>>>
>>>> Sample messages flown in indexing topic
>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>> 999158","sig_generator":"1"}
>>>>
>>>>
>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>> wrote:
>>>>
>>>>> Yes I am getting messages
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>
>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages
>>>>>> 10
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>
>>>&g
Re: Snort logs flow issue
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log .
I clicked the search and changed the date range too. Still no records. Do
we have to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
wrote:
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>>
>>> Sorry for the typo. Can you please help with the required configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>> wrote:
>>>
>>>> Are we missing any configuration? Initially elastic search was down. We
>>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>>> metron indexing but still those indices not created. So we created it
>>>> manually.Do we have to change any parser configuration . How logs will flow
>>>> into metron alerts dashboard and kibana dashboard..what is the required
>>>> congratulation
>>>>
>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>> wrote:
>>>>
>>>>> Sample messages flown in indexing topic
>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>> 52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort
>>>>> test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>> 999158","sig_generator":"1"}
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Yes I am getting messages
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> Do you get 10 records output to the CLI when you run the following
Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings'
was not found. Do we have to create that hbase table . Where to find the
hbase tables created. I could see only two namespace in hbase - default and
hbase. No tables created in that. Do I have to run metron rest in dev
profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
> Hi Michael,
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> If you see them in the dashboard you should be able to see them in the
>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>> default behavior is that the UI doesn't initiate a search at login, it's up
>> to the user to click search.
>>
>> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>>
>>> After recreating the index, now we are able to visualize the data in
>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>
>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>> wrote:
>>>
>>>> Sorry for the typo. Can you please help with the required
>>>> configuration.
>>>>
>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>> wrote:
>>>>
>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>> restarted metron indexing but still those indices not created. So we
>>>>> created it manually.Do we have to change any parser configuration . How
>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>> the required congratulation
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Sample messages flown in indexing topic
>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>>>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>>>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>>>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>>>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>&
RE: Snort logs flow issue
Hello Hema,
Unless I’m wrong, this must be setup in MySQL, the database you use for Metron
REST.
From: Hema malini [mailto:nhemamalin...@gmail.com]
Sent: Tuesday, April 09, 2019 09:42
To: user@metron.apache.org
Subject: Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings' was
not found. Do we have to create that hbase table . Where to find the hbase
tables created. I could see only two namespace in hbase - default and hbase. No
tables created in that. Do I have to run metron rest in dev profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log . I
clicked the search and changed the date range too. Still no records. Do we have
to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
If you see them in the dashboard you should be able to see them in the alerts
UI. Any errors in either the alerts UI or REST logs? Also, the new default
behavior is that the UI doesn't initiate a search at login, it's up to the user
to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently there
is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Are we missing any configuration? Initially elastic search was down. We figured
out the issue and fixed it .Now elastic search is up . We restarted metron
indexing but still those indices not created. So we created it manually.Do we
have to change any parser configuration . How logs will flow into metron alerts
dashboard and kibana dashboard..what is the required congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sample messages flown in indexing topic
{"msg":"'snort test
alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test
alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name<http://threat.triage.rules.0.name/>":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Yes I am getting messages
On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
Do you get 10 records output to the CLI when you run the following?
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10
On Fri,
Re: Snort logs flow issue
Hi Stephanie,
Issue got resolved by creating that table in hbase.
Thanks and regards,
Hema
On Tue, Apr 9, 2019, 1:31 PM wrote:
> Hello Hema,
>
>
>
> Unless I’m wrong, this must be setup in MySQL, the database you use for
> Metron REST.
>
>
>
>
>
> *From:* Hema malini [mailto:nhemamalin...@gmail.com]
> *Sent:* Tuesday, April 09, 2019 09:42
> *To:* user@metron.apache.org
> *Subject:* Re: Snort logs flow issue
>
>
>
> Hi Michael,
>
>
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
>
>
> Thanks & Regards
>
> Hema
>
>
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
>
> Hi Michael,
>
>
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
>
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
>
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
>
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
> Sorry for the typo. Can you please help with the required configuration.
>
>
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
>
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
> Sample messages flown in indexing topic
>
> {"msg":"'snort test
> alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>
>
>
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
> Yes I
Re: Snort logs flow issue
Hi Michael,
Issue got resolved after I manually created user settings table in hbase.
There are no contents in that table though it is working. Where are the
records getting stored then for alerts UI. Where can I find the records in
HDFS.
Thanks and regards,
Hema
On Tue, Apr 9, 2019, 1:12 PM Hema malini wrote:
> Hi Michael,
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
> Thanks & Regards
> Hema
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
>
>> Hi Michael,
>>
>> Thanks for your reply. I couldn't find any errors in metron alerts UI log
>> . I clicked the search and changed the date range too. Still no records. Do
>> we have to run metron rest in dev profile?
>>
>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> If you see them in the dashboard you should be able to see them in the
>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>> to the user to click search.
>>>
>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini
>>> wrote:
>>>
>>>> After recreating the index, now we are able to visualize the data in
>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>> Currently there is no data in alerts UI. How.to configure the logs as
>>>> alerts
>>>>
>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>>> wrote:
>>>>
>>>>> Sorry for the typo. Can you please help with the required
>>>>> configuration.
>>>>>
>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>>> the required congratulation
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> Sample messages flown in indexing topic
>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,
>>>>>>> 52,53248","parallelenricher.enrich.end.
>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>>>>>> 66.121","timestamp":1484148196104,"ethdst"
Re: Snort logs flow issue
That table should have been created by default as part of the Ambari
installation of Metron via our MPack -
https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
You shouldn't have to worry about this at all as an end user, but here is
where it happens if you're curious-
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
I believe that table will be empty by default. @Ryan Merriman, do you
happen to know if we currently save user UI data there?
On Tue, Apr 9, 2019 at 2:14 AM Hema malini wrote:
> Hi Michael,
>
> Issue got resolved after I manually created user settings table in hbase.
> There are no contents in that table though it is working. Where are the
> records getting stored then for alerts UI. Where can I find the records in
> HDFS.
>
> Thanks and regards,
> Hema
>
> On Tue, Apr 9, 2019, 1:12 PM Hema malini wrote:
>
>> Hi Michael,
>>
>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>> was not found. Do we have to create that hbase table . Where to find the
>> hbase tables created. I could see only two namespace in hbase - default and
>> hbase. No tables created in that. Do I have to run metron rest in dev
>> profile.
>>
>> Thanks & Regards
>> Hema
>>
>> On Tue, Apr 9, 2019, 12:44 PM Hema malini
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>> log . I clicked the search and changed the date range too. Still no
>>> records. Do we have to run metron rest in dev profile?
>>>
>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> If you see them in the dashboard you should be able to see them in the
>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>>> to the user to click search.
>>>>
>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini
>>>> wrote:
>>>>
>>>>> After recreating the index, now we are able to visualize the data in
>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>> Currently there is no data in alerts UI. How.to configure the logs as
>>>>> alerts
>>>>>
>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Sorry for the typo. Can you please help with the required
>>>>>> configuration.
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what
>>>>>>> is
>>>>>>> the required congratulation
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sample messages flown in indexing topic
>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>>> rules.0.score":&quo
Re: Snort logs flow issue
Thanks a lot Michael for your help. Will explore further.
On Wed, Apr 10, 2019, 3:37 AM Michael Miklavcic
wrote:
> That table should have been created by default as part of the Ambari
> installation of Metron via our MPack -
> https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
> You shouldn't have to worry about this at all as an end user, but here is
> where it happens if you're curious-
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
>
> I believe that table will be empty by default. @Ryan Merriman, do you
> happen to know if we currently save user UI data there?
>
> On Tue, Apr 9, 2019 at 2:14 AM Hema malini
> wrote:
>
>> Hi Michael,
>>
>> Issue got resolved after I manually created user settings table in hbase.
>> There are no contents in that table though it is working. Where are the
>> records getting stored then for alerts UI. Where can I find the records in
>> HDFS.
>>
>> Thanks and regards,
>> Hema
>>
>> On Tue, Apr 9, 2019, 1:12 PM Hema malini wrote:
>>
>>> Hi Michael,
>>>
>>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>>> was not found. Do we have to create that hbase table . Where to find the
>>> hbase tables created. I could see only two namespace in hbase - default and
>>> hbase. No tables created in that. Do I have to run metron rest in dev
>>> profile.
>>>
>>> Thanks & Regards
>>> Hema
>>>
>>> On Tue, Apr 9, 2019, 12:44 PM Hema malini
>>> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>>> log . I clicked the search and changed the date range too. Still no
>>>> records. Do we have to run metron rest in dev profile?
>>>>
>>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> If you see them in the dashboard you should be able to see them in the
>>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>>> default behavior is that the UI doesn't initiate a search at login, it's
>>>>> up
>>>>> to the user to click search.
>>>>>
>>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>> After recreating the index, now we are able to visualize the data in
>>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>>> Currently there is no data in alerts UI. How.to configure the logs as
>>>>>> alerts
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry for the typo. Can you please help with the required
>>>>>>> configuration.
>>>>>>>
>>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Are we missing any configuration? Initially elastic search was
>>>>>>>> down. We figured out the issue and fixed it .Now elastic search is up
>>>>>>>> . We
>>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what
>>>>>>>> is
>>>>>>>> the required congratulation
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sample messages flown in indexing topic
>>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>>> tcpseq":"0x8DF34F4B","threat.
Parser Error while Snort IDS usage
Hi, I am getting following errors when I am using snort in IDS mode. java.lang.IllegalStateException: Unable to parse message: 06/28-02:06:18.667820 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1 at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.time.format.DateTimeParseException: Text '06/28-02:06:18.667820' could not be parsed at index 5 at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at org.apache.metron.parsers.snort.BasicSnortParser.toEpoch(BasicSnortParser.java:194) at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:165) ... 12 more Following is the data i am getting in alerts.csv 06/28-02:00:39.145636 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,12590,1 06/28-02:00:39.145690 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,40061,32,32768,0,0,12590,1 06/28-02:00:49.949974 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,13210,1 06/28-02:00:49.950011 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41071,32,32768,0,0,13210,1 06/28-02:01:00.534199 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,55879,1 06/28-02:01:00.534224 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,43938,32,32768,0,0,55879,1 06/28-02:01:02.185767 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,4364,1 06/28-02:01:02.185812 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41648,32,32768,0,0,4364,1 06/28-02:01:03.946563 ,1,384,5,"ICMP PING",ICMP,37.187.231.251,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x5E,,61,0,0,80,81920,8,0,20112,56 06/28-02:01:03.946596 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,37.187.231.251,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x5E,,64,0,38270,80,81920,0,0,20112,56 06/28-02:01:05.015592 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,55637,1 06/28-02:01:05.015640 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.186.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,29998,32,32768,0,0,55637,1 06/28-02:01:08.820637 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,12,8,1,32,32768,8,0,10717,1 06/28-02:01:08.820684 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,167.114.37.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,51338,32,32768,0,0,10717,1 06/28-02:01:16.702204 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,12,8,1,32,32768,8,0,23094,1 06/28-02:01:16.702256 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,167.114.37.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,51904,32,32768,0,0,23094,1 06/28-02:01:18.322369 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,26763,1 06/28-02:01:18.322409 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,46352,32,32768,0,0,26763,1 06/28-02:01:20.123553 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:8
Re: Parser Error while Snort IDS usage
Admin, if you will go through this post and allow it to be responded it will be great. On Wed, Jun 27, 2018 at 11:16 PM, Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > Hi, > > I am getting following errors when I am using snort in IDS mode. > > java.lang.IllegalStateException: Unable to parse message: > 06/28-02:06:18.667820 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1 > at > org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180) > at org.apache.metron.parsers.interfaces.MessageParser. > parseOptional(MessageParser.java:45) at org.apache.metron.parsers. > bolt.ParserBolt.execute(ParserBolt.java:177) at org.apache.storm.daemon. > executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at > org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) > at > org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > at > org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) > at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at > clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) > Caused by: java.time.format.DateTimeParseException: Text > '06/28-02:06:18.667820' could not be parsed at index 5 at java.time.format. > DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at > java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at > java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at > org.apache.metron.parsers.snort.BasicSnortParser. > toEpoch(BasicSnortParser.java:194) at org.apache.metron.parsers. > snort.BasicSnortParser.parse(BasicSnortParser.java:165) ... 12 more > > > Following is the data i am getting in alerts.csv > > 06/28-02:00:39.145636 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 11,8,1,32,32768,8,0,12590,1 > 06/28-02:00:39.145690 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,40061,32,32768,0,0,12590,1 > 06/28-02:00:49.949974 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 10,8,1,32,32768,8,0,13210,1 > 06/28-02:00:49.950011 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,41071,32,32768,0,0,13210,1 > 06/28-02:01:00.534199 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 11,8,1,32,32768,8,0,55879,1 > 06/28-02:01:00.534224 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,43938,32,32768,0,0,55879,1 > 06/28-02:01:02.185767 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 10,8,1,32,32768,8,0,4364,1 > 06/28-02:01:02.185812 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,41648,32,32768,0,0,4364,1 > 06/28-02:01:03.946563 ,1,384,5,"ICMP PING",ICMP,37.187.231.251,, > 158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x5E,, > 61,0,0,80,81920,8,0,20112,56 > 06/28-02:01:03.946596 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 37.187.231.251,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x5E,, > 64,0,38270,80,81920,0,0,20112,56 > 06/28-02:01:05.015592 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 10,8,1,32,32768,8,0,55637,1 > 06/28-02:01:05.015640 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 92.222.186.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,29998,32,32768,0,0,55637,1 > 06/28-02:01:08.820637 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 12,8,1,32,32768,8,0,10717,1 > 06/28-02:01:08.820684 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,, > 167.114.37.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E, > ,64,8,51338,32,32768,0,0,10717,1 > 06/28-02:01:16.702204 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158. > 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,, > 12,
Re: Parser Error while Snort IDS usage
The snort parser by default supports dates in the following format:
Your dates are missing the ‘yy’. If I add that, your failing message
parses:
/**
06/28/18-02:06:18.667820 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
*/
@Multiline
public static String userMessage;
@Test
public void testUserIssue() {
BasicSnortParser parser = new BasicSnortParser();
parser.configure(new HashMap<>());
List msgs = parser.parse(userMessage.getBytes());
Assert.assertTrue(msgs != null && msgs.isEmpty() == false);
}
You need to configure the snort processor’s dateFormat
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html
shows
a sample. See ‘Sample configuration for a sensor’.
On June 28, 2018 at 02:16:42, Farrukh Naveed Anjum (anjum.farr...@gmail.com)
wrote:
Hi,
I am getting following errors when I am using snort in IDS mode.
java.lang.IllegalStateException: Unable to parse message:
06/28-02:06:18.667820 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
at
org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177)
at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: java.time.format.DateTimeParseException: Text
'06/28-02:06:18.667820' could not be parsed at index 5 at
java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at
java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at
org.apache.metron.parsers.snort.BasicSnortParser.toEpoch(BasicSnortParser.java:194)
at
org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:165)
... 12 more
Following is the data i am getting in alerts.csv
06/28-02:00:39.145636 ,1,384,5,"ICMP
PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,12590,1
06/28-02:00:39.145690 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,40061,32,32768,0,0,12590,1
06/28-02:00:49.949974 ,1,384,5,"ICMP
PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,13210,1
06/28-02:00:49.950011 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41071,32,32768,0,0,13210,1
06/28-02:01:00.534199 ,1,384,5,"ICMP
PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,55879,1
06/28-02:01:00.534224 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,43938,32,32768,0,0,55879,1
06/28-02:01:02.185767 ,1,384,5,"ICMP
PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,4364,1
06/28-02:01:02.185812 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41648,32,32768,0,0,4364,1
06/28-02:01:03.946563 ,1,384,5,"ICMP
PING",ICMP,37.187.231.251,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x5E,,61,0,0,80,81920,8,0,20112,56
06/28-02:01:03.946596 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,37.187.231.251,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x5E,,64,0,38270,80,81920,0,0,20112,56
06/28-02:01:05.015592 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,55637,1
06/28-02:01:05.015640 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.186.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,29998,32,32768,0,0,55637,1
06/28-02:01:08.820637 ,1,384,5,"ICMP
PING",ICMP,167.114.37.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,12,8,1,32,32768,8,0,10717,1
06/28-02:01:08.820684 ,1,408,5,"ICMP Echo
R
Re: Parser Error while Snort IDS usage
Forgot to put the default format in. It is : private static String
defaultDateFormat
= "MM/dd/yy-HH:mm:ss.SS";
On June 28, 2018 at 10:06:08, Otto Fowler (ottobackwa...@gmail.com) wrote:
The snort parser by default supports dates in the following format:
Your dates are missing the ‘yy’. If I add that, your failing message
parses:
/**
06/28/18-02:06:18.667820 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
*/
@Multiline
public static String userMessage;
@Test
public void testUserIssue() {
BasicSnortParser parser = new BasicSnortParser();
parser.configure(new HashMap<>());
List msgs = parser.parse(userMessage.getBytes());
Assert.assertTrue(msgs != null && msgs.isEmpty() == false);
}
You need to configure the snort processor’s dateFormat
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html
shows
a sample. See ‘Sample configuration for a sensor’.
On June 28, 2018 at 02:16:42, Farrukh Naveed Anjum (anjum.farr...@gmail.com)
wrote:
Hi,
I am getting following errors when I am using snort in IDS mode.
java.lang.IllegalStateException: Unable to parse message:
06/28-02:06:18.667820 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
at
org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180)
at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177)
at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: java.time.format.DateTimeParseException: Text
'06/28-02:06:18.667820' could not be parsed at index 5 at
java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at
java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at
org.apache.metron.parsers.snort.BasicSnortParser.toEpoch(BasicSnortParser.java:194)
at
org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:165)
... 12 more
Following is the data i am getting in alerts.csv
06/28-02:00:39.145636 ,1,384,5,"ICMP
PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,12590,1
06/28-02:00:39.145690 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,40061,32,32768,0,0,12590,1
06/28-02:00:49.949974 ,1,384,5,"ICMP
PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,13210,1
06/28-02:00:49.950011 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41071,32,32768,0,0,13210,1
06/28-02:01:00.534199 ,1,384,5,"ICMP
PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,55879,1
06/28-02:01:00.534224 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,43938,32,32768,0,0,55879,1
06/28-02:01:02.185767 ,1,384,5,"ICMP
PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,4364,1
06/28-02:01:02.185812 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41648,32,32768,0,0,4364,1
06/28-02:01:03.946563 ,1,384,5,"ICMP
PING",ICMP,37.187.231.251,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x5E,,61,0,0,80,81920,8,0,20112,56
06/28-02:01:03.946596 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,37.187.231.251,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x5E,,64,0,38270,80,81920,0,0,20112,56
06/28-02:01:05.015592 ,1,384,5,"ICMP
PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,55637,1
06/28-02:01:05.015640 ,1,408,5,"ICMP Echo
Reply",ICMP,158.69.118.104,,92.222.186.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,29998,32,32768,0,0,55637,1
06/28-02
Re: Parser Error while Snort IDS usage
Thanks, I resolved it by allowing the config show_year in snort.conf file
and the snort parser date format configurations.
On Thu, Jun 28, 2018 at 7:06 PM, Otto Fowler
wrote:
> Forgot to put the default format in. It is : private static String
> defaultDateFormat
> = "MM/dd/yy-HH:mm:ss.SS";
>
>
> On June 28, 2018 at 10:06:08, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> The snort parser by default supports dates in the following format:
>
> Your dates are missing the ‘yy’. If I add that, your failing message
> parses:
>
> /**
> 06/28/18-02:06:18.667820 ,1,384,5,"ICMP
> PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
> */
> @Multiline
> public static String userMessage;
>
>
> @Test
> public void testUserIssue() {
> BasicSnortParser parser = new BasicSnortParser();
> parser.configure(new HashMap<>());
> List msgs = parser.parse(userMessage.getBytes());
> Assert.assertTrue(msgs != null && msgs.isEmpty() == false);
> }
>
>
> You need to configure the snort processor’s dateFormat
>
> https://metron.apache.org/current-book/metron-platform/
> metron-parsers/index.html shows a sample. See ‘Sample configuration for
> a sensor’.
>
>
> On June 28, 2018 at 02:16:42, Farrukh Naveed Anjum (
> anjum.farr...@gmail.com) wrote:
>
> Hi,
>
> I am getting following errors when I am using snort in IDS mode.
>
> java.lang.IllegalStateException: Unable to parse message:
> 06/28-02:06:18.667820 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
> at
> org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180)
> at org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:177) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.time.format.DateTimeParseException: Text
> '06/28-02:06:18.667820' could not be parsed at index 5 at java.time.format.
> DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at
> java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at
> java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at
> org.apache.metron.parsers.snort.BasicSnortParser.
> toEpoch(BasicSnortParser.java:194) at org.apache.metron.parsers.
> snort.BasicSnortParser.parse(BasicSnortParser.java:165) ... 12 more
>
>
> Following is the data i am getting in alerts.csv
>
> 06/28-02:00:39.145636 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 11,8,1,32,32768,8,0,12590,1
> 06/28-02:00:39.145690 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,40061,32,32768,0,0,12590,1
> 06/28-02:00:49.949974 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 10,8,1,32,32768,8,0,13210,1
> 06/28-02:00:49.950011 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,41071,32,32768,0,0,13210,1
> 06/28-02:01:00.534199 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 11,8,1,32,32768,8,0,55879,1
> 06/28-02:01:00.534224 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,43938,32,32768,0,0,55879,1
> 06/28-02:01:02.185767 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 10,8,1,32,32768,8,0,4364,1
> 06/28-02:01:02.185812 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,41648,
