Snort Logs
Hello everyone, I have run snort independently on vagrant ssh and dumped the logs in tcpdump format. Now I want to bring them to metron to play with them a bit. Some of you already replied me with some solutions but thats lost in the inbox somewhere and engulfed by the elasticsearhc issue that I had. Please give me an easy to understand this solution for this problem. Regards.
Re: Snort Logs
If you have text snort logs you can use Apache nifi or the Kafka producer script as described in step 4 here[1] to push them to Metron's snort topic. You may also want to look at this [2]. 1: https://kafka.apache.org/quickstart 2: https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script Jon On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir wrote: > Hello everyone, > > I have run snort independently on vagrant ssh and dumped the logs in > tcpdump format. Now I want to bring them to metron to play with them a bit. > Some of you already replied me with some solutions but thats lost in the > inbox somewhere and engulfed by the elasticsearhc issue that I had. Please > give me an easy to understand this solution for this problem. > > Regards. > -- Jon
Re: Snort Logs
snort logs are in tcp dump format. I may have to convert them. bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test How to give file name or path in this command? On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com wrote: > If you have text snort logs you can use Apache nifi or the Kafka producer > script as described in step 4 here[1] to push them to Metron's snort > topic. You may also want to look at this [2]. > > 1: https://kafka.apache.org/quickstart > 2: https://stackoverflow.com/questions/38701179/kafka- > console-producer-and-bash-script > > Jon > > On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir > wrote: > >> Hello everyone, >> >> I have run snort independently on vagrant ssh and dumped the logs in >> tcpdump format. Now I want to bring them to metron to play with them a bit. >> Some of you already replied me with some solutions but thats lost in the >> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >> give me an easy to understand this solution for this problem. >> >> Regards. >> > -- > > Jon >
Re: Snort Logs
On the 25th I said: It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from memory) on node1, assuming you are running full dev. Jon Jon On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir wrote: > snort logs are in tcp dump format. I may have to convert them. > > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > > How to give file name or path in this command? > > On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com > wrote: > >> If you have text snort logs you can use Apache nifi or the Kafka producer >> script as described in step 4 here[1] to push them to Metron's snort >> topic. You may also want to look at this [2]. >> >> 1: https://kafka.apache.org/quickstart >> 2: >> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >> >> Jon >> >> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >> wrote: >> >>> Hello everyone, >>> >>> I have run snort independently on vagrant ssh and dumped the logs in >>> tcpdump format. Now I want to bring them to metron to play with them a bit. >>> Some of you already replied me with some solutions but thats lost in the >>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>> give me an easy to understand this solution for this problem. >>> >>> Regards. >>> >> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
I have found the kafka-console-producer.sh but I need to know how to make it read snort.log (tcp dump format) file. May be I am missing something in the plain sight but it would be awsome if you tell me that. Regards. On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com wrote: > On the 25th I said: > > It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from > memory) on node1, assuming you are running full dev. > > Jon > > > Jon > > On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir > wrote: > >> snort logs are in tcp dump format. I may have to convert them. >> >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >> >> How to give file name or path in this command? >> >> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >> wrote: >> >>> If you have text snort logs you can use Apache nifi or the Kafka >>> producer script as described in step 4 here[1] to push them to Metron's >>> snort topic. You may also want to look at this [2]. >>> >>> 1: https://kafka.apache.org/quickstart >>> 2: https://stackoverflow.com/questions/38701179/kafka- >>> console-producer-and-bash-script >>> >>> Jon >>> >>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>> wrote: >>> >>>> Hello everyone, >>>> >>>> I have run snort independently on vagrant ssh and dumped the logs in >>>> tcpdump format. Now I want to bring them to metron to play with them a bit. >>>> Some of you already replied me with some solutions but thats lost in the >>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>> give me an easy to understand this solution for this problem. >>>> >>>> Regards. >>>> >>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
You need text logs. Here's an example of some properly formatted logs - https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out Jon On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir wrote: > I have found the kafka-console-producer.sh but I need to know how to make > it read snort.log (tcp dump format) file. May be I am missing something in > the plain sight but it would be awsome if you tell me that. > > Regards. > > On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com > wrote: > >> On the 25th I said: >> >> It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from >> memory) on node1, assuming you are running full dev. >> >> Jon >> >> >> Jon >> >> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >> wrote: >> >>> snort logs are in tcp dump format. I may have to convert them. >>> >>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>> >>> How to give file name or path in this command? >>> >>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>> wrote: >>> >>>> If you have text snort logs you can use Apache nifi or the Kafka >>>> producer script as described in step 4 here[1] to push them to Metron's >>>> snort topic. You may also want to look at this [2]. >>>> >>>> 1: https://kafka.apache.org/quickstart >>>> 2: >>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>> bit. >>>>> Some of you already replied me with some solutions but thats lost in the >>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>>> give me an easy to understand this solution for this problem. >>>>> >>>>> Regards. >>>>> >>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
Yes, I have converted them to text but those logs are simply captured packet headers over the local network. Now I just push them via that kafka producer command under topic name of snort and they will be visible in metron? On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com wrote: > You need text logs. Here's an example of some properly formatted logs - > https://raw.githubusercontent.com/apache/metron/master/metron- > deployment/roles/sensor-stubs/files/snort.out > > Jon > > On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir > wrote: > >> I have found the kafka-console-producer.sh but I need to know how to >> make it read snort.log (tcp dump format) file. May be I am missing >> something in the plain sight but it would be awsome if you tell me that. >> >> Regards. >> >> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >> wrote: >> >>> On the 25th I said: >>> >>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>> (from memory) on node1, assuming you are running full dev. >>> >>> Jon >>> >>> >>> Jon >>> >>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >>> wrote: >>> >>>> snort logs are in tcp dump format. I may have to convert them. >>>> >>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>> >>>> How to give file name or path in this command? >>>> >>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>> snort topic. You may also want to look at this [2]. >>>>> >>>>> 1: https://kafka.apache.org/quickstart >>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>> console-producer-and-bash-script >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>> bit. >>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>> Please >>>>>> give me an easy to understand this solution for this problem. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
They need to meet the format of the logs I sent earlier. Look into the snort output options - may require you rerun snort, depending on your situation Jon On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir wrote: > Yes, I have converted them to text but those logs are simply captured > packet headers over the local network. Now I just push them via that kafka > producer command under topic name of snort and they will be visible in > metron? > > On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com > wrote: > >> You need text logs. Here's an example of some properly formatted logs - >> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out >> >> Jon >> >> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir >> wrote: >> >>> I have found the kafka-console-producer.sh but I need to know how to >>> make it read snort.log (tcp dump format) file. May be I am missing >>> something in the plain sight but it would be awsome if you tell me that. >>> >>> Regards. >>> >>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >>> wrote: >>> >>>> On the 25th I said: >>>> >>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>> (from memory) on node1, assuming you are running full dev. >>>> >>>> Jon >>>> >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >>>> wrote: >>>> >>>>> snort logs are in tcp dump format. I may have to convert them. >>>>> >>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>>> >>>>> How to give file name or path in this command? >>>>> >>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>>> wrote: >>>>> >>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>> snort topic. You may also want to look at this [2]. >>>>>> >>>>>> 1: https://kafka.apache.org/quickstart >>>>>> 2: >>>>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>>>> >>>>>> Jon >>>>>> >>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>>> wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>>> bit. >>>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>> Please >>>>>>> give me an easy to understand this solution for this problem. >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>> -- >>>>>> >>>>>> Jon >>>>>> >>>>> >>>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
I sent a random message to that kafka topic and got this [image: Inline image 1] I guess this is because I am not following the format of message I should send? Like those snort logs you showed. On Mon, Oct 30, 2017 at 5:24 PM, zeo...@gmail.com wrote: > They need to meet the format of the logs I sent earlier. Look into the > snort output options - may require you rerun snort, depending on your > situation > > Jon > > On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir > wrote: > >> Yes, I have converted them to text but those logs are simply captured >> packet headers over the local network. Now I just push them via that kafka >> producer command under topic name of snort and they will be visible in >> metron? >> >> On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com >> wrote: >> >>> You need text logs. Here's an example of some properly formatted logs - >>> https://raw.githubusercontent.com/apache/metron/master/metron- >>> deployment/roles/sensor-stubs/files/snort.out >>> >>> Jon >>> >>> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir >>> wrote: >>> >>>> I have found the kafka-console-producer.sh but I need to know how to >>>> make it read snort.log (tcp dump format) file. May be I am missing >>>> something in the plain sight but it would be awsome if you tell me that. >>>> >>>> Regards. >>>> >>>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> On the 25th I said: >>>>> >>>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>>> (from memory) on node1, assuming you are running full dev. >>>>> >>>>> Jon >>>>> >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> snort logs are in tcp dump format. I may have to convert them. >>>>>> >>>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic >>>>>> test >>>>>> >>>>>> How to give file name or path in this command? >>>>>> >>>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>>> snort topic. You may also want to look at this [2]. >>>>>>> >>>>>>> 1: https://kafka.apache.org/quickstart >>>>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>>>> console-producer-and-bash-script >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>>>> wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> I have run snort independently on vagrant ssh and dumped the logs >>>>>>>> in tcpdump format. Now I want to bring them to metron to play with >>>>>>>> them a >>>>>>>> bit. Some of you already replied me with some solutions but thats lost >>>>>>>> in >>>>>>>> the inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>>> Please give me an easy to understand this solution for this problem. >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jon >>>>>>> >>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
And how do I install elasticsearch head on the vagrant VM?
Re: Snort Logs
You can install it into the chrome web browser from the play store. On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) wrote: And how do I install elasticsearch head on the vagrant VM?
Fwd: Snort Logs
-- Forwarded message -- From: Syed Hammad Tahir Date: Fri, Nov 3, 2017 at 5:07 PM Subject: Re: Snort Logs To: Otto Fowler NVM, I have installed the elastic search head. Now where do I go in this to find out why I cant see the snort logs in kibana dashboard, pushed to snort topic via kafka producer? [image: Inline image 1] On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler wrote: > You can install it into the chrome web browser from the play store. > > > > On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) > wrote: > > And how do I install elasticsearch head on the vagrant VM? > >
Re: Snort Logs
It looks like your ES cluster has a health of Red, so there's your problem. I would go look in /var/log/elasticsearch/ at some logs. Jon On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir wrote: > > -- Forwarded message -- > From: Syed Hammad Tahir > Date: Fri, Nov 3, 2017 at 5:07 PM > Subject: Re: Snort Logs > To: Otto Fowler > > > NVM, I have installed the elastic search head. Now where do I go in this > to find out why I cant see the snort logs in kibana dashboard, pushed to > snort topic via kafka producer? > > [image: Inline image 1] > > On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler > wrote: > >> You can install it into the chrome web browser from the play store. >> >> >> >> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >> wrote: >> >> And how do I install elasticsearch head on the vagrant VM? >> >> > -- Jon
Re: Snort Logs
hi, I am back at work. lets see if i can find something in logs On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com wrote: > It looks like your ES cluster has a health of Red, so there's your > problem. I would go look in /var/log/elasticsearch/ at some logs. > > Jon > > On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir > wrote: > >> >> -- Forwarded message -- >> From: Syed Hammad Tahir >> Date: Fri, Nov 3, 2017 at 5:07 PM >> Subject: Re: Snort Logs >> To: Otto Fowler >> >> >> NVM, I have installed the elastic search head. Now where do I go in this >> to find out why I cant see the snort logs in kibana dashboard, pushed to >> snort topic via kafka producer? >> >> [image: Inline image 1] >> >> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler >> wrote: >> >>> You can install it into the chrome web browser from the play store. >>> >>> >>> >>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >>> wrote: >>> >>> And how do I install elasticsearch head on the vagrant VM? >>> >>> >> -- > > Jon >
Re: Snort Logs
gt;>>>>>>>>> >>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the logs >>>>>>>>>>>>> I am trying to push >>>>>>>>>>>>> >>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>> >>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>> file? Maybe I could see its formatting and try following the same >>>>>>>>>>>>> thing. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>> >>>>>>>>>>>>>> After running this command, I copy paste a few lines from >>>>>>>>>>>>>> here: https://raw.githubusercontent.com/apache/metron/master/ >>>>>>>>>>>>>> metron-deployment/roles/sensor-stubs/files/snort.out >>>>>>>>>>>>>> >>>>>>>>>>>>>> like this >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not getting any error here. I can also see these lines >>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This was the mechanism I am using to push the logs. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler < >>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> What I mean is this: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort >>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think >>>>>>>>>>>>>>> of as live. >>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see >>>>>>>>>>>>>>> this error. >>>>>>>>>>>>>>> So what I am asking is if you see the same errors for things >>>>>>>>>>>>>>> that are automatically pushed to kafka as you do when you >>>>>>>>>>>>>>> manual push them. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On November 7, 2017 at 08:51:41, Syed Hammad Tahir ( >>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "Yes, If the messages cannot be parsed then that would be a >>>>>>>>>>>>>>> problem. If you see this error with your ‘live’ messages as >>>>>>>>>>>>>>> well then that >>>>>>>>>>>>>>> could be it. >>>>>>>>>>>>>>> I wonder if the issue is with the date format?" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If by 'live' messages you mean the time I push them into >>>>>>>>>>>>>>> kafka topic then no, I dont see
Re: Snort Logs
t;>>>>>> >>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler ( >>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column >>>>>>>>>>>>>> is for a piece of data that is failing >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir ( >>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the logs >>>>>>>>>>>>>> I am trying to push >>>>>>>>>>>>>> >>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the >>>>>>>>>>>>>> same thing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After running this command, I copy paste a few lines from >>>>>>>>>>>>>>> here: https://raw.githubusercontent. >>>>>>>>>>>>>>> com/apache/metron/master/metron-deployment/roles/sensor- >>>>>>>>>>>>>>> stubs/files/snort.out >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> like this >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines >>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This was the mechanism I am using to push the logs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler < >>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What I mean is this: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort >>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think >>>>>>>>>>>>>>>> of as live. >>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see >>>>>>>>>>>>>>>> this error. >>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for >>>>>>>>>>>>>>>> things that are automatically pushed to kafka as you do when >>>>>>>>>>>>>>>> you manual >&g
Re: Snort Logs
>>>>>> private transient DateTimeFormatter dateTimeFormatter; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If your records are in dd/MM/yy- format, then you may see >>>>>>>>>>>>>>> this error I believe. >>>>>>>>>>>>>>> Can you verify the timestamp field’s format? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If this is the case, then you will need to modify the >>>>>>>>>>>>>>> default log timestamp format for snort in the short term. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler ( >>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column >>>>>>>>>>>>>>> is for a piece of data that is failing >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir ( >>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the >>>>>>>>>>>>>>> logs I am trying to push >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the >>>>>>>>>>>>>>> same thing. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from >>>>>>>>>>>>>>>> here: https://raw.githubusercontent. >>>>>>>>>>>>>>>> com/apache/metron/master/metro >>>>>>>>>>>>>>>> n-deployment/roles/sensor-stubs/files/snort.out >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> like this >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines >>>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This was the mechanism I am using to push the logs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler < >>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>&
Re: Snort Logs
problems >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 8, 2017, 06:59 Otto Fowler < >>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The snort parser is coded to support dates in this format: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> private static String defaultDateFormat = >>>>>>>>>>>>>>>> "MM/dd/yy-HH:mm:ss.SS"; >>>>>>>>>>>>>>>> private transient DateTimeFormatter dateTimeFormatter; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If your records are in dd/MM/yy- format, then you may see >>>>>>>>>>>>>>>> this error I believe. >>>>>>>>>>>>>>>> Can you verify the timestamp field’s format? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If this is the case, then you will need to modify the >>>>>>>>>>>>>>>> default log timestamp format for snort in the short term. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler ( >>>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column >>>>>>>>>>>>>>>> is for a piece of data that is failing >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir ( >>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the >>>>>>>>>>>>>>>> logs I am trying to push >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the >>>>>>>>>>>>>>>> same thing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from >>>>>>>>>>>>>>>>> here: >>>>>>>>>>>>>>>>> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> like this >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [image: Inline i
Re: Snort Logs
>>>>>>>>>> 27:00:00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC, >>>>>>>>>>>>>>> 0x56900BB6,,0x1000,64,10,23403,76,77824 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 5:30 PM, zeo...@gmail.com < >>>>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would download the entire snort.out file and run cat >>>>>>>>>>>>>>>> snort.out | kafka-console-producer.sh ... to make sure there >>>>>>>>>>>>>>>> are no copy >>>>>>>>>>>>>>>> paste problems >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Nov 8, 2017, 06:59 Otto Fowler < >>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The snort parser is coded to support dates in this format: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> private static String defaultDateFormat = >>>>>>>>>>>>>>>>> "MM/dd/yy-HH:mm:ss.SS"; >>>>>>>>>>>>>>>>> private transient DateTimeFormatter dateTimeFormatter; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If your records are in dd/MM/yy- format, then you may see >>>>>>>>>>>>>>>>> this error I believe. >>>>>>>>>>>>>>>>> Can you verify the timestamp field’s format? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If this is the case, then you will need to modify the >>>>>>>>>>>>>>>>> default log timestamp format for snort in the short term. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler ( >>>>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ >>>>>>>>>>>>>>>>> field/column is for a piece of data that is failing >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir ( >>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the >>>>>>>>>>>>>>>>> logs I am trying to push >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the >>>>>>>>>>>>>>>>> same thing. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> thats how I am pus
Re: Snort Logs
t;>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>>>>>>>>>>>>>>>>> Also, are you saying that you are no longer seeing >>>>>>>>>>>>>>>>>>>>>>>>> the parser topology error? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella ( >>>>>>>>>>>>>>>>>>>>>>>>> ceste...@gmail.com) wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> If you click on the port (6704) there in those >>>>>>>>>>>>>>>>>>>>>>>>> errors, what's the full stacktrace (that starts with >>>>>>>>>>>>>>>>>>>>>>>>> the suggestion you >>>>>>>>>>>>>>>>>>>>>>>>> file a JIRA)? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> What this means is that an exception is bleeding >>>>>>>>>>>>>>>>>>>>>>>>> from the individual writer into the writer component >>>>>>>>>>>>>>>>>>>>>>>>> (It should be handled >>>>>>>>>>>>>>>>>>>>>>>>> in the writer itself). The fact that it's happening >>>>>>>>>>>>>>>>>>>>>>>>> for both HDFS and ES >>>>>>>>>>>>>>>>>>>>>>>>> is telling as well and I'm very interested in the >>>>>>>>>>>>>>>>>>>>>>>>> full stacktrace there >>>>>>>>>>>>>>>>>>>>>>>>> because it'll have the wrapped exception from the >>>>>>>>>>>>>>>>>>>>>>>>> individual writer >>>>>>>>>>>>>>>>>>>>>>>>> included. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 11:24 AM, Syed Hammad Tahir >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> OK I did what Zeolla said, cat snort.out | kafka >>>>>>>>>>>>>>>>>>>>>>>>>> producer and now the error at storm parser >>>>>>>>>>>>>>>>>>>>>>>>>> topology is gone but I am >>>>>>>>>>>>>>>>>>>>>>>>>&g
Re: Snort Logs
t;>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> from indexingbolt in indexing topology >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>>>>>>>>>>>>>>>>> Also, are you saying that you are no longer seeing >>>>>>>>>>>>>>>>>>>>>>>>> the parser topology error? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella ( >>>>>>>>>>>>>>>>>>>>>>>>> ceste...@gmail.com) wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> If you click on the port (6704) there in those >>>>>>>>>>>>>>>>>>>>>>>>> errors, what's the full stacktrace (that starts with >>>>>>>>>>>>>>>>>>>>>>>>> the suggestion you >>>>>>>>>>>>>>>>>>>>>>>>> file a JIRA)? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> What this means is that an exception is bleeding >>>>>>>>>>>>>>>>>>>>>>>>> from the individual writer into the writer component >>>>>>>>>>>>>>>>>>>>>>>>> (It should be handled >>>>>>>>>>>>>>>>>>>>>>>>> in the writer itself). The fact that it's happening >>>>>>>>>>>>>>>>>>>>>>>>> for both HDFS and ES >>>>>>>>>>>>>>>>>>>>>>>>> is telling as well and I'm very interested in the >>>>>>>>>>>>>>>>>>>>>>>>> full stacktrace there >>>>>>>>>>>>>>>>>>>>>>>>> because it'll have the wrapped exception from the >>>>>>>>>>>>>>>>>>>>>>>>> individual writer >>>>>>>>>>>>>>>>>>>>>>>>> included. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 11:24 AM, Syed Hammad Tahir >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> OK I did what Zeolla said, cat snort.out | kafka >>>>>>>>>>>>>>>>>>>>>>>>>> producer and now the error at storm parser >>>>>>>>>>>>>>>>>>>>>>>>>> topology is gone but I am >>>>&
Re: Snort Logs
gt;>>>>>>>>>>>>>>>>>> the enriched map to the indexing topic. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> No I am no longer seeing the parsing topology >>>>>>>>>>>>>>>>>>>>>>>>> error, here is the full stack trace >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> from indexingbolt in indexing topology >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>>>>>>>>>>>>>>>>>> Also, are you saying that you are no longer >>>>>>>>>>>>>>>>>>>>>>>>>> seeing the parser topology error? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella ( >>>>>>>>>>>>>>>>>>>>>>>>>> ceste...@gmail.com) wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> If you click on the port (6704) there in those >>>>>>>>>>>>>>>>>>>>>>>>>> errors, what's the full stacktrace (that starts with >>>>>>>>>>>>>>>>>>>>>>>>>> the suggestion you >>>>>>>>>>>>>>>>>>>>>>>>>> file a JIRA)? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> What this means is that an exception is bleeding >>>>>>>>>>>>>>>>>>>>>>>>>> from the individual writer into the writer component >>>>>>>>>>>>>>>>>>>>>>>>>> (It should be handled >>>>>>>>>>>>>>>>>>>>>>>>>> in the writer itself). The fact that it's happening >>>>>>>>>>>>>>&g
Snort logs flow issue
Hi, We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui Thanks and Regards Hema
Re: Snort logs flow issue
How did you validate the logs are making it to the indexing topology? On Fri, Apr 5, 2019 at 8:12 AM Hema malini wrote: > > Hi, > > > > We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we > sent the sample snort logs copied from metron git repo to snort kafka > topic.We did the same for bro topic.Logs are getting parsed and reached > indexing topology . Elastic search indices are not getting created though > we gave elastic search template install from ambari. So manually created > the elastic search index using template available in metron repo. Though > elastic search index is present , data from indexing toplogy neither > reached elastic search nor hdfs path .There are no errors in storm toplogy > logs.We could see the sample log in Metron management ui. How we can send > the logs to alerts ui and kibana dashboard. In kibana dashboard we could > see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but > with no data. Elasticsearch health is yellow and we are able to insert data > via rest call. Any documentation on sending the smaple snort logs to metron > alerts ui will be helpful . Any configuration from metron management ui is > required to pass it to alerts –ui > > > > > > Thanks and Regards > > Hema > > > > > > >
Re: Snort logs flow issue
We verified it in Storm ui and in Storm topology logs On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic wrote: > How did you validate the logs are making it to the indexing topology? > > On Fri, Apr 5, 2019 at 8:12 AM Hema malini > wrote: > >> >> Hi, >> >> >> >> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >> sent the sample snort logs copied from metron git repo to snort kafka >> topic.We did the same for bro topic.Logs are getting parsed and reached >> indexing topology . Elastic search indices are not getting created though >> we gave elastic search template install from ambari. So manually created >> the elastic search index using template available in metron repo. Though >> elastic search index is present , data from indexing toplogy neither >> reached elastic search nor hdfs path .There are no errors in storm toplogy >> logs.We could see the sample log in Metron management ui. How we can send >> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >> with no data. Elasticsearch health is yellow and we are able to insert data >> via rest call. Any documentation on sending the smaple snort logs to metron >> alerts ui will be helpful . Any configuration from metron management ui is >> required to pass it to alerts –ui >> >> >> >> >> >> Thanks and Regards >> >> Hema >> >> >> >> >> >> >> >
Re: Snort logs flow issue
Do you get 10 records output to the CLI when you run the following? /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 On Fri, Apr 5, 2019 at 11:38 AM Hema malini wrote: > We verified it in Storm ui and in Storm topology logs > > On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> How did you validate the logs are making it to the indexing topology? >> >> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >> wrote: >> >>> >>> Hi, >>> >>> >>> >>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>> sent the sample snort logs copied from metron git repo to snort kafka >>> topic.We did the same for bro topic.Logs are getting parsed and reached >>> indexing topology . Elastic search indices are not getting created though >>> we gave elastic search template install from ambari. So manually created >>> the elastic search index using template available in metron repo. >>> Though elastic search index is present , data from indexing toplogy neither >>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>> logs.We could see the sample log in Metron management ui. How we can send >>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>> with no data. Elasticsearch health is yellow and we are able to insert data >>> via rest call. Any documentation on sending the smaple snort logs to metron >>> alerts ui will be helpful . Any configuration from metron management ui is >>> required to pass it to alerts –ui >>> >>> >>> >>> >>> >>> Thanks and Regards >>> >>> Hema >>> >>> >>> >>> >>> >>> >>> >>
Re: Snort logs flow issue
Yes I am getting messages On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic wrote: > Do you get 10 records output to the CLI when you run the following? > > /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper > $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 > > > On Fri, Apr 5, 2019 at 11:38 AM Hema malini > wrote: > >> We verified it in Storm ui and in Storm topology logs >> >> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> How did you validate the logs are making it to the indexing topology? >>> >>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>>> sent the sample snort logs copied from metron git repo to snort kafka >>>> topic.We did the same for bro topic.Logs are getting parsed and reached >>>> indexing topology . Elastic search indices are not getting created though >>>> we gave elastic search template install from ambari. So manually created >>>> the elastic search index using template available in metron repo. >>>> Though elastic search index is present , data from indexing toplogy neither >>>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>>> logs.We could see the sample log in Metron management ui. How we can send >>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>>> with no data. Elasticsearch health is yellow and we are able to insert data >>>> via rest call. Any documentation on sending the smaple snort logs to metron >>>> alerts ui will be helpful . Any configuration from metron management ui is >>>> required to pass it to alerts –ui >>>> >>>> >>>> >>>> >>>> >>>> Thanks and Regards >>>> >>>> Hema >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>
Re: Snort logs flow issue
Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> How did you validate the logs are making it to the indexing topology?
>>>>
>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>>>> sent the sample snort logs copied from metron git repo to snort kafka
>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>> indexing topology . Elastic search indices are not getting created though
>>>>> we gave elastic search template install from ambari. So manually created
>>>>> the elastic search index using template available in metron repo.
>>>>> Though elastic search index is present , data from indexing toplogy
>>>>> neither
>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>> data
>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>> metron
>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>> required to pass it to alerts –ui
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards
>>>>>
>>>>> Hema
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
Re: Snort logs flow issue
Are we missing any configuration? Initially elastic search was down. We
figured out the issue and fixed it .Now elastic search is up . We restarted
metron indexing but still those indices not created. So we created it
manually.Do we have to change any parser configuration . How logs will flow
into metron alerts dashboard and kibana dashboard..what is the required
congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>> wrote:
>>>
>>>> We verified it in Storm ui and in Storm topology logs
>>>>
>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>
>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>> the elastic search index using template available in metron repo.
>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>> neither
>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>> toplogy
>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>> data
>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>> metron
>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>> is
>>>>>> required to pass it to alerts –ui
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Hema
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
Re: Snort logs flow issue
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>> 0x1F5","parallelenricher.splitter.begin.ts":"
>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> Do you get 10 records output to the CLI when you run the following?
>>>>
>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>
>>>>
>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>> wrote:
>>>>
>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>
>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>> though
>>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>>> the elastic search index using template available in metron repo.
>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>> neither
>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>> toplogy
>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>> send
>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>> data
>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>> metron
>>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>>> is
>>>>>>> required to pass it to alerts –ui
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Hema
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
Re: Snort logs flow issue
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently
there is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
> Sorry for the typo. Can you please help with the required configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
>> Are we missing any configuration? Initially elastic search was down. We
>> figured out the issue and fixed it .Now elastic search is up . We restarted
>> metron indexing but still those indices not created. So we created it
>> manually.Do we have to change any parser configuration . How logs will flow
>> into metron alerts dashboard and kibana dashboard..what is the required
>> congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>> wrote:
>>>
>>>> Yes I am getting messages
>>>>
>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>
>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>>> we sent the sample snort logs copied from metron git repo to snort
>>>>>>>> kafka
>>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>>> though
>>>>>>>> we gave elastic search template install from ambari. So manually
>>>>>>>> created
>>>>>>>> the elastic search index using template available in metron repo.
>>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>>> neither
>>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>>> toplogy
>>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>>> send
>>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we
>>>>>>>> could
>>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created
>>>>>>>> but
>>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>>> data
>>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>>> metron
>>>>>>>> alerts ui will be helpful . Any configuration from metron management
>>>>>>>> ui is
>>>>>>>> required to pass it to alerts –ui
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Hema
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
Re: Snort logs flow issue
If you see them in the dashboard you should be able to see them in the
alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
default behavior is that the UI doesn't initiate a search at login, it's up
to the user to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
>> Sorry for the typo. Can you please help with the required configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down. We
>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>> metron indexing but still those indices not created. So we created it
>>> manually.Do we have to change any parser configuration . How logs will flow
>>> into metron alerts dashboard and kibana dashboard..what is the required
>>> congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>> wrote:
>>>
>>>> Sample messages flown in indexing topic
>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>> 999158","sig_generator":"1"}
>>>>
>>>>
>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>> wrote:
>>>>
>>>>> Yes I am getting messages
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>
>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages
>>>>>> 10
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>
>>>&g
Re: Snort logs flow issue
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log .
I clicked the search and changed the date range too. Still no records. Do
we have to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
wrote:
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>>
>>> Sorry for the typo. Can you please help with the required configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>> wrote:
>>>
>>>> Are we missing any configuration? Initially elastic search was down. We
>>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>>> metron indexing but still those indices not created. So we created it
>>>> manually.Do we have to change any parser configuration . How logs will flow
>>>> into metron alerts dashboard and kibana dashboard..what is the required
>>>> congratulation
>>>>
>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>> wrote:
>>>>
>>>>> Sample messages flown in indexing topic
>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>> 52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort
>>>>> test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>> 999158","sig_generator":"1"}
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Yes I am getting messages
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> Do you get 10 records output to the CLI when you run the following
Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings'
was not found. Do we have to create that hbase table . Where to find the
hbase tables created. I could see only two namespace in hbase - default and
hbase. No tables created in that. Do I have to run metron rest in dev
profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
> Hi Michael,
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> If you see them in the dashboard you should be able to see them in the
>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>> default behavior is that the UI doesn't initiate a search at login, it's up
>> to the user to click search.
>>
>> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>>
>>> After recreating the index, now we are able to visualize the data in
>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>
>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>> wrote:
>>>
>>>> Sorry for the typo. Can you please help with the required
>>>> configuration.
>>>>
>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>> wrote:
>>>>
>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>> restarted metron indexing but still those indices not created. So we
>>>>> created it manually.Do we have to change any parser configuration . How
>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>> the required congratulation
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Sample messages flown in indexing topic
>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>>>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>>>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>>>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>>>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>&
RE: Snort logs flow issue
Hello Hema,
Unless I’m wrong, this must be setup in MySQL, the database you use for Metron
REST.
From: Hema malini [mailto:nhemamalin...@gmail.com]
Sent: Tuesday, April 09, 2019 09:42
To: user@metron.apache.org
Subject: Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings' was
not found. Do we have to create that hbase table . Where to find the hbase
tables created. I could see only two namespace in hbase - default and hbase. No
tables created in that. Do I have to run metron rest in dev profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log . I
clicked the search and changed the date range too. Still no records. Do we have
to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
If you see them in the dashboard you should be able to see them in the alerts
UI. Any errors in either the alerts UI or REST logs? Also, the new default
behavior is that the UI doesn't initiate a search at login, it's up to the user
to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently there
is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Are we missing any configuration? Initially elastic search was down. We figured
out the issue and fixed it .Now elastic search is up . We restarted metron
indexing but still those indices not created. So we created it manually.Do we
have to change any parser configuration . How logs will flow into metron alerts
dashboard and kibana dashboard..what is the required congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sample messages flown in indexing topic
{"msg":"'snort test
alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test
alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name<http://threat.triage.rules.0.name/>":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Yes I am getting messages
On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
Do you get 10 records output to the CLI when you run the following?
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10
On Fri,
Re: Snort logs flow issue
Hi Stephanie,
Issue got resolved by creating that table in hbase.
Thanks and regards,
Hema
On Tue, Apr 9, 2019, 1:31 PM wrote:
> Hello Hema,
>
>
>
> Unless I’m wrong, this must be setup in MySQL, the database you use for
> Metron REST.
>
>
>
>
>
> *From:* Hema malini [mailto:nhemamalin...@gmail.com]
> *Sent:* Tuesday, April 09, 2019 09:42
> *To:* user@metron.apache.org
> *Subject:* Re: Snort logs flow issue
>
>
>
> Hi Michael,
>
>
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
>
>
> Thanks & Regards
>
> Hema
>
>
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
>
> Hi Michael,
>
>
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
>
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
>
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
>
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
> Sorry for the typo. Can you please help with the required configuration.
>
>
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
>
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
> Sample messages flown in indexing topic
>
> {"msg":"'snort test
> alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>
>
>
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
> Yes I
Re: Snort logs flow issue
;:"0A:00:27:00:00:00"," >>>>>>> threat.triage.rules.0.name":null,"is_alert": >>>>>>> "true","parallelenricher.enrich.begin.ts":" >>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter. >>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42" >>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":" >>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A"," >>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":" >>>>>>> 999158","sig_generator":"1"} >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini >>>>>>> wrote: >>>>>>> >>>>>>>> Yes I am getting messages >>>>>>>> >>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic < >>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Do you get 10 records output to the CLI when you run the following? >>>>>>>>> >>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh >>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning >>>>>>>>> --max-messages 10 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini < >>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> We verified it in Storm ui and in Storm topology logs >>>>>>>>>> >>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >>>>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> How did you validate the logs are making it to the indexing >>>>>>>>>>> topology? >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini < >>>>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using >>>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to >>>>>>>>>>>> snort >>>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed >>>>>>>>>>>> and >>>>>>>>>>>> reached indexing topology . Elastic search indices are not getting >>>>>>>>>>>> created >>>>>>>>>>>> though we gave elastic search template install from ambari. So >>>>>>>>>>>> manually >>>>>>>>>>>> created the elastic search index using template available in >>>>>>>>>>>> metron repo. Though elastic search index is present , data from >>>>>>>>>>>> indexing >>>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are no >>>>>>>>>>>> errors >>>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron >>>>>>>>>>>> management ui. >>>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In >>>>>>>>>>>> kibana >>>>>>>>>>>> dashboard we could see two dashboards - >>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data. >>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via >>>>>>>>>>>> rest >>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to metron >>>>>>>>>>>> alerts >>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui is >>>>>>>>>>>> required to pass it to alerts –ui >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks and Regards >>>>>>>>>>>> >>>>>>>>>>>> Hema >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>
Re: Snort logs flow issue
t;10","tcpack":"0x836687BD","protocol":"TCP"," >>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984 >>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121, >>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00, >>>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040, >>>>>>>> 52,53248","parallelenricher.enrich.end. >>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null," >>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin. >>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168. >>>>>>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00"," >>>>>>>> threat.triage.rules.0.name":null,"is_alert": >>>>>>>> "true","parallelenricher.enrich.begin.ts":" >>>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter. >>>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42" >>>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":" >>>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A"," >>>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":" >>>>>>>> 999158","sig_generator":"1"} >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes I am getting messages >>>>>>>>> >>>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic < >>>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Do you get 10 records output to the CLI when you run the >>>>>>>>>> following? >>>>>>>>>> >>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh >>>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning >>>>>>>>>> --max-messages 10 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini < >>>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> We verified it in Storm ui and in Storm topology logs >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >>>>>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> How did you validate the logs are making it to the indexing >>>>>>>>>>>> topology? >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini < >>>>>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using >>>>>>>>>>>>> Nifi we sent the sample snort logs copied from metron git repo to >>>>>>>>>>>>> snort >>>>>>>>>>>>> kafka topic.We did the same for bro topic.Logs are getting parsed >>>>>>>>>>>>> and >>>>>>>>>>>>> reached indexing topology . Elastic search indices are not >>>>>>>>>>>>> getting created >>>>>>>>>>>>> though we gave elastic search template install from ambari. So >>>>>>>>>>>>> manually >>>>>>>>>>>>> created the elastic search index using template available in >>>>>>>>>>>>> metron repo. Though elastic search index is present , data from >>>>>>>>>>>>> indexing >>>>>>>>>>>>> toplogy neither reached elastic search nor hdfs path .There are >>>>>>>>>>>>> no errors >>>>>>>>>>>>> in storm toplogy logs.We could see the sample log in Metron >>>>>>>>>>>>> management ui. >>>>>>>>>>>>> How we can send the logs to alerts ui and kibana dashboard. In >>>>>>>>>>>>> kibana >>>>>>>>>>>>> dashboard we could see two dashboards - >>>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data. >>>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data via >>>>>>>>>>>>> rest >>>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to >>>>>>>>>>>>> metron alerts >>>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui >>>>>>>>>>>>> is >>>>>>>>>>>>> required to pass it to alerts –ui >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks and Regards >>>>>>>>>>>>> >>>>>>>>>>>>> Hema >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>
Re: Snort logs flow issue
triage.score":10.0,"dgmlen":" >>>>>>>>> 52","adapter.hostfromjsonlistadapter.end. >>>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":" >>>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher. >>>>>>>>> splitter.begin.ts":"1554384505264","threat.triage. >>>>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP"," >>>>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984 >>>>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121, >>>>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00, >>>>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040, >>>>>>>>> 52,53248","parallelenricher.enrich.end. >>>>>>>>> ts":"1554384505342","threat.triage.rules.0.reason":null," >>>>>>>>> tos":"0","adapter.hostfromjsonlistadapter.begin. >>>>>>>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168. >>>>>>>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00"," >>>>>>>>> threat.triage.rules.0.name":null,"is_alert": >>>>>>>>> "true","parallelenricher.enrich.begin.ts":" >>>>>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter. >>>>>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42" >>>>>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":" >>>>>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A"," >>>>>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":" >>>>>>>>> 999158","sig_generator":"1"} >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes I am getting messages >>>>>>>>>> >>>>>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic < >>>>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Do you get 10 records output to the CLI when you run the >>>>>>>>>>> following? >>>>>>>>>>> >>>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh >>>>>>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning >>>>>>>>>>> --max-messages 10 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini < >>>>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> We verified it in Storm ui and in Storm topology logs >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >>>>>>>>>>>> michael.miklav...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> How did you validate the logs are making it to the indexing >>>>>>>>>>>>> topology? >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini < >>>>>>>>>>>>> nhemamalin...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We have installed Metron 0.7.1 in centos 7 using >>>>>>>>>>>>>> Amabari.Using Nifi we sent the sample snort logs copied from >>>>>>>>>>>>>> metron git >>>>>>>>>>>>>> repo to snort kafka topic.We did the same for bro topic.Logs are >>>>>>>>>>>>>> getting >>>>>>>>>>>>>> parsed and reached indexing topology . Elastic search indices >>>>>>>>>>>>>> are not >>>>>>>>>>>>>> getting created though we gave elastic search template install >>>>>>>>>>>>>> from ambari. >>>>>>>>>>>>>> So manually created the elastic search index using template >>>>>>>>>>>>>> available in metron repo. Though elastic search index is present >>>>>>>>>>>>>> , data >>>>>>>>>>>>>> from indexing toplogy neither reached elastic search nor hdfs >>>>>>>>>>>>>> path .There >>>>>>>>>>>>>> are no errors in storm toplogy logs.We could see the sample log >>>>>>>>>>>>>> in Metron >>>>>>>>>>>>>> management ui. How we can send the logs to alerts ui and kibana >>>>>>>>>>>>>> dashboard. >>>>>>>>>>>>>> In kibana dashboard we could see two dashboards - >>>>>>>>>>>>>> Metron-Dashboard,Metron-Error-Dashboard created but with no data. >>>>>>>>>>>>>> Elasticsearch health is yellow and we are able to insert data >>>>>>>>>>>>>> via rest >>>>>>>>>>>>>> call. Any documentation on sending the smaple snort logs to >>>>>>>>>>>>>> metron alerts >>>>>>>>>>>>>> ui will be helpful . Any configuration from metron management ui >>>>>>>>>>>>>> is >>>>>>>>>>>>>> required to pass it to alerts –ui >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks and Regards >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hema >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>
