hi, This problem still persists guys . On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:
> Any solution to these issues guys? > > On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> I have attached the output of this dump >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> >> >> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >>> What is the output of: >>> >>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>> >>> ? >>> >>> Jon >>> >>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir <mscs16...@itu.edu.pk> >>> wrote: >>> >>>> This is the script/command i used >>>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> sudo cat snort.out | >>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>> --broker-list node1:6667 --topic snort >>>>> >>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler <ottobackwa...@gmail.com> >>>>> wrote: >>>>> >>>>>> What topic? what are the parameters you are calling the script with? >>>>>> >>>>>> >>>>>> >>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>> mscs16...@itu.edu.pk) wrote: >>>>>> >>>>>> The metron installation I have (single node based vm install) comes >>>>>> with sensor stubs. I assume that everything has already been done for >>>>>> those >>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>>> the >>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella <ceste...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> How did you start the snort parser topology and what's the parser >>>>>>> config (in zookeeper)? >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> This is what I am doing >>>>>>>> >>>>>>>> sudo cat snort.out | >>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella <ceste...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>> records in >>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>> topic, >>>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>>> then >>>>>>>>> the enrichments topology enrich that map and emits the enriched map >>>>>>>>> to the >>>>>>>>> indexing topic. >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>>> full stack trace >>>>>>>>>> >>>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>>> >>>>>>>>>> [image: Inline image 1] >>>>>>>>>> >>>>>>>>>> from indexingbolt in indexing topology >>>>>>>>>> >>>>>>>>>> [image: Inline image 2] >>>>>>>>>> >>>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>>> Also, are you saying that you are no longer seeing the parser >>>>>>>>>>> topology error? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella ( >>>>>>>>>>> ceste...@gmail.com) wrote: >>>>>>>>>>> >>>>>>>>>>> If you click on the port (6704) there in those errors, what's >>>>>>>>>>> the full stacktrace (that starts with the suggestion you file a >>>>>>>>>>> JIRA)? >>>>>>>>>>> >>>>>>>>>>> What this means is that an exception is bleeding from the >>>>>>>>>>> individual writer into the writer component (It should be handled >>>>>>>>>>> in the >>>>>>>>>>> writer itself). The fact that it's happening for both HDFS and ES >>>>>>>>>>> is >>>>>>>>>>> telling as well and I'm very interested in the full stacktrace there >>>>>>>>>>> because it'll have the wrapped exception from the individual writer >>>>>>>>>>> included. >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 8, 2017 at 11:24 AM, Syed Hammad Tahir < >>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>> >>>>>>>>>>>> OK I did what Zeolla said, cat snort.out | kafka producer .... >>>>>>>>>>>> and now the error at storm parser topology is gone but I am now >>>>>>>>>>>> seeing this >>>>>>>>>>>> at the indexing toology >>>>>>>>>>>> >>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 8, 2017 at 8:25 PM, Syed Hammad Tahir < >>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> this is a single line I am trying to push >>>>>>>>>>>>> 01/11/17-20:49:18.107168 ,1,999158,0,"'snort test >>>>>>>>>>>>> alert'",TCP,192.168.66.1,49581,192.168.66.121,22,0A:00:27:00 >>>>>>>>>>>>> :00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC,0x56900BB6 >>>>>>>>>>>>> ,,0x1000,64,10,23403,76,77824,,,, >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 8, 2017 at 5:30 PM, zeo...@gmail.com < >>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I would download the entire snort.out file and run cat >>>>>>>>>>>>>> snort.out | kafka-console-producer.sh ... to make sure there are >>>>>>>>>>>>>> no copy >>>>>>>>>>>>>> paste problems >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Nov 8, 2017, 06:59 Otto Fowler < >>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The snort parser is coded to support dates in this format: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> private static String defaultDateFormat = >>>>>>>>>>>>>>> "MM/dd/yy-HH:mm:ss.SSSSSS"; >>>>>>>>>>>>>>> private transient DateTimeFormatter dateTimeFormatter; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If your records are in dd/MM/yy- format, then you may see >>>>>>>>>>>>>>> this error I believe. >>>>>>>>>>>>>>> Can you verify the timestamp field’s format? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If this is the case, then you will need to modify the >>>>>>>>>>>>>>> default log timestamp format for snort in the short term. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler ( >>>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column >>>>>>>>>>>>>>> is for a piece of data that is failing >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir ( >>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the >>>>>>>>>>>>>>> logs I am trying to push >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data >>>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the >>>>>>>>>>>>>>> same thing. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir < >>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> After running this command, I copy paste a few lines from >>>>>>>>>>>>>>>> here: https://raw.githubusercontent. >>>>>>>>>>>>>>>> com/apache/metron/master/metro >>>>>>>>>>>>>>>> n-deployment/roles/sensor-stubs/files/snort.out >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> like this >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines >>>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This was the mechanism I am using to push the logs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler < >>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What I mean is this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort >>>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think >>>>>>>>>>>>>>>>> of as live. >>>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see >>>>>>>>>>>>>>>>> this error. >>>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for >>>>>>>>>>>>>>>>> things that are automatically pushed to kafka as you do when >>>>>>>>>>>>>>>>> you manual >>>>>>>>>>>>>>>>> push them. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On November 7, 2017 at 08:51:41, Syed Hammad Tahir ( >>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "Yes, If the messages cannot be parsed then that would be >>>>>>>>>>>>>>>>> a problem. If you see this error with your ‘live’ messages >>>>>>>>>>>>>>>>> as well then >>>>>>>>>>>>>>>>> that could be it. >>>>>>>>>>>>>>>>> I wonder if the issue is with the date format?" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If by 'live' messages you mean the time I push them into >>>>>>>>>>>>>>>>> kafka topic then no, I dont see any error at that time. If >>>>>>>>>>>>>>>>> 'live' means >>>>>>>>>>>>>>>>> something else here then please tell me what could it be. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 5:07 PM, Otto Fowler < >>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yes, If the messages cannot be parsed then that would be >>>>>>>>>>>>>>>>>> a problem. If you see this error with your ‘live’ messages >>>>>>>>>>>>>>>>>> as well then >>>>>>>>>>>>>>>>>> that could be it. >>>>>>>>>>>>>>>>>> I wonder if the issue is with the date format? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> You need to confirm that you see these same errors with >>>>>>>>>>>>>>>>>> the live data or not. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Remember, the flow is like this >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> snort -> ??? -> Kafka -> Storm Parser Topology -> kafka >>>>>>>>>>>>>>>>>> -> Storm Enrichment Topology -> Kafka -> Storm Indexing >>>>>>>>>>>>>>>>>> Topology -> HDFS | >>>>>>>>>>>>>>>>>> ElasticSearch >>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>> Kibana <-> Elastic Search >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Any point in this chain could fail and result in Kibana >>>>>>>>>>>>>>>>>> not seeing things. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On November 7, 2017 at 01:57:19, Syed Hammad Tahir ( >>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> could this be related to why I am unable to see logs in >>>>>>>>>>>>>>>>>> kibana dashboard? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am copying a few lines from here >>>>>>>>>>>>>>>>>> https://raw.githubusercontent. >>>>>>>>>>>>>>>>>> com/apache/metron/master/metro >>>>>>>>>>>>>>>>>> n-deployment/roles/sensor-stubs/files/snort.out >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and then pushing them to snort kafka topic. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> THis is some error I am seeing in stormUI parser bolt in >>>>>>>>>>>>>>>>>> snort section: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 11:49 AM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I guess I have hit a dead end. I am not able to get the >>>>>>>>>>>>>>>>>>> snort logs in kibana dashboard. Any help will be >>>>>>>>>>>>>>>>>>> appreciated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 1:24 PM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I guess this (metron.log) in /var/log/elasticsearch/ is >>>>>>>>>>>>>>>>>>>> also relevant >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 11:46 AM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Cluster health by index shows this: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> looks like some shard is unassigned and that is >>>>>>>>>>>>>>>>>>>>> related to snort. Could it be the logs I was pushing to >>>>>>>>>>>>>>>>>>>>> kafka topic earlier? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:47 AM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> This is what I see here. What should I be looking at >>>>>>>>>>>>>>>>>>>>>> here? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:33 AM, Syed Hammad Tahir < >>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> hi, I am back at work. lets see if i can find >>>>>>>>>>>>>>>>>>>>>>> something in logs >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com < >>>>>>>>>>>>>>>>>>>>>>> zeo...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> It looks like your ES cluster has a health of Red, >>>>>>>>>>>>>>>>>>>>>>>> so there's your problem. I would go look in >>>>>>>>>>>>>>>>>>>>>>>> /var/log/elasticsearch/ at >>>>>>>>>>>>>>>>>>>>>>>> some logs. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir < >>>>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>>>>>>>>>>>>> From: Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>>>>>>>>>>>>>>>>>>>>>> Date: Fri, Nov 3, 2017 at 5:07 PM >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Snort Logs >>>>>>>>>>>>>>>>>>>>>>>>> To: Otto Fowler <ottobackwa...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> NVM, I have installed the elastic search head. Now >>>>>>>>>>>>>>>>>>>>>>>>> where do I go in this to find out why I cant see the >>>>>>>>>>>>>>>>>>>>>>>>> snort logs in kibana >>>>>>>>>>>>>>>>>>>>>>>>> dashboard, pushed to snort topic via kafka producer? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler < >>>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> You can install it into the chrome web browser >>>>>>>>>>>>>>>>>>>>>>>>>> from the play store. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On November 3, 2017 at 07:47:47, Syed Hammad >>>>>>>>>>>>>>>>>>>>>>>>>> Tahir (mscs16...@itu.edu.pk) wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> And how do I install elasticsearch head on the >>>>>>>>>>>>>>>>>>>>>>>>>> vagrant VM? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> -- >>> >>> Jon >>> >> >> >
