Snort logs flow issue
Hi, We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui Thanks and Regards Hema
Re: Snort logs flow issue
How did you validate the logs are making it to the indexing topology? On Fri, Apr 5, 2019 at 8:12 AM Hema malini wrote: > > Hi, > > > > We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we > sent the sample snort logs copied from metron git repo to snort kafka > topic.We did the same for bro topic.Logs are getting parsed and reached > indexing topology . Elastic search indices are not getting created though > we gave elastic search template install from ambari. So manually created > the elastic search index using template available in metron repo. Though > elastic search index is present , data from indexing toplogy neither > reached elastic search nor hdfs path .There are no errors in storm toplogy > logs.We could see the sample log in Metron management ui. How we can send > the logs to alerts ui and kibana dashboard. In kibana dashboard we could > see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but > with no data. Elasticsearch health is yellow and we are able to insert data > via rest call. Any documentation on sending the smaple snort logs to metron > alerts ui will be helpful . Any configuration from metron management ui is > required to pass it to alerts –ui > > > > > > Thanks and Regards > > Hema > > > > > > >
Re: Snort logs flow issue
We verified it in Storm ui and in Storm topology logs On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic wrote: > How did you validate the logs are making it to the indexing topology? > > On Fri, Apr 5, 2019 at 8:12 AM Hema malini > wrote: > >> >> Hi, >> >> >> >> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >> sent the sample snort logs copied from metron git repo to snort kafka >> topic.We did the same for bro topic.Logs are getting parsed and reached >> indexing topology . Elastic search indices are not getting created though >> we gave elastic search template install from ambari. So manually created >> the elastic search index using template available in metron repo. Though >> elastic search index is present , data from indexing toplogy neither >> reached elastic search nor hdfs path .There are no errors in storm toplogy >> logs.We could see the sample log in Metron management ui. How we can send >> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >> with no data. Elasticsearch health is yellow and we are able to insert data >> via rest call. Any documentation on sending the smaple snort logs to metron >> alerts ui will be helpful . Any configuration from metron management ui is >> required to pass it to alerts –ui >> >> >> >> >> >> Thanks and Regards >> >> Hema >> >> >> >> >> >> >> >
Re: Snort logs flow issue
Do you get 10 records output to the CLI when you run the following? /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 On Fri, Apr 5, 2019 at 11:38 AM Hema malini wrote: > We verified it in Storm ui and in Storm topology logs > > On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> How did you validate the logs are making it to the indexing topology? >> >> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >> wrote: >> >>> >>> Hi, >>> >>> >>> >>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>> sent the sample snort logs copied from metron git repo to snort kafka >>> topic.We did the same for bro topic.Logs are getting parsed and reached >>> indexing topology . Elastic search indices are not getting created though >>> we gave elastic search template install from ambari. So manually created >>> the elastic search index using template available in metron repo. >>> Though elastic search index is present , data from indexing toplogy neither >>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>> logs.We could see the sample log in Metron management ui. How we can send >>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>> with no data. Elasticsearch health is yellow and we are able to insert data >>> via rest call. Any documentation on sending the smaple snort logs to metron >>> alerts ui will be helpful . Any configuration from metron management ui is >>> required to pass it to alerts –ui >>> >>> >>> >>> >>> >>> Thanks and Regards >>> >>> Hema >>> >>> >>> >>> >>> >>> >>> >>
Re: Snort logs flow issue
Yes I am getting messages On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic wrote: > Do you get 10 records output to the CLI when you run the following? > > /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper > $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 > > > On Fri, Apr 5, 2019 at 11:38 AM Hema malini > wrote: > >> We verified it in Storm ui and in Storm topology logs >> >> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> How did you validate the logs are making it to the indexing topology? >>> >>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >>> wrote: >>> Hi, We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui Thanks and Regards Hema >>>
Re: Snort logs flow issue
Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
How did you validate the logs are making it to the indexing topology?
On Fri, Apr 5, 2019 at 8:12 AM Hema malini
wrote:
>
> Hi,
>
>
>
> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
> sent the sample snort logs copied from metron git repo to snort kafka
> topic.We did the same for bro topic.Logs are getting parsed and reached
> indexing topology . Elastic search indices are not getting created though
> we gave elastic search template install from ambari. So manually created
> the elastic search index using template available in metron repo.
> Though elastic search index is present , data from indexing toplogy
> neither
> reached elastic search nor hdfs path .There are no errors in storm toplogy
> logs.We could see the sample log in Metron management ui. How we can send
> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
> with no data. Elasticsearch health is yellow and we are able to insert
> data
> via rest call. Any documentation on sending the smaple snort logs to
> metron
> alerts ui will be helpful . Any configuration from metron management ui is
> required to pass it to alerts –ui
>
>
>
>
>
> Thanks and Regards
>
> Hema
>
>
>
>
>
>
>
