Re: Snort logs flow issue
Thanks a lot Michael for your help. Will explore further.
On Wed, Apr 10, 2019, 3:37 AM Michael Miklavcic
wrote:
> That table should have been created by default as part of the Ambari
> installation of Metron via our MPack -
> https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
> You shouldn't have to worry about this at all as an end user, but here is
> where it happens if you're curious-
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
>
> I believe that table will be empty by default. @Ryan Merriman, do you
> happen to know if we currently save user UI data there?
>
> On Tue, Apr 9, 2019 at 2:14 AM Hema malini
> wrote:
>
>> Hi Michael,
>>
>> Issue got resolved after I manually created user settings table in hbase.
>> There are no contents in that table though it is working. Where are the
>> records getting stored then for alerts UI. Where can I find the records in
>> HDFS.
>>
>> Thanks and regards,
>> Hema
>>
>> On Tue, Apr 9, 2019, 1:12 PM Hema malini wrote:
>>
>>> Hi Michael,
>>>
>>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>>> was not found. Do we have to create that hbase table . Where to find the
>>> hbase tables created. I could see only two namespace in hbase - default and
>>> hbase. No tables created in that. Do I have to run metron rest in dev
>>> profile.
>>>
>>> Thanks & Regards
>>> Hema
>>>
>>> On Tue, Apr 9, 2019, 12:44 PM Hema malini
>>> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>>> log . I clicked the search and changed the date range too. Still no
>>>> records. Do we have to run metron rest in dev profile?
>>>>
>>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> If you see them in the dashboard you should be able to see them in the
>>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>>> default behavior is that the UI doesn't initiate a search at login, it's
>>>>> up
>>>>> to the user to click search.
>>>>>
>>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>> After recreating the index, now we are able to visualize the data in
>>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>>> Currently there is no data in alerts UI. How.to configure the logs as
>>>>>> alerts
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry for the typo. Can you please help with the required
>>>>>>> configuration.
>>>>>>>
>>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Are we missing any configuration? Initially elastic search was
>>>>>>>> down. We figured out the issue and fixed it .Now elastic search is up
>>>>>>>> . We
>>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what
>>>>>>>> is
>>>>>>>> the required congratulation
>>>>>>>>
>>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sample messages flown in indexing topic
>>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":&q
Re: Snort logs flow issue
That table should have been created by default as part of the Ambari
installation of Metron via our MPack -
https://github.com/apache/metron/tree/master/metron-deployment#how-do-i-deploy-metron-with-ambari.
You shouldn't have to worry about this at all as an end user, but here is
where it happens if you're curious-
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py#L230
I believe that table will be empty by default. @Ryan Merriman, do you
happen to know if we currently save user UI data there?
On Tue, Apr 9, 2019 at 2:14 AM Hema malini wrote:
> Hi Michael,
>
> Issue got resolved after I manually created user settings table in hbase.
> There are no contents in that table though it is working. Where are the
> records getting stored then for alerts UI. Where can I find the records in
> HDFS.
>
> Thanks and regards,
> Hema
>
> On Tue, Apr 9, 2019, 1:12 PM Hema malini wrote:
>
>> Hi Michael,
>>
>> Sorry just noticed the error in metron rest logs - Table 'user settings'
>> was not found. Do we have to create that hbase table . Where to find the
>> hbase tables created. I could see only two namespace in hbase - default and
>> hbase. No tables created in that. Do I have to run metron rest in dev
>> profile.
>>
>> Thanks & Regards
>> Hema
>>
>> On Tue, Apr 9, 2019, 12:44 PM Hema malini
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks for your reply. I couldn't find any errors in metron alerts UI
>>> log . I clicked the search and changed the date range too. Still no
>>> records. Do we have to run metron rest in dev profile?
>>>
>>> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> If you see them in the dashboard you should be able to see them in the
>>>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>>>> default behavior is that the UI doesn't initiate a search at login, it's up
>>>> to the user to click search.
>>>>
>>>> On Mon, Apr 8, 2019, 6:38 AM Hema malini
>>>> wrote:
>>>>
>>>>> After recreating the index, now we are able to visualize the data in
>>>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>>>> Currently there is no data in alerts UI. How.to configure the logs as
>>>>> alerts
>>>>>
>>>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Sorry for the typo. Can you please help with the required
>>>>>> configuration.
>>>>>>
>>>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>>>> restarted metron indexing but still those indices not created. So we
>>>>>>> created it manually.Do we have to change any parser configuration . How
>>>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what
>>>>>>> is
>>>>>>> the required congratulation
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sample messages flown in indexing topic
>>>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>>>> rules.0.score":"10","tcpack":"0x836687B
Re: Snort logs flow issue
Hi Stephanie,
Issue got resolved by creating that table in hbase.
Thanks and regards,
Hema
On Tue, Apr 9, 2019, 1:31 PM wrote:
> Hello Hema,
>
>
>
> Unless I’m wrong, this must be setup in MySQL, the database you use for
> Metron REST.
>
>
>
>
>
> *From:* Hema malini [mailto:nhemamalin...@gmail.com]
> *Sent:* Tuesday, April 09, 2019 09:42
> *To:* user@metron.apache.org
> *Subject:* Re: Snort logs flow issue
>
>
>
> Hi Michael,
>
>
>
> Sorry just noticed the error in metron rest logs - Table 'user settings'
> was not found. Do we have to create that hbase table . Where to find the
> hbase tables created. I could see only two namespace in hbase - default and
> hbase. No tables created in that. Do I have to run metron rest in dev
> profile.
>
>
>
> Thanks & Regards
>
> Hema
>
>
>
> On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
>
> Hi Michael,
>
>
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
>
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
>
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
>
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
> Sorry for the typo. Can you please help with the required configuration.
>
>
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
>
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
> Sample messages flown in indexing topic
>
> {"msg":"'snort test
> alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
> threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>
>
>
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
> Yes I am getting messages
>
>
>
> On F
RE: Snort logs flow issue
Hello Hema,
Unless I’m wrong, this must be setup in MySQL, the database you use for Metron
REST.
From: Hema malini [mailto:nhemamalin...@gmail.com]
Sent: Tuesday, April 09, 2019 09:42
To: user@metron.apache.org
Subject: Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings' was
not found. Do we have to create that hbase table . Where to find the hbase
tables created. I could see only two namespace in hbase - default and hbase. No
tables created in that. Do I have to run metron rest in dev profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log . I
clicked the search and changed the date range too. Still no records. Do we have
to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
If you see them in the dashboard you should be able to see them in the alerts
UI. Any errors in either the alerts UI or REST logs? Also, the new default
behavior is that the UI doesn't initiate a search at login, it's up to the user
to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently there
is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Are we missing any configuration? Initially elastic search was down. We figured
out the issue and fixed it .Now elastic search is up . We restarted metron
indexing but still those indices not created. So we created it manually.Do we
have to change any parser configuration . How logs will flow into metron alerts
dashboard and kibana dashboard..what is the required congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Sample messages flown in indexing topic
{"msg":"'snort test
alert'","parallelenricher.splitter.end.ts":"1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.splitter.begin.ts":"1554384505264","threat.triage.rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test
alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.ts":"1554384505342","threat.triage.rules.0.reason":null,"tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"62040","ip_src_addr":"192.168.66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name<http://threat.triage.rules.0.name/>":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"1554384505264","ttl":"64","source.type":"snort","adapter.geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.threatinteladapter.begin.ts":"1554384505264","ip_src_port":"8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini
mailto:nhemamalin...@gmail.com>> wrote:
Yes I am getting messages
On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic
mailto:michael.miklav...@gmail.com>> wrote:
Do you get 10 records output to the CLI when you run the following?
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
$ZOOKEEPER --topic indexing --from-beginning --max-messages 10
On Fri, Apr 5, 2019 at 11:38 AM Hema malini
mailto:n
Re: Snort logs flow issue
Hi Michael,
Sorry just noticed the error in metron rest logs - Table 'user settings'
was not found. Do we have to create that hbase table . Where to find the
hbase tables created. I could see only two namespace in hbase - default and
hbase. No tables created in that. Do I have to run metron rest in dev
profile.
Thanks & Regards
Hema
On Tue, Apr 9, 2019, 12:44 PM Hema malini wrote:
> Hi Michael,
>
> Thanks for your reply. I couldn't find any errors in metron alerts UI log
> . I clicked the search and changed the date range too. Still no records. Do
> we have to run metron rest in dev profile?
>
> On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> If you see them in the dashboard you should be able to see them in the
>> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
>> default behavior is that the UI doesn't initiate a search at login, it's up
>> to the user to click search.
>>
>> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>>
>>> After recreating the index, now we are able to visualize the data in
>>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>>
>>> On Sat, Apr 6, 2019, 9:21 PM Hema malini
>>> wrote:
>>>
>>>> Sorry for the typo. Can you please help with the required
>>>> configuration.
>>>>
>>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>>> wrote:
>>>>
>>>>> Are we missing any configuration? Initially elastic search was down.
>>>>> We figured out the issue and fixed it .Now elastic search is up . We
>>>>> restarted metron indexing but still those indices not created. So we
>>>>> created it manually.Do we have to change any parser configuration . How
>>>>> logs will flow into metron alerts dashboard and kibana dashboard..what is
>>>>> the required congratulation
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Sample messages flown in indexing topic
>>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>>> 52","adapter.hostfromjsonlistadapter.end.
>>>>>> ts":"1554384503452","adapter.geoadapter.begin.ts":"
>>>>>> 1554384503452","tcpwindow":"0x1F5","parallelenricher.
>>>>>> splitter.begin.ts":"1554384505264","threat.triage.
>>>>>> rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
>>>>>> ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
>>>>>> ,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
>>>>>> 8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
>>>>>> 0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","
>>>>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>>>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>>>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>>>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>>>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>>>>> 8080","tcpflags&quo
Re: Snort logs flow issue
Hi Michael,
Thanks for your reply. I couldn't find any errors in metron alerts UI log .
I clicked the search and changed the date range too. Still no records. Do
we have to run metron rest in dev profile?
On Mon, Apr 8, 2019, 7:50 PM Michael Miklavcic
wrote:
> If you see them in the dashboard you should be able to see them in the
> alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
> default behavior is that the UI doesn't initiate a search at login, it's up
> to the user to click search.
>
> On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
>
>> After recreating the index, now we are able to visualize the data in
>> kibana metron dashboard. How we can pass alerts to metron alerts UI.
>> Currently there is no data in alerts UI. How.to configure the logs as alerts
>>
>> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>>
>>> Sorry for the typo. Can you please help with the required configuration.
>>>
>>> On Sat, Apr 6, 2019, 5:39 PM Hema malini
>>> wrote:
>>>
>>>> Are we missing any configuration? Initially elastic search was down. We
>>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>>> metron indexing but still those indices not created. So we created it
>>>> manually.Do we have to change any parser configuration . How logs will flow
>>>> into metron alerts dashboard and kibana dashboard..what is the required
>>>> congratulation
>>>>
>>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>>> wrote:
>>>>
>>>>> Sample messages flown in indexing topic
>>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"
>>>>> 52","adapter.hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort
>>>>> test alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>>> 999158","sig_generator":"1"}
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>>> wrote:
>>>>>
>>>>>> Yes I am getting messages
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>>
>
Re: Snort logs flow issue
If you see them in the dashboard you should be able to see them in the
alerts UI. Any errors in either the alerts UI or REST logs? Also, the new
default behavior is that the UI doesn't initiate a search at login, it's up
to the user to click search.
On Mon, Apr 8, 2019, 6:38 AM Hema malini wrote:
> After recreating the index, now we are able to visualize the data in
> kibana metron dashboard. How we can pass alerts to metron alerts UI.
> Currently there is no data in alerts UI. How.to configure the logs as alerts
>
> On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
>
>> Sorry for the typo. Can you please help with the required configuration.
>>
>> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>>
>>> Are we missing any configuration? Initially elastic search was down. We
>>> figured out the issue and fixed it .Now elastic search is up . We restarted
>>> metron indexing but still those indices not created. So we created it
>>> manually.Do we have to change any parser configuration . How logs will flow
>>> into metron alerts dashboard and kibana dashboard..what is the required
>>> congratulation
>>>
>>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>>> wrote:
>>>
>>>> Sample messages flown in indexing topic
>>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","
>>>> parallelenricher.enrich.end.ts":"1554384505342","threat.
>>>> triage.rules.0.reason":null,"tos":"0","adapter.
>>>> hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
>>>> 62040","ip_src_addr":"192.168.66.121","timestamp":
>>>> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
>>>> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
>>>> 1554384505264","ttl":"64","source.type":"snort","adapter.
>>>> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
>>>> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
>>>> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
>>>> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
>>>> 999158","sig_generator":"1"}
>>>>
>>>>
>>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>>> wrote:
>>>>
>>>>> Yes I am getting messages
>>>>>
>>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>>
>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages
>>>>>> 10
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>>
>>>>>>> On Fri, Apr
Re: Snort logs flow issue
After recreating the index, now we are able to visualize the data in kibana
metron dashboard. How we can pass alerts to metron alerts UI. Currently
there is no data in alerts UI. How.to configure the logs as alerts
On Sat, Apr 6, 2019, 9:21 PM Hema malini wrote:
> Sorry for the typo. Can you please help with the required configuration.
>
> On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
>
>> Are we missing any configuration? Initially elastic search was down. We
>> figured out the issue and fixed it .Now elastic search is up . We restarted
>> metron indexing but still those indices not created. So we created it
>> manually.Do we have to change any parser configuration . How logs will flow
>> into metron alerts dashboard and kibana dashboard..what is the required
>> congratulation
>>
>> On Fri, Apr 5, 2019, 11:52 PM Hema malini
>> wrote:
>>
>>> Sample messages flown in indexing topic
>>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>>> 0x1F5","parallelenricher.splitter.begin.ts":"
>>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>>> enrich.begin.ts":"1554384505264","ttl":"64","
>>> source.type":"snort","adapter.geoadapter.end.ts":"
>>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>>
>>>
>>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>>> wrote:
>>>
>>>> Yes I am getting messages
>>>>
>>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> Do you get 10 records output to the CLI when you run the following?
>>>>>
>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh
>>>>> --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>>
>>>>>
>>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>>
>>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>>
>>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>>> we sent the sample snort logs copied from metron git repo to snort
>>>>>>>> kafka
>>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>>> though
>>>>>>>> we gave elastic search template install from ambari. So manually
>>>>>>>> created
>>>>>>>> the elastic search index using template available in metron repo.
>>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>>> neither
>>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>>> toplogy
>>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>>> send
>>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we
>>>>>>>> could
>>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created
>>>>>>>> but
>>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>>> data
>>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>>> metron
>>>>>>>> alerts ui will be helpful . Any configuration from metron management
>>>>>>>> ui is
>>>>>>>> required to pass it to alerts –ui
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks and Regards
>>>>>>>>
>>>>>>>> Hema
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
Re: Snort logs flow issue
Sorry for the typo. Can you please help with the required configuration.
On Sat, Apr 6, 2019, 5:39 PM Hema malini wrote:
> Are we missing any configuration? Initially elastic search was down. We
> figured out the issue and fixed it .Now elastic search is up . We restarted
> metron indexing but still those indices not created. So we created it
> manually.Do we have to change any parser configuration . How logs will flow
> into metron alerts dashboard and kibana dashboard..what is the required
> congratulation
>
> On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
>
>> Sample messages flown in indexing topic
>> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
>> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
>> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
>> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
>> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
>> geoadapter.begin.ts":"1554384503452","tcpwindow":"
>> 0x1F5","parallelenricher.splitter.begin.ts":"
>> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
>> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
>> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
>> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
>> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
>> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
>> ts":"1554384505342","threat.triage.rules.0.reason":null,"
>> tos":"0","adapter.hostfromjsonlistadapter.begin.
>> ts":"1554384503452","id":"62040","ip_src_addr":"192.168.
>> 66.121","timestamp":1484148196104,"ethdst":"0A:00:27:00:00:00","
>> threat.triage.rules.0.name":null,"is_alert":"true","parallelenricher.
>> enrich.begin.ts":"1554384505264","ttl":"64","
>> source.type":"snort","adapter.geoadapter.end.ts":"
>> 1554384503453","ethlen":"0x42","iplen":"53248","adapter.
>> threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
>> 8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
>> aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
>>
>>
>> On Fri, Apr 5, 2019, 11:43 PM Hema malini
>> wrote:
>>
>>> Yes I am getting messages
>>>
>>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> Do you get 10 records output to the CLI when you run the following?
>>>>
>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>>
>>>>
>>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>>> wrote:
>>>>
>>>>> We verified it in Storm ui and in Storm topology logs
>>>>>
>>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>>> michael.miklav...@gmail.com> wrote:
>>>>>
>>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>>
>>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>>> indexing topology . Elastic search indices are not getting created
>>>>>>> though
>>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>>> the elastic search index using template available in metron repo.
>>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>>> neither
>>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>>> toplogy
>>>>>>> logs.We could see the sample log in Metron management ui. How we can
>>>>>>> send
>>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>>> data
>>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>>> metron
>>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>>> is
>>>>>>> required to pass it to alerts –ui
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Hema
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
Re: Snort logs flow issue
Are we missing any configuration? Initially elastic search was down. We
figured out the issue and fixed it .Now elastic search is up . We restarted
metron indexing but still those indices not created. So we created it
manually.Do we have to change any parser configuration . How logs will flow
into metron alerts dashboard and kibana dashboard..what is the required
congratulation
On Fri, Apr 5, 2019, 11:52 PM Hema malini wrote:
> Sample messages flown in indexing topic
> {"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
> 1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
> "08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
> tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
> hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
> geoadapter.begin.ts":"1554384503452","tcpwindow":"
> 0x1F5","parallelenricher.splitter.begin.ts":"
> 1554384505264","threat.triage.rules.0.score":"10","tcpack":"
> 0x836687BD","protocol":"TCP","ip_dst_addr":"192.168.66.1","
> original_string":"01\/11\/17-20:53:16.104984 ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.66.121,8080,192.168.66.1,50183,08:00:
> 27:E8:B0:7A,0A:00:27:00:00:00,0x42,***A,0x8DF34F4B,
> 0x836687BD,,0x1F5,64,0,62040,52,53248","parallelenricher.enrich.end.
> ts":"1554384505342","threat.triage.rules.0.reason":null,"
> tos":"0","adapter.hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
> 62040","ip_src_addr":"192.168.66.121","timestamp":
> 1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
> ":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
> 1554384505264","ttl":"64","source.type":"snort","adapter.
> geoadapter.end.ts":"1554384503453","ethlen":"0x42"
> ,"iplen":"53248","adapter.threatinteladapter.begin.ts":"
> 1554384505264","ip_src_port":"8080","tcpflags":"***A","
> guid":"2f6f3f3c-7739-47fe-aa04-3c62425fbcbf","sig_id":"
> 999158","sig_generator":"1"}
>
>
> On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
>
>> Yes I am getting messages
>>
>> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Do you get 10 records output to the CLI when you run the following?
>>>
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>>
>>>
>>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>>> wrote:
>>>
>>>> We verified it in Storm ui and in Storm topology logs
>>>>
>>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>>> michael.miklav...@gmail.com> wrote:
>>>>
>>>>> How did you validate the logs are making it to the indexing topology?
>>>>>
>>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi
>>>>>> we sent the sample snort logs copied from metron git repo to snort kafka
>>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>>> indexing topology . Elastic search indices are not getting created though
>>>>>> we gave elastic search template install from ambari. So manually created
>>>>>> the elastic search index using template available in metron repo.
>>>>>> Though elastic search index is present , data from indexing toplogy
>>>>>> neither
>>>>>> reached elastic search nor hdfs path .There are no errors in storm
>>>>>> toplogy
>>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>>> data
>>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>>> metron
>>>>>> alerts ui will be helpful . Any configuration from metron management ui
>>>>>> is
>>>>>> required to pass it to alerts –ui
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>>
>>>>>> Hema
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
Re: Snort logs flow issue
Sample messages flown in indexing topic
{"msg":"'snort test alert'","parallelenricher.splitter.end.ts":"
1554384505264","sig_rev":"0","ip_dst_port":"50183","ethsrc":
"08:00:27:E8:B0:7A","threat.triage.rules.0.comment":null,"
tcpseq":"0x8DF34F4B","threat.triage.score":10.0,"dgmlen":"52","adapter.
hostfromjsonlistadapter.end.ts":"1554384503452","adapter.
geoadapter.begin.ts":"1554384503452","tcpwindow":"0x1F5","parallelenricher.
splitter.begin.ts":"1554384505264","threat.triage.
rules.0.score":"10","tcpack":"0x836687BD","protocol":"TCP","
ip_dst_addr":"192.168.66.1","original_string":"01\/11\/17-20:53:16.104984
,1,999158,0,\"'snort test alert'\",TCP,192.168.66.121,
8080,192.168.66.1,50183,08:00:27:E8:B0:7A,0A:00:27:00:00:00,
0x42,***A,0x8DF34F4B,0x836687BD,,0x1F5,64,0,62040,52,53248","
parallelenricher.enrich.end.ts":"1554384505342","threat.
triage.rules.0.reason":null,"tos":"0","adapter.
hostfromjsonlistadapter.begin.ts":"1554384503452","id":"
62040","ip_src_addr":"192.168.66.121","timestamp":
1484148196104,"ethdst":"0A:00:27:00:00:00","threat.triage.rules.0.name
":null,"is_alert":"true","parallelenricher.enrich.begin.ts":"
1554384505264","ttl":"64","source.type":"snort","adapter.
geoadapter.end.ts":"1554384503453","ethlen":"0x42","iplen":"53248","adapter.
threatinteladapter.begin.ts":"1554384505264","ip_src_port":"
8080","tcpflags":"***A","guid":"2f6f3f3c-7739-47fe-
aa04-3c62425fbcbf","sig_id":"999158","sig_generator":"1"}
On Fri, Apr 5, 2019, 11:43 PM Hema malini wrote:
> Yes I am getting messages
>
> On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Do you get 10 records output to the CLI when you run the following?
>>
>> /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
>> $ZOOKEEPER --topic indexing --from-beginning --max-messages 10
>>
>>
>> On Fri, Apr 5, 2019 at 11:38 AM Hema malini
>> wrote:
>>
>>> We verified it in Storm ui and in Storm topology logs
>>>
>>> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>>> How did you validate the logs are making it to the indexing topology?
>>>>
>>>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini
>>>> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we
>>>>> sent the sample snort logs copied from metron git repo to snort kafka
>>>>> topic.We did the same for bro topic.Logs are getting parsed and reached
>>>>> indexing topology . Elastic search indices are not getting created though
>>>>> we gave elastic search template install from ambari. So manually created
>>>>> the elastic search index using template available in metron repo.
>>>>> Though elastic search index is present , data from indexing toplogy
>>>>> neither
>>>>> reached elastic search nor hdfs path .There are no errors in storm toplogy
>>>>> logs.We could see the sample log in Metron management ui. How we can send
>>>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could
>>>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but
>>>>> with no data. Elasticsearch health is yellow and we are able to insert
>>>>> data
>>>>> via rest call. Any documentation on sending the smaple snort logs to
>>>>> metron
>>>>> alerts ui will be helpful . Any configuration from metron management ui is
>>>>> required to pass it to alerts –ui
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Regards
>>>>>
>>>>> Hema
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
Re: Snort logs flow issue
Yes I am getting messages On Fri, Apr 5, 2019, 11:17 PM Michael Miklavcic wrote: > Do you get 10 records output to the CLI when you run the following? > > /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper > $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 > > > On Fri, Apr 5, 2019 at 11:38 AM Hema malini > wrote: > >> We verified it in Storm ui and in Storm topology logs >> >> On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> How did you validate the logs are making it to the indexing topology? >>> >>> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> >>>> >>>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>>> sent the sample snort logs copied from metron git repo to snort kafka >>>> topic.We did the same for bro topic.Logs are getting parsed and reached >>>> indexing topology . Elastic search indices are not getting created though >>>> we gave elastic search template install from ambari. So manually created >>>> the elastic search index using template available in metron repo. >>>> Though elastic search index is present , data from indexing toplogy neither >>>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>>> logs.We could see the sample log in Metron management ui. How we can send >>>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>>> with no data. Elasticsearch health is yellow and we are able to insert data >>>> via rest call. Any documentation on sending the smaple snort logs to metron >>>> alerts ui will be helpful . Any configuration from metron management ui is >>>> required to pass it to alerts –ui >>>> >>>> >>>> >>>> >>>> >>>> Thanks and Regards >>>> >>>> Hema >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>
Re: Snort logs flow issue
Do you get 10 records output to the CLI when you run the following? /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic indexing --from-beginning --max-messages 10 On Fri, Apr 5, 2019 at 11:38 AM Hema malini wrote: > We verified it in Storm ui and in Storm topology logs > > On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> How did you validate the logs are making it to the indexing topology? >> >> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >> wrote: >> >>> >>> Hi, >>> >>> >>> >>> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >>> sent the sample snort logs copied from metron git repo to snort kafka >>> topic.We did the same for bro topic.Logs are getting parsed and reached >>> indexing topology . Elastic search indices are not getting created though >>> we gave elastic search template install from ambari. So manually created >>> the elastic search index using template available in metron repo. >>> Though elastic search index is present , data from indexing toplogy neither >>> reached elastic search nor hdfs path .There are no errors in storm toplogy >>> logs.We could see the sample log in Metron management ui. How we can send >>> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >>> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >>> with no data. Elasticsearch health is yellow and we are able to insert data >>> via rest call. Any documentation on sending the smaple snort logs to metron >>> alerts ui will be helpful . Any configuration from metron management ui is >>> required to pass it to alerts –ui >>> >>> >>> >>> >>> >>> Thanks and Regards >>> >>> Hema >>> >>> >>> >>> >>> >>> >>> >>
Re: Snort logs flow issue
We verified it in Storm ui and in Storm topology logs On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic wrote: > How did you validate the logs are making it to the indexing topology? > > On Fri, Apr 5, 2019 at 8:12 AM Hema malini > wrote: > >> >> Hi, >> >> >> >> We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we >> sent the sample snort logs copied from metron git repo to snort kafka >> topic.We did the same for bro topic.Logs are getting parsed and reached >> indexing topology . Elastic search indices are not getting created though >> we gave elastic search template install from ambari. So manually created >> the elastic search index using template available in metron repo. Though >> elastic search index is present , data from indexing toplogy neither >> reached elastic search nor hdfs path .There are no errors in storm toplogy >> logs.We could see the sample log in Metron management ui. How we can send >> the logs to alerts ui and kibana dashboard. In kibana dashboard we could >> see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but >> with no data. Elasticsearch health is yellow and we are able to insert data >> via rest call. Any documentation on sending the smaple snort logs to metron >> alerts ui will be helpful . Any configuration from metron management ui is >> required to pass it to alerts –ui >> >> >> >> >> >> Thanks and Regards >> >> Hema >> >> >> >> >> >> >> >
Re: Snort logs flow issue
How did you validate the logs are making it to the indexing topology? On Fri, Apr 5, 2019 at 8:12 AM Hema malini wrote: > > Hi, > > > > We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we > sent the sample snort logs copied from metron git repo to snort kafka > topic.We did the same for bro topic.Logs are getting parsed and reached > indexing topology . Elastic search indices are not getting created though > we gave elastic search template install from ambari. So manually created > the elastic search index using template available in metron repo. Though > elastic search index is present , data from indexing toplogy neither > reached elastic search nor hdfs path .There are no errors in storm toplogy > logs.We could see the sample log in Metron management ui. How we can send > the logs to alerts ui and kibana dashboard. In kibana dashboard we could > see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but > with no data. Elasticsearch health is yellow and we are able to insert data > via rest call. Any documentation on sending the smaple snort logs to metron > alerts ui will be helpful . Any configuration from metron management ui is > required to pass it to alerts –ui > > > > > > Thanks and Regards > > Hema > > > > > > >
Snort logs flow issue
Hi, We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we sent the sample snort logs copied from metron git repo to snort kafka topic.We did the same for bro topic.Logs are getting parsed and reached indexing topology . Elastic search indices are not getting created though we gave elastic search template install from ambari. So manually created the elastic search index using template available in metron repo. Though elastic search index is present , data from indexing toplogy neither reached elastic search nor hdfs path .There are no errors in storm toplogy logs.We could see the sample log in Metron management ui. How we can send the logs to alerts ui and kibana dashboard. In kibana dashboard we could see two dashboards - Metron-Dashboard,Metron-Error-Dashboard created but with no data. Elasticsearch health is yellow and we are able to insert data via rest call. Any documentation on sending the smaple snort logs to metron alerts ui will be helpful . Any configuration from metron management ui is required to pass it to alerts –ui Thanks and Regards Hema
Re: Parser Error while Snort IDS usage
Thanks, I resolved it by allowing the config show_year in snort.conf file
and the snort parser date format configurations.
On Thu, Jun 28, 2018 at 7:06 PM, Otto Fowler
wrote:
> Forgot to put the default format in. It is : private static String
> defaultDateFormat
> = "MM/dd/yy-HH:mm:ss.SS";
>
>
> On June 28, 2018 at 10:06:08, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> The snort parser by default supports dates in the following format:
>
> Your dates are missing the ‘yy’. If I add that, your failing message
> parses:
>
> /**
> 06/28/18-02:06:18.667820 ,1,384,5,"ICMP
> PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
> */
> @Multiline
> public static String userMessage;
>
>
> @Test
> public void testUserIssue() {
> BasicSnortParser parser = new BasicSnortParser();
> parser.configure(new HashMap<>());
> List msgs = parser.parse(userMessage.getBytes());
> Assert.assertTrue(msgs != null && msgs.isEmpty() == false);
> }
>
>
> You need to configure the snort processor’s dateFormat
>
> https://metron.apache.org/current-book/metron-platform/
> metron-parsers/index.html shows a sample. See ‘Sample configuration for
> a sensor’.
>
>
> On June 28, 2018 at 02:16:42, Farrukh Naveed Anjum (
> anjum.farr...@gmail.com) wrote:
>
> Hi,
>
> I am getting following errors when I am using snort in IDS mode.
>
> java.lang.IllegalStateException: Unable to parse message:
> 06/28-02:06:18.667820 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1
> at
> org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180)
> at org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:177) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.time.format.DateTimeParseException: Text
> '06/28-02:06:18.667820' could not be parsed at index 5 at java.time.format.
> DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at
> java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at
> java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at
> org.apache.metron.parsers.snort.BasicSnortParser.
> toEpoch(BasicSnortParser.java:194) at org.apache.metron.parsers.
> snort.BasicSnortParser.parse(BasicSnortParser.java:165) ... 12 more
>
>
> Following is the data i am getting in alerts.csv
>
> 06/28-02:00:39.145636 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 11,8,1,32,32768,8,0,12590,1
> 06/28-02:00:39.145690 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,40061,32,32768,0,0,12590,1
> 06/28-02:00:49.949974 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 10,8,1,32,32768,8,0,13210,1
> 06/28-02:00:49.950011 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,41071,32,32768,0,0,13210,1
> 06/28-02:01:00.534199 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 11,8,1,32,32768,8,0,55879,1
> 06/28-02:01:00.534224 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,43938,32,32768,0,0,55879,1
> 06/28-02:01:02.185767 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.
> 69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,
> 10,8,1,32,32768,8,0,4364,1
> 06/28-02:01:02.185812 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,
> 92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,
> ,64,8,41648,32,32768,0
Parser Error while Snort IDS usage
Hi, I am getting following errors when I am using snort in IDS mode. java.lang.IllegalStateException: Unable to parse message: 06/28-02:06:18.667820 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,21914,1 at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:180) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.time.format.DateTimeParseException: Text '06/28-02:06:18.667820' could not be parsed at index 5 at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at java.time.ZonedDateTime.parse(ZonedDateTime.java:597) at org.apache.metron.parsers.snort.BasicSnortParser.toEpoch(BasicSnortParser.java:194) at org.apache.metron.parsers.snort.BasicSnortParser.parse(BasicSnortParser.java:165) ... 12 more Following is the data i am getting in alerts.csv 06/28-02:00:39.145636 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,12590,1 06/28-02:00:39.145690 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,40061,32,32768,0,0,12590,1 06/28-02:00:49.949974 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,13210,1 06/28-02:00:49.950011 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41071,32,32768,0,0,13210,1 06/28-02:01:00.534199 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,55879,1 06/28-02:01:00.534224 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,43938,32,32768,0,0,55879,1 06/28-02:01:02.185767 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,4364,1 06/28-02:01:02.185812 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.185.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,41648,32,32768,0,0,4364,1 06/28-02:01:03.946563 ,1,384,5,"ICMP PING",ICMP,37.187.231.251,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x5E,,61,0,0,80,81920,8,0,20112,56 06/28-02:01:03.946596 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,37.187.231.251,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x5E,,64,0,38270,80,81920,0,0,20112,56 06/28-02:01:05.015592 ,1,384,5,"ICMP PING",ICMP,92.222.186.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,10,8,1,32,32768,8,0,55637,1 06/28-02:01:05.015640 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.186.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,29998,32,32768,0,0,55637,1 06/28-02:01:08.820637 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,12,8,1,32,32768,8,0,10717,1 06/28-02:01:08.820684 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,167.114.37.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,51338,32,32768,0,0,10717,1 06/28-02:01:16.702204 ,1,384,5,"ICMP PING",ICMP,167.114.37.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,12,8,1,32,32768,8,0,23094,1 06/28-02:01:16.702256 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,167.114.37.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,51904,32,32768,0,0,23094,1 06/28-02:01:18.322369 ,1,384,5,"ICMP PING",ICMP,92.222.184.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,,11,8,1,32,32768,8,0,26763,1 06/28-02:01:18.322409 ,1,408,5,"ICMP Echo Reply",ICMP,158.69.118.104,,92.222.184.1,,0C:C4:7A:79:83:7C,00:FF:FF:FF:FF:FF,0x2E,,64,8,46352,32,32768,0,0,26763,1 06/28-02:01:20.123553 ,1,384,5,"ICMP PING",ICMP,92.222.185.1,,158.69.118.104,,00:FF:FF:FF:FF:FD,0C:C4:7A:79:83:7C,0x3C,
Re: Snort enrichment issue
ANd I dint load anything. It was supposed to be loaded during installation? My installation is ambari based single node VM install on ubuntu host. On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Here you go, the error part of the log is in the attachment. > > On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Did you setup and load the geo enrichment database? https://metron.apach >> e.org/current-book/metron-platform/metron-data-managemen >> t/index.html#GeoLite2_Loader >> >> Also, we can’t really see the error from screenshots, please send log >> entries. >> >> Simon >> >> On 17 Nov 2017, at 07:11, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: >> >> Hi all, I am starting it again. Last one got a bit messy >> >> Ok, Now I have started everything again from scratch (redeployed single >> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and >> now when I execute this command: >> >> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date >> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> (format of ths command was taken from: https://github.com/apach >> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te >> mplates/start-snort-stub) >> >> I get this under enrichment storm topology : >> >> >> >> >> >> I have come this far, please help me push these dummy preformatted snort >> logs into kibana dashboard. >> >> Regards. >> >> >> >
Re: Snort enrichment issue
Here you go, the error part of the log is in the attachment. On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Did you setup and load the geo enrichment database? https://metron. > apache.org/current-book/metron-platform/metron-data-management/index.html# > GeoLite2_Loader > > Also, we can’t really see the error from screenshots, please send log > entries. > > Simon > > On 17 Nov 2017, at 07:11, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > > Hi all, I am starting it again. Last one got a bit messy > > Ok, Now I have started everything again from scratch (redeployed single > node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and > now when I execute this command: > > shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date > +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | > /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh > --broker-list node1:6667 --topic snort > > (format of ths command was taken from: https://github.com/apach > e/metron/blob/master/metron-deployment/roles/sensor-stubs/ > templates/start-snort-stub) > > I get this under enrichment storm topology : > > > > > > I have come this far, please help me push these dummy preformatted snort > logs into kibana dashboard. > > Regards. > > > Enrichment Description: Binary data
Re: Snort enrichment issue
Did you setup and load the geo enrichment database? https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader <https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader> Also, we can’t really see the error from screenshots, please send log entries. Simon > On 17 Nov 2017, at 07:11, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > > Hi all, I am starting it again. Last one got a bit messy > > Ok, Now I have started everything again from scratch (redeployed single node > based ambari metron cluster with ansibleSkipTags = 'quick-dev') and now when > I execute this command: > > shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date > +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | > /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list > node1:6667 --topic snort > > (format of ths command was taken from: > https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub > > <https://github.com/apache/metron/blob/master/metron-deployment/roles/sensor-stubs/templates/start-snort-stub>) > > I get this under enrichment storm topology : > > > > > > I have come this far, please help me push these dummy preformatted snort logs > into kibana dashboard. > > Regards. >
Re: Snort Logs
ok, Doing it. On Mon, Nov 13, 2017 at 3:07 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > Can you restart storm and give it another shot? > > Jon > > On Mon, Nov 13, 2017, 00:30 Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> hi, This problem still persists guys . >> >> On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> Any solution to these issues guys? >>> >>> On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> >>> wrote: >>> >>>> I have attached the output of this dump >>>> >>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>> >>>> >>>> >>>> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com <zeo...@gmail.com> >>>> wrote: >>>> >>>>> What is the output of: >>>>> >>>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>>> >>>>> ? >>>>> >>>>> Jon >>>>> >>>>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>> wrote: >>>>> >>>>>> This is the script/command i used >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> sudo cat snort.out | >>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>> --broker-list node1:6667 --topic snort >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler < >>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>> >>>>>>>> What topic? what are the parameters you are calling the script >>>>>>>> with? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>> >>>>>>>> The metron installation I have (single node based vm install) comes >>>>>>>> with sensor stubs. I assume that everything has already been done for >>>>>>>> those >>>>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>>>> directly pushing the preformatted canned data to kafka topic. I can >>>>>>>> see the >>>>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella <ceste...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> How did you start the snort parser topology and what's the parser >>>>>>>>> config (in zookeeper)? >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> This is what I am doing >>>>>>>>>> >>>>>>>>>> sudo cat snort.out | >>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella <ceste...@gmail.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>>>> records in >>>>>>>>>>> kafka that are not JSON. By the time it gets to the i
Re: Snort Logs
hi, This problem still persists guys . On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Any solution to these issues guys? > > On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> I have attached the output of this dump >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> >> >> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >>> What is the output of: >>> >>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>> >>> ? >>> >>> Jon >>> >>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir <mscs16...@itu.edu.pk> >>> wrote: >>> >>>> This is the script/command i used >>>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> sudo cat snort.out | >>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>> --broker-list node1:6667 --topic snort >>>>> >>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler <ottobackwa...@gmail.com> >>>>> wrote: >>>>> >>>>>> What topic? what are the parameters you are calling the script with? >>>>>> >>>>>> >>>>>> >>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>> mscs16...@itu.edu.pk) wrote: >>>>>> >>>>>> The metron installation I have (single node based vm install) comes >>>>>> with sensor stubs. I assume that everything has already been done for >>>>>> those >>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>>> the >>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella <ceste...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> How did you start the snort parser topology and what's the parser >>>>>>> config (in zookeeper)? >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> This is what I am doing >>>>>>>> >>>>>>>> sudo cat snort.out | >>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella <ceste...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>> records in >>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>> topic, >>>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>>> then >>>>>>>>> the enrichments topology enrich that map and emits the enriched map >>>>>>>>> to the >>>>>>>>> indexing topic. >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>>> full stack trace >>>>>>>>>> >>>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>>>
Re: Snort Logs
I have attached the output of this dump /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com <zeo...@gmail.com> wrote: > What is the output of: > > /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP > > ? > > Jon > > On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> This is the script/command i used >> >> sudo cat snort.out | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> sudo cat snort.out | >>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>> --broker-list node1:6667 --topic snort >>> >>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>>> What topic? what are the parameters you are calling the script with? >>>> >>>> >>>> >>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> The metron installation I have (single node based vm install) comes >>>> with sensor stubs. I assume that everything has already been done for those >>>> stub sensors to push the canned data. I am doing the similar thing, >>>> directly pushing the preformatted canned data to kafka topic. I can see the >>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>> push the same logs myself, those errors pop that I have shown earlier. >>>> >>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella <ceste...@gmail.com> >>>> wrote: >>>> >>>>> How did you start the snort parser topology and what's the parser >>>>> config (in zookeeper)? >>>>> >>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> This is what I am doing >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> >>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella <ceste...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>> parser or from some other source? It looks like there are some records >>>>>>> in >>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>> topic, >>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>> then >>>>>>> the enrichments topology enrich that map and emits the enriched map to >>>>>>> the >>>>>>> indexing topic. >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>> full stack trace >>>>>>>> >>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 1] >>>>>>>> >>>>>>>> from indexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 2] >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>> >>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>> Also, are you saying that you are no longer seeing the parser >>>>>>>>> topology error? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella (ceste...@gmail.com) >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>
Re: Snort Logs
hi, I am back at work. lets see if i can find something in logs On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > It looks like your ES cluster has a health of Red, so there's your > problem. I would go look in /var/log/elasticsearch/ at some logs. > > Jon > > On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> >> -- Forwarded message -- >> From: Syed Hammad Tahir <mscs16...@itu.edu.pk> >> Date: Fri, Nov 3, 2017 at 5:07 PM >> Subject: Re: Snort Logs >> To: Otto Fowler <ottobackwa...@gmail.com> >> >> >> NVM, I have installed the elastic search head. Now where do I go in this >> to find out why I cant see the snort logs in kibana dashboard, pushed to >> snort topic via kafka producer? >> >> [image: Inline image 1] >> >> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> You can install it into the chrome web browser from the play store. >>> >>> >>> >>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >>> wrote: >>> >>> And how do I install elasticsearch head on the vagrant VM? >>> >>> >> -- > > Jon >
Re: Snort Logs
You can install it into the chrome web browser from the play store. On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) wrote: And how do I install elasticsearch head on the vagrant VM?
Re: Snort Logs
And how do I install elasticsearch head on the vagrant VM?
Re: Snort Logs
They need to meet the format of the logs I sent earlier. Look into the snort output options - may require you rerun snort, depending on your situation Jon On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Yes, I have converted them to text but those logs are simply captured > packet headers over the local network. Now I just push them via that kafka > producer command under topic name of snort and they will be visible in > metron? > > On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > >> You need text logs. Here's an example of some properly formatted logs - >> https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out >> >> Jon >> >> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> I have found the kafka-console-producer.sh but I need to know how to >>> make it read snort.log (tcp dump format) file. May be I am missing >>> something in the plain sight but it would be awsome if you tell me that. >>> >>> Regards. >>> >>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com <zeo...@gmail.com> >>> wrote: >>> >>>> On the 25th I said: >>>> >>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>> (from memory) on node1, assuming you are running full dev. >>>> >>>> Jon >>>> >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>> wrote: >>>> >>>>> snort logs are in tcp dump format. I may have to convert them. >>>>> >>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>>> >>>>> How to give file name or path in this command? >>>>> >>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com <zeo...@gmail.com> >>>>> wrote: >>>>> >>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>> snort topic. You may also want to look at this [2]. >>>>>> >>>>>> 1: https://kafka.apache.org/quickstart >>>>>> 2: >>>>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>>>> >>>>>> Jon >>>>>> >>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>>> wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>>> bit. >>>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>> Please >>>>>>> give me an easy to understand this solution for this problem. >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>> -- >>>>>> >>>>>> Jon >>>>>> >>>>> >>>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Logs
Yes, I have converted them to text but those logs are simply captured packet headers over the local network. Now I just push them via that kafka producer command under topic name of snort and they will be visible in metron? On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > You need text logs. Here's an example of some properly formatted logs - > https://raw.githubusercontent.com/apache/metron/master/metron- > deployment/roles/sensor-stubs/files/snort.out > > Jon > > On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> I have found the kafka-console-producer.sh but I need to know how to >> make it read snort.log (tcp dump format) file. May be I am missing >> something in the plain sight but it would be awsome if you tell me that. >> >> Regards. >> >> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >>> On the 25th I said: >>> >>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>> (from memory) on node1, assuming you are running full dev. >>> >>> Jon >>> >>> >>> Jon >>> >>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir <mscs16...@itu.edu.pk> >>> wrote: >>> >>>> snort logs are in tcp dump format. I may have to convert them. >>>> >>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>> >>>> How to give file name or path in this command? >>>> >>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com <zeo...@gmail.com> >>>> wrote: >>>> >>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>> snort topic. You may also want to look at this [2]. >>>>> >>>>> 1: https://kafka.apache.org/quickstart >>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>> console-producer-and-bash-script >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>> bit. >>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>> Please >>>>>> give me an easy to understand this solution for this problem. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
You need text logs. Here's an example of some properly formatted logs - https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/sensor-stubs/files/snort.out Jon On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > I have found the kafka-console-producer.sh but I need to know how to make > it read snort.log (tcp dump format) file. May be I am missing something in > the plain sight but it would be awsome if you tell me that. > > Regards. > > On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > >> On the 25th I said: >> >> It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from >> memory) on node1, assuming you are running full dev. >> >> Jon >> >> >> Jon >> >> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> snort logs are in tcp dump format. I may have to convert them. >>> >>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>> >>> How to give file name or path in this command? >>> >>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com <zeo...@gmail.com> >>> wrote: >>> >>>> If you have text snort logs you can use Apache nifi or the Kafka >>>> producer script as described in step 4 here[1] to push them to Metron's >>>> snort topic. You may also want to look at this [2]. >>>> >>>> 1: https://kafka.apache.org/quickstart >>>> 2: >>>> https://stackoverflow.com/questions/38701179/kafka-console-producer-and-bash-script >>>> >>>> Jon >>>> >>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>> bit. >>>>> Some of you already replied me with some solutions but thats lost in the >>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>>> give me an easy to understand this solution for this problem. >>>>> >>>>> Regards. >>>>> >>>> -- >>>> >>>> Jon >>>> >>> >>> -- >> >> Jon >> > > -- Jon
Re: Snort Installation
All I did was install snort separately on vagrant ssh console. The ran it to collect logs. Now I need to bring those logs to metron. On Wed, Oct 25, 2017 at 9:50 AM, Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > Hi Syed Hammed, > > Can you share the steps how did you connected snort with external source ? > (Metron Snort ?) > > On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen <n...@nickallen.org> wrote: > >> Take a look at `kafka-console-producer.sh`, which is installed as part of >> Kafka. >> >> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> Ok, I have fixed everything on my own. Now that I have snort logs saved >>> in a file, I need to get them to metron. Can anyone help me on that? >>> >>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk >>> > wrote: >>> >>>> yes nut I am a bit confused here. Let me ask them as well then. >>>> >>>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com <zeo...@gmail.com> >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> Just to clarify, this a snort issue you are having? If so I suggest >>>>> looking at their documentation (https://snort.org/documents) or >>>>> reaching out to their community (https://snort.org/community), as >>>>> they have more expertise in this area. >>>>> >>>>> Jon >>>>> >>>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I tried to add another network interface in order to bridge it to >>>>>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up >>>>>> after that, there was no bridged interface. Can anyone help me on this? >>>>>> >>>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>>> >>>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets <laur...@daemon.be> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> See inline. >>>>>>>> >>>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>>> >>>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>>> traffic? >>>>>>>>> >>>>>>>> >>>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>>> of the project. Have a look at the documentation at >>>>>>>> https://www.snort.org/. >>>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>>> some >>>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>>> >>>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>>> preprocessing >>>>>>>>> >>>>>>>> >>>>>>>> Have a look at the Metron documentation at >>>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>>> >>>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>>> >>>>>>>> >>>>>>>> I can't help you with this :) >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> >>> >> > > > -- > With Regards > Farrukh Naveed Anjum >
Re: Snort Installation
Take a look at `kafka-console-producer.sh`, which is installed as part of Kafka. On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Ok, I have fixed everything on my own. Now that I have snort logs saved in > a file, I need to get them to metron. Can anyone help me on that? > > On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> yes nut I am a bit confused here. Let me ask them as well then. >> >> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >>> Hi Syed, >>> >>> Just to clarify, this a snort issue you are having? If so I suggest >>> looking at their documentation (https://snort.org/documents) or >>> reaching out to their community (https://snort.org/community), as they >>> have more expertise in this area. >>> >>> Jon >>> >>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>> wrote: >>> >>>> Hi guys, >>>> >>>> I tried to add another network interface in order to bridge it to LAN. >>>> I tried to do it on virtualbox vm settings and when i did vagrant up after >>>> that, there was no bridged interface. Can anyone help me on this? >>>> >>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>>>> in the given configuration, might be helpful for others. I will then try >>>>> to >>>>> do that kafka topic and will ask if any help is needed. >>>>> >>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets <laur...@daemon.be> >>>>> wrote: >>>>> >>>>>> Hi Syed, >>>>>> >>>>>> See inline. >>>>>> >>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>> >>>>>>> I have installed the snort manually. Now I need help with : >>>>>>> >>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>>>> >>>>>> >>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>> of the project. Have a look at the documentation at >>>>>> https://www.snort.org/. >>>>>> You will probably have to add a 2nd network interface bridged to your >>>>>> LAN in promiscuous mode. Additionally, I think most of us expect some >>>>>> basic >>>>>> Linux & network administration knowledge when using Metron. >>>>>> >>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>> preprocessing >>>>>>> >>>>>> >>>>>> Have a look at the Metron documentation at >>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>> >>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>> >>>>>> >>>>>> I can't help you with this :) >>>>>> >>>>> >>>>> >>>> -- >>> >>> Jon >>> >> >> >
Re: Snort Installation
yes nut I am a bit confused here. Let me ask them as well then. On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > Hi Syed, > > Just to clarify, this a snort issue you are having? If so I suggest > looking at their documentation (https://snort.org/documents) or reaching > out to their community (https://snort.org/community), as they have more > expertise in this area. > > Jon > > On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> Hi guys, >> >> I tried to add another network interface in order to bridge it to LAN. I >> tried to do it on virtualbox vm settings and when i did vagrant up after >> that, there was no bridged interface. Can anyone help me on this? >> >> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk >> > wrote: >> >>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>> in the given configuration, might be helpful for others. I will then try to >>> do that kafka topic and will ask if any help is needed. >>> >>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets <laur...@daemon.be> wrote: >>> >>>> Hi Syed, >>>> >>>> See inline. >>>> >>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>> >>>>> I have installed the snort manually. Now I need help with : >>>>> >>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>> >>>> >>>> To be honest, configuring Snort to work on your LAN is out of scope of >>>> the project. Have a look at the documentation at https://www.snort.org/ >>>> . >>>> You will probably have to add a 2nd network interface bridged to your >>>> LAN in promiscuous mode. Additionally, I think most of us expect some basic >>>> Linux & network administration knowledge when using Metron. >>>> >>>> 2- Making a kafka topic to push those saved logs in metron for >>>>> preprocessing >>>>> >>>> >>>> Have a look at the Metron documentation at https://metron.apache.org/ >>>> current-book/index.html. Adding a new sensor in the Metron UI will >>>> create the Kafka iirc. >>>> >>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>> >>>> >>>> I can't help you with this :) >>>> >>> >>> >> -- > > Jon >
Re: Snort Installation
Hi Syed, Just to clarify, this a snort issue you are having? If so I suggest looking at their documentation (https://snort.org/documents) or reaching out to their community (https://snort.org/community), as they have more expertise in this area. Jon On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Hi guys, > > I tried to add another network interface in order to bridge it to LAN. I > tried to do it on virtualbox vm settings and when i did vagrant up after > that, there was no bridged interface. Can anyone help me on this? > > On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> Ok, thankyou. I will let you know once I make snort sniff the traffic in >> the given configuration, might be helpful for others. I will then try to do >> that kafka topic and will ask if any help is needed. >> >> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets <laur...@daemon.be> wrote: >> >>> Hi Syed, >>> >>> See inline. >>> >>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>> >>>> I have installed the snort manually. Now I need help with : >>>> >>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>> >>> >>> To be honest, configuring Snort to work on your LAN is out of scope of >>> the project. Have a look at the documentation at https://www.snort.org/. >>> You will probably have to add a 2nd network interface bridged to your >>> LAN in promiscuous mode. Additionally, I think most of us expect some basic >>> Linux & network administration knowledge when using Metron. >>> >>> 2- Making a kafka topic to push those saved logs in metron for >>>> preprocessing >>>> >>> >>> Have a look at the Metron documentation at >>> https://metron.apache.org/current-book/index.html. Adding a new sensor >>> in the Metron UI will create the Kafka iirc. >>> >>> 3- Applying a basic Machine learning algorithm on the captured data. >>>> >>> >>> I can't help you with this :) >>> >> >> > -- Jon
Re: Snort Installation
Hi guys, I tried to add another network interface in order to bridge it to LAN. I tried to do it on virtualbox vm settings and when i did vagrant up after that, there was no bridged interface. Can anyone help me on this? On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Ok, thankyou. I will let you know once I make snort sniff the traffic in > the given configuration, might be helpful for others. I will then try to do > that kafka topic and will ask if any help is needed. > > On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets <laur...@daemon.be> wrote: > >> Hi Syed, >> >> See inline. >> >> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >> >>> I have installed the snort manually. Now I need help with : >>> >>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see >>> the traffic outside vagrant vm, how do I make it see that traffic? >>> >> >> To be honest, configuring Snort to work on your LAN is out of scope of >> the project. Have a look at the documentation at https://www.snort.org/. >> You will probably have to add a 2nd network interface bridged to your LAN >> in promiscuous mode. Additionally, I think most of us expect some basic >> Linux & network administration knowledge when using Metron. >> >> 2- Making a kafka topic to push those saved logs in metron for >>> preprocessing >>> >> >> Have a look at the Metron documentation at https://metron.apache.org/curr >> ent-book/index.html. Adding a new sensor in the Metron UI will create >> the Kafka iirc. >> >> 3- Applying a basic Machine learning algorithm on the captured data. >>> >> >> I can't help you with this :) >> > >
Re: Snort
Should I edit the vagrant file using text editor and what exactly should I edit there? On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > I would recommend just using a text editor if you’re not familiar with > sed. To solve your sed problem… > > sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" > metron-deployment/vagrant/full-dev-platform/Vagrantfile > > sed -i means run the sed command (in this case a find replace) inplace on > the file, the text following the -i is the name to append to a backup > version (ie the original file unchanged). > > Metron does tend to assume a good knowledge of linux admin, you’ll find we > have a lot of shell gurus in the community, but if you’re struggling with > this, maybe a simple text editor would be easier. All you’re trying to do > here is change a config value. > > Simon > > On 19 Oct 2017, at 11:46, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > > Ran it without -i swtich, gives this: > > > > On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > >> The sed command is falling. It's written for a Mac so it will need an >> alteration to be portable. Run it without the '' after -i, from >> ~/metron-master >> >> Jon >> >> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> I did what this guide said to install the original sensor: >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> Still didnt work. How do I install snort into this? >>> >>> >>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Maybe I did something wrong >>>> >>>> >>>> >>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>> >>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com <zeo...@gmail.com> >>>>> wrote: >>>>> >>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>> set up snort for you. I have a sed one liner in my bro security patch pr >>>>>> to do this, just need to do it before vagrant up. >>>>>> >>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>> vagrant up >>>>>> >>>>>> Jon >>>>>> >>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir <mscs16...@itu.edu.pk> >>>>>> wrote: >>>>>> >>>>>>> I followed this guide exactly: https://cwiki.apache >>>>>>> .org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>> >>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>> returns error of not able to find the snort command. >>>>>>> >>>>>>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets <laur...@daemon.be> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> I was under the impression that you installed the full-dev >>>>>>>> environment? If so, snort should already be installed... >>>>>>>> >>>>>>>> On 2017-10-18 09:45, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>> It has become a mess. Apparently snort is released for centos 7 >>>>>>>> whereas metron one is centos 6.8. Whenever I try to install snort it >>>>>>>> gives >>>>>>>> me this: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen <n...@nickallen.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just use those as a guide to run the commands yourself. >>>>>>
Re: Snort
I am so noob in all of this. I am using full-dev vm metron install to do my research. So I have 2 options to install snort: as per my understanding 1- Install it in a usual way (like that on a regular linux machine) and then make its kafka topic 2- Use ansible role to do all of that. Read the content of those yml files given in main.yml to understand the procedure? Which one do you suggest? On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen <n...@nickallen.org> wrote: > No special commands. Install and configure Snort however you like and get > those logs into a Kafka topic. Metron is completely agnostic to how sensor > telemetry lands in Kafka. > > We also have an Ansible role that will install Snort along with a simple > mechanism to transport its logs to Kafka. This is only useful for > development environments; not a production install. > > Using the Ansible role directly may be beyond the knowledge level of > some. I only offer this as a guide that you can use to follow along and > manually install it yourself. > > https://github.com/apache/metron/blob/master/metron- > deployment/roles/snort/tasks/main.yml > > > If you are not familiar with how Ansible roles are defined, just start at > the main.yml, then follow through each of the other files as they are > included. It is pretty readable once you get use to the layout. > > On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal >> way snort is usually install on a linux distro or do I need to run some >> special commands again? >> >> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen <n...@nickallen.org> wrote: >> >>> In the Full Dev environment, Snort is not installed. We install "Sensor >>> Stubs" which is just a mechanism that continually replays canned telemetry >>> logs repetitively to mimic real sensors. We have to do this because of >>> resource constraints when running all of Metron on a single VM. See the >>> following for more information. >>> >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> >>> >>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >>>> installed where as it can be seen working in metron. Due to that reason I >>>> am confused because James Sirota said to install snort. >>>> >>>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen <n...@nickallen.org> wrote: >>>> >>>>> From Metron's perspective, Snort is just another sensor. Snort is >>>>> installed, managed and executed completely independent of Metron itself. >>>>> As >>>>> with any sensor, you are responsible for getting the telemetry produced by >>>>> Snort into Kafka. Metron can then consume that telemetry from Kafka and >>>>> do >>>>> wonderful things with it. :) >>>>> >>>>> >>>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> And I am sorry about one confusion but isnt snort builtin into the >>>>>> metron framework? If so then cant we access that snort and do the tasks >>>>>> you >>>>>> mentioned earlier? >>>>>> >>>>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for the support. Can it be performed both on dumped log and >>>>>>> real time data? >>>>>>> Regards. >>>>>>> >>>>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> What I mean is that you should install snort, load the appropriate >>>>>>>> Snort rules for your use case, set Snort to log to a directory, and >>>>>>>> send >>>>>>>> traffic to the network interface where Snort is listening. That will >>>>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities
Re: Snort
No special commands. Install and configure Snort however you like and get those logs into a Kafka topic. Metron is completely agnostic to how sensor telemetry lands in Kafka. We also have an Ansible role that will install Snort along with a simple mechanism to transport its logs to Kafka. This is only useful for development environments; not a production install. Using the Ansible role directly may be beyond the knowledge level of some. I only offer this as a guide that you can use to follow along and manually install it yourself. https://github.com/apache/metron/blob/master/metron-deployment/roles/snort/tasks/main.yml If you are not familiar with how Ansible roles are defined, just start at the main.yml, then follow through each of the other files as they are included. It is pretty readable once you get use to the layout. On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Ok, Now I get it. Now should I install snort in vagrant ssh in the normal > way snort is usually install on a linux distro or do I need to run some > special commands again? > > On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen <n...@nickallen.org> wrote: > >> In the Full Dev environment, Snort is not installed. We install "Sensor >> Stubs" which is just a mechanism that continually replays canned telemetry >> logs repetitively to mimic real sensors. We have to do this because of >> resource constraints when running all of Metron on a single VM. See the >> following for more information. >> >> https://github.com/apache/metron/tree/master/metron-deployme >> nt/roles/sensor-stubs >> >> >> >> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk >> > wrote: >> >>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >>> installed where as it can be seen working in metron. Due to that reason I >>> am confused because James Sirota said to install snort. >>> >>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen <n...@nickallen.org> wrote: >>> >>>> From Metron's perspective, Snort is just another sensor. Snort is >>>> installed, managed and executed completely independent of Metron itself. As >>>> with any sensor, you are responsible for getting the telemetry produced by >>>> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >>>> wonderful things with it. :) >>>> >>>> >>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> And I am sorry about one confusion but isnt snort builtin into the >>>>> metron framework? If so then cant we access that snort and do the tasks >>>>> you >>>>> mentioned earlier? >>>>> >>>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks for the support. Can it be performed both on dumped log and >>>>>> real time data? >>>>>> Regards. >>>>>> >>>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> What I mean is that you should install snort, load the appropriate >>>>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>>>> traffic to the network interface where Snort is listening. That will >>>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>>> command line producer. This should be pushed to a Kafka topic called >>>>>>> Snort >>>>>>> where each message is a log line of the Snort file. Does that make >>>>>>> sense? >>>>>>> >>>>>>> Thanks, >>>>>>> James >>>>>>> >>>>>>> >>>>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>>>> >>>>>>> You mean that I must start snort from terminal by doing snort -v and >>>>>>> then push it to kafka topic? I need to start snort in packet capture >>>>>>> mode. >>>>>>> >>>>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>>>> the box. You have to setup Snort on your own and push the output into a >>>>>>> kafka topic (most likely using NiFi). From there on you can use the >>>>>>> output >>>>>>> of Snort in Metron. >>>>>>> >>>>>>> >>>>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Can I use snort in packet capture mode with metron? By default it >>>>>>> works in IDS mode only. >>>>>>> >>>>>>> Regards. >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> Thank you, >>>>>>> >>>>>>> James Sirota >>>>>>> PMC- Apache Metron >>>>>>> jsirota AT apache DOT org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> Thank you, >>>>>>> >>>>>>> James Sirota >>>>>>> PMC- Apache Metron >>>>>>> jsirota AT apache DOT org >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Snort
Ok, Now I get it. Now should I install snort in vagrant ssh in the normal way snort is usually install on a linux distro or do I need to run some special commands again? On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen <n...@nickallen.org> wrote: > In the Full Dev environment, Snort is not installed. We install "Sensor > Stubs" which is just a mechanism that continually replays canned telemetry > logs repetitively to mimic real sensors. We have to do this because of > resource constraints when running all of Metron on a single VM. See the > following for more information. > > https://github.com/apache/metron/tree/master/metron- > deployment/roles/sensor-stubs > > > > On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >> installed where as it can be seen working in metron. Due to that reason I >> am confused because James Sirota said to install snort. >> >> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen <n...@nickallen.org> wrote: >> >>> From Metron's perspective, Snort is just another sensor. Snort is >>> installed, managed and executed completely independent of Metron itself. As >>> with any sensor, you are responsible for getting the telemetry produced by >>> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >>> wonderful things with it. :) >>> >>> >>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk >>> > wrote: >>> >>>> And I am sorry about one confusion but isnt snort builtin into the >>>> metron framework? If so then cant we access that snort and do the tasks you >>>> mentioned earlier? >>>> >>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Hi, >>>>> >>>>> Thanks for the support. Can it be performed both on dumped log and >>>>> real time data? >>>>> Regards. >>>>> >>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> >>>>> wrote: >>>>> >>>>>> What I mean is that you should install snort, load the appropriate >>>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>>> traffic to the network interface where Snort is listening. That will >>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>> command line producer. This should be pushed to a Kafka topic called >>>>>> Snort >>>>>> where each message is a log line of the Snort file. Does that make sense? >>>>>> >>>>>> Thanks, >>>>>> James >>>>>> >>>>>> >>>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>>> >>>>>> You mean that I must start snort from terminal by doing snort -v and >>>>>> then push it to kafka topic? I need to start snort in packet capture >>>>>> mode. >>>>>> >>>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> >>>>>> wrote: >>>>>> >>>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>>> the box. You have to setup Snort on your own and push the output into a >>>>>> kafka topic (most likely using NiFi). From there on you can use the >>>>>> output >>>>>> of Snort in Metron. >>>>>> >>>>>> >>>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Can I use snort in packet capture mode with metron? By default it >>>>>> works in IDS mode only. >>>>>> >>>>>> Regards. >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Snort
In the Full Dev environment, Snort is not installed. We install "Sensor Stubs" which is just a mechanism that continually replays canned telemetry logs repetitively to mimic real sensors. We have to do this because of resource constraints when running all of Metron on a single VM. See the following for more information. https://github.com/apache/metron/tree/master/metron-deployment/roles/sensor-stubs On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > yes,, but when i do snort -v in vagrant ssh console it says snort isnt > installed where as it can be seen working in metron. Due to that reason I > am confused because James Sirota said to install snort. > > On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen <n...@nickallen.org> wrote: > >> From Metron's perspective, Snort is just another sensor. Snort is >> installed, managed and executed completely independent of Metron itself. As >> with any sensor, you are responsible for getting the telemetry produced by >> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >> wonderful things with it. :) >> >> >> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> >> wrote: >> >>> And I am sorry about one confusion but isnt snort builtin into the >>> metron framework? If so then cant we access that snort and do the tasks you >>> mentioned earlier? >>> >>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Hi, >>>> >>>> Thanks for the support. Can it be performed both on dumped log and real >>>> time data? >>>> Regards. >>>> >>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> >>>> wrote: >>>> >>>>> What I mean is that you should install snort, load the appropriate >>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>> traffic to the network interface where Snort is listening. That will >>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>> command line producer. This should be pushed to a Kafka topic called Snort >>>>> where each message is a log line of the Snort file. Does that make sense? >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>> >>>>> You mean that I must start snort from terminal by doing snort -v and >>>>> then push it to kafka topic? I need to start snort in packet capture mode. >>>>> >>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> >>>>> wrote: >>>>> >>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>> the box. You have to setup Snort on your own and push the output into a >>>>> kafka topic (most likely using NiFi). From there on you can use the output >>>>> of Snort in Metron. >>>>> >>>>> >>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>>> >>>>> Hi, >>>>> >>>>> Can I use snort in packet capture mode with metron? By default it >>>>> works in IDS mode only. >>>>> >>>>> Regards. >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PMC- Apache Metron >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PMC- Apache Metron >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>> >>> >> >
Re: Snort
yes,, but when i do snort -v in vagrant ssh console it says snort isnt installed where as it can be seen working in metron. Due to that reason I am confused because James Sirota said to install snort. On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen <n...@nickallen.org> wrote: > From Metron's perspective, Snort is just another sensor. Snort is > installed, managed and executed completely independent of Metron itself. As > with any sensor, you are responsible for getting the telemetry produced by > Snort into Kafka. Metron can then consume that telemetry from Kafka and do > wonderful things with it. :) > > > On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> And I am sorry about one confusion but isnt snort builtin into the metron >> framework? If so then cant we access that snort and do the tasks you >> mentioned earlier? >> >> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk >> > wrote: >> >>> Hi, >>> >>> Thanks for the support. Can it be performed both on dumped log and real >>> time data? >>> Regards. >>> >>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> >>> wrote: >>> >>>> What I mean is that you should install snort, load the appropriate >>>> Snort rules for your use case, set Snort to log to a directory, and send >>>> traffic to the network interface where Snort is listening. That will >>>> produce Snort log files. Then you can push the contents of Snort logs >>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>> command line producer. This should be pushed to a Kafka topic called Snort >>>> where each message is a log line of the Snort file. Does that make sense? >>>> >>>> Thanks, >>>> James >>>> >>>> >>>> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>> >>>> You mean that I must start snort from terminal by doing snort -v and >>>> then push it to kafka topic? I need to start snort in packet capture mode. >>>> >>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> >>>> wrote: >>>> >>>> Yes, you can use Snort. Metron can consume Snort telemetries out of the >>>> box. You have to setup Snort on your own and push the output into a kafka >>>> topic (most likely using NiFi). From there on you can use the output of >>>> Snort in Metron. >>>> >>>> >>>> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>>> >>>> Hi, >>>> >>>> Can I use snort in packet capture mode with metron? By default it works >>>> in IDS mode only. >>>> >>>> Regards. >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>> >> >
Re: Snort
>From Metron's perspective, Snort is just another sensor. Snort is installed, managed and executed completely independent of Metron itself. As with any sensor, you are responsible for getting the telemetry produced by Snort into Kafka. Metron can then consume that telemetry from Kafka and do wonderful things with it. :) On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > And I am sorry about one confusion but isnt snort builtin into the metron > framework? If so then cant we access that snort and do the tasks you > mentioned earlier? > > On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> > wrote: > >> Hi, >> >> Thanks for the support. Can it be performed both on dumped log and real >> time data? >> Regards. >> >> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> wrote: >> >>> What I mean is that you should install snort, load the appropriate Snort >>> rules for your use case, set Snort to log to a directory, and send traffic >>> to the network interface where Snort is listening. That will produce Snort >>> log files. Then you can push the contents of Snort logs either to Kafka >>> using NiFi (preferred) or using Kafka utilities such as command line >>> producer. This should be pushed to a Kafka topic called Snort where each >>> message is a log line of the Snort file. Does that make sense? >>> >>> Thanks, >>> James >>> >>> >>> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>> >>> You mean that I must start snort from terminal by doing snort -v and >>> then push it to kafka topic? I need to start snort in packet capture mode. >>> >>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> >>> wrote: >>> >>> Yes, you can use Snort. Metron can consume Snort telemetries out of the >>> box. You have to setup Snort on your own and push the output into a kafka >>> topic (most likely using NiFi). From there on you can use the output of >>> Snort in Metron. >>> >>> >>> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >>> >>> Hi, >>> >>> Can I use snort in packet capture mode with metron? By default it works >>> in IDS mode only. >>> >>> Regards. >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> >> >
Re: Snort
And I am sorry about one confusion but isnt snort builtin into the metron framework? If so then cant we access that snort and do the tasks you mentioned earlier? On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote: > Hi, > > Thanks for the support. Can it be performed both on dumped log and real > time data? > Regards. > > On Tue, Oct 17, 2017 at 1:02 AM, James Sirota <jsir...@apache.org> wrote: > >> What I mean is that you should install snort, load the appropriate Snort >> rules for your use case, set Snort to log to a directory, and send traffic >> to the network interface where Snort is listening. That will produce Snort >> log files. Then you can push the contents of Snort logs either to Kafka >> using NiFi (preferred) or using Kafka utilities such as command line >> producer. This should be pushed to a Kafka topic called Snort where each >> message is a log line of the Snort file. Does that make sense? >> >> Thanks, >> James >> >> >> 11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >> >> You mean that I must start snort from terminal by doing snort -v and then >> push it to kafka topic? I need to start snort in packet capture mode. >> >> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> wrote: >> >> Yes, you can use Snort. Metron can consume Snort telemetries out of the >> box. You have to setup Snort on your own and push the output into a kafka >> topic (most likely using NiFi). From there on you can use the output of >> Snort in Metron. >> >> >> 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: >> >> Hi, >> >> Can I use snort in packet capture mode with metron? By default it works >> in IDS mode only. >> >> Regards. >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >
Re: Snort
What I mean is that you should install snort, load the appropriate Snort rules for your use case, set Snort to log to a directory, and send traffic to the network interface where Snort is listening. That will produce Snort log files. Then you can push the contents of Snort logs either to Kafka using NiFi (preferred) or using Kafka utilities such as command line producer. This should be pushed to a Kafka topic called Snort where each message is a log line of the Snort file. Does that make sense? Thanks,James11.10.2017, 23:08, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>:You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode.On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> wrote:Yes, you can use Snort. Metron can consume Snort telemetries out of the box. You have to setup Snort on your own and push the output into a kafka topic (most likely using NiFi). From there on you can use the output of Snort in Metron.10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>:Hi,Can I use snort in packet capture mode with metron? By default it works in IDS mode only.Regards. --- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org --- Thank you, James SirotaPMC- Apache Metronjsirota AT apache DOT org
Re: Snort
You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode. On Tue, Oct 10, 2017 at 9:52 PM, James Sirota <jsir...@apache.org> wrote: > Yes, you can use Snort. Metron can consume Snort telemetries out of the > box. You have to setup Snort on your own and push the output into a kafka > topic (most likely using NiFi). From there on you can use the output of > Snort in Metron. > > > 10.10.2017, 00:48, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>: > > Hi, > > Can I use snort in packet capture mode with metron? By default it works in > IDS mode only. > > Regards. > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Suricata support for snort parsers
Hello, In the info discovery phase, and I'm just curious if anyone has tried/had any problems with leveraging existing snort parsers for suricata logs. Cheers, Ian 'z0r0' Abreu
