Re: Initial Testing
If you have never worked with HDFS before here is a quick tutorial:https://hortonworks.com/hadoop-tutorial/using-commandline-manage-files-hdfs/05.10.2017, 01:26, "Simon Elliston Ball" :Try the ambari files view.On 5 Oct 2017, at 09:24, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:THanks again, also how can I access the snort log via hdfs? Is there any web based hdfs portal or will I have to sneak into the vagrant VM file system to access that?On Thu, Oct 5, 2017 at 1:21 PM, Umesh Kaushik <umesh.kaus...@bhujang.net> wrote:I am sorry I will not be able to provide you the exact tutorials. However, I believe you can find something here:https://cwiki.apache.org/confluence/display/METRON/Metron+ArchitectureIf not exact answer you will the enough idea to do R&D to achieve your goals.On 5 October 2017 at 13:43, Syed Hammad Tahir <mscs16...@itu.edu.pk> wrote:Thanks for the information. Can I get any tutorial or guide on that enrichment and labelling phase in metron?On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik <umesh.kaus...@bhujang.net> wrote:Yes, after passing your data from enrichment and labelling phase you can further take it do data modelling phase where you can use python kind of language to apply different modelling techniques on your data.Cheers,Umesh Kaushik9620023458Sent from mobile device, kindly ignore the typographical errors.On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" <mscs16...@itu.edu.pk> wrote:Hi,Lets say I have dumped snort data. Can I apply some machine learning on it in metron?On Thu, Oct 5, 2017 at 12:54 AM, James Sirota <jsir...@apache.org> wrote:1 - It us up to you to install and configure snort however you want. Metron simply consumes the Snort telemetry, but is not opinionated about how you setup your sensors. I would recommend starting with the community rule set: https://www.snort.org/faq/what-are-community-rules 2 - Again, this is outside of scope of Metron. You can view this video to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw 3 - Metron is not a network mapping tool (although support for graph databases is not too far in the future). Today, the best way to generate a network map (graph) is by using kibana. I would refer you to the following article: https://www.elastic.co/products/x-pack/graph 4 - The snort generated data would be indexed in Elasticsearch and/or stored on HDFS, depending on how you configured the system Thanks,James04.10.2017, 03:23, "Syed Hammad Tahir" <mscs16...@itu.edu.pk>:Hi all,Now that I have installed metron (single node installation on ubuntu machine), I want to do some initial testing on snort data. I have a few questions regarding this:1- In how many configurations can I use snort with metron (for ex packet capture in sniffing mode etc)?2- How can I change the rules in snort3- Can I map the network using metron?4- Is snort generated data stored somewhere?KIndly also give me some tutorial to follow for better understanding.Regards. --- Thank you, James SirotaPPMC- Apache Metron (Incubating)jsirota AT apache DOT org -- Cheers,Umesh Kaushik(Full Stack Developer- Cyber security analyst: Bhujang Innovations)(9620023458) --- Thank you, James SirotaPPMC- Apache Metron (Incubating)jsirota AT apache DOT org
Re: Initial Testing
Try the ambari files view. > On 5 Oct 2017, at 09:24, Syed Hammad Tahir wrote: > > THanks again, also how can I access the snort log via hdfs? Is there any web > based hdfs portal or will I have to sneak into the vagrant VM file system to > access that? > >> On Thu, Oct 5, 2017 at 1:21 PM, Umesh Kaushik >> wrote: >> I am sorry I will not be able to provide you the exact tutorials. However, I >> believe you can find something here: >> https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture >> >> If not exact answer you will the enough idea to do R&D to achieve your goals. >> >>> On 5 October 2017 at 13:43, Syed Hammad Tahir wrote: >>> Thanks for the information. Can I get any tutorial or guide on that >>> enrichment and labelling phase in metron? >>> >>>> On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik >>>> wrote: >>>> Yes, after passing your data from enrichment and labelling phase you can >>>> further take it do data modelling phase where you can use python kind of >>>> language to apply different modelling techniques on your data. >>>> >>>> Cheers, >>>> Umesh Kaushik >>>> 9620023458 >>>> >>>> Sent from mobile device, kindly ignore the typographical errors. >>>> >>>>> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" wrote: >>>>> Hi, >>>>> >>>>> Lets say I have dumped snort data. Can I apply some machine learning on >>>>> it in metron? >>>>> >>>>>> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: >>>>>> 1 - It us up to you to install and configure snort however you want. >>>>>> Metron simply consumes the Snort telemetry, but is not opinionated about >>>>>> how you setup your sensors. I would recommend starting with the >>>>>> community rule set: https://www.snort.org/faq/what-are-community-rules >>>>>> >>>>>> 2 - Again, this is outside of scope of Metron. You can view this video >>>>>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>>>>> >>>>>> 3 - Metron is not a network mapping tool (although support for graph >>>>>> databases is not too far in the future). Today, the best way to generate >>>>>> a network map (graph) is by using kibana. I would refer you to the >>>>>> following article: https://www.elastic.co/products/x-pack/graph >>>>>> >>>>>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>>>>> stored on HDFS, depending on how you configured the system >>>>>> >>>>>> Thanks, >>>>>> James >>>>>> >>>>>> >>>>>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>>>>>> Hi all, >>>>>>> >>>>>>> Now that I have installed metron (single node installation on ubuntu >>>>>>> machine), I want to do some initial testing on snort data. I have a few >>>>>>> questions regarding this: >>>>>>> >>>>>>> 1- In how many configurations can I use snort with metron (for ex >>>>>>> packet capture in sniffing mode etc)? >>>>>>> >>>>>>> 2- How can I change the rules in snort >>>>>>> >>>>>>> 3- Can I map the network using metron? >>>>>>> >>>>>>> 4- Is snort generated data stored somewhere? >>>>>>> >>>>>>> KIndly also give me some tutorial to follow for better understanding. >>>>>>> Regards. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PPMC- Apache Metron (Incubating) >>>>>> jsirota AT apache DOT org >>>>>> >>>>> >>> >> >> >> >> -- >> Cheers, >> Umesh Kaushik >> (Full Stack Developer- Cyber security analyst: Bhujang Innovations) >> (9620023458) >
Re: Initial Testing
THanks again, also how can I access the snort log via hdfs? Is there any web based hdfs portal or will I have to sneak into the vagrant VM file system to access that? On Thu, Oct 5, 2017 at 1:21 PM, Umesh Kaushik wrote: > I am sorry I will not be able to provide you the exact tutorials. However, > I believe you can find something here: > https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture > > If not exact answer you will the enough idea to do R&D to achieve your > goals. > > On 5 October 2017 at 13:43, Syed Hammad Tahir > wrote: > >> Thanks for the information. Can I get any tutorial or guide on that >> enrichment and labelling phase in metron? >> >> On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik >> wrote: >> >>> Yes, after passing your data from enrichment and labelling phase you can >>> further take it do data modelling phase where you can use python kind of >>> language to apply different modelling techniques on your data. >>> >>> Cheers, >>> Umesh Kaushik >>> 9620023458 >>> >>> Sent from mobile device, kindly ignore the typographical errors. >>> >>> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" >>> wrote: >>> >>>> Hi, >>>> >>>> Lets say I have dumped snort data. Can I apply some machine learning on >>>> it in metron? >>>> >>>> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota >>>> wrote: >>>> >>>>> 1 - It us up to you to install and configure snort however you want. >>>>> Metron simply consumes the Snort telemetry, but is not opinionated about >>>>> how you setup your sensors. I would recommend starting with the community >>>>> rule set: https://www.snort.org/faq/what-are-community-rules >>>>> >>>>> 2 - Again, this is outside of scope of Metron. You can view this video >>>>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>>>> >>>>> 3 - Metron is not a network mapping tool (although support for graph >>>>> databases is not too far in the future). Today, the best way to generate a >>>>> network map (graph) is by using kibana. I would refer you to the following >>>>> article: https://www.elastic.co/products/x-pack/graph >>>>> >>>>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>>>> stored on HDFS, depending on how you configured the system >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>>>> >>>>> Hi all, >>>>> >>>>> Now that I have installed metron (single node installation on ubuntu >>>>> machine), I want to do some initial testing on snort data. I have a few >>>>> questions regarding this: >>>>> >>>>> 1- In how many configurations can I use snort with metron (for ex >>>>> packet capture in sniffing mode etc)? >>>>> >>>>> 2- How can I change the rules in snort >>>>> >>>>> 3- Can I map the network using metron? >>>>> >>>>> 4- Is snort generated data stored somewhere? >>>>> >>>>> KIndly also give me some tutorial to follow for better understanding. >>>>> Regards. >>>>> >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PPMC- Apache Metron (Incubating) >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>> >> > > > -- > Cheers, > Umesh Kaushik > (Full Stack Developer- Cyber security analyst: Bhujang Innovations) > (9620023458) >
Re: Initial Testing
I am sorry I will not be able to provide you the exact tutorials. However, I believe you can find something here: https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture If not exact answer you will the enough idea to do R&D to achieve your goals. On 5 October 2017 at 13:43, Syed Hammad Tahir wrote: > Thanks for the information. Can I get any tutorial or guide on that > enrichment and labelling phase in metron? > > On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik > wrote: > >> Yes, after passing your data from enrichment and labelling phase you can >> further take it do data modelling phase where you can use python kind of >> language to apply different modelling techniques on your data. >> >> Cheers, >> Umesh Kaushik >> 9620023458 >> >> Sent from mobile device, kindly ignore the typographical errors. >> >> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" >> wrote: >> >>> Hi, >>> >>> Lets say I have dumped snort data. Can I apply some machine learning on >>> it in metron? >>> >>> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota >>> wrote: >>> >>>> 1 - It us up to you to install and configure snort however you want. >>>> Metron simply consumes the Snort telemetry, but is not opinionated about >>>> how you setup your sensors. I would recommend starting with the community >>>> rule set: https://www.snort.org/faq/what-are-community-rules >>>> >>>> 2 - Again, this is outside of scope of Metron. You can view this video >>>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>>> >>>> 3 - Metron is not a network mapping tool (although support for graph >>>> databases is not too far in the future). Today, the best way to generate a >>>> network map (graph) is by using kibana. I would refer you to the following >>>> article: https://www.elastic.co/products/x-pack/graph >>>> >>>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>>> stored on HDFS, depending on how you configured the system >>>> >>>> Thanks, >>>> James >>>> >>>> >>>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>>> >>>> Hi all, >>>> >>>> Now that I have installed metron (single node installation on ubuntu >>>> machine), I want to do some initial testing on snort data. I have a few >>>> questions regarding this: >>>> >>>> 1- In how many configurations can I use snort with metron (for ex >>>> packet capture in sniffing mode etc)? >>>> >>>> 2- How can I change the rules in snort >>>> >>>> 3- Can I map the network using metron? >>>> >>>> 4- Is snort generated data stored somewhere? >>>> >>>> KIndly also give me some tutorial to follow for better understanding. >>>> Regards. >>>> >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PPMC- Apache Metron (Incubating) >>>> jsirota AT apache DOT org >>>> >>>> >>> > -- Cheers, Umesh Kaushik (Full Stack Developer- Cyber security analyst: Bhujang Innovations) (9620023458)
Re: Initial Testing
Syed, I would strongly suggest you go through the Squid based tutorial to get an idea of how enrichment and indexing works. See: https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application <https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application> > On 5 Oct 2017, at 09:13, Syed Hammad Tahir wrote: > > Thanks for the information. Can I get any tutorial or guide on that > enrichment and labelling phase in metron? > > On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik <mailto:umesh.kaus...@bhujang.net>> wrote: > Yes, after passing your data from enrichment and labelling phase you can > further take it do data modelling phase where you can use python kind of > language to apply different modelling techniques on your data. > > Cheers, > Umesh Kaushik > 9620023458 > > Sent from mobile device, kindly ignore the typographical errors. > > On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" <mailto:mscs16...@itu.edu.pk>> wrote: > Hi, > > Lets say I have dumped snort data. Can I apply some machine learning on it in > metron? > > On Thu, Oct 5, 2017 at 12:54 AM, James Sirota <mailto:jsir...@apache.org>> wrote: > 1 - It us up to you to install and configure snort however you want. Metron > simply consumes the Snort telemetry, but is not opinionated about how you > setup your sensors. I would recommend starting with the community rule set: > https://www.snort.org/faq/what-are-community-rules > <https://www.snort.org/faq/what-are-community-rules> > > 2 - Again, this is outside of scope of Metron. You can view this video to get > you started: https://www.youtube.com/watch?v=RUmYojxy3Xw > <https://www.youtube.com/watch?v=RUmYojxy3Xw> > > 3 - Metron is not a network mapping tool (although support for graph > databases is not too far in the future). Today, the best way to generate a > network map (graph) is by using kibana. I would refer you to the following > article: https://www.elastic.co/products/x-pack/graph > <https://www.elastic.co/products/x-pack/graph> > > 4 - The snort generated data would be indexed in Elasticsearch and/or stored > on HDFS, depending on how you configured the system > > Thanks, > James > > > 04.10.2017, 03:23, "Syed Hammad Tahir" <mailto:mscs16...@itu.edu.pk>>: >> Hi all, >> >> Now that I have installed metron (single node installation on ubuntu >> machine), I want to do some initial testing on snort data. I have a few >> questions regarding this: >> >> 1- In how many configurations can I use snort with metron (for ex packet >> capture in sniffing mode etc)? >> >> 2- How can I change the rules in snort >> >> 3- Can I map the network using metron? >> >> 4- Is snort generated data stored somewhere? >> >> KIndly also give me some tutorial to follow for better understanding. >> Regards. >> >> > > > --- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org > > >
Re: Initial Testing
Thanks for the information. Can I get any tutorial or guide on that enrichment and labelling phase in metron? On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik wrote: > Yes, after passing your data from enrichment and labelling phase you can > further take it do data modelling phase where you can use python kind of > language to apply different modelling techniques on your data. > > Cheers, > Umesh Kaushik > 9620023458 > > Sent from mobile device, kindly ignore the typographical errors. > > On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" wrote: > >> Hi, >> >> Lets say I have dumped snort data. Can I apply some machine learning on >> it in metron? >> >> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: >> >>> 1 - It us up to you to install and configure snort however you want. >>> Metron simply consumes the Snort telemetry, but is not opinionated about >>> how you setup your sensors. I would recommend starting with the community >>> rule set: https://www.snort.org/faq/what-are-community-rules >>> >>> 2 - Again, this is outside of scope of Metron. You can view this video >>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>> >>> 3 - Metron is not a network mapping tool (although support for graph >>> databases is not too far in the future). Today, the best way to generate a >>> network map (graph) is by using kibana. I would refer you to the following >>> article: https://www.elastic.co/products/x-pack/graph >>> >>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>> stored on HDFS, depending on how you configured the system >>> >>> Thanks, >>> James >>> >>> >>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>> >>> Hi all, >>> >>> Now that I have installed metron (single node installation on ubuntu >>> machine), I want to do some initial testing on snort data. I have a few >>> questions regarding this: >>> >>> 1- In how many configurations can I use snort with metron (for ex packet >>> capture in sniffing mode etc)? >>> >>> 2- How can I change the rules in snort >>> >>> 3- Can I map the network using metron? >>> >>> 4- Is snort generated data stored somewhere? >>> >>> KIndly also give me some tutorial to follow for better understanding. >>> Regards. >>> >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org >>> >>> >>
Re: Initial Testing
Yes, after passing your data from enrichment and labelling phase you can further take it do data modelling phase where you can use python kind of language to apply different modelling techniques on your data. Cheers, Umesh Kaushik 9620023458 Sent from mobile device, kindly ignore the typographical errors. On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" wrote: > Hi, > > Lets say I have dumped snort data. Can I apply some machine learning on it > in metron? > > On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: > >> 1 - It us up to you to install and configure snort however you want. >> Metron simply consumes the Snort telemetry, but is not opinionated about >> how you setup your sensors. I would recommend starting with the community >> rule set: https://www.snort.org/faq/what-are-community-rules >> >> 2 - Again, this is outside of scope of Metron. You can view this video to >> get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >> >> 3 - Metron is not a network mapping tool (although support for graph >> databases is not too far in the future). Today, the best way to generate a >> network map (graph) is by using kibana. I would refer you to the following >> article: https://www.elastic.co/products/x-pack/graph >> >> 4 - The snort generated data would be indexed in Elasticsearch and/or >> stored on HDFS, depending on how you configured the system >> >> Thanks, >> James >> >> >> 04.10.2017, 03:23, "Syed Hammad Tahir" : >> >> Hi all, >> >> Now that I have installed metron (single node installation on ubuntu >> machine), I want to do some initial testing on snort data. I have a few >> questions regarding this: >> >> 1- In how many configurations can I use snort with metron (for ex packet >> capture in sniffing mode etc)? >> >> 2- How can I change the rules in snort >> >> 3- Can I map the network using metron? >> >> 4- Is snort generated data stored somewhere? >> >> KIndly also give me some tutorial to follow for better understanding. >> Regards. >> >> >> >> >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org >> >> >
Re: Initial Testing
Hi, Lets say I have dumped snort data. Can I apply some machine learning on it in metron? On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: > 1 - It us up to you to install and configure snort however you want. > Metron simply consumes the Snort telemetry, but is not opinionated about > how you setup your sensors. I would recommend starting with the community > rule set: https://www.snort.org/faq/what-are-community-rules > > 2 - Again, this is outside of scope of Metron. You can view this video to > get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw > > 3 - Metron is not a network mapping tool (although support for graph > databases is not too far in the future). Today, the best way to generate a > network map (graph) is by using kibana. I would refer you to the following > article: https://www.elastic.co/products/x-pack/graph > > 4 - The snort generated data would be indexed in Elasticsearch and/or > stored on HDFS, depending on how you configured the system > > Thanks, > James > > > 04.10.2017, 03:23, "Syed Hammad Tahir" : > > Hi all, > > Now that I have installed metron (single node installation on ubuntu > machine), I want to do some initial testing on snort data. I have a few > questions regarding this: > > 1- In how many configurations can I use snort with metron (for ex packet > capture in sniffing mode etc)? > > 2- How can I change the rules in snort > > 3- Can I map the network using metron? > > 4- Is snort generated data stored somewhere? > > KIndly also give me some tutorial to follow for better understanding. > Regards. > > > > > --- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org > >
Re: Initial Testing
1 - It us up to you to install and configure snort however you want. Metron simply consumes the Snort telemetry, but is not opinionated about how you setup your sensors. I would recommend starting with the community rule set: https://www.snort.org/faq/what-are-community-rules 2 - Again, this is outside of scope of Metron. You can view this video to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw 3 - Metron is not a network mapping tool (although support for graph databases is not too far in the future). Today, the best way to generate a network map (graph) is by using kibana. I would refer you to the following article: https://www.elastic.co/products/x-pack/graph 4 - The snort generated data would be indexed in Elasticsearch and/or stored on HDFS, depending on how you configured the system Thanks,James04.10.2017, 03:23, "Syed Hammad Tahir" :Hi all,Now that I have installed metron (single node installation on ubuntu machine), I want to do some initial testing on snort data. I have a few questions regarding this:1- In how many configurations can I use snort with metron (for ex packet capture in sniffing mode etc)?2- How can I change the rules in snort3- Can I map the network using metron?4- Is snort generated data stored somewhere?KIndly also give me some tutorial to follow for better understanding.Regards. --- Thank you, James SirotaPPMC- Apache Metron (Incubating)jsirota AT apache DOT org
Initial Testing
Hi all, Now that I have installed metron (single node installation on ubuntu machine), I want to do some initial testing on snort data. I have a few questions regarding this: 1- In how many configurations can I use snort with metron (for ex packet capture in sniffing mode etc)? 2- How can I change the rules in snort 3- Can I map the network using metron? 4- Is snort generated data stored somewhere? KIndly also give me some tutorial to follow for better understanding. Regards.
