MaaS
Hi all, I need a layman example of how to deploy my python or R model on metron. Do I have to develop the model separately and then deploy? I want to write the model on the go using the snort data collected in hdfs. Maybe I am over-expecting here but correct me if I am wrong. This http://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html is too difficult for me to understand Regards.
Re: Basic analysis
How do I do it using kibana dashboard? What would be the most easy way? On Wed, Dec 6, 2017 at 7:35 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Agreed… for the users list I would just say use the Install Notebooks > action, and look at the squid example on the wiki, but since it was you who > asked for links, Otto, I went a bit dev list ;) > > Simon > > > On 6 Dec 2017, at 14:33, Otto Fowler wrote: > > The issue is the requirement for people on the user list to go to the > source. > > > On December 6, 2017 at 09:16:39, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > No problem, I’ll grant you it’s not in the most intuitive part of the > source tree to go digging in, but you can also get to the zeppelin bits via > the actions button on the Metron config section (Install Notebooks) > > If anyone has any good ideas (or code!) for sample zeppelin notebooks that > would be useful, you can add them to a specific instance of the platform > via the config/zeppelin/metron location and run the action again I believe, > and this would be a great place for more security people to contribute > sample run books for example. There are also efforts by commercial support > providers I believe to add more samples of both dashboards and use cases. > > Simon > > On 6 Dec 2017, at 14:12, Otto Fowler wrote: > > Thanks Simon > > > On December 6, 2017 at 09:11:50, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > In product… Install Zeppelin Notebooks, and the samples including > notebooks at https://github.com/apache/metron/tree/master/metron- > platform/metron-indexing/src/main/config/zeppelin/metron > > as of course there are similar Kibana dashboards included, which are > examples of custom visualisation of metron data, there is also the run book > for visualising squid data in kibana on the docs wiki > https://cwiki.apache.org/confluence/display/METRON/ > Enhancing+Metron+Dashboard > > Should at least get us started. > > Simon > > On 6 Dec 2017, at 14:00, Otto Fowler wrote: > > Links? > > > On December 6, 2017 at 08:18:23, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > We do already have a number of example of exactly this, but sure if > someone feels like adding to those that would be great. > > Simon > > On 6 Dec 2017, at 13:14, Otto Fowler wrote: > > Maybe a Jira logged for an ‘example’ notebook for this would be > appropriate as well? > > > On December 6, 2017 at 07:06:30, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Yes. Consider a zeppelin notebook, or kibana dashboard for this. > > If you want to use these values for detection, consider building a profile > based on the stats objects (see the profiler section of the documentation > under analytics. > > Simon > > > On 6 Dec 2017, at 07:42, Syed Hammad Tahir wrote: > > > > > Hi, > > > > Can I setup custom visualization to show lets say the peak netrwork > usage traffic in a certain time? > > > > Regards. > > >
Basic analysis
Hi, Can I setup custom visualization to show lets say the peak netrwork usage traffic in a certain time? Regards.
Re: ML in Metron
hi, Yes. I need to start with basics. Can you helo me in deploying a model that can detect arp spoofing attacks using data from snort? Regards On Tue, Nov 28, 2017 at 4:08 AM, James Sirota wrote: > Do you currently have any models we can help you deploy? > > Thanks, > James > > > 21.11.2017, 04:44, "Simon Elliston Ball" : > > Use MaaS: > http://metron.apache.org/current-book/metron-analytics/ > metron-maas-service/index.html > > > On 21 Nov 2017, at 11:43, Syed Hammad Tahir wrote: > > HI all, > > I have succesfully pushed real snort logs in to metron, now I need to > apply a machine learning or data science algorithm on it. How could I do > that? I want to code in python/R and then apply it in metron. > > Regards. > > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Metron Use Case
Hi guys, Now that I am ready to work on my research problem and start working on metron, I need to see a use case where a POC has been developed using metron. Just need to get familiar with what we can potentially do on this platform. Regards.
ML in Metron
HI all, I have succesfully pushed real snort logs in to metron, now I need to apply a machine learning or data science algorithm on it. How could I do that? I want to code in python/R and then apply it in metron. Regards.
Re: Snort enrichment issue
ANd I dint load anything. It was supposed to be loaded during installation? My installation is ambari based single node VM install on ubuntu host. On Fri, Nov 17, 2017 at 3:55 PM, Syed Hammad Tahir wrote: > Here you go, the error part of the log is in the attachment. > > On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Did you setup and load the geo enrichment database? https://metron.apach >> e.org/current-book/metron-platform/metron-data-managemen >> t/index.html#GeoLite2_Loader >> >> Also, we can’t really see the error from screenshots, please send log >> entries. >> >> Simon >> >> On 17 Nov 2017, at 07:11, Syed Hammad Tahir wrote: >> >> Hi all, I am starting it again. Last one got a bit messy >> >> Ok, Now I have started everything again from scratch (redeployed single >> node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and >> now when I execute this command: >> >> shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date >> +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> (format of ths command was taken from: https://github.com/apach >> e/metron/blob/master/metron-deployment/roles/sensor-stubs/te >> mplates/start-snort-stub) >> >> I get this under enrichment storm topology : >> >> >> >> >> >> I have come this far, please help me push these dummy preformatted snort >> logs into kibana dashboard. >> >> Regards. >> >> >> >
Re: Snort enrichment issue
Here you go, the error part of the log is in the attachment. On Fri, Nov 17, 2017 at 3:48 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Did you setup and load the geo enrichment database? https://metron. > apache.org/current-book/metron-platform/metron-data-management/index.html# > GeoLite2_Loader > > Also, we can’t really see the error from screenshots, please send log > entries. > > Simon > > On 17 Nov 2017, at 07:11, Syed Hammad Tahir wrote: > > Hi all, I am starting it again. Last one got a bit messy > > Ok, Now I have started everything again from scratch (redeployed single > node based ambari metron cluster with ansibleSkipTags = 'quick-dev') and > now when I execute this command: > > shuf -n 10 snort.out | sed -e "s/[^,]\+ ,/`date > +'%m\/%d\/%y-%H:%M:%S'`.00 ,/g" | > /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh > --broker-list node1:6667 --topic snort > > (format of ths command was taken from: https://github.com/apach > e/metron/blob/master/metron-deployment/roles/sensor-stubs/ > templates/start-snort-stub) > > I get this under enrichment storm topology : > > > > > > I have come this far, please help me push these dummy preformatted snort > logs into kibana dashboard. > > Regards. > > > Enrichment Description: Binary data
Monit and sensor stubs
Hi, I re deployed single node ambari based metron cluster and this time with ansibleSkipTags= 'quick_dev' and now monit and sersor stubs are gone. I run sudo service monit status and it says monit: unrecognized service
Re: HDFS SIze
Sorry for this noobish question. I didnt understand "If you can add more disks to your node then do so". You mean add physical drive in the machine? I already have plenty of free space. Just dont know how to expand hdfs over it On Thu, Nov 16, 2017 at 12:03 PM, Aaron Harris wrote: > Syed, > > > Check what you have set for the dfs.datanode.data.dir parameter in HDFS > config. > > If you can add more disks to your node then do so and update the above > parameter so it references them. > > Your other option is to add a complete new node, then install the datanode > service on it through Ambari. > > > Regards, > > Aaron > -- > *From:* Syed Hammad Tahir > *Sent:* Thursday, November 16, 2017 5:47:49 AM > *To:* user@metron.apache.org > *Subject:* HDFS SIze > > HI, > > I there anyway I could alot more space to hdfs? I am redeploying single > node based ambari Metron cluster > > Regards. >
HDFS SIze
HI, I there anyway I could alot more space to hdfs? I am redeploying single node based ambari Metron cluster Regards.
Re: Snort Logs
ok, Doing it. On Mon, Nov 13, 2017 at 3:07 PM, zeo...@gmail.com wrote: > Can you restart storm and give it another shot? > > Jon > > On Mon, Nov 13, 2017, 00:30 Syed Hammad Tahir > wrote: > >> hi, This problem still persists guys . >> >> On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir >> wrote: >> >>> Any solution to these issues guys? >>> >>> On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir >>> wrote: >>> >>>> I have attached the output of this dump >>>> >>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>> >>>> >>>> >>>> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com >>>> wrote: >>>> >>>>> What is the output of: >>>>> >>>>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>>>> >>>>> ? >>>>> >>>>> Jon >>>>> >>>>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> This is the script/command i used >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> sudo cat snort.out | >>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>> --broker-list node1:6667 --topic snort >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler < >>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>> >>>>>>>> What topic? what are the parameters you are calling the script >>>>>>>> with? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>>>> mscs16...@itu.edu.pk) wrote: >>>>>>>> >>>>>>>> The metron installation I have (single node based vm install) comes >>>>>>>> with sensor stubs. I assume that everything has already been done for >>>>>>>> those >>>>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>>>> directly pushing the preformatted canned data to kafka topic. I can >>>>>>>> see the >>>>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>>>>> wrote: >>>>>>>> >>>>>>>>> How did you start the snort parser topology and what's the parser >>>>>>>>> config (in zookeeper)? >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> This is what I am doing >>>>>>>>>> >>>>>>>>>> sudo cat snort.out | >>>>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>>>> records in >>>>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>>>> topic, >>>>>>>>>>> it should be a JSON map. The parser topology emits that JSON map >>>>>>>>&
Re: Snort Logs
hi, This problem still persists guys . On Thu, Nov 9, 2017 at 11:13 PM, Syed Hammad Tahir wrote: > Any solution to these issues guys? > > On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir > wrote: > >> I have attached the output of this dump >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> >> >> On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com >> wrote: >> >>> What is the output of: >>> >>> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >>> >>> ? >>> >>> Jon >>> >>> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >>> wrote: >>> >>>> This is the script/command i used >>>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> sudo cat snort.out | >>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>> --broker-list node1:6667 --topic snort >>>>> >>>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>>>> wrote: >>>>> >>>>>> What topic? what are the parameters you are calling the script with? >>>>>> >>>>>> >>>>>> >>>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>>> mscs16...@itu.edu.pk) wrote: >>>>>> >>>>>> The metron installation I have (single node based vm install) comes >>>>>> with sensor stubs. I assume that everything has already been done for >>>>>> those >>>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>>> the >>>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>>> >>>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>>> wrote: >>>>>> >>>>>>> How did you start the snort parser topology and what's the parser >>>>>>> config (in zookeeper)? >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> This is what I am doing >>>>>>>> >>>>>>>> sudo cat snort.out | >>>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>>> --broker-list node1:6667 --topic snort >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>>> parser or from some other source? It looks like there are some >>>>>>>>> records in >>>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>>> topic, >>>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>>> then >>>>>>>>> the enrichments topology enrich that map and emits the enriched map >>>>>>>>> to the >>>>>>>>> indexing topic. >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>> >>>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>>> full stack trace >>>>>>>>>> >>>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>>> >>>>>>>>>> [image: Inline image 1] >>>>>>>>>> >>>>>>>>>> from indexingbolt in indexing topology >>>>
Re: Snort Logs
Any solution to these issues guys? On Thu, Nov 9, 2017 at 6:01 AM, Syed Hammad Tahir wrote: > I have attached the output of this dump > > /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP > > > > On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com > wrote: > >> What is the output of: >> >> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP >> >> ? >> >> Jon >> >> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir >> wrote: >> >>> This is the script/command i used >>> >>> sudo cat snort.out | >>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>> --broker-list node1:6667 --topic snort >>> >>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir >> > wrote: >>> >>>> sudo cat snort.out | >>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>> --broker-list node1:6667 --topic snort >>>> >>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>>> wrote: >>>> >>>>> What topic? what are the parameters you are calling the script with? >>>>> >>>>> >>>>> >>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>>> mscs16...@itu.edu.pk) wrote: >>>>> >>>>> The metron installation I have (single node based vm install) comes >>>>> with sensor stubs. I assume that everything has already been done for >>>>> those >>>>> stub sensors to push the canned data. I am doing the similar thing, >>>>> directly pushing the preformatted canned data to kafka topic. I can see >>>>> the >>>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>>> push the same logs myself, those errors pop that I have shown earlier. >>>>> >>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>>> wrote: >>>>> >>>>>> How did you start the snort parser topology and what's the parser >>>>>> config (in zookeeper)? >>>>>> >>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> This is what I am doing >>>>>>> >>>>>>> sudo cat snort.out | >>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>>> --broker-list node1:6667 --topic snort >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>>> wrote: >>>>>>> >>>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>>> parser or from some other source? It looks like there are some >>>>>>>> records in >>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>>> topic, >>>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>>> then >>>>>>>> the enrichments topology enrich that map and emits the enriched map to >>>>>>>> the >>>>>>>> indexing topic. >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>> >>>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>>> full stack trace >>>>>>>>> >>>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>>> >>>>>>>>> [image: Inline image 1] >>>>>>>>> >>>>>>>>> from indexingbolt in indexing topology >>>>>>>>> >>>>>>>>> [image: Inline image 2] >>>>>>>>> >>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>>> Also, are you saying that you are no longer seeing the parser >&
Re: Snort Logs
I have attached the output of this dump /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com wrote: > What is the output of: > > /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP > > ? > > Jon > > On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir > wrote: > >> This is the script/command i used >> >> sudo cat snort.out | >> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >> --broker-list node1:6667 --topic snort >> >> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir >> wrote: >> >>> sudo cat snort.out | >>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>> --broker-list node1:6667 --topic snort >>> >>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler >>> wrote: >>> >>>> What topic? what are the parameters you are calling the script with? >>>> >>>> >>>> >>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir ( >>>> mscs16...@itu.edu.pk) wrote: >>>> >>>> The metron installation I have (single node based vm install) comes >>>> with sensor stubs. I assume that everything has already been done for those >>>> stub sensors to push the canned data. I am doing the similar thing, >>>> directly pushing the preformatted canned data to kafka topic. I can see the >>>> logs in kibana dashboard when I start stub sensor from monit but then I >>>> push the same logs myself, those errors pop that I have shown earlier. >>>> >>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella >>>> wrote: >>>> >>>>> How did you start the snort parser topology and what's the parser >>>>> config (in zookeeper)? >>>>> >>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> This is what I am doing >>>>>> >>>>>> sudo cat snort.out | >>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh >>>>>> --broker-list node1:6667 --topic snort >>>>>> >>>>>> >>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella >>>>>> wrote: >>>>>> >>>>>>> Are you directly writing to the "indexing" kafka topic from the >>>>>>> parser or from some other source? It looks like there are some records >>>>>>> in >>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka >>>>>>> topic, >>>>>>> it should be a JSON map. The parser topology emits that JSON map and >>>>>>> then >>>>>>> the enrichments topology enrich that map and emits the enriched map to >>>>>>> the >>>>>>> indexing topic. >>>>>>> >>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> No I am no longer seeing the parsing topology error, here is the >>>>>>>> full stack trace >>>>>>>> >>>>>>>> from hdfsindexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 1] >>>>>>>> >>>>>>>> from indexingbolt in indexing topology >>>>>>>> >>>>>>>> [image: Inline image 2] >>>>>>>> >>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler < >>>>>>>> ottobackwa...@gmail.com> wrote: >>>>>>>> >>>>>>>>> What Casey said. We need the whole stack trace. >>>>>>>>> Also, are you saying that you are no longer seeing the parser >>>>>>>>> topology error? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella (ceste...@gmail.com) >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> If you click on the port (6704) there in those errors, what's the >>>>>>>>> full stacktrace (that starts with the suggesti
Re: Snort Logs
hi, I am back at work. lets see if i can find something in logs On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com wrote: > It looks like your ES cluster has a health of Red, so there's your > problem. I would go look in /var/log/elasticsearch/ at some logs. > > Jon > > On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir > wrote: > >> >> -- Forwarded message -- >> From: Syed Hammad Tahir >> Date: Fri, Nov 3, 2017 at 5:07 PM >> Subject: Re: Snort Logs >> To: Otto Fowler >> >> >> NVM, I have installed the elastic search head. Now where do I go in this >> to find out why I cant see the snort logs in kibana dashboard, pushed to >> snort topic via kafka producer? >> >> [image: Inline image 1] >> >> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler >> wrote: >> >>> You can install it into the chrome web browser from the play store. >>> >>> >>> >>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) >>> wrote: >>> >>> And how do I install elasticsearch head on the vagrant VM? >>> >>> >> -- > > Jon >
Fwd: Snort Logs
-- Forwarded message -- From: Syed Hammad Tahir Date: Fri, Nov 3, 2017 at 5:07 PM Subject: Re: Snort Logs To: Otto Fowler NVM, I have installed the elastic search head. Now where do I go in this to find out why I cant see the snort logs in kibana dashboard, pushed to snort topic via kafka producer? [image: Inline image 1] On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler wrote: > You can install it into the chrome web browser from the play store. > > > > On November 3, 2017 at 07:47:47, Syed Hammad Tahir (mscs16...@itu.edu.pk) > wrote: > > And how do I install elasticsearch head on the vagrant VM? > >
Re: Snort Logs
And how do I install elasticsearch head on the vagrant VM?
memory issue
How do I increase vagrant vm`s RAM. I have plenty of RAM to allocate to it. [image: Inline image 1]
Re: Snort Logs
I sent a random message to that kafka topic and got this [image: Inline image 1] I guess this is because I am not following the format of message I should send? Like those snort logs you showed. On Mon, Oct 30, 2017 at 5:24 PM, zeo...@gmail.com wrote: > They need to meet the format of the logs I sent earlier. Look into the > snort output options - may require you rerun snort, depending on your > situation > > Jon > > On Mon, Oct 30, 2017, 06:53 Syed Hammad Tahir > wrote: > >> Yes, I have converted them to text but those logs are simply captured >> packet headers over the local network. Now I just push them via that kafka >> producer command under topic name of snort and they will be visible in >> metron? >> >> On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com >> wrote: >> >>> You need text logs. Here's an example of some properly formatted logs - >>> https://raw.githubusercontent.com/apache/metron/master/metron- >>> deployment/roles/sensor-stubs/files/snort.out >>> >>> Jon >>> >>> On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir >>> wrote: >>> >>>> I have found the kafka-console-producer.sh but I need to know how to >>>> make it read snort.log (tcp dump format) file. May be I am missing >>>> something in the plain sight but it would be awsome if you tell me that. >>>> >>>> Regards. >>>> >>>> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> On the 25th I said: >>>>> >>>>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>>>> (from memory) on node1, assuming you are running full dev. >>>>> >>>>> Jon >>>>> >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> snort logs are in tcp dump format. I may have to convert them. >>>>>> >>>>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic >>>>>> test >>>>>> >>>>>> How to give file name or path in this command? >>>>>> >>>>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>>>> snort topic. You may also want to look at this [2]. >>>>>>> >>>>>>> 1: https://kafka.apache.org/quickstart >>>>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>>>> console-producer-and-bash-script >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>>>> wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> I have run snort independently on vagrant ssh and dumped the logs >>>>>>>> in tcpdump format. Now I want to bring them to metron to play with >>>>>>>> them a >>>>>>>> bit. Some of you already replied me with some solutions but thats lost >>>>>>>> in >>>>>>>> the inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>>>> Please give me an easy to understand this solution for this problem. >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jon >>>>>>> >>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
Yes, I have converted them to text but those logs are simply captured packet headers over the local network. Now I just push them via that kafka producer command under topic name of snort and they will be visible in metron? On Mon, Oct 30, 2017 at 2:41 PM, zeo...@gmail.com wrote: > You need text logs. Here's an example of some properly formatted logs - > https://raw.githubusercontent.com/apache/metron/master/metron- > deployment/roles/sensor-stubs/files/snort.out > > Jon > > On Mon, Oct 30, 2017, 01:34 Syed Hammad Tahir > wrote: > >> I have found the kafka-console-producer.sh but I need to know how to >> make it read snort.log (tcp dump format) file. May be I am missing >> something in the plain sight but it would be awsome if you tell me that. >> >> Regards. >> >> On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com >> wrote: >> >>> On the 25th I said: >>> >>> It should be in /usr/hdp/current/kafka-broker/bin/ or similar >>> (from memory) on node1, assuming you are running full dev. >>> >>> Jon >>> >>> >>> Jon >>> >>> On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir >>> wrote: >>> >>>> snort logs are in tcp dump format. I may have to convert them. >>>> >>>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >>>> >>>> How to give file name or path in this command? >>>> >>>> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> If you have text snort logs you can use Apache nifi or the Kafka >>>>> producer script as described in step 4 here[1] to push them to Metron's >>>>> snort topic. You may also want to look at this [2]. >>>>> >>>>> 1: https://kafka.apache.org/quickstart >>>>> 2: https://stackoverflow.com/questions/38701179/kafka- >>>>> console-producer-and-bash-script >>>>> >>>>> Jon >>>>> >>>>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I have run snort independently on vagrant ssh and dumped the logs in >>>>>> tcpdump format. Now I want to bring them to metron to play with them a >>>>>> bit. >>>>>> Some of you already replied me with some solutions but thats lost in the >>>>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. >>>>>> Please >>>>>> give me an easy to understand this solution for this problem. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
I have found the kafka-console-producer.sh but I need to know how to make it read snort.log (tcp dump format) file. May be I am missing something in the plain sight but it would be awsome if you tell me that. Regards. On Fri, Oct 27, 2017 at 5:09 PM, zeo...@gmail.com wrote: > On the 25th I said: > > It should be in /usr/hdp/current/kafka-broker/bin/ or similar (from > memory) on node1, assuming you are running full dev. > > Jon > > > Jon > > On Fri, Oct 27, 2017 at 6:25 AM Syed Hammad Tahir > wrote: > >> snort logs are in tcp dump format. I may have to convert them. >> >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >> >> How to give file name or path in this command? >> >> On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com >> wrote: >> >>> If you have text snort logs you can use Apache nifi or the Kafka >>> producer script as described in step 4 here[1] to push them to Metron's >>> snort topic. You may also want to look at this [2]. >>> >>> 1: https://kafka.apache.org/quickstart >>> 2: https://stackoverflow.com/questions/38701179/kafka- >>> console-producer-and-bash-script >>> >>> Jon >>> >>> On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir >>> wrote: >>> >>>> Hello everyone, >>>> >>>> I have run snort independently on vagrant ssh and dumped the logs in >>>> tcpdump format. Now I want to bring them to metron to play with them a bit. >>>> Some of you already replied me with some solutions but thats lost in the >>>> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >>>> give me an easy to understand this solution for this problem. >>>> >>>> Regards. >>>> >>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Snort Logs
snort logs are in tcp dump format. I may have to convert them. bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test How to give file name or path in this command? On Fri, Oct 27, 2017 at 2:53 PM, zeo...@gmail.com wrote: > If you have text snort logs you can use Apache nifi or the Kafka producer > script as described in step 4 here[1] to push them to Metron's snort > topic. You may also want to look at this [2]. > > 1: https://kafka.apache.org/quickstart > 2: https://stackoverflow.com/questions/38701179/kafka- > console-producer-and-bash-script > > Jon > > On Fri, Oct 27, 2017, 02:15 Syed Hammad Tahir > wrote: > >> Hello everyone, >> >> I have run snort independently on vagrant ssh and dumped the logs in >> tcpdump format. Now I want to bring them to metron to play with them a bit. >> Some of you already replied me with some solutions but thats lost in the >> inbox somewhere and engulfed by the elasticsearhc issue that I had. Please >> give me an easy to understand this solution for this problem. >> >> Regards. >> > -- > > Jon >
Snort Logs
Hello everyone, I have run snort independently on vagrant ssh and dumped the logs in tcpdump format. Now I want to bring them to metron to play with them a bit. Some of you already replied me with some solutions but thats lost in the inbox somewhere and engulfed by the elasticsearhc issue that I had. Please give me an easy to understand this solution for this problem. Regards.
Re: Kibana Error
I don know where this is going. I will restart the PC tomorrow to reload
everything from scratch. I repeat, I have done the single node vagrant
based installation on my ubuntu pc (core i7/32GB/1TB). I was supposed to
push snort logs (tcpdump format) to metron but then I was hit by this
issue. My domain is mid-level machine learning or datascience hence I might
not even know things which are required.
On Wed, Oct 25, 2017 at 10:40 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:
> Health should look something like this, if you grab it from the Head
> plugin.
>
> {
> "cluster_name": "metron",
> *"status": "yellow",*
> "timed_out": false,
> *"number_of_nodes": 1,*
> *"number_of_data_nodes": 1*,
> "active_primary_shards": 15,
> "active_shards": 15,
> "relocating_shards": 0,
> "initializing_shards": 0,
> "unassigned_shards": 15,
> "delayed_unassigned_shards": 0,
> "number_of_pending_tasks": 0,
> "number_of_in_flight_fetch": 0,
> "task_max_waiting_in_queue_millis": 0,
> "active_shards_percent_as_number": 50
> }
>
> On Wed, Oct 25, 2017 at 11:38 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> What do you see when you go here?
>> http://node1:9200/_cat/health?v
>>
>> You can also get the Elasticsearch Head Plugin for Chrome, which is very
>> useful and will be compatible with 5.x versions of Elasticsearch when
>> Metron upgrades (plugins from 2.x are no longer available in v5.6).
>> https://chrome.google.com/webstore/detail/elasticsear
>> ch-head/ffmkiejjmecolpfloofpjologoblkegm
>>
>> Just plugin the address http://node1:9200/ and hit connect. I believe
>> our default status is "yellow." But that should be sufficient.
>>
>> I also second Simon's comments about reading up on Elasticsearch.
>>
>> Best,
>> Mike Miklavcic
>>
>>
>> On Wed, Oct 25, 2017 at 11:13 AM, Syed Hammad Tahir > > wrote:
>>
>>> I killed it via terminal and then restarted it. Still the same thing,
>>> cant load the page when I go to elasticsearch health shortlink in ambari.
>>>
>>> On Wed, Oct 25, 2017 at 5:16 PM, Simon Elliston Ball <
>>> si...@simonellistonball.com> wrote:
>>>
>>>> Ok, this is an elastic problem which prevents it shutting down. Find
>>>> the elastic processes, kill them, and start it up again.
>>>>
>>>>
>>>> On 25 Oct 2017, at 13:15, Syed Hammad Tahir
>>>> wrote:
>>>>
>>>> Just gave the command but its stuck here. I restart it earleir via
>>>> ambari after changing heapsize. Now doing it via console
>>>>
>>>>
>>>>
>>>> On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball <
>>>> si...@simonellistonball.com> wrote:
>>>>
>>>>> That just shows running, not health. The problem is that it is not
>>>>> responding. I assume you have tried restarting elastic.
>>>>>
>>>>> On 25 Oct 2017, at 13:12, Syed Hammad Tahir
>>>>> wrote:
>>>>>
>>>>> It shows healthy
>>>>>
>>>>>
>>>>> But when I click in any quick link it shows this
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball <
>>>>> si...@simonellistonball.com> wrote:
>>>>>
>>>>>> Did you check the elastic service was running and healthy with the
>>>>>> health checks. Try a few of the quick links from the elastic section in
>>>>>> ambari.
>>>>>>
>>>>>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir
>>>>>> wrote:
>>>>>>
>>>>>> I have increased size to 2048mb. Still seeing it
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball <
>>>>>> si...@simonellistonball.com> wrote:
>>>>>>
>>>>>>> I strongly suggest you spend some time learning about elastic search
>>>>>>> and some of the basic components. This is not a bug, it’s that elastic
>>>>>>> is
>>>>>>> down. The default heap (use the ambari search in the elastic section) is
>>>>>>> probably set too low. The default is 128m. C
Re: Kibana Error
I killed it via terminal and then restarted it. Still the same thing, cant load the page when I go to elasticsearch health shortlink in ambari. On Wed, Oct 25, 2017 at 5:16 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Ok, this is an elastic problem which prevents it shutting down. Find the > elastic processes, kill them, and start it up again. > > > On 25 Oct 2017, at 13:15, Syed Hammad Tahir wrote: > > Just gave the command but its stuck here. I restart it earleir via ambari > after changing heapsize. Now doing it via console > > > > On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> That just shows running, not health. The problem is that it is not >> responding. I assume you have tried restarting elastic. >> >> On 25 Oct 2017, at 13:12, Syed Hammad Tahir wrote: >> >> It shows healthy >> >> >> But when I click in any quick link it shows this >> >> >> >> On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> Did you check the elastic service was running and healthy with the >>> health checks. Try a few of the quick links from the elastic section in >>> ambari. >>> >>> On 25 Oct 2017, at 13:05, Syed Hammad Tahir >>> wrote: >>> >>> I have increased size to 2048mb. Still seeing it >>> >>> >>> >>> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> I strongly suggest you spend some time learning about elastic search >>>> and some of the basic components. This is not a bug, it’s that elastic is >>>> down. The default heap (use the ambari search in the elastic section) is >>>> probably set too low. The default is 128m. Change this to more, probably >>>> more like 2048m. >>>> >>>> Essential background reading for metron is an understanding of elastic >>>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume >>>> you have at least some familiarity with those technologies. >>>> >>>> Simon >>>> >>>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir >>>> wrote: >>>> >>>> Sorry, I didnt understand. Which baremetal guide should I look into? >>>> And I googled it and found no help. Please help me guys, there are bigger >>>> issues at hand and I cant afford to waste much time on this problem :( >>>> >>>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum < >>>> anjum.farr...@gmail.com> wrote: >>>> >>>>> Its a bug reported in metron, >>>>> >>>>> Look into barematel guide, Turn Red to green Cluster google it. >>>>> >>>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" >>>>> wrote: >>>>> >>>>>> SHould I do it from here? If yes then please guide me how to >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball < >>>>>> si...@simonellistonball.com> wrote: >>>>>> >>>>>>> Your elastic search instance has died. Try given it more heap size >>>>>>> in the elastic section on ambari. >>>>>>> >>>>>>> >>>>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >>>>>>> wrote: >>>>>>> > >>>>>>> > When I try to open node1:5000 I see this. >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > What could be the problem and its solution? >>>>>>> >>>>>>> >>>>>> >>>> >>> >>> >> >> > >
Re: Kibana Error
Just gave the command but its stuck here. I restart it earleir via ambari after changing heapsize. Now doing it via console [image: Inline image 1] On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > That just shows running, not health. The problem is that it is not > responding. I assume you have tried restarting elastic. > > On 25 Oct 2017, at 13:12, Syed Hammad Tahir wrote: > > It shows healthy > > > But when I click in any quick link it shows this > > > > On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Did you check the elastic service was running and healthy with the health >> checks. Try a few of the quick links from the elastic section in ambari. >> >> On 25 Oct 2017, at 13:05, Syed Hammad Tahir wrote: >> >> I have increased size to 2048mb. Still seeing it >> >> >> >> On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> I strongly suggest you spend some time learning about elastic search and >>> some of the basic components. This is not a bug, it’s that elastic is down. >>> The default heap (use the ambari search in the elastic section) is probably >>> set too low. The default is 128m. Change this to more, probably more like >>> 2048m. >>> >>> Essential background reading for metron is an understanding of elastic >>> search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume >>> you have at least some familiarity with those technologies. >>> >>> Simon >>> >>> On 25 Oct 2017, at 11:40, Syed Hammad Tahir >>> wrote: >>> >>> Sorry, I didnt understand. Which baremetal guide should I look into? And >>> I googled it and found no help. Please help me guys, there are bigger >>> issues at hand and I cant afford to waste much time on this problem :( >>> >>> On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum < >>> anjum.farr...@gmail.com> wrote: >>> >>>> Its a bug reported in metron, >>>> >>>> Look into barematel guide, Turn Red to green Cluster google it. >>>> >>>> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" >>>> wrote: >>>> >>>>> SHould I do it from here? If yes then please guide me how to >>>>> >>>>> >>>>> >>>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball < >>>>> si...@simonellistonball.com> wrote: >>>>> >>>>>> Your elastic search instance has died. Try given it more heap size in >>>>>> the elastic section on ambari. >>>>>> >>>>>> >>>>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >>>>>> wrote: >>>>>> > >>>>>> > When I try to open node1:5000 I see this. >>>>>> > >>>>>> > >>>>>> > >>>>>> > What could be the problem and its solution? >>>>>> >>>>>> >>>>> >>> >> >> > >
Re: Kibana Error
I have increased size to 2048mb. Still seeing it [image: Inline image 1] On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > I strongly suggest you spend some time learning about elastic search and > some of the basic components. This is not a bug, it’s that elastic is down. > The default heap (use the ambari search in the elastic section) is probably > set too low. The default is 128m. Change this to more, probably more like > 2048m. > > Essential background reading for metron is an understanding of elastic > search, kafka, hadoop (hdfs in particular) and Linux. Our docs will assume > you have at least some familiarity with those technologies. > > Simon > > On 25 Oct 2017, at 11:40, Syed Hammad Tahir wrote: > > Sorry, I didnt understand. Which baremetal guide should I look into? And I > googled it and found no help. Please help me guys, there are bigger issues > at hand and I cant afford to waste much time on this problem :( > > On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum < > anjum.farr...@gmail.com> wrote: > >> Its a bug reported in metron, >> >> Look into barematel guide, Turn Red to green Cluster google it. >> >> On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" >> wrote: >> >>> SHould I do it from here? If yes then please guide me how to >>> >>> >>> >>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> Your elastic search instance has died. Try given it more heap size in >>>> the elastic section on ambari. >>>> >>>> >>>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >>>> wrote: >>>> > >>>> > When I try to open node1:5000 I see this. >>>> > >>>> > >>>> > >>>> > What could be the problem and its solution? >>>> >>>> >>> >
Re: Kibana Error
Sorry, I didnt understand. Which baremetal guide should I look into? And I googled it and found no help. Please help me guys, there are bigger issues at hand and I cant afford to waste much time on this problem :( On Wed, Oct 25, 2017 at 1:31 PM, Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > Its a bug reported in metron, > > Look into barematel guide, Turn Red to green Cluster google it. > > On Oct 25, 2017 1:21 PM, "Syed Hammad Tahir" wrote: > >> SHould I do it from here? If yes then please guide me how to >> >> [image: Inline image 1] >> >> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> Your elastic search instance has died. Try given it more heap size in >>> the elastic section on ambari. >>> >>> >>> > On 25 Oct 2017, at 09:16, Syed Hammad Tahir >>> wrote: >>> > >>> > When I try to open node1:5000 I see this. >>> > >>> > >>> > >>> > What could be the problem and its solution? >>> >>> >>
Re: Kibana Error
SHould I do it from here? If yes then please guide me how to [image: Inline image 1] On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Your elastic search instance has died. Try given it more heap size in the > elastic section on ambari. > > > > On 25 Oct 2017, at 09:16, Syed Hammad Tahir > wrote: > > > > When I try to open node1:5000 I see this. > > > > > > > > What could be the problem and its solution? > >
Kibana Error
When I try to open node1:5000 I see this. [image: Inline image 1] What could be the problem and its solution?
Re: Snort Installation
All I did was install snort separately on vagrant ssh console. The ran it to collect logs. Now I need to bring those logs to metron. On Wed, Oct 25, 2017 at 9:50 AM, Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > Hi Syed Hammed, > > Can you share the steps how did you connected snort with external source ? > (Metron Snort ?) > > On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen wrote: > >> Take a look at `kafka-console-producer.sh`, which is installed as part of >> Kafka. >> >> On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir >> wrote: >> >>> Ok, I have fixed everything on my own. Now that I have snort logs saved >>> in a file, I need to get them to metron. Can anyone help me on that? >>> >>> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> > wrote: >>> >>>> yes nut I am a bit confused here. Let me ask them as well then. >>>> >>>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> Just to clarify, this a snort issue you are having? If so I suggest >>>>> looking at their documentation (https://snort.org/documents) or >>>>> reaching out to their community (https://snort.org/community), as >>>>> they have more expertise in this area. >>>>> >>>>> Jon >>>>> >>>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I tried to add another network interface in order to bridge it to >>>>>> LAN. I tried to do it on virtualbox vm settings and when i did vagrant up >>>>>> after that, there was no bridged interface. Can anyone help me on this? >>>>>> >>>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>>> >>>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> See inline. >>>>>>>> >>>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>>> >>>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>>> traffic? >>>>>>>>> >>>>>>>> >>>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>>> of the project. Have a look at the documentation at >>>>>>>> https://www.snort.org/. >>>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>>> some >>>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>>> >>>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>>> preprocessing >>>>>>>>> >>>>>>>> >>>>>>>> Have a look at the Metron documentation at >>>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>>> >>>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>>> >>>>>>>> >>>>>>>> I can't help you with this :) >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> >>> >> > > > -- > With Regards > Farrukh Naveed Anjum >
Re: Snort Installation
Where do I find this file kafka-console-producer.sh? On Tue, Oct 24, 2017 at 8:27 PM, Nick Allen wrote: > Take a look at `kafka-console-producer.sh`, which is installed as part of > Kafka. > > On Tue, Oct 24, 2017 at 2:11 AM, Syed Hammad Tahir > wrote: > >> Ok, I have fixed everything on my own. Now that I have snort logs saved >> in a file, I need to get them to metron. Can anyone help me on that? >> >> On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir >> wrote: >> >>> yes nut I am a bit confused here. Let me ask them as well then. >>> >>> On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com >>> wrote: >>> >>>> Hi Syed, >>>> >>>> Just to clarify, this a snort issue you are having? If so I suggest >>>> looking at their documentation (https://snort.org/documents) or >>>> reaching out to their community (https://snort.org/community), as they >>>> have more expertise in this area. >>>> >>>> Jon >>>> >>>> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >>>> wrote: >>>> >>>>> Hi guys, >>>>> >>>>> I tried to add another network interface in order to bridge it to LAN. >>>>> I tried to do it on virtualbox vm settings and when i did vagrant up after >>>>> that, there was no bridged interface. Can anyone help me on this? >>>>> >>>>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Ok, thankyou. I will let you know once I make snort sniff the >>>>>> traffic in the given configuration, might be helpful for others. I will >>>>>> then try to do that kafka topic and will ask if any help is needed. >>>>>> >>>>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>>>> wrote: >>>>>> >>>>>>> Hi Syed, >>>>>>> >>>>>>> See inline. >>>>>>> >>>>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>>>> >>>>>>>> I have installed the snort manually. Now I need help with : >>>>>>>> >>>>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort >>>>>>>> cant see the traffic outside vagrant vm, how do I make it see that >>>>>>>> traffic? >>>>>>>> >>>>>>> >>>>>>> To be honest, configuring Snort to work on your LAN is out of scope >>>>>>> of the project. Have a look at the documentation at >>>>>>> https://www.snort.org/. >>>>>>> You will probably have to add a 2nd network interface bridged to >>>>>>> your LAN in promiscuous mode. Additionally, I think most of us expect >>>>>>> some >>>>>>> basic Linux & network administration knowledge when using Metron. >>>>>>> >>>>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>>>> preprocessing >>>>>>>> >>>>>>> >>>>>>> Have a look at the Metron documentation at >>>>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>>>> sensor in the Metron UI will create the Kafka iirc. >>>>>>> >>>>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>>>> >>>>>>> >>>>>>> I can't help you with this :) >>>>>>> >>>>>> >>>>>> >>>>> -- >>>> >>>> Jon >>>> >>> >>> >> >
Re: Snort Installation
Ok, I have fixed everything on my own. Now that I have snort logs saved in a file, I need to get them to metron. Can anyone help me on that? On Mon, Oct 23, 2017 at 3:44 PM, Syed Hammad Tahir wrote: > yes nut I am a bit confused here. Let me ask them as well then. > > On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com > wrote: > >> Hi Syed, >> >> Just to clarify, this a snort issue you are having? If so I suggest >> looking at their documentation (https://snort.org/documents) or reaching >> out to their community (https://snort.org/community), as they have more >> expertise in this area. >> >> Jon >> >> On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir >> wrote: >> >>> Hi guys, >>> >>> I tried to add another network interface in order to bridge it to LAN. I >>> tried to do it on virtualbox vm settings and when i did vagrant up after >>> that, there was no bridged interface. Can anyone help me on this? >>> >>> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>>> in the given configuration, might be helpful for others. I will then try to >>>> do that kafka topic and will ask if any help is needed. >>>> >>>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets >>>> wrote: >>>> >>>>> Hi Syed, >>>>> >>>>> See inline. >>>>> >>>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>>> >>>>>> I have installed the snort manually. Now I need help with : >>>>>> >>>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>>> >>>>> >>>>> To be honest, configuring Snort to work on your LAN is out of scope of >>>>> the project. Have a look at the documentation at >>>>> https://www.snort.org/. >>>>> You will probably have to add a 2nd network interface bridged to your >>>>> LAN in promiscuous mode. Additionally, I think most of us expect some >>>>> basic >>>>> Linux & network administration knowledge when using Metron. >>>>> >>>>> 2- Making a kafka topic to push those saved logs in metron for >>>>>> preprocessing >>>>>> >>>>> >>>>> Have a look at the Metron documentation at >>>>> https://metron.apache.org/current-book/index.html. Adding a new >>>>> sensor in the Metron UI will create the Kafka iirc. >>>>> >>>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>>> >>>>> >>>>> I can't help you with this :) >>>>> >>>> >>>> >>> -- >> >> Jon >> > >
Re: Snort Installation
yes nut I am a bit confused here. Let me ask them as well then. On Mon, Oct 23, 2017 at 3:35 PM, zeo...@gmail.com wrote: > Hi Syed, > > Just to clarify, this a snort issue you are having? If so I suggest > looking at their documentation (https://snort.org/documents) or reaching > out to their community (https://snort.org/community), as they have more > expertise in this area. > > Jon > > On Mon, Oct 23, 2017, 03:52 Syed Hammad Tahir > wrote: > >> Hi guys, >> >> I tried to add another network interface in order to bridge it to LAN. I >> tried to do it on virtualbox vm settings and when i did vagrant up after >> that, there was no bridged interface. Can anyone help me on this? >> >> On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir > > wrote: >> >>> Ok, thankyou. I will let you know once I make snort sniff the traffic >>> in the given configuration, might be helpful for others. I will then try to >>> do that kafka topic and will ask if any help is needed. >>> >>> On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: >>> >>>> Hi Syed, >>>> >>>> See inline. >>>> >>>> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >>>> >>>>> I have installed the snort manually. Now I need help with : >>>>> >>>>> 1- Capturing the data of my lan and dumping it via snort :Snort cant >>>>> see the traffic outside vagrant vm, how do I make it see that traffic? >>>>> >>>> >>>> To be honest, configuring Snort to work on your LAN is out of scope of >>>> the project. Have a look at the documentation at https://www.snort.org/ >>>> . >>>> You will probably have to add a 2nd network interface bridged to your >>>> LAN in promiscuous mode. Additionally, I think most of us expect some basic >>>> Linux & network administration knowledge when using Metron. >>>> >>>> 2- Making a kafka topic to push those saved logs in metron for >>>>> preprocessing >>>>> >>>> >>>> Have a look at the Metron documentation at https://metron.apache.org/ >>>> current-book/index.html. Adding a new sensor in the Metron UI will >>>> create the Kafka iirc. >>>> >>>> 3- Applying a basic Machine learning algorithm on the captured data. >>>>> >>>> >>>> I can't help you with this :) >>>> >>> >>> >> -- > > Jon >
Re: Snort Installation
Hi guys, I tried to add another network interface in order to bridge it to LAN. I tried to do it on virtualbox vm settings and when i did vagrant up after that, there was no bridged interface. Can anyone help me on this? On Sun, Oct 22, 2017 at 11:44 AM, Syed Hammad Tahir wrote: > Ok, thankyou. I will let you know once I make snort sniff the traffic in > the given configuration, might be helpful for others. I will then try to do > that kafka topic and will ask if any help is needed. > > On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: > >> Hi Syed, >> >> See inline. >> >> On 2017-10-20 00:32, Syed Hammad Tahir wrote: >> >>> I have installed the snort manually. Now I need help with : >>> >>> 1- Capturing the data of my lan and dumping it via snort :Snort cant see >>> the traffic outside vagrant vm, how do I make it see that traffic? >>> >> >> To be honest, configuring Snort to work on your LAN is out of scope of >> the project. Have a look at the documentation at https://www.snort.org/. >> You will probably have to add a 2nd network interface bridged to your LAN >> in promiscuous mode. Additionally, I think most of us expect some basic >> Linux & network administration knowledge when using Metron. >> >> 2- Making a kafka topic to push those saved logs in metron for >>> preprocessing >>> >> >> Have a look at the Metron documentation at https://metron.apache.org/curr >> ent-book/index.html. Adding a new sensor in the Metron UI will create >> the Kafka iirc. >> >> 3- Applying a basic Machine learning algorithm on the captured data. >>> >> >> I can't help you with this :) >> > >
Re: Snort Installation
Ok, thankyou. I will let you know once I make snort sniff the traffic in the given configuration, might be helpful for others. I will then try to do that kafka topic and will ask if any help is needed. On Sun, Oct 22, 2017 at 6:10 AM, Laurens Vets wrote: > Hi Syed, > > See inline. > > On 2017-10-20 00:32, Syed Hammad Tahir wrote: > >> I have installed the snort manually. Now I need help with : >> >> 1- Capturing the data of my lan and dumping it via snort :Snort cant see >> the traffic outside vagrant vm, how do I make it see that traffic? >> > > To be honest, configuring Snort to work on your LAN is out of scope of the > project. Have a look at the documentation at https://www.snort.org/. > You will probably have to add a 2nd network interface bridged to your LAN > in promiscuous mode. Additionally, I think most of us expect some basic > Linux & network administration knowledge when using Metron. > > 2- Making a kafka topic to push those saved logs in metron for >> preprocessing >> > > Have a look at the Metron documentation at https://metron.apache.org/curr > ent-book/index.html. Adding a new sensor in the Metron UI will create the > Kafka iirc. > > 3- Applying a basic Machine learning algorithm on the captured data. >> > > I can't help you with this :) >
Re: Snort Installation
Help guys !!! On Fri, Oct 20, 2017 at 12:32 PM, Syed Hammad Tahir wrote: > I have installed the snort manually. Now I need help with : > > 1- Capturing the data of my lan and dumping it via snort :Snort cant see > the traffic outside vagrant vm, how do I make it see that traffic? > > 2- Making a kafka topic to push those saved logs in metron for > preprocessing > > 3- Applying a basic Machine learning algorithm on the captured data. > > Regards. >
Snort Installation
I have installed the snort manually. Now I need help with : 1- Capturing the data of my lan and dumping it via snort :Snort cant see the traffic outside vagrant vm, how do I make it see that traffic? 2- Making a kafka topic to push those saved logs in metron for preprocessing 3- Applying a basic Machine learning algorithm on the captured data. Regards.
Re: Snort
I did all of that and then did vagrant up again. Snort is still not installed. Will I have to vagrant destroy and then vagrant up again in order for it to work? On Thu, Oct 19, 2017 at 8:58 PM, Syed Hammad Tahir wrote: > would I need to vagrant destroy and then vagrant up again after this or > will vagrant halt and vagrant up will do the job? > > On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com > wrote: > >> In the Vagrantfile for full-dev, edit the line that starts with >> ansibleSkipTags (this line >> <https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20>) >> to be exactly the following: >> >> ansibleSkipTags='quick_dev' >> >> Jon >> >> On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir >> wrote: >> >>> Should I edit the vagrant file using text editor and what exactly should >>> I edit there? >>> >>> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> I would recommend just using a text editor if you’re not familiar with >>>> sed. To solve your sed problem… >>>> >>>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>> >>>> sed -i means run the sed command (in this case a find replace) inplace >>>> on the file, the text following the -i is the name to append to a backup >>>> version (ie the original file unchanged). >>>> >>>> Metron does tend to assume a good knowledge of linux admin, you’ll find >>>> we have a lot of shell gurus in the community, but if you’re struggling >>>> with this, maybe a simple text editor would be easier. All you’re trying to >>>> do here is change a config value. >>>> >>>> Simon >>>> >>>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir >>>> wrote: >>>> >>>> Ran it without -i swtich, gives this: >>>> >>>> >>>> >>>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> The sed command is falling. It's written for a Mac so it will need an >>>>> alteration to be portable. Run it without the '' after -i, from >>>>> ~/metron-master >>>>> >>>>> Jon >>>>> >>>>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> I did what this guide said to install the original sensor: >>>>>> https://github.com/apache/metron/tree/master/metron-deployme >>>>>> nt/roles/sensor-stubs >>>>>> >>>>>> Still didnt work. How do I install snort into this? >>>>>> >>>>>> >>>>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Maybe I did something wrong >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>> >>>>>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>>>>> >>>>>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>>>> .com> wrote: >>>>>>>> >>>>>>>>> When you set up full dev if you remove the sensors skip tag it >>>>>>>>> will set up snort for you. I have a sed one liner in my bro security >>>>>>>>> patch >>>>>>>>> pr to do this, just need to do it before vagrant up. >>>>>>>>> >>>>>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>>>>> vagrant up >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir < >>>>>>>>> mscs16...@itu.edu.pk&g
Re: Snort
would I need to vagrant destroy and then vagrant up again after this or will vagrant halt and vagrant up will do the job? On Thu, Oct 19, 2017 at 5:23 PM, zeo...@gmail.com wrote: > In the Vagrantfile for full-dev, edit the line that starts with > ansibleSkipTags (this line > <https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20>) > to be exactly the following: > > ansibleSkipTags='quick_dev' > > Jon > > On Thu, Oct 19, 2017 at 7:59 AM Syed Hammad Tahir > wrote: > >> Should I edit the vagrant file using text editor and what exactly should >> I edit there? >> >> On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> I would recommend just using a text editor if you’re not familiar with >>> sed. To solve your sed problem… >>> >>> sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>> >>> sed -i means run the sed command (in this case a find replace) inplace >>> on the file, the text following the -i is the name to append to a backup >>> version (ie the original file unchanged). >>> >>> Metron does tend to assume a good knowledge of linux admin, you’ll find >>> we have a lot of shell gurus in the community, but if you’re struggling >>> with this, maybe a simple text editor would be easier. All you’re trying to >>> do here is change a config value. >>> >>> Simon >>> >>> On 19 Oct 2017, at 11:46, Syed Hammad Tahir >>> wrote: >>> >>> Ran it without -i swtich, gives this: >>> >>> >>> >>> On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com >>> wrote: >>> >>>> The sed command is falling. It's written for a Mac so it will need an >>>> alteration to be portable. Run it without the '' after -i, from >>>> ~/metron-master >>>> >>>> Jon >>>> >>>> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >>>> wrote: >>>> >>>>> I did what this guide said to install the original sensor: >>>>> https://github.com/apache/metron/tree/master/metron- >>>>> deployment/roles/sensor-stubs >>>>> >>>>> Still didnt work. How do I install snort into this? >>>>> >>>>> >>>>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Maybe I did something wrong >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>>>> >>>>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>>> > wrote: >>>>>>> >>>>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>>>> set up snort for you. I have a sed one liner in my bro security patch >>>>>>>> pr >>>>>>>> to do this, just need to do it before vagrant up. >>>>>>>> >>>>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>>>> vagrant up >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I followed this guide exactly: https://cwiki. >>>>>>>>> apache.org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>>>> >>>>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>>>> returns error of not able to find the snort command. >>>>>>>>> >>>>>>>>> On Wed, Oct 18,
Re: Snort
Should I edit the vagrant file using text editor and what exactly should I edit there? On Thu, Oct 19, 2017 at 3:54 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > I would recommend just using a text editor if you’re not familiar with > sed. To solve your sed problem… > > sed -i.bak "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" > metron-deployment/vagrant/full-dev-platform/Vagrantfile > > sed -i means run the sed command (in this case a find replace) inplace on > the file, the text following the -i is the name to append to a backup > version (ie the original file unchanged). > > Metron does tend to assume a good knowledge of linux admin, you’ll find we > have a lot of shell gurus in the community, but if you’re struggling with > this, maybe a simple text editor would be easier. All you’re trying to do > here is change a config value. > > Simon > > On 19 Oct 2017, at 11:46, Syed Hammad Tahir wrote: > > Ran it without -i swtich, gives this: > > > > On Thu, Oct 19, 2017 at 2:56 PM, zeo...@gmail.com > wrote: > >> The sed command is falling. It's written for a Mac so it will need an >> alteration to be portable. Run it without the '' after -i, from >> ~/metron-master >> >> Jon >> >> On Thu, Oct 19, 2017, 04:07 Syed Hammad Tahir >> wrote: >> >>> I did what this guide said to install the original sensor: >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> Still didnt work. How do I install snort into this? >>> >>> >>> On Thu, Oct 19, 2017 at 10:26 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Maybe I did something wrong >>>> >>>> >>>> >>>> On Thu, Oct 19, 2017 at 6:03 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Ok, thankyou. It will install all the sensors (bro, snort etc) ? >>>>> >>>>> On Thu, Oct 19, 2017 at 12:30 AM, zeo...@gmail.com >>>>> wrote: >>>>> >>>>>> When you set up full dev if you remove the sensors skip tag it will >>>>>> set up snort for you. I have a sed one liner in my bro security patch pr >>>>>> to do this, just need to do it before vagrant up. >>>>>> >>>>>> sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" >>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile >>>>>> cd metron-deployment/vagrant/full-dev-platform/ >>>>>> vagrant up >>>>>> >>>>>> Jon >>>>>> >>>>>> On Wed, Oct 18, 2017, 14:51 Syed Hammad Tahir >>>>>> wrote: >>>>>> >>>>>>> I followed this guide exactly: https://cwiki.apache >>>>>>> .org/confluence/pages/viewpage.action?pageId=68718548 >>>>>>> >>>>>>> And the did vagrant up in full-development-platform folder. And >>>>>>> Snort is not installed because when I type snort -v in vagrant ssh, it >>>>>>> returns error of not able to find the snort command. >>>>>>> >>>>>>> On Wed, Oct 18, 2017 at 10:12 PM, Laurens Vets >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Syed, >>>>>>>> >>>>>>>> I was under the impression that you installed the full-dev >>>>>>>> environment? If so, snort should already be installed... >>>>>>>> >>>>>>>> On 2017-10-18 09:45, Syed Hammad Tahir wrote: >>>>>>>> >>>>>>>> It has become a mess. Apparently snort is released for centos 7 >>>>>>>> whereas metron one is centos 6.8. Whenever I try to install snort it >>>>>>>> gives >>>>>>>> me this: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 18, 2017 at 5:58 PM, Nick Allen >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just use those as a guide to run the commands yourself. >>>>>>>>> >>>>>>>>> On Wed, Oct 18, 2017 at 7:23 AM Syed Hammad Tahir < >>>>>>>>> msc
Re: Snort
I am so noob in all of this. I am using full-dev vm metron install to do my research. So I have 2 options to install snort: as per my understanding 1- Install it in a usual way (like that on a regular linux machine) and then make its kafka topic 2- Use ansible role to do all of that. Read the content of those yml files given in main.yml to understand the procedure? Which one do you suggest? On Tue, Oct 17, 2017 at 9:22 PM, Nick Allen wrote: > No special commands. Install and configure Snort however you like and get > those logs into a Kafka topic. Metron is completely agnostic to how sensor > telemetry lands in Kafka. > > We also have an Ansible role that will install Snort along with a simple > mechanism to transport its logs to Kafka. This is only useful for > development environments; not a production install. > > Using the Ansible role directly may be beyond the knowledge level of > some. I only offer this as a guide that you can use to follow along and > manually install it yourself. > > https://github.com/apache/metron/blob/master/metron- > deployment/roles/snort/tasks/main.yml > > > If you are not familiar with how Ansible roles are defined, just start at > the main.yml, then follow through each of the other files as they are > included. It is pretty readable once you get use to the layout. > > On Tue, Oct 17, 2017 at 12:05 PM, Syed Hammad Tahir > wrote: > >> Ok, Now I get it. Now should I install snort in vagrant ssh in the normal >> way snort is usually install on a linux distro or do I need to run some >> special commands again? >> >> On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote: >> >>> In the Full Dev environment, Snort is not installed. We install "Sensor >>> Stubs" which is just a mechanism that continually replays canned telemetry >>> logs repetitively to mimic real sensors. We have to do this because of >>> resource constraints when running all of Metron on a single VM. See the >>> following for more information. >>> >>> https://github.com/apache/metron/tree/master/metron-deployme >>> nt/roles/sensor-stubs >>> >>> >>> >>> On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >>>> installed where as it can be seen working in metron. Due to that reason I >>>> am confused because James Sirota said to install snort. >>>> >>>> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: >>>> >>>>> From Metron's perspective, Snort is just another sensor. Snort is >>>>> installed, managed and executed completely independent of Metron itself. >>>>> As >>>>> with any sensor, you are responsible for getting the telemetry produced by >>>>> Snort into Kafka. Metron can then consume that telemetry from Kafka and >>>>> do >>>>> wonderful things with it. :) >>>>> >>>>> >>>>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> And I am sorry about one confusion but isnt snort builtin into the >>>>>> metron framework? If so then cant we access that snort and do the tasks >>>>>> you >>>>>> mentioned earlier? >>>>>> >>>>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for the support. Can it be performed both on dumped log and >>>>>>> real time data? >>>>>>> Regards. >>>>>>> >>>>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>>>>> wrote: >>>>>>> >>>>>>>> What I mean is that you should install snort, load the appropriate >>>>>>>> Snort rules for your use case, set Snort to log to a directory, and >>>>>>>> send >>>>>>>> traffic to the network interface where Snort is listening. That will >>>>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>>>> command line producer. This should be pushed to a Kafka topic c
Re: Snort
Ok, Now I get it. Now should I install snort in vagrant ssh in the normal way snort is usually install on a linux distro or do I need to run some special commands again? On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote: > In the Full Dev environment, Snort is not installed. We install "Sensor > Stubs" which is just a mechanism that continually replays canned telemetry > logs repetitively to mimic real sensors. We have to do this because of > resource constraints when running all of Metron on a single VM. See the > following for more information. > > https://github.com/apache/metron/tree/master/metron- > deployment/roles/sensor-stubs > > > > On Tue, Oct 17, 2017 at 10:16 AM, Syed Hammad Tahir > wrote: > >> yes,, but when i do snort -v in vagrant ssh console it says snort isnt >> installed where as it can be seen working in metron. Due to that reason I >> am confused because James Sirota said to install snort. >> >> On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: >> >>> From Metron's perspective, Snort is just another sensor. Snort is >>> installed, managed and executed completely independent of Metron itself. As >>> with any sensor, you are responsible for getting the telemetry produced by >>> Snort into Kafka. Metron can then consume that telemetry from Kafka and do >>> wonderful things with it. :) >>> >>> >>> On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir >> > wrote: >>> >>>> And I am sorry about one confusion but isnt snort builtin into the >>>> metron framework? If so then cant we access that snort and do the tasks you >>>> mentioned earlier? >>>> >>>> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Hi, >>>>> >>>>> Thanks for the support. Can it be performed both on dumped log and >>>>> real time data? >>>>> Regards. >>>>> >>>>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>>>> wrote: >>>>> >>>>>> What I mean is that you should install snort, load the appropriate >>>>>> Snort rules for your use case, set Snort to log to a directory, and send >>>>>> traffic to the network interface where Snort is listening. That will >>>>>> produce Snort log files. Then you can push the contents of Snort logs >>>>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>>>> command line producer. This should be pushed to a Kafka topic called >>>>>> Snort >>>>>> where each message is a log line of the Snort file. Does that make sense? >>>>>> >>>>>> Thanks, >>>>>> James >>>>>> >>>>>> >>>>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>>>> >>>>>> You mean that I must start snort from terminal by doing snort -v and >>>>>> then push it to kafka topic? I need to start snort in packet capture >>>>>> mode. >>>>>> >>>>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>>>> wrote: >>>>>> >>>>>> Yes, you can use Snort. Metron can consume Snort telemetries out of >>>>>> the box. You have to setup Snort on your own and push the output into a >>>>>> kafka topic (most likely using NiFi). From there on you can use the >>>>>> output >>>>>> of Snort in Metron. >>>>>> >>>>>> >>>>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>>>> >>>>>> Hi, >>>>>> >>>>>> Can I use snort in packet capture mode with metron? By default it >>>>>> works in IDS mode only. >>>>>> >>>>>> Regards. >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> Thank you, >>>>>> >>>>>> James Sirota >>>>>> PMC- Apache Metron >>>>>> jsirota AT apache DOT org >>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Snort
yes,, but when i do snort -v in vagrant ssh console it says snort isnt installed where as it can be seen working in metron. Due to that reason I am confused because James Sirota said to install snort. On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote: > From Metron's perspective, Snort is just another sensor. Snort is > installed, managed and executed completely independent of Metron itself. As > with any sensor, you are responsible for getting the telemetry produced by > Snort into Kafka. Metron can then consume that telemetry from Kafka and do > wonderful things with it. :) > > > On Tue, Oct 17, 2017 at 4:00 AM, Syed Hammad Tahir > wrote: > >> And I am sorry about one confusion but isnt snort builtin into the metron >> framework? If so then cant we access that snort and do the tasks you >> mentioned earlier? >> >> On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir > > wrote: >> >>> Hi, >>> >>> Thanks for the support. Can it be performed both on dumped log and real >>> time data? >>> Regards. >>> >>> On Tue, Oct 17, 2017 at 1:02 AM, James Sirota >>> wrote: >>> >>>> What I mean is that you should install snort, load the appropriate >>>> Snort rules for your use case, set Snort to log to a directory, and send >>>> traffic to the network interface where Snort is listening. That will >>>> produce Snort log files. Then you can push the contents of Snort logs >>>> either to Kafka using NiFi (preferred) or using Kafka utilities such as >>>> command line producer. This should be pushed to a Kafka topic called Snort >>>> where each message is a log line of the Snort file. Does that make sense? >>>> >>>> Thanks, >>>> James >>>> >>>> >>>> 11.10.2017, 23:08, "Syed Hammad Tahir" : >>>> >>>> You mean that I must start snort from terminal by doing snort -v and >>>> then push it to kafka topic? I need to start snort in packet capture mode. >>>> >>>> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota >>>> wrote: >>>> >>>> Yes, you can use Snort. Metron can consume Snort telemetries out of the >>>> box. You have to setup Snort on your own and push the output into a kafka >>>> topic (most likely using NiFi). From there on you can use the output of >>>> Snort in Metron. >>>> >>>> >>>> 10.10.2017, 00:48, "Syed Hammad Tahir" : >>>> >>>> Hi, >>>> >>>> Can I use snort in packet capture mode with metron? By default it works >>>> in IDS mode only. >>>> >>>> Regards. >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>>> >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> >>> >> >
Re: Snort
And I am sorry about one confusion but isnt snort builtin into the metron framework? If so then cant we access that snort and do the tasks you mentioned earlier? On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir wrote: > Hi, > > Thanks for the support. Can it be performed both on dumped log and real > time data? > Regards. > > On Tue, Oct 17, 2017 at 1:02 AM, James Sirota wrote: > >> What I mean is that you should install snort, load the appropriate Snort >> rules for your use case, set Snort to log to a directory, and send traffic >> to the network interface where Snort is listening. That will produce Snort >> log files. Then you can push the contents of Snort logs either to Kafka >> using NiFi (preferred) or using Kafka utilities such as command line >> producer. This should be pushed to a Kafka topic called Snort where each >> message is a log line of the Snort file. Does that make sense? >> >> Thanks, >> James >> >> >> 11.10.2017, 23:08, "Syed Hammad Tahir" : >> >> You mean that I must start snort from terminal by doing snort -v and then >> push it to kafka topic? I need to start snort in packet capture mode. >> >> On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: >> >> Yes, you can use Snort. Metron can consume Snort telemetries out of the >> box. You have to setup Snort on your own and push the output into a kafka >> topic (most likely using NiFi). From there on you can use the output of >> Snort in Metron. >> >> >> 10.10.2017, 00:48, "Syed Hammad Tahir" : >> >> Hi, >> >> Can I use snort in packet capture mode with metron? By default it works >> in IDS mode only. >> >> Regards. >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >
Re: Snort
Hi, Thanks for the support. Can it be performed both on dumped log and real time data? Regards. On Tue, Oct 17, 2017 at 1:02 AM, James Sirota wrote: > What I mean is that you should install snort, load the appropriate Snort > rules for your use case, set Snort to log to a directory, and send traffic > to the network interface where Snort is listening. That will produce Snort > log files. Then you can push the contents of Snort logs either to Kafka > using NiFi (preferred) or using Kafka utilities such as command line > producer. This should be pushed to a Kafka topic called Snort where each > message is a log line of the Snort file. Does that make sense? > > Thanks, > James > > > 11.10.2017, 23:08, "Syed Hammad Tahir" : > > You mean that I must start snort from terminal by doing snort -v and then > push it to kafka topic? I need to start snort in packet capture mode. > > On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: > > Yes, you can use Snort. Metron can consume Snort telemetries out of the > box. You have to setup Snort on your own and push the output into a kafka > topic (most likely using NiFi). From there on you can use the output of > Snort in Metron. > > > 10.10.2017, 00:48, "Syed Hammad Tahir" : > > Hi, > > Can I use snort in packet capture mode with metron? By default it works in > IDS mode only. > > Regards. > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Re: Snort
You mean that I must start snort from terminal by doing snort -v and then push it to kafka topic? I need to start snort in packet capture mode. On Tue, Oct 10, 2017 at 9:52 PM, James Sirota wrote: > Yes, you can use Snort. Metron can consume Snort telemetries out of the > box. You have to setup Snort on your own and push the output into a kafka > topic (most likely using NiFi). From there on you can use the output of > Snort in Metron. > > > 10.10.2017, 00:48, "Syed Hammad Tahir" : > > Hi, > > Can I use snort in packet capture mode with metron? By default it works in > IDS mode only. > > Regards. > > > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Snort
Hi, Can I use snort in packet capture mode with metron? By default it works in IDS mode only. Regards.
Re: Initial Testing
THanks again, also how can I access the snort log via hdfs? Is there any web based hdfs portal or will I have to sneak into the vagrant VM file system to access that? On Thu, Oct 5, 2017 at 1:21 PM, Umesh Kaushik wrote: > I am sorry I will not be able to provide you the exact tutorials. However, > I believe you can find something here: > https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture > > If not exact answer you will the enough idea to do R&D to achieve your > goals. > > On 5 October 2017 at 13:43, Syed Hammad Tahir > wrote: > >> Thanks for the information. Can I get any tutorial or guide on that >> enrichment and labelling phase in metron? >> >> On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik >> wrote: >> >>> Yes, after passing your data from enrichment and labelling phase you can >>> further take it do data modelling phase where you can use python kind of >>> language to apply different modelling techniques on your data. >>> >>> Cheers, >>> Umesh Kaushik >>> 9620023458 >>> >>> Sent from mobile device, kindly ignore the typographical errors. >>> >>> On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" >>> wrote: >>> >>>> Hi, >>>> >>>> Lets say I have dumped snort data. Can I apply some machine learning on >>>> it in metron? >>>> >>>> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota >>>> wrote: >>>> >>>>> 1 - It us up to you to install and configure snort however you want. >>>>> Metron simply consumes the Snort telemetry, but is not opinionated about >>>>> how you setup your sensors. I would recommend starting with the community >>>>> rule set: https://www.snort.org/faq/what-are-community-rules >>>>> >>>>> 2 - Again, this is outside of scope of Metron. You can view this video >>>>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>>>> >>>>> 3 - Metron is not a network mapping tool (although support for graph >>>>> databases is not too far in the future). Today, the best way to generate a >>>>> network map (graph) is by using kibana. I would refer you to the following >>>>> article: https://www.elastic.co/products/x-pack/graph >>>>> >>>>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>>>> stored on HDFS, depending on how you configured the system >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>>>> >>>>> Hi all, >>>>> >>>>> Now that I have installed metron (single node installation on ubuntu >>>>> machine), I want to do some initial testing on snort data. I have a few >>>>> questions regarding this: >>>>> >>>>> 1- In how many configurations can I use snort with metron (for ex >>>>> packet capture in sniffing mode etc)? >>>>> >>>>> 2- How can I change the rules in snort >>>>> >>>>> 3- Can I map the network using metron? >>>>> >>>>> 4- Is snort generated data stored somewhere? >>>>> >>>>> KIndly also give me some tutorial to follow for better understanding. >>>>> Regards. >>>>> >>>>> >>>>> >>>>> >>>>> --- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PPMC- Apache Metron (Incubating) >>>>> jsirota AT apache DOT org >>>>> >>>>> >>>> >> > > > -- > Cheers, > Umesh Kaushik > (Full Stack Developer- Cyber security analyst: Bhujang Innovations) > (9620023458) >
Re: Initial Testing
Thanks for the information. Can I get any tutorial or guide on that enrichment and labelling phase in metron? On Thu, Oct 5, 2017 at 1:05 PM, Umesh Kaushik wrote: > Yes, after passing your data from enrichment and labelling phase you can > further take it do data modelling phase where you can use python kind of > language to apply different modelling techniques on your data. > > Cheers, > Umesh Kaushik > 9620023458 > > Sent from mobile device, kindly ignore the typographical errors. > > On 05-Oct-2017 10:55 AM, "Syed Hammad Tahir" wrote: > >> Hi, >> >> Lets say I have dumped snort data. Can I apply some machine learning on >> it in metron? >> >> On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: >> >>> 1 - It us up to you to install and configure snort however you want. >>> Metron simply consumes the Snort telemetry, but is not opinionated about >>> how you setup your sensors. I would recommend starting with the community >>> rule set: https://www.snort.org/faq/what-are-community-rules >>> >>> 2 - Again, this is outside of scope of Metron. You can view this video >>> to get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw >>> >>> 3 - Metron is not a network mapping tool (although support for graph >>> databases is not too far in the future). Today, the best way to generate a >>> network map (graph) is by using kibana. I would refer you to the following >>> article: https://www.elastic.co/products/x-pack/graph >>> >>> 4 - The snort generated data would be indexed in Elasticsearch and/or >>> stored on HDFS, depending on how you configured the system >>> >>> Thanks, >>> James >>> >>> >>> 04.10.2017, 03:23, "Syed Hammad Tahir" : >>> >>> Hi all, >>> >>> Now that I have installed metron (single node installation on ubuntu >>> machine), I want to do some initial testing on snort data. I have a few >>> questions regarding this: >>> >>> 1- In how many configurations can I use snort with metron (for ex packet >>> capture in sniffing mode etc)? >>> >>> 2- How can I change the rules in snort >>> >>> 3- Can I map the network using metron? >>> >>> 4- Is snort generated data stored somewhere? >>> >>> KIndly also give me some tutorial to follow for better understanding. >>> Regards. >>> >>> >>> >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org >>> >>> >>
Re: Initial Testing
Hi, Lets say I have dumped snort data. Can I apply some machine learning on it in metron? On Thu, Oct 5, 2017 at 12:54 AM, James Sirota wrote: > 1 - It us up to you to install and configure snort however you want. > Metron simply consumes the Snort telemetry, but is not opinionated about > how you setup your sensors. I would recommend starting with the community > rule set: https://www.snort.org/faq/what-are-community-rules > > 2 - Again, this is outside of scope of Metron. You can view this video to > get you started: https://www.youtube.com/watch?v=RUmYojxy3Xw > > 3 - Metron is not a network mapping tool (although support for graph > databases is not too far in the future). Today, the best way to generate a > network map (graph) is by using kibana. I would refer you to the following > article: https://www.elastic.co/products/x-pack/graph > > 4 - The snort generated data would be indexed in Elasticsearch and/or > stored on HDFS, depending on how you configured the system > > Thanks, > James > > > 04.10.2017, 03:23, "Syed Hammad Tahir" : > > Hi all, > > Now that I have installed metron (single node installation on ubuntu > machine), I want to do some initial testing on snort data. I have a few > questions regarding this: > > 1- In how many configurations can I use snort with metron (for ex packet > capture in sniffing mode etc)? > > 2- How can I change the rules in snort > > 3- Can I map the network using metron? > > 4- Is snort generated data stored somewhere? > > KIndly also give me some tutorial to follow for better understanding. > Regards. > > > > > --- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org > >
Initial Testing
Hi all, Now that I have installed metron (single node installation on ubuntu machine), I want to do some initial testing on snort data. I have a few questions regarding this: 1- In how many configurations can I use snort with metron (for ex packet capture in sniffing mode etc)? 2- How can I change the rules in snort 3- Can I map the network using metron? 4- Is snort generated data stored somewhere? KIndly also give me some tutorial to follow for better understanding. Regards.
Metron Services
Hi, After installing all the services, I put them on start since yesterday. It took all the resources and I couldnt do anything. THe power outage caused system, to restart so that process was interrupted. Now when I try to start all services again I get this error: [image: Inline image 1]
Metron Installation
WHat services are necessary to run metron? [image: Inline image 1]
Guide
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 Does this installaion guide work any more? I am trying to to it on my 32gb ram ubuntu PC. Please let me know if there are any changes to be made in this.
help
WHat do I do now? [image: Inline image 1]
Re: Metron Installation error
I guess it failed on these steps, trying to install hadoop_2_5_3_0_37-yarn as shown at the end of log: [image: Inline image 1] Then I tried manually installing it: [image: Inline image 2] ANd its complete: [image: Inline image 3] Now what should I do? How do I install remaining packages? SHould I try vagrant provision again? On Thu, Sep 28, 2017 at 4:28 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > There are a lot of packages involved in installation. If you have a > failure here, you are likely to fail some of the others too. Often you can > retry in these scenarios, or go to ambari and in hosts view try > reinstalling (on the install failed services). The ensile method of install > is not the method I would recommend personally, but it should be retryable. > > I would honestly look at what’s blocking your package downloads. > > Simon > > > On 28 Sep 2017, at 11:38, Syed Hammad Tahir wrote: > > Ok, I guess it failed to install package: hadoop_2_5_3_0_37-yarn > If I do it succesfully then should I do vagrant provision again or > anything else? > > On Thu, Sep 28, 2017 at 3:32 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Just try a yum install of the package manually. >> >> On 28 Sep 2017, at 11:29, Syed Hammad Tahir wrote: >> >> My internet connection seems to be ok but to remove the doubt, is there >> any way to install the failed package manually? From where do I get the >> python script it ran before failure. The script which tries to download the >> packages >> >> On Thu, Sep 28, 2017 at 3:23 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> It looks like you do not have access to the internet, or at least your >>> connection is not good enough to download the packages. >>> >>> Verify that you're not getting rpms blocked by a corporate proxy (a >>> common problem) or something of the sort, or use a clean connection. >>> >>> Simon >>> >>> On 28 Sep 2017, at 11:17, Syed Hammad Tahir >>> wrote: >>> >>> OK, I reran everything on my machine (destroyed vagrant and then doing >>> vagrant up) >>> Screenshots are in the order: >>> >>> >>> >>> >>> The Ambari page soon after getting the above error: >>> >>> >>> Clicked ops on top left >>> >>> >>> >>> Clicked "install components on host node 1" >>> >>> >>> >>> >>> Clicked node1: >>> >>> >>> >>> clicked App timeline server Install >>> >>> >>> >>> here is the pastebin of output of this error log: >>> >>> https://pastebin.com/eFqHTbxQ >>> >>> Please let me know whats the real issue here. WHy cant it install these >>> services. >>> >>> >>> >>> On Thu, Sep 28, 2017 at 10:25 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Here is the ambari-agent.log >>>> >>>> On Thu, Sep 28, 2017 at 10:22 AM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Hello All, >>>>> >>>>> This is what I see >>>>> >>>>> >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> >>>>> [image: Inline image 2] >>>>> >>>>> [image: Inline image 3] >>>>> >>>>> Now going to last error >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> I think the connection got refused because these components never got >>>>> installed. What could be the reason? >>>>> >>>>> >>>>> >>>>> On Wed, Sep 27, 2017 at 11:26 PM, Dima Kovalyov < >>>>> dima.koval...@sstech.us> wrote: >>>>> >>>>>> I agree with Jon, the most effective way to start troubleshooting is >>>>>> to look at error in Ambari UI. I have shown in red rectangle in the top >>>>>> left corner where you should click on the ATT2.png. >>>>>> >>>>>> There will be list of ambari tasks (ambari_background.png), you can >>>>>> see the in my case everything went just fine. What do you see in there on >>>>>> your side? And if there are errors, can you drill down to the exact >>>>>> message >>>>>> of the error? >>>>>> Thank you. >>>>>> >>>>>> - Dima >>>>>> >>>>>> On 09/27/2017 05:41 PM, David Lyle wrote: >>>>>> >>>>>> Is there any chance your VM can't reach the internet? >>>>>> >>>>>> Each component failed to install, not just Metron. The Ambari Server >>>>>> log clearly states that, but doesn't give the reason. The reasons should >>>>>> be >>>>>> in the Ambari Agent log and and are definitely accessible from the Ambari >>>>>> page if you click on ops directly to the right of your cluster name. It >>>>>> should show failed operations and give you more information. You can >>>>>> capture the command line Ambari used and run that on node1 if you want to >>>>>> try to recreate the failure. >>>>>> >>>>>> -D... >>>>>> >>>>>> >>>>>> On Wed, Sep 27, 2017 at 10:25 AM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> I re ran everything, still getting this error: >>>>>>> >>>>>>> >>>>>>> >>>>>>> I ahve also attached the ambari server log file >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > >
Re: Metron Installation error
Ok, I guess it failed to install package: hadoop_2_5_3_0_37-yarn If I do it succesfully then should I do vagrant provision again or anything else? On Thu, Sep 28, 2017 at 3:32 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Just try a yum install of the package manually. > > On 28 Sep 2017, at 11:29, Syed Hammad Tahir wrote: > > My internet connection seems to be ok but to remove the doubt, is there > any way to install the failed package manually? From where do I get the > python script it ran before failure. The script which tries to download the > packages > > On Thu, Sep 28, 2017 at 3:23 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> It looks like you do not have access to the internet, or at least your >> connection is not good enough to download the packages. >> >> Verify that you're not getting rpms blocked by a corporate proxy (a >> common problem) or something of the sort, or use a clean connection. >> >> Simon >> >> On 28 Sep 2017, at 11:17, Syed Hammad Tahir wrote: >> >> OK, I reran everything on my machine (destroyed vagrant and then doing >> vagrant up) >> Screenshots are in the order: >> >> >> >> >> The Ambari page soon after getting the above error: >> >> >> Clicked ops on top left >> >> >> >> Clicked "install components on host node 1" >> >> >> >> >> Clicked node1: >> >> >> >> clicked App timeline server Install >> >> >> >> here is the pastebin of output of this error log: >> >> https://pastebin.com/eFqHTbxQ >> >> Please let me know whats the real issue here. WHy cant it install these >> services. >> >> >> >> On Thu, Sep 28, 2017 at 10:25 AM, Syed Hammad Tahir > > wrote: >> >>> Here is the ambari-agent.log >>> >>> On Thu, Sep 28, 2017 at 10:22 AM, Syed Hammad Tahir < >>> mscs16...@itu.edu.pk> wrote: >>> >>>> Hello All, >>>> >>>> This is what I see >>>> >>>> >>>> >>>> [image: Inline image 1] >>>> >>>> >>>> [image: Inline image 2] >>>> >>>> [image: Inline image 3] >>>> >>>> Now going to last error >>>> >>>> >>>> >>>> >>>> >>>> I think the connection got refused because these components never got >>>> installed. What could be the reason? >>>> >>>> >>>> >>>> On Wed, Sep 27, 2017 at 11:26 PM, Dima Kovalyov < >>>> dima.koval...@sstech.us> wrote: >>>> >>>>> I agree with Jon, the most effective way to start troubleshooting is >>>>> to look at error in Ambari UI. I have shown in red rectangle in the top >>>>> left corner where you should click on the ATT2.png. >>>>> >>>>> There will be list of ambari tasks (ambari_background.png), you can >>>>> see the in my case everything went just fine. What do you see in there on >>>>> your side? And if there are errors, can you drill down to the exact >>>>> message >>>>> of the error? >>>>> Thank you. >>>>> >>>>> - Dima >>>>> >>>>> On 09/27/2017 05:41 PM, David Lyle wrote: >>>>> >>>>> Is there any chance your VM can't reach the internet? >>>>> >>>>> Each component failed to install, not just Metron. The Ambari Server >>>>> log clearly states that, but doesn't give the reason. The reasons should >>>>> be >>>>> in the Ambari Agent log and and are definitely accessible from the Ambari >>>>> page if you click on ops directly to the right of your cluster name. It >>>>> should show failed operations and give you more information. You can >>>>> capture the command line Ambari used and run that on node1 if you want to >>>>> try to recreate the failure. >>>>> >>>>> -D... >>>>> >>>>> >>>>> On Wed, Sep 27, 2017 at 10:25 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> I re ran everything, still getting this error: >>>>>> >>>>>> >>>>>> >>>>>> I ahve also attached the ambari server log file >>>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
Re: Installation Issues
Here is the output of platform-info.sh Metron Version 0.4.1 ansible 2.0.0.2 config file = /etc/ansible/ansible.cfg configured module search path = Default w/o overrides -- Vagrant 1.9.1 -- Python 2.7.12 -- Apache Maven 3.3.9 Maven home: /usr/share/maven Java version: 1.8.0_144, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-8-oracle/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.10.0-35-generic", arch: "amd64", family: "unix" -- Docker version 1.12.6, build 78d1802 -- node ./platform-info.sh: line 69: node: command not found -- npm ./platform-info.sh: line 74: npm: command not found -- Linux everyone 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 09:02:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux -- Total System Memory = 7946.98 MB Processor Model: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz Processor Speed: 3158.087 MHz Processor Speed: 3114.001 MHz Processor Speed: 2981.933 MHz Processor Speed: 2458.770 MHz Total Physical Processors: 4 Total cores: 16 Disk information: /dev/sda1 268G 21G 234G 9% / This CPU appears to support virtualization On Wed, Sep 27, 2017 at 1:06 PM, Syed Hammad Tahir wrote: > yes, which one should I pursue in order to find the issue? > > On Wed, Sep 27, 2017 at 12:50 PM, tkg_cangkul > wrote: > >> what alert that you see on ambari? there are 24 alert on your screenshot >> below. >> >> >> On 27/09/17 13:50, Syed Hammad Tahir wrote: >> >> Ambari server and agent both are running >> >> On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul >> wrote: >> >>> Maybe you can check the ambari-agent service first from the terminal. >>> If it stopped, just start it manually and then you can check the ambari >>> again. >>> >>> On 27/09/17 13:16, Syed Hammad Tahir wrote: >>> >>> This is what I see when I login into ambari. How do I check where >>> cluster deployment failed? >>> >>> [image: Inline image 1] >>> >>> On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris < >>> aaron.s.har...@outlook.com> wrote: >>> >>>> Syed, >>>> >>>> >>>> Have you checked if Ambari is running on the node? And if it is can you >>>> login and check what part the cluster deploy failed at. >>>> >>>> >>>> Regards, >>>> >>>> Aaron >>>> >>>> >>>> From: Syed Hammad Tahir >>>> Sent: Wednesday, 27 September, 06:28 >>>> Subject: Installation Issues >>>> To: user@metron.apache.org >>>> Cc: Muhammad Umar Janjua >>>> >>>> >>>> Ok, Re-did every thing again and got this error. This time on 12 GB RAM >>>> >>>> Will try on 16GB ram next time but is it actually related to RAM? >>>> >>>> >>>> >>> >>> >> >> >
Re: Installation Issues
yes, which one should I pursue in order to find the issue? On Wed, Sep 27, 2017 at 12:50 PM, tkg_cangkul wrote: > what alert that you see on ambari? there are 24 alert on your screenshot > below. > > > On 27/09/17 13:50, Syed Hammad Tahir wrote: > > Ambari server and agent both are running > > On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul > wrote: > >> Maybe you can check the ambari-agent service first from the terminal. >> If it stopped, just start it manually and then you can check the ambari >> again. >> >> On 27/09/17 13:16, Syed Hammad Tahir wrote: >> >> This is what I see when I login into ambari. How do I check where cluster >> deployment failed? >> >> [image: Inline image 1] >> >> On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris < >> aaron.s.har...@outlook.com> wrote: >> >>> Syed, >>> >>> >>> Have you checked if Ambari is running on the node? And if it is can you >>> login and check what part the cluster deploy failed at. >>> >>> >>> Regards, >>> >>> Aaron >>> >>> >>> From: Syed Hammad Tahir >>> Sent: Wednesday, 27 September, 06:28 >>> Subject: Installation Issues >>> To: user@metron.apache.org >>> Cc: Muhammad Umar Janjua >>> >>> >>> Ok, Re-did every thing again and got this error. This time on 12 GB RAM >>> >>> Will try on 16GB ram next time but is it actually related to RAM? >>> >>> >>> >> >> > >
Re: Installation Issues
Ambari server and agent both are running On Wed, Sep 27, 2017 at 11:49 AM, tkg_cangkul wrote: > Maybe you can check the ambari-agent service first from the terminal. > If it stopped, just start it manually and then you can check the ambari > again. > > On 27/09/17 13:16, Syed Hammad Tahir wrote: > > This is what I see when I login into ambari. How do I check where cluster > deployment failed? > > [image: Inline image 1] > > On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris > wrote: > >> Syed, >> >> >> Have you checked if Ambari is running on the node? And if it is can you >> login and check what part the cluster deploy failed at. >> >> >> Regards, >> >> Aaron >> >> >> From: Syed Hammad Tahir >> Sent: Wednesday, 27 September, 06:28 >> Subject: Installation Issues >> To: user@metron.apache.org >> Cc: Muhammad Umar Janjua >> >> >> Ok, Re-did every thing again and got this error. This time on 12 GB RAM >> >> Will try on 16GB ram next time but is it actually related to RAM? >> >> >> > >
Re: Installation Issues
This is what I see when I login into ambari. How do I check where cluster deployment failed? [image: Inline image 1] On Wed, Sep 27, 2017 at 10:54 AM, Aaron Harris wrote: > Syed, > > > Have you checked if Ambari is running on the node? And if it is can you > login and check what part the cluster deploy failed at. > > > Regards, > > Aaron > > > From: Syed Hammad Tahir > Sent: Wednesday, 27 September, 06:28 > Subject: Installation Issues > To: user@metron.apache.org > Cc: Muhammad Umar Janjua > > > Ok, Re-did every thing again and got this error. This time on 12 GB RAM > > Will try on 16GB ram next time but is it actually related to RAM? > > >
Re: Installation Issues
Provisioning a server grade machine is impossible at the moment. The current resources are the maximum I have to run metron or atleast just start it. On Tue, Sep 26, 2017 at 10:15 AM, Khurram Ahmed wrote: > Dear Hammad > Without getting into specifics of technical problems you are facing right > now. Based on my experience getting Metron to work on anything lower than > 32GB of RAM and a quad-core processor is usually difficult and leads to > unpredictable problems. I learned this the hard way and am willing to stand > by that assertion unless proven otherwise. Your best bet is to requisition > some server grade hardware from your university to test metron even if it's > just the dev version. > > > > On Tue, Sep 26, 2017 at 9:50 AM, Syed Hammad Tahir > wrote: > >> Hello everyone, any Idea how I can resolve this? >> >> [image: Inline image 1] >> > >
Installation Issues
Hello everyone, any Idea how I can resolve this? [image: Inline image 1]
Re: Metron Installation
I have increased the ram to 12 GB. The os I use is ubuntu so I guess metron might be able to get 8GB of ram. Have left the installation running back at office. Last command I left running was vagrant provision and will check tomorrow the outcome. On Mon, Sep 25, 2017 at 5:53 PM, zeo...@gmail.com wrote: > You need 8GB to dedicate 100% to the VM. Any other processes, the OS, > etc. will often absorb many more GB of RAM. In my scenario I usually test > on a VM on a ESXi host because my laptop only has 16 GB and more than half > of that is usually in use (leaving < 8GB for Metron testing). I don't > recall the specifics of your system, are you making sure you have over 8GB > *free* when you start spinning this up? > > Jon > > On Mon, Sep 25, 2017, 03:25 Syed Hammad Tahir > wrote: > >> But this guide says that 8gb ram is required (which I have) to run single >> node VM version >> >> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >> >> I am able to get into ambari and see this: >> >> [image: Inline image 1] >> >> From where can I see the error logs on whats going on? I just need to run >> metron . Please help me with that. >> >> Regards >> >> On Mon, Sep 25, 2017 at 12:19 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> This looks like it’s probably a timeout. From your other posts it sounds >>> like the machine you’re using is really not up to running the base platform >>> for Metron. I would strongly recommend going for something cloud based. >>> >>> I would also consider using the mpack method on an existing ambari, and >>> avoiding the ansible method, that will be a little less brittle. >>> >>> Simon >>> >>> >>> > On 25 Sep 2017, at 06:49, Syed Hammad Tahir >>> wrote: >>> > >>> > Any fix for this? >>> > >>> > >>> >>> >> -- > > Jon >
Re: Metron Installation
But this guide says that 8gb ram is required (which I have) to run single node VM version https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install I am able to get into ambari and see this: [image: Inline image 1] >From where can I see the error logs on whats going on? I just need to run metron . Please help me with that. Regards On Mon, Sep 25, 2017 at 12:19 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > This looks like it’s probably a timeout. From your other posts it sounds > like the machine you’re using is really not up to running the base platform > for Metron. I would strongly recommend going for something cloud based. > > I would also consider using the mpack method on an existing ambari, and > avoiding the ansible method, that will be a little less brittle. > > Simon > > > > On 25 Sep 2017, at 06:49, Syed Hammad Tahir > wrote: > > > > Any fix for this? > > > > > >
Metron Installation
Any fix for this? [image: Inline image 2]
Metron Installation
Hello everyone, I have been trying to install metron for over 2 weeks already and I havent got any success so far. I am doing it on my core i5 machine and have followed this guide so far: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 Please help me as its getting quite frustrating. I need this for my research. Regards
Help
Please help, I cant even find any folder named ambari in log files [image: Inline image 1] This is when I do vagrant provision
Re: System Requrements
1- The nodes are endpoints (desktops and laptops connected in lan and using shared internet) 2- They are behind NAT 3- They are for one primary user each. 4- These nodes are deployed in our university labs so there is no internet exposed service. On Wed, Sep 20, 2017 at 3:55 PM, zeo...@gmail.com wrote: > Okay, so I have some more questions then, but I'm still not sure how > helpful I can be. Maybe someone else with a similar environment can chime > in. > > These nodes, are they servers or endpoints (laptop/desktops used for > productivity - internet use, email, etc.)? Are they behind network > firewalls or NAT, or are they exposed? Are they shared machines or one > primary user each? If there are any internet exposed services, what are > they? > > Jon > > On Wed, Sep 20, 2017, 06:50 Syed Hammad Tahir > wrote: > >> Actually I need to forward the specs for my IT department as soon as >> possible, I was thinking to get a rough idea. >> Regards. >> >> On Wed, Sep 20, 2017 at 3:43 PM, zeo...@gmail.com >> wrote: >> >>> This is very much something Metron can do, but scoping hardware requires >>> more detail about the data and work to be done on the data. I would focus >>> on setting up the sensors (custom IDS, snort) and then either gather >>> metrics and scope Metron or just spin it up by default/with whatever you >>> have and see how it works. >>> >>> Jon >>> >>> On Wed, Sep 20, 2017, 06:23 Syed Hammad Tahir >>> wrote: >>> >>>> Hi, >>>> >>>> 1- I want to focus more on real time analysis but lets say we start >>>> with pcap dump, I dont know at this point that how much data it can dump in >>>> 24hr period given the lan environment of 100 nodes. You can assert your >>>> assumption to answer. >>>> >>>> 2- Snort data most probably and dont know about the nukber of events >>>> yes. You can also assert your assumption here for a hypothetical scenerio >>>> to guide me. >>>> >>>> 3- I want to build an intrusion detection system and apply some machine >>>> learning algorithm on it so Guess profiling is the answer to the third >>>> question. >>>> >>>> Based on those partial answers and your insight into this domain, >>>> kindly reply with most suitable solution with assumptions where necessary. >>>> >>>> If you think that I am expecting something from metron which it cant do >>>> then kindly let me know. >>>> >>>> Regards >>>> >>>> Regards. >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Sep 20, 2017 at 3:11 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> Full dev is intended for testing, not actual use. That said, to >>>>> answer your question it is more important to know (1) will you be storing >>>>> pcap, (1b) if so, how much per day and for how long, (2) what data will >>>>> you >>>>> be sending into Metron (bro, yaf, snort, asa, etc.) and how many events >>>>> per >>>>> second is it, and (3) what are you planning to do with the data >>>>> (profiling, >>>>> MaaS, enrichments, etc.)? >>>>> >>>>> Jon >>>>> >>>>> On Wed, Sep 20, 2017, 04:04 Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> What would be the system required in order to run metron and analyzy >>>>>> a LAN environment of almost 100 nodes using single node full development >>>>>> depoloyment. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: System Requrements
Actually I need to forward the specs for my IT department as soon as possible, I was thinking to get a rough idea. Regards. On Wed, Sep 20, 2017 at 3:43 PM, zeo...@gmail.com wrote: > This is very much something Metron can do, but scoping hardware requires > more detail about the data and work to be done on the data. I would focus > on setting up the sensors (custom IDS, snort) and then either gather > metrics and scope Metron or just spin it up by default/with whatever you > have and see how it works. > > Jon > > On Wed, Sep 20, 2017, 06:23 Syed Hammad Tahir > wrote: > >> Hi, >> >> 1- I want to focus more on real time analysis but lets say we start with >> pcap dump, I dont know at this point that how much data it can dump in 24hr >> period given the lan environment of 100 nodes. You can assert your >> assumption to answer. >> >> 2- Snort data most probably and dont know about the nukber of events yes. >> You can also assert your assumption here for a hypothetical scenerio to >> guide me. >> >> 3- I want to build an intrusion detection system and apply some machine >> learning algorithm on it so Guess profiling is the answer to the third >> question. >> >> Based on those partial answers and your insight into this domain, kindly >> reply with most suitable solution with assumptions where necessary. >> >> If you think that I am expecting something from metron which it cant do >> then kindly let me know. >> >> Regards >> >> Regards. >> >> >> >> >> >> On Wed, Sep 20, 2017 at 3:11 PM, zeo...@gmail.com >> wrote: >> >>> Full dev is intended for testing, not actual use. That said, to answer >>> your question it is more important to know (1) will you be storing pcap, >>> (1b) if so, how much per day and for how long, (2) what data will you be >>> sending into Metron (bro, yaf, snort, asa, etc.) and how many events per >>> second is it, and (3) what are you planning to do with the data (profiling, >>> MaaS, enrichments, etc.)? >>> >>> Jon >>> >>> On Wed, Sep 20, 2017, 04:04 Syed Hammad Tahir >>> wrote: >>> >>>> Hello, >>>> >>>> What would be the system required in order to run metron and analyzy a >>>> LAN environment of almost 100 nodes using single node full development >>>> depoloyment. >>>> >>>> Regards. >>>> >>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: System Requrements
Hi, 1- I want to focus more on real time analysis but lets say we start with pcap dump, I dont know at this point that how much data it can dump in 24hr period given the lan environment of 100 nodes. You can assert your assumption to answer. 2- Snort data most probably and dont know about the nukber of events yes. You can also assert your assumption here for a hypothetical scenerio to guide me. 3- I want to build an intrusion detection system and apply some machine learning algorithm on it so Guess profiling is the answer to the third question. Based on those partial answers and your insight into this domain, kindly reply with most suitable solution with assumptions where necessary. If you think that I am expecting something from metron which it cant do then kindly let me know. Regards Regards. On Wed, Sep 20, 2017 at 3:11 PM, zeo...@gmail.com wrote: > Full dev is intended for testing, not actual use. That said, to answer > your question it is more important to know (1) will you be storing pcap, > (1b) if so, how much per day and for how long, (2) what data will you be > sending into Metron (bro, yaf, snort, asa, etc.) and how many events per > second is it, and (3) what are you planning to do with the data (profiling, > MaaS, enrichments, etc.)? > > Jon > > On Wed, Sep 20, 2017, 04:04 Syed Hammad Tahir > wrote: > >> Hello, >> >> What would be the system required in order to run metron and analyzy a >> LAN environment of almost 100 nodes using single node full development >> depoloyment. >> >> Regards. >> > -- > > Jon >
System Requrements
Hello, What would be the system required in order to run metron and analyzy a LAN environment of almost 100 nodes using single node full development depoloyment. Regards.
Re: Metron Installation
Is this guide also similar to one you provided? https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 On Fri, Sep 15, 2017 at 8:07 PM, Ryan Merriman wrote: > I'm not sure, have only built it with Mac OS. Others may have guidance > for other OSs. > > On Fri, Sep 15, 2017 at 10:01 AM, Syed Hammad Tahir > wrote: > >> Can it be done on ubuntu or some other linux distribution or MAC OS is a >> must? >> >> >> On Fri, Sep 15, 2017 at 7:37 PM, Ryan Merriman >> wrote: >> >>> Instructions for starting full dev are on that page: >>> https://github.com/apache/metron/tree/master/metron-deploym >>> ent/vagrant/full-dev-platform#deploy-metron. >>> >>> On Fri, Sep 15, 2017 at 9:26 AM, Syed Hammad Tahir >> > wrote: >>> >>>> yes but that link just states the prerequisites and not a guide, can I >>>> find a full guide somewhere to install the full dev environment? >>>> >>>> On Fri, Sep 15, 2017 at 7:10 PM, Ryan Merriman >>>> wrote: >>>> >>>>> If you are just looking to demo it (which I assume is true because >>>>> you're installing on a desktop), our full dev environment might be a >>>>> better >>>>> approach for you and get you up and running faster: >>>>> https://github.com/apache/metron/tree/master/metron-deploym >>>>> ent/vagrant/full-dev-platform. >>>>> >>>>> Ryan >>>>> >>>>> On Fri, Sep 15, 2017 at 9:06 AM, Syed Hammad Tahir < >>>>> mscs16...@itu.edu.pk> wrote: >>>>> >>>>>> Hi, THis guide seems to be for 3 nodes but would it also work if I >>>>>> install cent OS 7 on my desktop? >>>>>> >>>>>> On Fri, Sep 15, 2017 at 3:30 PM, kotipalli venkatesh < >>>>>> kotipallivenkates...@gmail.com> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> [image: Inline image 1] >>>>>>> >>>>>>> On Fri, Sep 15, 2017 at 3:33 PM, kotipalli venkatesh < >>>>>>> kotipallivenkates...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> Please go through the below link, >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Metron+0. >>>>>>>> 4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB >>>>>>>> +for+Metron+REST >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Venkatesh >>>>>>>> >>>>>>>> On Fri, Sep 15, 2017 at 3:27 PM, Khurram Ahmed < >>>>>>>> khurramah...@gmail.com> wrote: >>>>>>>> >>>>>>>>> My experience was extremely painful and I gave up shifting to a >>>>>>>>> server machine with loads of RAM and processing power. >>>>>>>>> >>>>>>>>> On Sep 15, 2017 2:51 PM, "Syed Hammad Tahir" >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Even a basic VM install wont work? It says that 8Gb ram might >>>>>>>>>> work. >>>>>>>>>> >>>>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >>>>>>>>>> >>>>>>>>>> On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed < >>>>>>>>>> khurramah...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Your specs might not be enough to run Metron. Anything below >>>>>>>>>>> 32gb of RAM will not work. You need server grade machines for >>>>>>>>>>> Metron to >>>>>>>>>>> work reliably. >>>>>>>>>>> >>>>>>>>>>> On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" < >>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>> >>>>>>>>>>> My PC is core i5, 8GB RAM and a few hundred GBs of disk space. >>>>>>>>>>> It doesnt have any OS as I will install it as per the >>>>>>>>>>> recommendations in >>>>>>>>>>> the guide. >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < >>>>>>>>>>> kotipallivenkates...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi >>>>>>>>>>>> >>>>>>>>>>>> please share the more information, which operating system your >>>>>>>>>>>> PC. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Venkatesh >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir < >>>>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> I need a guide to install metron on my PC from scratch. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Metron Installation
Can it be done on ubuntu or some other linux distribution or MAC OS is a must? On Fri, Sep 15, 2017 at 7:37 PM, Ryan Merriman wrote: > Instructions for starting full dev are on that page: > https://github.com/apache/metron/tree/master/metron- > deployment/vagrant/full-dev-platform#deploy-metron. > > On Fri, Sep 15, 2017 at 9:26 AM, Syed Hammad Tahir > wrote: > >> yes but that link just states the prerequisites and not a guide, can I >> find a full guide somewhere to install the full dev environment? >> >> On Fri, Sep 15, 2017 at 7:10 PM, Ryan Merriman >> wrote: >> >>> If you are just looking to demo it (which I assume is true because >>> you're installing on a desktop), our full dev environment might be a better >>> approach for you and get you up and running faster: >>> https://github.com/apache/metron/tree/master/metron-deploym >>> ent/vagrant/full-dev-platform. >>> >>> Ryan >>> >>> On Fri, Sep 15, 2017 at 9:06 AM, Syed Hammad Tahir >> > wrote: >>> >>>> Hi, THis guide seems to be for 3 nodes but would it also work if I >>>> install cent OS 7 on my desktop? >>>> >>>> On Fri, Sep 15, 2017 at 3:30 PM, kotipalli venkatesh < >>>> kotipallivenkates...@gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> [image: Inline image 1] >>>>> >>>>> On Fri, Sep 15, 2017 at 3:33 PM, kotipalli venkatesh < >>>>> kotipallivenkates...@gmail.com> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> Please go through the below link, >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/METRON/Metron+0. >>>>>> 4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB >>>>>> +for+Metron+REST >>>>>> >>>>>> Thanks, >>>>>> Venkatesh >>>>>> >>>>>> On Fri, Sep 15, 2017 at 3:27 PM, Khurram Ahmed < >>>>>> khurramah...@gmail.com> wrote: >>>>>> >>>>>>> My experience was extremely painful and I gave up shifting to a >>>>>>> server machine with loads of RAM and processing power. >>>>>>> >>>>>>> On Sep 15, 2017 2:51 PM, "Syed Hammad Tahir" >>>>>>> wrote: >>>>>>> >>>>>>>> Even a basic VM install wont work? It says that 8Gb ram might work. >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >>>>>>>> >>>>>>>> On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed < >>>>>>>> khurramah...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Your specs might not be enough to run Metron. Anything below 32gb >>>>>>>>> of RAM will not work. You need server grade machines for Metron to >>>>>>>>> work >>>>>>>>> reliably. >>>>>>>>> >>>>>>>>> On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It >>>>>>>>> doesnt have any OS as I will install it as per the recommendations in >>>>>>>>> the >>>>>>>>> guide. >>>>>>>>> >>>>>>>>> On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < >>>>>>>>> kotipallivenkates...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> please share the more information, which operating system your PC. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Venkatesh >>>>>>>>>> >>>>>>>>>> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir < >>>>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I need a guide to install metron on my PC from scratch. >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Metron Installation
yes but that link just states the prerequisites and not a guide, can I find a full guide somewhere to install the full dev environment? On Fri, Sep 15, 2017 at 7:10 PM, Ryan Merriman wrote: > If you are just looking to demo it (which I assume is true because you're > installing on a desktop), our full dev environment might be a better > approach for you and get you up and running faster: > https://github.com/apache/metron/tree/master/metron- > deployment/vagrant/full-dev-platform. > > Ryan > > On Fri, Sep 15, 2017 at 9:06 AM, Syed Hammad Tahir > wrote: > >> Hi, THis guide seems to be for 3 nodes but would it also work if I >> install cent OS 7 on my desktop? >> >> On Fri, Sep 15, 2017 at 3:30 PM, kotipalli venkatesh < >> kotipallivenkates...@gmail.com> wrote: >> >>> Hi >>> >>> [image: Inline image 1] >>> >>> On Fri, Sep 15, 2017 at 3:33 PM, kotipalli venkatesh < >>> kotipallivenkates...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> Please go through the below link, >>>> >>>> https://cwiki.apache.org/confluence/display/METRON/Metron+0. >>>> 4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB >>>> +for+Metron+REST >>>> >>>> Thanks, >>>> Venkatesh >>>> >>>> On Fri, Sep 15, 2017 at 3:27 PM, Khurram Ahmed >>>> wrote: >>>> >>>>> My experience was extremely painful and I gave up shifting to a server >>>>> machine with loads of RAM and processing power. >>>>> >>>>> On Sep 15, 2017 2:51 PM, "Syed Hammad Tahir" >>>>> wrote: >>>>> >>>>>> Even a basic VM install wont work? It says that 8Gb ram might work. >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >>>>>> >>>>>> On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed < >>>>>> khurramah...@gmail.com> wrote: >>>>>> >>>>>>> Your specs might not be enough to run Metron. Anything below 32gb of >>>>>>> RAM will not work. You need server grade machines for Metron to work >>>>>>> reliably. >>>>>>> >>>>>>> On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" >>>>>>> wrote: >>>>>>> >>>>>>> My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It >>>>>>> doesnt have any OS as I will install it as per the recommendations in >>>>>>> the >>>>>>> guide. >>>>>>> >>>>>>> On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < >>>>>>> kotipallivenkates...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> please share the more information, which operating system your PC. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Venkatesh >>>>>>>> >>>>>>>> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir < >>>>>>>> mscs16...@itu.edu.pk> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I need a guide to install metron on my PC from scratch. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> >> >
Re: Metron Installation
Hi, THis guide seems to be for 3 nodes but would it also work if I install cent OS 7 on my desktop? On Fri, Sep 15, 2017 at 3:30 PM, kotipalli venkatesh < kotipallivenkates...@gmail.com> wrote: > Hi > > [image: Inline image 1] > > On Fri, Sep 15, 2017 at 3:33 PM, kotipalli venkatesh < > kotipallivenkates...@gmail.com> wrote: > >> Hi >> >> Please go through the below link, >> >> https://cwiki.apache.org/confluence/display/METRON/Metron+0. >> 4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+ >> MariaDB+for+Metron+REST >> >> Thanks, >> Venkatesh >> >> On Fri, Sep 15, 2017 at 3:27 PM, Khurram Ahmed >> wrote: >> >>> My experience was extremely painful and I gave up shifting to a server >>> machine with loads of RAM and processing power. >>> >>> On Sep 15, 2017 2:51 PM, "Syed Hammad Tahir" >>> wrote: >>> >>>> Even a basic VM install wont work? It says that 8Gb ram might work. >>>> >>>> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >>>> >>>> On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed >>>> wrote: >>>> >>>>> Your specs might not be enough to run Metron. Anything below 32gb of >>>>> RAM will not work. You need server grade machines for Metron to work >>>>> reliably. >>>>> >>>>> On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" >>>>> wrote: >>>>> >>>>> My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It >>>>> doesnt have any OS as I will install it as per the recommendations in the >>>>> guide. >>>>> >>>>> On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < >>>>> kotipallivenkates...@gmail.com> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> please share the more information, which operating system your PC. >>>>>> >>>>>> Thanks, >>>>>> Venkatesh >>>>>> >>>>>> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir < >>>>>> mscs16...@itu.edu.pk> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I need a guide to install metron on my PC from scratch. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >> >
Re: Metron Installation
ok, I am doing it for my graduate level research. I will try to do it on a PC first and then move to a more powerful workstation if needed. Kindly give me the guide to install it on a PC. On Fri, Sep 15, 2017 at 2:57 PM, Khurram Ahmed wrote: > My experience was extremely painful and I gave up shifting to a server > machine with loads of RAM and processing power. > > On Sep 15, 2017 2:51 PM, "Syed Hammad Tahir" wrote: > >> Even a basic VM install wont work? It says that 8Gb ram might work. >> >> https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install >> >> On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed >> wrote: >> >>> Your specs might not be enough to run Metron. Anything below 32gb of RAM >>> will not work. You need server grade machines for Metron to work reliably. >>> >>> On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" >>> wrote: >>> >>> My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It doesnt >>> have any OS as I will install it as per the recommendations in the guide. >>> >>> On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < >>> kotipallivenkates...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> please share the more information, which operating system your PC. >>>> >>>> Thanks, >>>> Venkatesh >>>> >>>> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir < >>>> mscs16...@itu.edu.pk> wrote: >>>> >>>>> Hello, >>>>> >>>>> I need a guide to install metron on my PC from scratch. >>>>> >>>>> Regards >>>>> >>>> >>>> >>> >>> >>
Re: Metron Installation
Even a basic VM install wont work? It says that 8Gb ram might work. https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install On Fri, Sep 15, 2017 at 2:48 PM, Khurram Ahmed wrote: > Your specs might not be enough to run Metron. Anything below 32gb of RAM > will not work. You need server grade machines for Metron to work reliably. > > On Sep 15, 2017 2:41 PM, "Syed Hammad Tahir" wrote: > > My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It doesnt > have any OS as I will install it as per the recommendations in the guide. > > On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < > kotipallivenkates...@gmail.com> wrote: > >> Hi >> >> please share the more information, which operating system your PC. >> >> Thanks, >> Venkatesh >> >> On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir >> wrote: >> >>> Hello, >>> >>> I need a guide to install metron on my PC from scratch. >>> >>> Regards >>> >> >> > >
Re: Metron Installation
My PC is core i5, 8GB RAM and a few hundred GBs of disk space. It doesnt have any OS as I will install it as per the recommendations in the guide. On Fri, Sep 15, 2017 at 2:29 PM, kotipalli venkatesh < kotipallivenkates...@gmail.com> wrote: > Hi > > please share the more information, which operating system your PC. > > Thanks, > Venkatesh > > On Fri, Sep 15, 2017 at 2:57 PM, Syed Hammad Tahir > wrote: > >> Hello, >> >> I need a guide to install metron on my PC from scratch. >> >> Regards >> > >
Metron Installation
Hello, I need a guide to install metron on my PC from scratch. Regards
Re: Getting Started
Thankyou. I will start with the VM and will ask if I need any further assistance. On Thursday, September 7, 2017, zeo...@gmail.com wrote: > When I say sensors I'm referring to tools that would feed into Metron like > bro, yaf, snort, etc. > > Jon > > On Thu, Sep 7, 2017, 09:13 Syed Hammad Tahir > wrote: > >> I will confirm about batch or streaming data. The sensors you mentioned, >> are they some particular devices or you are referring to sniffers or >> builtin Metron tools? >> >> On Thursday, September 7, 2017, zeo...@gmail.com > > wrote: >> >>> Okay so that sounds much easier - will it be done in batches or >>> streaming (the network data processing, not the analytics)? I assume the >>> former, given your situation. If that's true and you don't have huge >>> amounts of data you may be able to do everything in full dev or an >>> equivalent VM. A lot of this depends on what you will be feeding into >>> Metron, and to know that you need to set up the sensors and get the network >>> traffic first. >>> >>> Jon >>> >>> On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir >>> wrote: >>> >>>> Hi, >>>> >>>> What I wanted to do with this is the following: >>>> >>>> 1- Gather Network Data >>>> >>>> 2- Analyse it >>>> >>>> 3- Apply some machine learning algorithm to detect intrusion >>>> >>>> >>>> Now by seeking the use of Metron framework, am I following the right >>>> track here? >>>> >>>> >>>> Regards. >>>> >>>> On Wed, Sep 6, 2017 at 6:10 PM, zeo...@gmail.com >>>> wrote: >>>> >>>>> I would start with getting the data sources (syslog, bro data, snort >>>>> logs, etc.) first. Without knowing the architecture of those tools makes >>>>> it very difficult to suggest an install method, although for prod use I >>>>> would always default to a bare metal install. In your case you don't seem >>>>> interested in PCAP, which means you _may_ be able to get away with >>>>> something in EC2 or similar. >>>>> >>>>> Jon >>>>> >>>>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Thankyou for answering my call to help. >>>>>> >>>>>> I am going to use it for the purpose of research at graduate level, >>>>>> and may scale it on a production level. I am targeting a few labs on this >>>>>> floor , that approximately accumulates upto 30-40 people using the >>>>>> network. >>>>>> I am open to options of using YAF, BRO, SNORT and others. Once started >>>>>> then I may also expand it in the future. What are your recommendations on >>>>>> the stated requirements. >>>>>> >>>>>> Best Regards. >>>>>> >>>>>> On Wed, Sep 6, 2017 at 3:06 PM, zeo...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> There are a few questions that need to be answered first. How do >>>>>>> you plan to monitor the LAN? Are you going to run YAF, Bro, Snort, >>>>>>> others? How big is your LAN, how much traffic traverses it, what is the >>>>>>> traffic composition (heavily impacts the amount of logs from >>>>>>> Bro/YAF/Snort), how much retention of data do you want, do you plan to >>>>>>> store PCAP? >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I intend to use Apache Metron framework for the analysis of our >>>>>>>> local area network. What is the best way to get started? Which >>>>>>>> installation >>>>>>>> is most suitable for me as listed in the following link: >>>>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation >>>>>>>> >>>>>>>> Kindly help me with this. >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jon >>>>>>> >>>>>> >>>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> -- > > Jon >
Re: Getting Started
I will confirm about batch or streaming data. The sensors you mentioned, are they some particular devices or you are referring to sniffers or builtin Metron tools? On Thursday, September 7, 2017, zeo...@gmail.com wrote: > Okay so that sounds much easier - will it be done in batches or streaming > (the network data processing, not the analytics)? I assume the former, > given your situation. If that's true and you don't have huge amounts of > data you may be able to do everything in full dev or an equivalent VM. A > lot of this depends on what you will be feeding into Metron, and to know > that you need to set up the sensors and get the network traffic first. > > Jon > > On Thu, Sep 7, 2017, 00:40 Syed Hammad Tahir > wrote: > >> Hi, >> >> What I wanted to do with this is the following: >> >> 1- Gather Network Data >> >> 2- Analyse it >> >> 3- Apply some machine learning algorithm to detect intrusion >> >> >> Now by seeking the use of Metron framework, am I following the right >> track here? >> >> >> Regards. >> >> On Wed, Sep 6, 2017 at 6:10 PM, zeo...@gmail.com > > wrote: >> >>> I would start with getting the data sources (syslog, bro data, snort >>> logs, etc.) first. Without knowing the architecture of those tools makes >>> it very difficult to suggest an install method, although for prod use I >>> would always default to a bare metal install. In your case you don't seem >>> interested in PCAP, which means you _may_ be able to get away with >>> something in EC2 or similar. >>> >>> Jon >>> >>> On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir >> > wrote: >>> >>>> Hello, >>>> >>>> Thankyou for answering my call to help. >>>> >>>> I am going to use it for the purpose of research at graduate level, and >>>> may scale it on a production level. I am targeting a few labs on this floor >>>> , that approximately accumulates upto 30-40 people using the network. I am >>>> open to options of using YAF, BRO, SNORT and others. Once started then I >>>> may also expand it in the future. What are your recommendations on the >>>> stated requirements. >>>> >>>> Best Regards. >>>> >>>> On Wed, Sep 6, 2017 at 3:06 PM, zeo...@gmail.com >>> > wrote: >>>> >>>>> There are a few questions that need to be answered first. How do you >>>>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others? >>>>> How big is your LAN, how much traffic traverses it, what is the traffic >>>>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how >>>>> much retention of data do you want, do you plan to store PCAP? >>>>> >>>>> Jon >>>>> >>>>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir >>>> > wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I intend to use Apache Metron framework for the analysis of our local >>>>>> area network. What is the best way to get started? Which installation is >>>>>> most suitable for me as listed in the following link: >>>>>> https://cwiki.apache.org/confluence/display/METRON/Installation >>>>>> >>>>>> Kindly help me with this. >>>>>> >>>>>> Regards. >>>>>> >>>>> -- >>>>> >>>>> Jon >>>>> >>>> >>>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Getting Started
Hi, What I wanted to do with this is the following: 1- Gather Network Data 2- Analyse it 3- Apply some machine learning algorithm to detect intrusion Now by seeking the use of Metron framework, am I following the right track here? Regards. On Wed, Sep 6, 2017 at 6:10 PM, zeo...@gmail.com wrote: > I would start with getting the data sources (syslog, bro data, snort logs, > etc.) first. Without knowing the architecture of those tools makes it very > difficult to suggest an install method, although for prod use I would > always default to a bare metal install. In your case you don't seem > interested in PCAP, which means you _may_ be able to get away with > something in EC2 or similar. > > Jon > > On Wed, Sep 6, 2017 at 6:41 AM Syed Hammad Tahir > wrote: > >> Hello, >> >> Thankyou for answering my call to help. >> >> I am going to use it for the purpose of research at graduate level, and >> may scale it on a production level. I am targeting a few labs on this floor >> , that approximately accumulates upto 30-40 people using the network. I am >> open to options of using YAF, BRO, SNORT and others. Once started then I >> may also expand it in the future. What are your recommendations on the >> stated requirements. >> >> Best Regards. >> >> On Wed, Sep 6, 2017 at 3:06 PM, zeo...@gmail.com >> wrote: >> >>> There are a few questions that need to be answered first. How do you >>> plan to monitor the LAN? Are you going to run YAF, Bro, Snort, others? >>> How big is your LAN, how much traffic traverses it, what is the traffic >>> composition (heavily impacts the amount of logs from Bro/YAF/Snort), how >>> much retention of data do you want, do you plan to store PCAP? >>> >>> Jon >>> >>> On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir >>> wrote: >>> >>>> Hello, >>>> >>>> I intend to use Apache Metron framework for the analysis of our local >>>> area network. What is the best way to get started? Which installation is >>>> most suitable for me as listed in the following link: >>>> https://cwiki.apache.org/confluence/display/METRON/Installation >>>> >>>> Kindly help me with this. >>>> >>>> Regards. >>>> >>> -- >>> >>> Jon >>> >> >> -- > > Jon >
Re: Getting Started
Hello, Thankyou for answering my call to help. I am going to use it for the purpose of research at graduate level, and may scale it on a production level. I am targeting a few labs on this floor , that approximately accumulates upto 30-40 people using the network. I am open to options of using YAF, BRO, SNORT and others. Once started then I may also expand it in the future. What are your recommendations on the stated requirements. Best Regards. On Wed, Sep 6, 2017 at 3:06 PM, zeo...@gmail.com wrote: > There are a few questions that need to be answered first. How do you plan > to monitor the LAN? Are you going to run YAF, Bro, Snort, others? How big > is your LAN, how much traffic traverses it, what is the traffic composition > (heavily impacts the amount of logs from Bro/YAF/Snort), how much retention > of data do you want, do you plan to store PCAP? > > Jon > > On Wed, Sep 6, 2017, 01:59 Syed Hammad Tahir wrote: > >> Hello, >> >> I intend to use Apache Metron framework for the analysis of our local >> area network. What is the best way to get started? Which installation is >> most suitable for me as listed in the following link: >> https://cwiki.apache.org/confluence/display/METRON/Installation >> >> Kindly help me with this. >> >> Regards. >> > -- > > Jon >
Getting Started
Hello, I intend to use Apache Metron framework for the analysis of our local area network. What is the best way to get started? Which installation is most suitable for me as listed in the following link: https://cwiki.apache.org/confluence/display/METRON/Installation Kindly help me with this. Regards.
