Yes, make sure the disk is local and not low latency shared one (e.g. SAN). Also SSD will probably fix all your pains.
-- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Lucene.NET committer and PMC member On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa <iyuv...@gmail.com> wrote: > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > special configurations I should do in such a case? > > Thanks, > Yuval. > > On Thursday, February 12, 2015, Itamar Syn-Hershko <ita...@code972.com> > wrote: > >> Yes - can you try using the bulk API? Also, are you running on a cloud >> server? >> >> -- >> >> Itamar Syn-Hershko >> http://code972.com | @synhershko <https://twitter.com/synhershko> >> Freelance Developer & Consultant >> Lucene.NET committer and PMC member >> >> On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa <iyuv...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I wrote that program and ran it and it did managed to keep a steady rate >>> of about 1,000 events per minute even when the Kibana's total events per >>> minute dropped from 60,000 to 6,000. However, when the >>> Kibana's total events per minute dropped to zero, my program got a >>> "connection refused" exception. I ran netstat -s and found out that every >>> time the Kibana's line hit zero the number of RX-DRP increased. At that >>> point I understood that I forgot to mention that this server has a 10GbE >>> nic. Is it possible that the packets are being dropped because of some >>> bufferis filling up? If so, how can I test it and verify that this is >>> actually the case? If it is, how can I solve it? >>> >>> Thanks, >>> Yuval. >>> On Wednesday, February 11, 2015, Yuval Khalifa <iyuv...@gmail.com> >>> wrote: >>> >>>> Hi. >>>> >>>> When you say "see how the file behaves" I'm not quite sure what you >>>> mean by that... As I mentioned earlier, it's not that events do not appear >>>> at all but instead, the RATE at which they come decreases, so how can I >>>> measure the events rate in a file? I thought that there's another way that >>>> I can test this: I'll write a quick-and-dirty program that will send an >>>> event to the ELK via TCP every 12ms which should result in events rate of >>>> about 5,000 events per minute and I'll let you know if the events rate >>>> continues to drop or not... >>>> >>>> >>>> Thanks, >>>> Yuval. >>>> >>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko <ita...@code972.com> >>>> wrote: >>>> >>>>> I'd start by using logstash with input tcp and output fs and see how >>>>> the file behaves. Same for the fs inputs - see how their files behave. And >>>>> take it from there. >>>>> >>>>> -- >>>>> >>>>> Itamar Syn-Hershko >>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>> Freelance Developer & Consultant >>>>> Lucene.NET committer and PMC member >>>>> >>>>> On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa <iyuv...@gmail.com> >>>>> wrote: >>>>> >>>>>> Great! How can I check that? >>>>>> >>>>>> >>>>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko <ita...@code972.com> >>>>>> wrote: >>>>>> >>>>>>> The graphic you sent suggests the issue is with logstash - since the >>>>>>> @timestamp field is being populated by logstash and is the one that is >>>>>>> used >>>>>>> to display the date histogram graphics in Kibana. I would start there. >>>>>>> I.e. >>>>>>> maybe SecurityOnion buffers writes etc, and then to check the logstash >>>>>>> shipper process stats. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Itamar Syn-Hershko >>>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>>>> Freelance Developer & Consultant >>>>>>> Lucene.NET committer and PMC member >>>>>>> >>>>>>> On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa <iyuv...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi. >>>>>>>> >>>>>>>> Absolutely (but since that in the past I also worked at >>>>>>>> the helpdesk dept. I certainly understand why it is important to ask >>>>>>>> those >>>>>>>> "Are you sure it's plugged in?" questions...). One of the logs is >>>>>>>> comming >>>>>>>> from SecurityOnion which logs (via bro-conn) all the connections so it >>>>>>>> must >>>>>>>> be sending data 24x7x365. >>>>>>>> >>>>>>>> Thanks for the quick reply, >>>>>>>> Yuval. >>>>>>>> >>>>>>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko < >>>>>>>> ita...@code972.com> wrote: >>>>>>>> >>>>>>>>> Are you sure your logs are generated linearly without bursts? >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Itamar Syn-Hershko >>>>>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>>>>>> Freelance Developer & Consultant >>>>>>>>> Lucene.NET committer and PMC member >>>>>>>>> >>>>>>>>> On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa <iyuv...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> We just installed an ELK server and configured the logstash >>>>>>>>>> configuration to match the data that we send to it and until last >>>>>>>>>> month it >>>>>>>>>> seems to be working fine but since then we see very strange behavior >>>>>>>>>> in the >>>>>>>>>> Kibana, the event over time histogram shows the event rate at the >>>>>>>>>> normal >>>>>>>>>> level for about a half an hour, then drops to about 20% of the >>>>>>>>>> normal rate >>>>>>>>>> and then it continues to drop slowly for about two hours and then >>>>>>>>>> stops and >>>>>>>>>> after a minute or two it returns to normal for the next half an hour >>>>>>>>>> or so >>>>>>>>>> and the same behavior repeats. Needless to say that both the >>>>>>>>>> /var/log/logstash and /var/log/elasticsearch both show nothing since >>>>>>>>>> the >>>>>>>>>> service started and by using tcpdump we can verify that events keep >>>>>>>>>> coming >>>>>>>>>> in at the same rate all time. I attached our logstash configuration, >>>>>>>>>> the >>>>>>>>>> /var/logstash/logstash.log, the >>>>>>>>>> /var/log/elasticsearch/clustername.log and >>>>>>>>>> a screenshot of our Kibana with no filter applied so that you can >>>>>>>>>> see the >>>>>>>>>> weird behavior that we see. >>>>>>>>>> >>>>>>>>>> Is there someone/somewhere that we can turn to to get some help >>>>>>>>>> on the subject? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks a lot, >>>>>>>>>> Yuval. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "elasticsearch" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> בברכה, >>>>>>>> >>>>>>>> *יובל כליפא* >>>>>>>> >>>>>>>> CTO >>>>>>>> תחום מערכות מידע | מגדל סוכנויות. >>>>>>>> נייד: 052-3336098 >>>>>>>> משרד: 03-7966565 >>>>>>>> פקס: 03-7976565 >>>>>>>> בלוג: http://www.artifex.co.il >>>>>>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>>>>>> >>>>>>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> בברכה, >>>>>> >>>>>> *יובל כליפא* >>>>>> >>>>>> CTO >>>>>> תחום מערכות מידע | מגדל סוכנויות. >>>>>> נייד: 052-3336098 >>>>>> משרד: 03-7966565 >>>>>> פקס: 03-7976565 >>>>>> בלוג: http://www.artifex.co.il >>>>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>>>> >>>>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> >>>> בברכה, >>>> >>>> *יובל כליפא* >>>> >>>> CTO >>>> תחום מערכות מידע | מגדל סוכנויות. >>>> נייד: 052-3336098 >>>> משרד: 03-7966565 >>>> פקס: 03-7976565 >>>> בלוג: http://www.artifex.co.il >>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>> >>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>> >>>> >>> >>> -- >>> >>> בברכה, >>> >>> *יובל כליפא* >>> >>> CTO >>> תחום מערכות מידע | מגדל סוכנויות. >>> נייד: 052-3336098 >>> משרד: 03-7966565 >>> פקס: 03-7976565 >>> בלוג: http://www.artifex.co.il >>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>> >>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > > בברכה, > > *יובל כליפא* > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: http://www.artifex.co.il > <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> > > *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
