Well SSD would also fix all the pains for my bank too... (-; Are you sure it's caused by disk latency and not some sort of mis-tuned TCP driver? I've read some blogs that recommeded to increase some of the buffers at the sysctl.conf. Do you think so too?
On Thursday, February 12, 2015, Itamar Syn-Hershko <ita...@code972.com> wrote: > Yes, make sure the disk is local and not low latency shared one (e.g. > SAN). Also SSD will probably fix all your pains. > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Lucene.NET committer and PMC member > > On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa <iyuv...@gmail.com > <javascript:_e(%7B%7D,'cvml','iyuv...@gmail.com');>> wrote: > >> Sort of... The ELK is running as a VM on a dedicated ESXi. Are there >> special configurations I should do in such a case? >> >> Thanks, >> Yuval. >> >> On Thursday, February 12, 2015, Itamar Syn-Hershko <ita...@code972.com >> <javascript:_e(%7B%7D,'cvml','ita...@code972.com');>> wrote: >> >>> Yes - can you try using the bulk API? Also, are you running on a cloud >>> server? >>> >>> -- >>> >>> Itamar Syn-Hershko >>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>> Freelance Developer & Consultant >>> Lucene.NET committer and PMC member >>> >>> On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa <iyuv...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I wrote that program and ran it and it did managed to keep a steady >>>> rate of about 1,000 events per minute even when the Kibana's total events >>>> per minute dropped from 60,000 to 6,000. However, when the >>>> Kibana's total events per minute dropped to zero, my program got a >>>> "connection refused" exception. I ran netstat -s and found out that every >>>> time the Kibana's line hit zero the number of RX-DRP increased. At that >>>> point I understood that I forgot to mention that this server has a 10GbE >>>> nic. Is it possible that the packets are being dropped because of some >>>> bufferis filling up? If so, how can I test it and verify that this is >>>> actually the case? If it is, how can I solve it? >>>> >>>> Thanks, >>>> Yuval. >>>> On Wednesday, February 11, 2015, Yuval Khalifa <iyuv...@gmail.com> >>>> wrote: >>>> >>>>> Hi. >>>>> >>>>> When you say "see how the file behaves" I'm not quite sure what you >>>>> mean by that... As I mentioned earlier, it's not that events do not appear >>>>> at all but instead, the RATE at which they come decreases, so how can I >>>>> measure the events rate in a file? I thought that there's another way that >>>>> I can test this: I'll write a quick-and-dirty program that will send an >>>>> event to the ELK via TCP every 12ms which should result in events rate of >>>>> about 5,000 events per minute and I'll let you know if the events rate >>>>> continues to drop or not... >>>>> >>>>> >>>>> Thanks, >>>>> Yuval. >>>>> >>>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko <ita...@code972.com> >>>>> wrote: >>>>> >>>>>> I'd start by using logstash with input tcp and output fs and see how >>>>>> the file behaves. Same for the fs inputs - see how their files behave. >>>>>> And >>>>>> take it from there. >>>>>> >>>>>> -- >>>>>> >>>>>> Itamar Syn-Hershko >>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>>> Freelance Developer & Consultant >>>>>> Lucene.NET committer and PMC member >>>>>> >>>>>> On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa <iyuv...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Great! How can I check that? >>>>>>> >>>>>>> >>>>>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko < >>>>>>> ita...@code972.com> wrote: >>>>>>> >>>>>>>> The graphic you sent suggests the issue is with logstash - since >>>>>>>> the @timestamp field is being populated by logstash and is the one >>>>>>>> that is >>>>>>>> used to display the date histogram graphics in Kibana. I would start >>>>>>>> there. >>>>>>>> I.e. maybe SecurityOnion buffers writes etc, and then to check the >>>>>>>> logstash >>>>>>>> shipper process stats. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Itamar Syn-Hershko >>>>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>>>>> Freelance Developer & Consultant >>>>>>>> Lucene.NET committer and PMC member >>>>>>>> >>>>>>>> On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa <iyuv...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi. >>>>>>>>> >>>>>>>>> Absolutely (but since that in the past I also worked at >>>>>>>>> the helpdesk dept. I certainly understand why it is important to ask >>>>>>>>> those >>>>>>>>> "Are you sure it's plugged in?" questions...). One of the logs is >>>>>>>>> comming >>>>>>>>> from SecurityOnion which logs (via bro-conn) all the connections so >>>>>>>>> it must >>>>>>>>> be sending data 24x7x365. >>>>>>>>> >>>>>>>>> Thanks for the quick reply, >>>>>>>>> Yuval. >>>>>>>>> >>>>>>>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko < >>>>>>>>> ita...@code972.com> wrote: >>>>>>>>> >>>>>>>>>> Are you sure your logs are generated linearly without bursts? >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Itamar Syn-Hershko >>>>>>>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>>>>>>> Freelance Developer & Consultant >>>>>>>>>> Lucene.NET committer and PMC member >>>>>>>>>> >>>>>>>>>> On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa <iyuv...@gmail.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> We just installed an ELK server and configured the logstash >>>>>>>>>>> configuration to match the data that we send to it and until last >>>>>>>>>>> month it >>>>>>>>>>> seems to be working fine but since then we see very strange >>>>>>>>>>> behavior in the >>>>>>>>>>> Kibana, the event over time histogram shows the event rate at the >>>>>>>>>>> normal >>>>>>>>>>> level for about a half an hour, then drops to about 20% of the >>>>>>>>>>> normal rate >>>>>>>>>>> and then it continues to drop slowly for about two hours and then >>>>>>>>>>> stops and >>>>>>>>>>> after a minute or two it returns to normal for the next half an >>>>>>>>>>> hour or so >>>>>>>>>>> and the same behavior repeats. Needless to say that both the >>>>>>>>>>> /var/log/logstash and /var/log/elasticsearch both show nothing >>>>>>>>>>> since the >>>>>>>>>>> service started and by using tcpdump we can verify that events keep >>>>>>>>>>> coming >>>>>>>>>>> in at the same rate all time. I attached our logstash >>>>>>>>>>> configuration, the >>>>>>>>>>> /var/logstash/logstash.log, the >>>>>>>>>>> /var/log/elasticsearch/clustername.log and >>>>>>>>>>> a screenshot of our Kibana with no filter applied so that you can >>>>>>>>>>> see the >>>>>>>>>>> weird behavior that we see. >>>>>>>>>>> >>>>>>>>>>> Is there someone/somewhere that we can turn to to get some help >>>>>>>>>>> on the subject? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks a lot, >>>>>>>>>>> Yuval. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "elasticsearch" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "elasticsearch" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>>>>>>> . >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> בברכה, >>>>>>>>> >>>>>>>>> *יובל כליפא* >>>>>>>>> >>>>>>>>> CTO >>>>>>>>> תחום מערכות מידע | מגדל סוכנויות. >>>>>>>>> נייד: 052-3336098 >>>>>>>>> משרד: 03-7966565 >>>>>>>>> פקס: 03-7976565 >>>>>>>>> בלוג: http://www.artifex.co.il >>>>>>>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>>>>>>> >>>>>>>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> בברכה, >>>>>>> >>>>>>> *יובל כליפא* >>>>>>> >>>>>>> CTO >>>>>>> תחום מערכות מידע | מגדל סוכנויות. >>>>>>> נייד: 052-3336098 >>>>>>> משרד: 03-7966565 >>>>>>> פקס: 03-7976565 >>>>>>> בלוג: http://www.artifex.co.il >>>>>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>>>>> >>>>>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "elasticsearch" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> בברכה, >>>>> >>>>> *יובל כליפא* >>>>> >>>>> CTO >>>>> תחום מערכות מידע | מגדל סוכנויות. >>>>> נייד: 052-3336098 >>>>> משרד: 03-7966565 >>>>> פקס: 03-7976565 >>>>> בלוג: http://www.artifex.co.il >>>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>>> >>>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>>> >>>>> >>>> >>>> -- >>>> >>>> בברכה, >>>> >>>> *יובל כליפא* >>>> >>>> CTO >>>> תחום מערכות מידע | מגדל סוכנויות. >>>> נייד: 052-3336098 >>>> משרד: 03-7966565 >>>> פקס: 03-7976565 >>>> בלוג: http://www.artifex.co.il >>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>> >>>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> >> בברכה, >> >> *יובל כליפא* >> >> CTO >> תחום מערכות מידע | מגדל סוכנויות. >> נייד: 052-3336098 >> משרד: 03-7966565 >> פקס: 03-7976565 >> בלוג: http://www.artifex.co.il >> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >> >> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com >> <javascript:_e(%7B%7D,'cvml','elasticsearch%2bunsubscr...@googlegroups.com');> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','elasticsearch%2bunsubscr...@googlegroups.com');> > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- בברכה, *יובל כליפא* CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: http://www.artifex.co.il <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADtR2A_v%2Bo_XcF%2Bcb6ywhsLotSyQ%3DKPZRgRY9P8%2BJ7GSNeJCqw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
