I'd start by using logstash with input tcp and output fs and see how the file behaves. Same for the fs inputs - see how their files behave. And take it from there.
-- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Lucene.NET committer and PMC member On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa <iyuv...@gmail.com> wrote: > Great! How can I check that? > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko <ita...@code972.com> > wrote: > >> The graphic you sent suggests the issue is with logstash - since the >> @timestamp field is being populated by logstash and is the one that is used >> to display the date histogram graphics in Kibana. I would start there. I.e. >> maybe SecurityOnion buffers writes etc, and then to check the logstash >> shipper process stats. >> >> -- >> >> Itamar Syn-Hershko >> http://code972.com | @synhershko <https://twitter.com/synhershko> >> Freelance Developer & Consultant >> Lucene.NET committer and PMC member >> >> On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa <iyuv...@gmail.com> wrote: >> >>> Hi. >>> >>> Absolutely (but since that in the past I also worked at the helpdesk >>> dept. I certainly understand why it is important to ask those "Are you sure >>> it's plugged in?" questions...). One of the logs is comming from >>> SecurityOnion which logs (via bro-conn) all the connections so it must be >>> sending data 24x7x365. >>> >>> Thanks for the quick reply, >>> Yuval. >>> >>> On Tuesday, February 10, 2015, Itamar Syn-Hershko <ita...@code972.com> >>> wrote: >>> >>>> Are you sure your logs are generated linearly without bursts? >>>> >>>> -- >>>> >>>> Itamar Syn-Hershko >>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>> Freelance Developer & Consultant >>>> Lucene.NET committer and PMC member >>>> >>>> On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa <iyuv...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> We just installed an ELK server and configured the logstash >>>>> configuration to match the data that we send to it and until last month it >>>>> seems to be working fine but since then we see very strange behavior in >>>>> the >>>>> Kibana, the event over time histogram shows the event rate at the normal >>>>> level for about a half an hour, then drops to about 20% of the normal rate >>>>> and then it continues to drop slowly for about two hours and then stops >>>>> and >>>>> after a minute or two it returns to normal for the next half an hour or so >>>>> and the same behavior repeats. Needless to say that both the >>>>> /var/log/logstash and /var/log/elasticsearch both show nothing since the >>>>> service started and by using tcpdump we can verify that events keep coming >>>>> in at the same rate all time. I attached our logstash configuration, the >>>>> /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and >>>>> a screenshot of our Kibana with no filter applied so that you can see the >>>>> weird behavior that we see. >>>>> >>>>> Is there someone/somewhere that we can turn to to get some help on the >>>>> subject? >>>>> >>>>> >>>>> Thanks a lot, >>>>> Yuval. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> >>> בברכה, >>> >>> *יובל כליפא* >>> >>> CTO >>> תחום מערכות מידע | מגדל סוכנויות. >>> נייד: 052-3336098 >>> משרד: 03-7966565 >>> פקס: 03-7976565 >>> בלוג: http://www.artifex.co.il >>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>> >>> *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > > בברכה, > > *יובל כליפא* > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: http://www.artifex.co.il > <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> > > *[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]* > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
