Re: [graylog2] limit is too low: [-1]
Hey Cornelius, cornelius.r...@gmail.com [Thu, Jul 03, 2014 at 09:12:48AM -0700] wrote: >2014-07-03 18:04:39,251 WARN : >org.graylog2.periodical.IndexerClusterCheckerThread - Indexer node > open file limit is too low: [-1]. Set it to at least >64000. >Where does "-1" come from? >I even rebooted the systems, didn't make any difference :-( > >Does anybody have a clue, what is wrong? > that's odd. Can you please provide the following details? * Operating System (Distribution, Version) * Java JDK/JRE version (java -version) * Graylog2 version Thank you! Regards, Bernd -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANNOUNCE] Graylog2 v0.20.4 has been released
Hey Robert,
Robert Logan [Fri, Jul 04, 2014 at 02:19:31AM -0700] wrote:
>Fired this up today on two systems 0.20.4 server and web, both show the
>same error on any search, coming from the application.log of the web
>interface:
>
> [...]
>
>Caused by: java.lang.RuntimeException: No highlight ranges for field:
>message
>at
>models.api.results.HighlightedField.getChunks(HighlightedField.java:41)
>
> [...]
>
that looks like a bug in the new message highlighting implementation
that was introduced in 0.20.4 to avoid a XSS problem.
There is a GitHub issue related to this.
https://github.com/Graylog2/graylog2-web-interface/issues/827
Until this is fixed, you have to disable highlighting unfortunately.
("allow_highlighting = false" in graylog2.conf)
Thank you for the report!
Regards,
Bernd
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH
Steckelhörn 11
20457 Hamburg
Germany
https://www.torch.sh/
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: limit is too low: [-1]
Great, thanks for the update! Bernd cornelius.r...@gmail.com [Mon, Jul 07, 2014 at 09:17:41AM -0700] wrote: >Hi Ankit, hi Bernd, > >thanks, that was the right direction! > >It wasn't the parameter you mentioned Ankit, for me it works without explicit >setting "-Des.max-open-files=true" but with the right jdk ;-) >I don't not right know why, but on the complaining systems elasticsearch was >started with ibm-jdk (any graylog-component was started with oracle-jdk 1.7.51 >though)... >Now I have ensured that elasticsearch-nodes are started with oracle-jdk - and >no complaint any more! > >regards, Cornelius > >Am Montag, 7. Juli 2014 11:50:30 UTC+2 schrieb Ankit Mittal: > >Hi Cornelius, > > > I faced the same issue, elasticsearch will does not use the full limit >until you do the below setting : > >pass the parameter in elasticsearch run time $ bin/elasticsearch -f >-Des.max-open-files=true > >Let me know if you need more details >Regards >Ankit Mittal > > > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an email >to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: New to Graylog - events coming in but cannot be seen?!
Hey, please make sure your clocks on the server and message sender are synced. Maybe the message timestamps are in the future. Regards, Bernd Alfonso Graña [Tue, Jul 08, 2014 at 12:59:58PM -0700] wrote: > >I have the same issue with 0.20.5 on debian 7. >Do you manage to get around this issue? > >Best Regards. > >El sábado, 28 de junio de 2014 23:48:59 UTC-3, Nathan escribió: >> >> Just finished setting up Graylog2 (0.20.3) using Mitchell Anicas's >> fantastic walkthrough on DigitalOcean. >> >> Everything went well and it all seems to test out fine but I cannot for >> the life of me see any of the events. I am seeing that the event counters >> are increasing but when I simply press on the green search button with no >> conditions, it get nothing! Currently I have an input that is taking in >> simple UDP syslog messages from routers and firewalls. >> >> If I am seeing the events increment, why can I not see the actual events? >> Could I have missed something in the configuration? >> >> When I show the metrics of the input that I created I currently see 100 or >> so incomingMessages, 0 incompleteMessages, 100 or so proccessedMessages. >> That all tells me that the messages are good and are being indexed. >> >> Thoughts? >> >> Thanks!!! >> > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANNOUNCE] Graylog2 v0.20.6 has been released
Hey everybody, a new bug fix release, Graylog2 v0.20.6, has been released. This server release includes a fix for a notification problem for deleted streams, a resource leak fix as well as some better defaults for AMQP inputs. For the web interface this release fixes a XSS vulnerability in the extractor preview and a notification issue regarding to streams. Please find the releases at Github: * https://github.com/Graylog2/graylog2-server/releases/tag/0.20.6 * https://github.com/Graylog2/graylog2-web-interface/releases/tag/0.20.6 The corresponding milestones are: * https://github.com/Graylog2/graylog2-server/issues?milestone=29&state=closed * https://github.com/Graylog2/graylog2-web-interface/issues?milestone=31&state=closed Thank you, Bernd -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANNOUNCE] Graylog2 v0.20.6 has been released
Hey Ankit, Ankit Mittal [Sat, Jul 19, 2014 at 05:35:34AM -0700] wrote: >I report a issue few days ago. That we are getting messages of second >stream in mail alert for stream first.if the > is this issue #628 or some earlier one? >Please let me know if the above issue is resolved or not. > Issue #628 isn't fixed yet because that was opened after the 0.20.6 release, sorry. >I have reported a memory utilization of graylog 0.20.5 server is higher >that its previous. > The new 0.20.6 release fixed a resource leak. So please check if the new release works for you now. Thank you! Regards, Bernd -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANNOUNCE] Graylog2 v0.20.6 has been released
Hey Denny, Denny Gebel [Wed, Jul 30, 2014 at 03:07:49AM -0700] wrote: >is it possible do update my current instance of graylog2 (0.20.1) directly >to 0.20.6 without losing any data/configuration? >Is there anything I have to be aware of? > Yes, there should be no problems doing that. If you use AMQP or Kafka Inputs (Radio) you might want to change the prefetch (AMQP) and threads (Kafka) settings to the new defaults. (prefetch=100, threads=2) Regards, Bernd -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANNOUNCE] Graylog2 v0.20.6 has been released
Hey Cornelius, cornelius.r...@gmail.com [Fri, Aug 29, 2014 at 01:44:20AM -0700] wrote: >is there a way to change the settings for existing Kafka-Inputs? Or do I >have to terminate and create a new Kafka-Input? > that's not possible at the moment. You have to terminate and re-create. >The hint when creating a Kafka-Input within GUI states >Number of processor threads to spawn. Use one thread per Kafka topic >partition. >I have setup a Kafka-Cluster with 4 partitions and 4 Cluster-Members, >initially each Member being the leader of 1 partition. Should I set threads >to 2 or 4 or 8? > I would say set it to 4 if you only have one Graylog2 server consuming the messages from Kafka. If you have more than one, divide equally between the Graylog2 server nodes. Regards, Bernd -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANNOUNCE] Graylog2 v0.90.1 and v0.91.1 have been released
Hey everybody, we just released two bug fix releases of Graylog2. Please find all information about the changes in the release announcement: http://www.graylog2.org/news/post/0006-two-new-graylog2-releases Thanks, Bernd (In the name of the whole Graylog2 team) -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANN] Graylog v1.0-rc.3 released
Hi everyone, we just released the third release candidate of Graylog v1.0. (1.0.0-rc.3) Changes since 1.0.0-rc.2: - Fixed compatibility with MongoDB version 2.2. SERVER#941 - Fixed performance regression in process buffer handling. SERVER#944 - Fixed data type for the max_size_per_index config option value. WEB#1100 - Fixed problem with indexer error page. WEB#1102 Please find the complete release notes on the Graylog blog. https://www.graylog.org/graylog-v1-0-rc-3-has-been-released/ Regards, Bernd (in the name of the complete Graylog team) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] [ANN] Graylog v1.0-rc.4 released
Hi everyone, we just released the fourth release candidate of Graylog v1.0. (1.0.0-rc.4) Changes since 1.0.0.-rc3: - Default configuration file locations have changed. SERVER#950 - Improved error handling on search errors. SERVER#954 - Dynamically update dashboard widgets with keyword range. SERVER#956 WEB#958 - Prevent duplicate loading of plugins. SERVER#948 - Fixed password handling when editing inputs. WEB#1103 - Fixed issues getting Elasticsearch cluster health. SERVER#953 - Better error handling for extractor imports. SERVER#942 - Fixed structured syslog parsing of keys containing special characters. SERVER#845 - Improved layout on Grok patterns page. WEB#1109 - Improved formatting large numbers. WEB# - New Graylog logo. https://www.graylog.org/graylog-v1-0-rc-4-has-been-released/ Regards, Bernd (in the name of the complete Graylog team) -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANN] Graylog v1.0 has been released
Arie, you mean it actually deleted the old files (/etc/graylog2.con and files in /etc/graylog2/server) even though you modified them? Bernd Arie [Thu, Feb 19, 2015 at 11:39:48PM -0800] wrote: >Congrats,, happy too, > > but updating my rpms throwed my old graylog configs away. > on centos the old versions are considered obsolete. > >Arie > > > >On Thursday, February 19, 2015 at 8:38:23 PM UTC+1, lennart wrote: >> >> We are very happy to announce that we released Graylog v1.0 today: >> >> https://www.graylog.org/announcing-graylog-v1-0-ga/ >> >> We'd like you all for the immense support we got over the last 5 1/2 >> years and look forward to build on top of this foundation now. >> >> Cheers, >> Lennart (In behalf of the whole Graylog, Inc team) >> > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Graylog 1.0 startup error
Do you have any old instances of Graylog running? Also please make sure there are no directories in /var/lib/graylog-server/journal other than the ones created by Graylog. Bernd On 20 February 2015 at 13:40, Arie wrote: > And found this to in the output: > > Caused by: java.io.IOException: Directory '/etc/graylog2/server' could not > be created > > I delete the 'old' dirs on the server in /etc/ > > > > On Friday, February 20, 2015 at 1:36:33 PM UTC+1, Arie wrote: >> >> Hi All >> >> After succesfully updating to 1.0 from the latest 0.9 and starting up >> after a reboot all was fine in our test environment. >> >> Now after a resatrt of the graylog-server service we have the following >> error: >> >> 2015-02-20T13:26:25.572+01:00 ERROR [CmdLineTool] Guice error (more detail >> on log level debug): Error injecting constructor, >> java.lang.RuntimeException: kafka.common.KafkaException: Failed to acquire >> lock on file .lock in /var/lib/graylog-server/journal. A Kafka instance in >> another process or thread is using this directory. >> >> Cleaning up the directory does not solve this startup error. >> >> What can be wrong/ >> >> running on centos 6.6 >> latest java >> es 1.4.3 >> >> > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Graylog 1.0 startup error
I think you have to adjust the node-id setting in your /etc/graylog/server/server.conf to point to the new directory. (/etc/graylog/server/) Bernd On 20 February 2015 at 13:51, Arie wrote: > Problem "solved" partially. > > graylog seems to rely on an old directory as mentioned earlier. > (/etc/graylog2/server/ and coping the node-id into there.) > > Removed everything in the journal directory an I am running fine again. > > hth > > > > > On Friday, February 20, 2015 at 1:44:42 PM UTC+1, Arie wrote: >> >> Solverd the problem partially bij creating >> >> /etc/graylog2/server/ and coping the node-id into there. >> >> Now only the kafaka exeption remains >> >> On Friday, February 20, 2015 at 1:36:33 PM UTC+1, Arie wrote: >>> >>> Hi All >>> >>> After succesfully updating to 1.0 from the latest 0.9 and starting up >>> after a reboot all was fine in our test environment. >>> >>> Now after a resatrt of the graylog-server service we have the following >>> error: >>> >>> 2015-02-20T13:26:25.572+01:00 ERROR [CmdLineTool] Guice error (more >>> detail on log level debug): Error injecting constructor, >>> java.lang.RuntimeException: kafka.common.KafkaException: Failed to acquire >>> lock on file .lock in /var/lib/graylog-server/journal. A Kafka instance in >>> another process or thread is using this directory. >>> >>> Cleaning up the directory does not solve this startup error. >>> >>> What can be wrong/ >>> >>> running on centos 6.6 >>> latest java >>> es 1.4.3 >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Graylog 1.0 startup error
You're welcome! :) Bernd On 20 February 2015 at 14:01, Arie wrote: > You are absolutely right about that, missed that in the diff > > thank you. > > On Friday, February 20, 2015 at 1:56:20 PM UTC+1, Bernd Ahlers wrote: >> >> I think you have to adjust the node-id setting in your >> /etc/graylog/server/server.conf to point to the new directory. >> (/etc/graylog/server/) >> >> Bernd >> >> On 20 February 2015 at 13:51, Arie wrote: >> > Problem "solved" partially. >> > >> > graylog seems to rely on an old directory as mentioned earlier. >> > (/etc/graylog2/server/ and coping the node-id into there.) >> > >> > Removed everything in the journal directory an I am running fine again. >> > >> > hth >> > >> > >> > >> > >> > On Friday, February 20, 2015 at 1:44:42 PM UTC+1, Arie wrote: >> >> >> >> Solverd the problem partially bij creating >> >> >> >> /etc/graylog2/server/ and coping the node-id into there. >> >> >> >> Now only the kafaka exeption remains >> >> >> >> On Friday, February 20, 2015 at 1:36:33 PM UTC+1, Arie wrote: >> >>> >> >>> Hi All >> >>> >> >>> After succesfully updating to 1.0 from the latest 0.9 and starting up >> >>> after a reboot all was fine in our test environment. >> >>> >> >>> Now after a resatrt of the graylog-server service we have the >> >>> following >> >>> error: >> >>> >> >>> 2015-02-20T13:26:25.572+01:00 ERROR [CmdLineTool] Guice error (more >> >>> detail on log level debug): Error injecting constructor, >> >>> java.lang.RuntimeException: kafka.common.KafkaException: Failed to >> >>> acquire >> >>> lock on file .lock in /var/lib/graylog-server/journal. A Kafka >> >>> instance in >> >>> another process or thread is using this directory. >> >>> >> >>> Cleaning up the directory does not solve this startup error. >> >>> >> >>> What can be wrong/ >> >>> >> >>> running on centos 6.6 >> >>> latest java >> >>> es 1.4.3 >> >>> >> >>> >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to graylog2+u...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH - A Graylog company >> Steckelhörn 11 >> 20457 Hamburg >> Germany >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Stream matcher code
Maciek, a regex match for a field value is not possible at the moment, sorry. Bernd On 20 February 2015 at 16:13, Arie wrote: > At leas in 09.3 you can > > Create a stream rule, and the first possibility on top is the field, > after that select regex > and in that you put (cvp) > > > > > On Friday, February 20, 2015 at 3:51:51 PM UTC+1, Maciej Strömich wrote: >> >> Hi, >> >> >> I'm trying to create a stream which will catch log entries containging >> "cvp" string. It's not a problem if there's only one field which needs to be >> checked. I've several places where this value can be found and I'm wondering >> is it possible to use a regex inside a "Field" input. AFAIK seperate stream >> rules will use AND and not OR to match messages. >> >> I've tried so far: >> >> (?:url|message) >> [url|message] >> url||message >> >> Is this even possible? >> >> br, >> Maciek > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Fresh start Graylog v1.0 with v0.92 logs
Santiago, why are you unable to run curl to restore the snapshot? Using Elasticsearch snapshots to backup/restore the indices is the way to go. Regards, Bernd On 20 February 2015 at 16:04, Santiago Cordone wrote: > Hi, > Im about to migrate from v 0.92 to v 1.0 but i installed the v1.0 from the > ova file, and i can't find a way to restore the snapshot from elasticsearch > containing all my logs from v0.92 to the v1.0 since elasticsearch is not > standalone and i can't run curl command to restore it. > Can you please tell me if there is any way to migrate logs from one > installation to the other? i can't affort to lose all this data. > Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Extractors and reverse dns
Dale, there is currently no way to do DNS reverse lookups on arbitrary fields. Sorry! You can always do that with a custom plugin, but that requires writing one in Java. (http://docs.graylog.org/en/1.0/pages/plugins.html) Regards, Bernd On 18 February 2015 at 20:35, DH wrote: > Hi Everyone, > > I have an input with a dozen or more extractors that work excellent. Now I > want some icing on the cake. I have an extractor that extracts a local ip > address from message. I'd like to do a reverse dns lookup on that ip to > give it a real name and use these quick values to create custom dashboards > for the info I want to look at. Is there a way to do a reverse dns lookup > on an extracted value? > > Dale > > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Upgrade path to Graylog2 1.0?
Curtis, that depends which version you are currently running. Anything from 0.20, 0.90, 0.91, and 0.92 should be fine. Please see the Upgrade section in our release announcement. https://www.graylog.org/announcing-graylog-v1-0-ga/ Regards, Bernd On 19 February 2015 at 23:47, Curtis Starnes wrote: > Is there an upgrade path from earlier versions of Graylog2 to the 1.0 train? > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] installation document
Hey, please see the manual setup documentation on how to setup Graylog. http://docs.graylog.org/en/1.0/pages/installation.html#the-manual-setup You might also try the OS packages or one of the virtual machine images. http://docs.graylog.org/en/1.0/pages/installation.html#operating-system-packages http://docs.graylog.org/en/1.0/pages/installation.html#virtual-machine-appliances Regards, Bernd On 21 February 2015 at 18:03, Abdüllatif ERKAYA wrote: > There's something missing in the installation documentation. I graduated > from the MongoDB installation. I downloaded Graylog. I did unzip. I'm stuck > after this stage .. > > There's something missing in the installation documentation. I finished the > elasticsearch and MongoDB installation. I downloaded Graylog. I extracted > it. I stuck after this stage .. > > /etc/graylog/server - There is no such directory. > > I created it, an configured. > > And I want To start Graylog. > > ~$ cd bin/ > ~$ ./graylogctl start > > > But it is not. I guess there's something missing in the document. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Alert questions
Josh, the current alerting implementation does not support that unfortunately. There are some possibilities to achive that functionality. 1. Use a HTTP alarm callback to send the alert including some messages to a custom HTTP server that handles the alerting. 2. Use something like riemann to handle the alerting and use our output plugin to send data to it. (https://github.com/Graylog2/graylog2-plugin-output-riemann) 3. Write a custom alert callback or output plugin. Hope that helps. Regards, Bernd On 23 February 2015 at 04:22, Tristan Rhodes wrote: > Josh, > > This type of functionality might be better provided by a tool like Observium > (http://observium.org/) Observium will autodiscover your network equipment, > create graphs for all interfaces and sensors, and you can easily setup > alerts for any problems, such as a failed power supply. > > Cheers, > > Tristan > > On Tue, Feb 17, 2015 at 1:48 PM, Josh Scott wrote: >> >> Is there an easy way to set up alerts on a per device basis without >> setting up streams for every device in my infrastructure? >> >> Here is my situation. I have over 200 switches deployed. I wan't to alert >> on any syslog level 1 events and then supress for 12 hours any subsequent >> events of the same type from the same IP address. For example we recently >> had a switch that had a bad power supply. Currently I have a stream set up >> to catch any Syslog level 1 events, send an SMTP alert and supress any >> additional ones for 12 hours. If a level 1 event happens on a different >> device or a different event on the same device I won't get the alert since >> the stream is set for any level 1 events >> >> Is there a way to set up the alerts to supress based on IP address or on >> message field content? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to graylog2+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Tristan Rhodes > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Johan, this sounds very strange indeed. Can you provide us with some more details? - What kind of messages are you pouring into Graylog via UDP? (GELF, raw, syslog?) - Do you have any extractors or grok filters running for the messages coming in via UDP? - Any other differences between the TCP and UDP messages? - Can you show us your input configuration? - Are you using reverse DNS lookups? Thank you! Regards, Bernd On 24 February 2015 at 16:45, wrote: > Well that could be a suspect if it wasn't for the fact that the old nodes > running on old hardware handle it just fine, along with the fact that the > traffic seems to reach the nodes just fine(i.e it actually fills the journal > up just fine, and the input buffer never breaks a sweat). And it's really > not that much traffic, even spread across four nodes those ~1000 messages > per second will cause this whereas the old nodes are just two and can handle > it just fine. > > About disk tuning, I haven't done much of that, and I realize I forgot to > mention that the Elasticsearch cluster is on separate physical hardware so > there's a minuscule amount of disk I/O happening on the Graylog nodes. > > It's really very strange since it seems like UDP itself isn't to blame, > after all the messages get into Graylog just fine and fills up the journal > rapidly. The screenshot from I linked was from after I had stopped sending > logs, i.e there was no longer any ingress traffic so the Graylog process had > nothing to do except emptying it's journal so it should all be internal > processing and egress traffic to Elasticsearch. And as can be seen in the > screenshot it seems like it's doing it in small bursts. > > In the exact same scenario(i.e when I just streamed a large file into the > system as fast as it could receive it) but with the logs having come over > TCP, it'll still store up a sizable number of messages in the journal, but > the processing of the journaled messages is both more even and vastly > faster. > > So in short it doesn't appear to be the communication itself, but something > happening "inside" the Graylog process, but that only happens when the > messages have been delivered over UDP. > > Regards > Johan > > > On Tuesday, February 24, 2015 at 3:07:47 PM UTC+1, Henrik Johansen wrote: >> >> Could this simply be because TCP avoids (or tries to avoid) congestion >> while UDP does not? >> >> /HJ >> >> On 24 Feb 2015, at 13:50, sun...@sunner.com wrote: >> >> Hello, >> >> With the release of 1.0 we've started moving towards a new cluster of GL >> hosts. These are working very well, with one exception. >> For some reason any reasonably significant UDP traffic will choke the >> message processor, fill up and process buffers on all four hosts, and >> effectively choke up all other message processing as well. >> Normally we do around 2k messages per second, split roughly 50/50 between >> TCP and UDP. Sending the entire TCP load to one host doesn't present a >> problem, it doesn't break a sweat. >> >> I've also experimented a little with sending a large text file using >> rsyslog's imfile module, sending it via TCP will bottleneck us at the ES >> side of things and cause the disk journal fill up fairly rapidly, but it's >> still working at at ~9k messages per second so that's fine. Sending it via >> UDP just causes GL to choke again, fill up the journal to a certain point >> and slowly slowly process the journal at little bursts of a few thousand >> messages followed by several seconds of apparent sleeping(i.e pretty much no >> CPU usage). >> >> During all of this the input buffer never fills up more than at most >> single digit percentages, using TCP the output buffer sometimes moves up to >> 20-30%, with UDP it never moves at all. It's all in the process buffer. >> Sending a large burst of messages and then stopping doesn't seem to affect >> this behavior either, even after the inbound messages stop it still takes a >> long time to process the messages that are already in the journal and >> process buffer. >> I'm using VisualVM to look at the CPU and memory usage, this is a >> screenshot of a UDP session: >> http://i59.tinypic.com/x23xfl.png >> >> I've tried mucking around with various knobs, processbuffer_processors, >> JVM settings, etc, with no results whatsoever, good or bad. >> There's nothing to suggest a problem in neither the graylog nor system >> logs. >> >> Pertinent specs and settings: >> ring_size = 16384 (CPU's have 20 MB L3) >> processbuffer_processors = 5 >> >> Java 8u31 >> Using G1GC with StringDeduplication, I've tried without the latter and >> just using CMC as well, no difference. >> 4 GB Xmx/Xms. >> Linux 3.16.0 >> net.core.rmem_max = 8388608 >> >> These are virtual machines, VMware, 8 GB / 8 vCPU's, Xeon E5-2690's. >> >> Software wise the old nodes are running the same setup more or less, >> except kernel 3.2.0, same JVM, G1GC, etc. Hardware wise, they're physical >> boxes, old Dell 2950's with dual quad core E5440's. That's
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Johan, the only thing that changed from 0.92 to 1.0 is that the DNS lookup is now done when the messages are read from the journal and not in the input path where the messages are received. Otherwise, nothing has changed in that regard. We do not do any manual caching of the DNS lookups, but the JVM caches them by default. Check http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. Regards, Bernd On 25 February 2015 at 08:56, wrote: > This is strange, I went through all of the settings for my reply, and we are > indeed using rdns, and it seems to be the culprit. The strangeness is that > it works fine on the old servers even though they're on the same networks, > and using the same DNS's and resolver settings. > Did something regarding reverse DNS change between 0.92 and 1.0? I'm > thinking perhaps the server is trying to do one lookup per message instead > of caching reverse lookups, seeing as the latter would result in very little > DNS traffic since most of the logs will be coming from a small number of > hosts. > > Regards > Johan > > On Tuesday, February 24, 2015 at 5:08:54 PM UTC+1, Bernd Ahlers wrote: >> >> Johan, >> >> this sounds very strange indeed. Can you provide us with some more >> details? >> >> - What kind of messages are you pouring into Graylog via UDP? (GELF, >> raw, syslog?) >> - Do you have any extractors or grok filters running for the messages >> coming in via UDP? >> - Any other differences between the TCP and UDP messages? >> - Can you show us your input configuration? >> - Are you using reverse DNS lookups? >> >> Thank you! >> >> Regards, >> Bernd >> >> On 24 February 2015 at 16:45, wrote: >> > Well that could be a suspect if it wasn't for the fact that the old >> > nodes >> > running on old hardware handle it just fine, along with the fact that >> > the >> > traffic seems to reach the nodes just fine(i.e it actually fills the >> > journal >> > up just fine, and the input buffer never breaks a sweat). And it's >> > really >> > not that much traffic, even spread across four nodes those ~1000 >> > messages >> > per second will cause this whereas the old nodes are just two and can >> > handle >> > it just fine. >> > >> > About disk tuning, I haven't done much of that, and I realize I forgot >> > to >> > mention that the Elasticsearch cluster is on separate physical hardware >> > so >> > there's a minuscule amount of disk I/O happening on the Graylog nodes. >> > >> > It's really very strange since it seems like UDP itself isn't to blame, >> > after all the messages get into Graylog just fine and fills up the >> > journal >> > rapidly. The screenshot from I linked was from after I had stopped >> > sending >> > logs, i.e there was no longer any ingress traffic so the Graylog process >> > had >> > nothing to do except emptying it's journal so it should all be internal >> > processing and egress traffic to Elasticsearch. And as can be seen in >> > the >> > screenshot it seems like it's doing it in small bursts. >> > >> > In the exact same scenario(i.e when I just streamed a large file into >> > the >> > system as fast as it could receive it) but with the logs having come >> > over >> > TCP, it'll still store up a sizable number of messages in the journal, >> > but >> > the processing of the journaled messages is both more even and vastly >> > faster. >> > >> > So in short it doesn't appear to be the communication itself, but >> > something >> > happening "inside" the Graylog process, but that only happens when the >> > messages have been delivered over UDP. >> > >> > Regards >> > Johan >> > >> > >> > On Tuesday, February 24, 2015 at 3:07:47 PM UTC+1, Henrik Johansen >> > wrote: >> >> >> >> Could this simply be because TCP avoids (or tries to avoid) congestion >> >> while UDP does not? >> >> >> >> /HJ >> >> >> >> On 24 Feb 2015, at 13:50, sun...@sunner.com wrote: >> >> >> >> Hello, >> >> >> >> With the release of 1.0 we've started moving towards a new cluster of >> >> GL >> >> hosts. These are working very well, with one exception. >> >
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Henrik, uh, okay. I suppose it worked for you in 0.92 as well? I will create an issue on GitHub for that. Bernd On 25 February 2015 at 17:14, Henrik Johansen wrote: > Bernd, > > We saw the exact same issue - here is a graph over the CPU idle > percentage across a few of the cluster nodes during the upgrade : > > http://5.9.37.177/graylog_cluster_cpu_idle.png > > We went from ~20% CPU utilisation to ~100% CPU utilisation across > ~200 cores and things only settled down after disabling force_rdns. > > > On 25 Feb 2015, at 11:55, Bernd Ahlers wrote: > > Johan, > > the only thing that changed from 0.92 to 1.0 is that the DNS lookup is > now done when the messages are read from the journal and not in the > input path where the messages are received. Otherwise, nothing has > changed in that regard. > > We do not do any manual caching of the DNS lookups, but the JVM caches > them by default. Check > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. > > Regards, > Bernd > > On 25 February 2015 at 08:56, wrote: > > This is strange, I went through all of the settings for my reply, and we are > indeed using rdns, and it seems to be the culprit. The strangeness is that > it works fine on the old servers even though they're on the same networks, > and using the same DNS's and resolver settings. > Did something regarding reverse DNS change between 0.92 and 1.0? I'm > thinking perhaps the server is trying to do one lookup per message instead > of caching reverse lookups, seeing as the latter would result in very little > DNS traffic since most of the logs will be coming from a small number of > hosts. > > Regards > Johan > > On Tuesday, February 24, 2015 at 5:08:54 PM UTC+1, Bernd Ahlers wrote: > > > Johan, > > this sounds very strange indeed. Can you provide us with some more > details? > > - What kind of messages are you pouring into Graylog via UDP? (GELF, > raw, syslog?) > - Do you have any extractors or grok filters running for the messages > coming in via UDP? > - Any other differences between the TCP and UDP messages? > - Can you show us your input configuration? > - Are you using reverse DNS lookups? > > Thank you! > > Regards, > Bernd > > On 24 February 2015 at 16:45, wrote: > > Well that could be a suspect if it wasn't for the fact that the old > nodes > running on old hardware handle it just fine, along with the fact that > the > traffic seems to reach the nodes just fine(i.e it actually fills the > journal > up just fine, and the input buffer never breaks a sweat). And it's > really > not that much traffic, even spread across four nodes those ~1000 > messages > per second will cause this whereas the old nodes are just two and can > handle > it just fine. > > About disk tuning, I haven't done much of that, and I realize I forgot > to > mention that the Elasticsearch cluster is on separate physical hardware > so > there's a minuscule amount of disk I/O happening on the Graylog nodes. > > It's really very strange since it seems like UDP itself isn't to blame, > after all the messages get into Graylog just fine and fills up the > journal > rapidly. The screenshot from I linked was from after I had stopped > sending > logs, i.e there was no longer any ingress traffic so the Graylog process > had > nothing to do except emptying it's journal so it should all be internal > processing and egress traffic to Elasticsearch. And as can be seen in > the > screenshot it seems like it's doing it in small bursts. > > In the exact same scenario(i.e when I just streamed a large file into > the > system as fast as it could receive it) but with the logs having come > over > TCP, it'll still store up a sizable number of messages in the journal, > but > the processing of the journaled messages is both more even and vastly > faster. > > So in short it doesn't appear to be the communication itself, but > something > happening "inside" the Graylog process, but that only happens when the > messages have been delivered over UDP. > > Regards > Johan > > > On Tuesday, February 24, 2015 at 3:07:47 PM UTC+1, Henrik Johansen > wrote: > > > Could this simply be because TCP avoids (or tries to avoid) congestion > while UDP does not? > > /HJ > > On 24 Feb 2015, at 13:50, sun...@sunner.com wrote: > > Hello, > > With the release of 1.0 we've started moving towards a new cluster of > GL > hosts. These are working very well, with one exception. > For some reason any reason
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Johan, Henrik, thanks for the details. I created an issue on GitHub and will investigate. https://github.com/Graylog2/graylog2-server/issues/999 Regards, Bernd On 25 February 2015 at 17:48, Henrik Johansen wrote: > Bernd, > > Correct - that issue started after 0.92.x. > > We are still seeing evaluated CPU utilisation but we are attributing that > to the fact that 0.92 was loosing messages in our setup. > > >> On 25 Feb 2015, at 17:37, Bernd Ahlers wrote: >> >> Henrik, >> >> uh, okay. I suppose it worked for you in 0.92 as well? >> >> I will create an issue on GitHub for that. >> >> Bernd >> >> On 25 February 2015 at 17:14, Henrik Johansen wrote: >>> Bernd, >>> >>> We saw the exact same issue - here is a graph over the CPU idle >>> percentage across a few of the cluster nodes during the upgrade : >>> >>> http://5.9.37.177/graylog_cluster_cpu_idle.png >>> >>> We went from ~20% CPU utilisation to ~100% CPU utilisation across >>> ~200 cores and things only settled down after disabling force_rdns. >>> >>> >>> On 25 Feb 2015, at 11:55, Bernd Ahlers wrote: >>> >>> Johan, >>> >>> the only thing that changed from 0.92 to 1.0 is that the DNS lookup is >>> now done when the messages are read from the journal and not in the >>> input path where the messages are received. Otherwise, nothing has >>> changed in that regard. >>> >>> We do not do any manual caching of the DNS lookups, but the JVM caches >>> them by default. Check >>> http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html >>> for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. >>> >>> Regards, >>> Bernd >>> >>> On 25 February 2015 at 08:56, wrote: >>> >>> This is strange, I went through all of the settings for my reply, and we are >>> indeed using rdns, and it seems to be the culprit. The strangeness is that >>> it works fine on the old servers even though they're on the same networks, >>> and using the same DNS's and resolver settings. >>> Did something regarding reverse DNS change between 0.92 and 1.0? I'm >>> thinking perhaps the server is trying to do one lookup per message instead >>> of caching reverse lookups, seeing as the latter would result in very little >>> DNS traffic since most of the logs will be coming from a small number of >>> hosts. >>> >>> Regards >>> Johan >>> >>> On Tuesday, February 24, 2015 at 5:08:54 PM UTC+1, Bernd Ahlers wrote: >>> >>> >>> Johan, >>> >>> this sounds very strange indeed. Can you provide us with some more >>> details? >>> >>> - What kind of messages are you pouring into Graylog via UDP? (GELF, >>> raw, syslog?) >>> - Do you have any extractors or grok filters running for the messages >>> coming in via UDP? >>> - Any other differences between the TCP and UDP messages? >>> - Can you show us your input configuration? >>> - Are you using reverse DNS lookups? >>> >>> Thank you! >>> >>> Regards, >>> Bernd >>> >>> On 24 February 2015 at 16:45, wrote: >>> >>> Well that could be a suspect if it wasn't for the fact that the old >>> nodes >>> running on old hardware handle it just fine, along with the fact that >>> the >>> traffic seems to reach the nodes just fine(i.e it actually fills the >>> journal >>> up just fine, and the input buffer never breaks a sweat). And it's >>> really >>> not that much traffic, even spread across four nodes those ~1000 >>> messages >>> per second will cause this whereas the old nodes are just two and can >>> handle >>> it just fine. >>> >>> About disk tuning, I haven't done much of that, and I realize I forgot >>> to >>> mention that the Elasticsearch cluster is on separate physical hardware >>> so >>> there's a minuscule amount of disk I/O happening on the Graylog nodes. >>> >>> It's really very strange since it seems like UDP itself isn't to blame, >>> after all the messages get into Graylog just fine and fills up the >>> journal >>> rapidly. The screenshot from I linked was from after I had stopped >>> sending >>> logs, i.e there was no longer any ingress traffic so the Graylog process >>> h
Re: [graylog2] journal broken
Ed, as Tristan already said, if you constantly sending in more messages than Graylog or Elasticsearch can process, you will always fill up your journal. Disabling the journal does not really fix the problem, because you will now lose messages. Please check the node details page (System -> Nodes -> click on the node name) and check the disk journal stats. If you writing more into the journal than reading from it, you have a problem with processing throughput. Regards, Bernd On 26 February 2015 at 00:50, Tristan Rhodes wrote: > Ed, > > I had this same problem. However, increasing the journal size will only > help if your rate of messages periodically decreases below what your system > can process. (For example, you will grow the journal during peak hours of > the day, and drain the journal when fewer logs are being sent to Graylog). > > If you are always sending more messages than your Elasticsearch can ingest, > the journal will not help. I increased my Elasticsearch ingesting > performance by changing this setting in elasticsearch.yml: > > index.refresh_interval: 30s > > You can read more about this setting here: > > http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ > http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/ > > Disclaimer: I am new to graylog+elastisearch and barely know what I am > doing. :) > > Cheers! > > Tristan > > On Mon, Feb 23, 2015 at 10:41 AM, Ed Totman wrote: >> >> I deployed the latest appliance from the ova file. Graylog2 worked fine >> for several days, but then the journal files grew to 5GB which is the >> default limit and search returns no current results. On the System page >> this error appeared: >> >> Journal utilization is too high a few seconds ago >> Journal utilization is too high and may go over the limit soon. Please >> verify that your Elasticsearch cluster is healthy and fast enough. You may >> also want to review your Graylog journal settings and set a higher limit. >> (Node: 43a9cc82-dc5a-4492-936b-418e1bc98f5e, journal utilization: 96.0%) >> >> I increased the journal limit to 10GB but this did not fix the problem. I >> restarted all services and checked the logs, but could not find any obvious >> problem. The VM is running on very fast storage with lots of CPU and >> memory. I set "message_journal_enabled = false" which seems to have >> temporarily resolved the problem. >> >> How do I troubleshoot the journal? All of the other components are >> working fine. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to graylog2+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Tristan Rhodes > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Journal settings
Hey, you can tweak the "message_journal_max_age" and "message_journal_max_size" settings in your graylog.conf. (see https://github.com/Graylog2/graylog2-server/blob/master/misc/graylog2.conf#L250-L254) That said, if you constantly writing more messages into the journal than you read from it, the journal will always fill up. This indicates that your Graylog server or Elasticsearch server is not able to keep up with the incoming message rate. The journal helps with temporary load spikes or if Elasticsearch is down for a short period. It does not help if you send more messages than you can process. Since your Elasticsearch seems to be bored, you might check the CPU usage of your Graylog server. Do you have lots of extractors? Reverse DNS lookups enabled? Regards, Bernd On 24 February 2015 at 17:16, wrote: > I have upgraded to 1.0, I am seeing errors regarding Journaling useage > being too high and deletion of messages due to journaling params being too > low. what params can I change? I only see on or off when it comes to > journaling. I do have a cluster for ES, running two nodes 32gb of ram each, > 4 cores. they seem bored looking at the metrics. I have increased the batch > size to 1500, but journaling is still running at ~95% and higher. > > Thanks in advance for any suggestions. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Switching to Graylog over LogAnalyzer
Clayton, as long as there are no port conflicts between LogAnalyzer and Graylog, you do not need to uninstall LogAnalyzer. Graylog by default is using ports 9000 (web), 12900 (Graylog server API), 9350 (Graylog Elasticsearch client), MongoDB uses 27017 and Elasticsearch is using 9200 and 9300. You also might have to reconfigure your syslog senders to send the syslog data to a different port where a Graylog Syslog input is listening. See our documentation for details: http://docs.graylog.org/en/1.0/pages/installation.html http://docs.graylog.org/en/1.0/pages/sending_data.html#syslog Regards, Bernd On 23 February 2015 at 17:34, Clayton Tavernier wrote: > Sorry, I'm not very good at this. I set up a syslog server with LogAnalyzer > and I would like to try Graylog instead. What, if anything, do I need to > uninstall from the server before installing Graylog? Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Message manipulation
Joanes, Graylog does not modify the messages anymore once they have been stored in Elasticsearch. So I think what you ask for is currently not possible. Sorry. Regards, Bernd On 25 February 2015 at 15:59, Joanes Errea wrote: > Hi, > > I would like to be able to manipulate some messages (via REST api or web > app) after receiving them. > For instance, I would like to set a flag to those messages that have been > reviewed/handled. Obviously this cannot be processed on receiving the > message by extractors but at any time. > The question is, does graylog already support this king of post-processing > messages? Any plugin somebody knows about? > > Thanks in advance, > Joanes > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Johan, Henrik, I tried to track this problem down.The problem is that the JVM does not cache reverse DNS lookups. The available JVM DNS cache settings like "networkaddress.cache.ttl" only affect forward DNS lookups. The code for doing the reverse lookups in Graylog did not change in a long time, so this problem is not new in 1.0. I my test setup enabling "force_rdns" for a syslog input reduced the throughput from around 7000 msg/s to 300 msg/s. This was without a local DNS cache. Once I installed a DNS cache on the Graylog server, the throughput went up to around 3000 msg/s. We will investigate if there is a sane way to cache the reverse lookups ourselves. In the meantime I suggest to test with a DNS cache installed on the Graylog server nodes to see if that helps or to disable the "force_rdns" setting. Regards, Bernd On 25 February 2015 at 18:00, Bernd Ahlers wrote: > Johan, Henrik, > > thanks for the details. I created an issue on GitHub and will investigate. > > https://github.com/Graylog2/graylog2-server/issues/999 > > Regards, > Bernd > > On 25 February 2015 at 17:48, Henrik Johansen wrote: >> Bernd, >> >> Correct - that issue started after 0.92.x. >> >> We are still seeing evaluated CPU utilisation but we are attributing that >> to the fact that 0.92 was loosing messages in our setup. >> >> >>> On 25 Feb 2015, at 17:37, Bernd Ahlers wrote: >>> >>> Henrik, >>> >>> uh, okay. I suppose it worked for you in 0.92 as well? >>> >>> I will create an issue on GitHub for that. >>> >>> Bernd >>> >>> On 25 February 2015 at 17:14, Henrik Johansen wrote: >>>> Bernd, >>>> >>>> We saw the exact same issue - here is a graph over the CPU idle >>>> percentage across a few of the cluster nodes during the upgrade : >>>> >>>> http://5.9.37.177/graylog_cluster_cpu_idle.png >>>> >>>> We went from ~20% CPU utilisation to ~100% CPU utilisation across >>>> ~200 cores and things only settled down after disabling force_rdns. >>>> >>>> >>>> On 25 Feb 2015, at 11:55, Bernd Ahlers wrote: >>>> >>>> Johan, >>>> >>>> the only thing that changed from 0.92 to 1.0 is that the DNS lookup is >>>> now done when the messages are read from the journal and not in the >>>> input path where the messages are received. Otherwise, nothing has >>>> changed in that regard. >>>> >>>> We do not do any manual caching of the DNS lookups, but the JVM caches >>>> them by default. Check >>>> http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html >>>> for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. >>>> >>>> Regards, >>>> Bernd >>>> >>>> On 25 February 2015 at 08:56, wrote: >>>> >>>> This is strange, I went through all of the settings for my reply, and we >>>> are >>>> indeed using rdns, and it seems to be the culprit. The strangeness is that >>>> it works fine on the old servers even though they're on the same networks, >>>> and using the same DNS's and resolver settings. >>>> Did something regarding reverse DNS change between 0.92 and 1.0? I'm >>>> thinking perhaps the server is trying to do one lookup per message instead >>>> of caching reverse lookups, seeing as the latter would result in very >>>> little >>>> DNS traffic since most of the logs will be coming from a small number of >>>> hosts. >>>> >>>> Regards >>>> Johan >>>> >>>> On Tuesday, February 24, 2015 at 5:08:54 PM UTC+1, Bernd Ahlers wrote: >>>> >>>> >>>> Johan, >>>> >>>> this sounds very strange indeed. Can you provide us with some more >>>> details? >>>> >>>> - What kind of messages are you pouring into Graylog via UDP? (GELF, >>>> raw, syslog?) >>>> - Do you have any extractors or grok filters running for the messages >>>> coming in via UDP? >>>> - Any other differences between the TCP and UDP messages? >>>> - Can you show us your input configuration? >>>> - Are you using reverse DNS lookups? >>>> >>>> Thank you! >>>> >>>> Regards, >>>> Bernd >>>> >>>> On 24 February 2015 at 16:45, wrote: >>>> >>>> Wel
Re: [graylog2] Logs from Cisco ASA with bad "source" field
Roberto, the Cisco ASA does not send valid Syslog, unfortunately. You have to create a "Raw" input and create extractors. There is a blog post about this here: http://spottedhyena.co.uk/2015/01/graylog2-cisco-asa-cisco-catalyst/ Hope that helps! Regards, Bernd On 27 February 2015 at 15:57, wrote: > Dear, I have a Graylog2 version 0.20.6 as our syslog server of our company. > > I defined an INPUT "Syslog UDP" running on port UDP/10514, and after that we > point several Windows and Linux servers to the Graylog2 with no problems. > > But in the case of the Cisco ASA firewalls, we have a problem because the > source sometimes matches something like: > > :%ASA-session-6-302013: > > In the Cisco ASA's I setup: > > logging enable > logging emblem > logging trap informational > logging history debugging > logging asdm debugging > logging device-id hostname > logging host inside_Frontend 10.1.1.1 format emblem > > I want to have the original hostname in the "source" field, so what can I > do??? > > Regards, > > Roberto > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Logs from Cisco ASA with bad "source" field
Roberto, you replace the Syslog input with a Raw input. The extractors are applied to the Raw input to parse the logs then. In your setup, remove the Syslog input and start a Raw input on the same port. Then add the extractors as described in the blog post I sent you earlier. Regards, Bernd On 27 February 2015 at 20:17, wrote: > Dear Bernd, thanks for your helpful respondebut now I have a new > question. > > I have a Graylog2 server with just one INPUT "Syslog UDP" listening on port > UDP/10514, and the tutorial said I have to create another INPUT "Raw" > suppose listening on port UDP/. > > How can I connect the raw input with the syslog input ??? I got lost... > > Thanks in advance, > > Roberto > > El viernes, 27 de febrero de 2015, 13:57:08 (UTC-3), Bernd Ahlers escribió: >> >> Roberto, >> >> the Cisco ASA does not send valid Syslog, unfortunately. You have to >> create a "Raw" input and create extractors. >> >> There is a blog post about this here: >> http://spottedhyena.co.uk/2015/01/graylog2-cisco-asa-cisco-catalyst/ >> >> Hope that helps! >> >> Regards, >> Bernd >> >> On 27 February 2015 at 15:57, wrote: >> > Dear, I have a Graylog2 version 0.20.6 as our syslog server of our >> > company. >> > >> > I defined an INPUT "Syslog UDP" running on port UDP/10514, and after >> > that we >> > point several Windows and Linux servers to the Graylog2 with no >> > problems. >> > >> > But in the case of the Cisco ASA firewalls, we have a problem because >> > the >> > source sometimes matches something like: >> > >> > :%ASA-session-6-302013: >> > >> > In the Cisco ASA's I setup: >> > >> > logging enable >> > logging emblem >> > logging trap informational >> > logging history debugging >> > logging asdm debugging >> > logging device-id hostname >> > logging host inside_Frontend 10.1.1.1 format emblem >> > >> > I want to have the original hostname in the "source" field, so what can >> > I >> > do??? >> > >> > Regards, >> > >> > Roberto >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to graylog2+u...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH - A Graylog company >> Steckelhörn 11 >> 20457 Hamburg >> Germany >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] journal broken
Ed, if you want to delete all of the journal, stop the server, delete the journal dir (see "message_journal_dir" setting in graylog.conf) and start the server again. Bernd On 26 February 2015 at 16:13, Ed Totman wrote: > Thanks for the reply. How do I clear the journal of old messages before I > restart it? > > On Wednesday, February 25, 2015 at 10:54:42 PM UTC-8, Bernd Ahlers wrote: >> >> Ed, >> >> as Tristan already said, if you constantly sending in more messages >> than Graylog or Elasticsearch can process, you will always fill up >> your journal. >> Disabling the journal does not really fix the problem, because you >> will now lose messages. >> >> Please check the node details page (System -> Nodes -> click on the >> node name) and check the disk journal stats. If you writing more into >> the journal than reading from it, you have a problem with processing >> throughput. >> >> Regards, >> Bernd >> >> On 26 February 2015 at 00:50, Tristan Rhodes wrote: >> > Ed, >> > >> > I had this same problem. However, increasing the journal size will only >> > help if your rate of messages periodically decreases below what your >> > system >> > can process. (For example, you will grow the journal during peak hours >> > of >> > the day, and drain the journal when fewer logs are being sent to >> > Graylog). >> > >> > If you are always sending more messages than your Elasticsearch can >> > ingest, >> > the journal will not help. I increased my Elasticsearch ingesting >> > performance by changing this setting in elasticsearch.yml: >> > >> > index.refresh_interval: 30s >> > >> > You can read more about this setting here: >> > >> > >> > http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ >> > >> > http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/ >> > >> > Disclaimer: I am new to graylog+elastisearch and barely know what I am >> > doing. :) >> > >> > Cheers! >> > >> > Tristan >> > >> > On Mon, Feb 23, 2015 at 10:41 AM, Ed Totman wrote: >> >> >> >> I deployed the latest appliance from the ova file. Graylog2 worked >> >> fine >> >> for several days, but then the journal files grew to 5GB which is the >> >> default limit and search returns no current results. On the System >> >> page >> >> this error appeared: >> >> >> >> Journal utilization is too high a few seconds ago >> >> Journal utilization is too high and may go over the limit soon. Please >> >> verify that your Elasticsearch cluster is healthy and fast enough. You >> >> may >> >> also want to review your Graylog journal settings and set a higher >> >> limit. >> >> (Node: 43a9cc82-dc5a-4492-936b-418e1bc98f5e, journal utilization: >> >> 96.0%) >> >> >> >> I increased the journal limit to 10GB but this did not fix the problem. >> >> I >> >> restarted all services and checked the logs, but could not find any >> >> obvious >> >> problem. The VM is running on very fast storage with lots of CPU and >> >> memory. I set "message_journal_enabled = false" which seems to have >> >> temporarily resolved the problem. >> >> >> >> How do I troubleshoot the journal? All of the other components are >> >> working fine. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "graylog2" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> >> an >> >> email to graylog2+u...@googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > >> > >> > -- >> > Tristan Rhodes >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to graylog2+u...@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH - A Graylog company >> Steckelhörn 11 >> 20457 Hamburg >> Germany >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Problem generating/loading chunked Gelf message in graylog2
Hey,
if you want to send GELF messages from your PHP application, you might
want to look at https://github.com/bzikarsky/gelf-php/.
This is a ready to use PHP GELF library which also supports chunking.
Hope that helps!
Regards,
Bernd
On 1 March 2015 at 19:31, Jesús Alberto Vidal Cortés
wrote:
> Can anyone write a detailed sample of a a chunked message?
>
> Thank you very much
>
>
> On Friday, February 27, 2015 at 6:32:46 PM UTC+1, Jesús Alberto Vidal Cortés
> wrote:
>>
>> Hi, I'm trying to process with gawk a PHP log for loading it graylog2 (I
>> have many log lines really big). I'm not able of send the correct
>> information to graylog2 input UDP 12200
>>
>> If I want to send the next log (is gelf formated) entry to graylog2 using
>> two chunks how could I do it? What information must have exactly each chunk?
>>
>> {\n \"version\": \"1.1\",\n \"host\":\"phcaeproma01\",\n
>> \"short_message\":\"Chunked message\",\n \"timestamp\": 123455134,\n
>> \"level\":1,\n \"_remote_addr\":\"10.1.104.57\",\n
>> \"_idf\":\"987297342\",\n \"_process\":\"Process\",\n
>> \"_uid\":\"9798742.938292\",\n \"_idcert\":\"9386101233\" \n}
>>
>> I'm able of loading this log line without using chunks (it's a simple log
>> line sample) I'm trying to send the next two chunks to graylog2:
>>
>> 1.
>> \x1e\x0f000102{\n \"version\": \"1.1\",\n
>> \"host\":\"phcaeproma01\",\n \"short_message\":\"%s\",\n \"timestamp\":
>> %d,\n \"level\":%d,\n \"_remote_addr\":\"%s\",\n \"_idf\":\"%s\",\n
>> \"_process\":\"%s\",\n
>>
>> 2.
>> \x1e\x0f000112\"_uid\":\"%s\",\n \"_idcert\":\"%s\" \n}
>>
>> and I obtain the next trace in graylog2 server log
>>
>> 2015-02-26 16:59:05,389 DEBUG:
>> org.graylog2.plugin.inputs.transports.NettyTransport - More chunks necessary
>> to complete this message
>> 2015-02-26 16:59:05,390 DEBUG:
>> org.graylog2.inputs.codecs.GelfChunkAggregator - Dumping GELF chunk map
>> [chunks for 1 messages]:
>> Message <3030303030303031> Chunks:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ID: 3030303030303031Sequence: 49/50 Arrival:
>> 1424966345389 Data size: 212
>>
>>
>> 2015-02-26 16:59:05,390 DEBUG:
>> org.graylog2.plugin.inputs.transports.NettyTransport - More chunks necessary
>> to complete this message
>>
>>
>> What I'm doing wrong?
>>
>> I'm using the next sentences to send the information from gawk server to
>> graylog2 server:
>>
>> printf "\x1e\x0f%s%c%c%s","0001",48,50,substr(v_cad,1,200) |&
>> "/inet/udp/0/10.253.114.218/12200";
>> printf "\x1e\x0f%s%c%c%s","0001",49,50,substr(v_cad,201) |&
>> "/inet/udp/0/10.253.114.218/12200";
>>
>> Thank you very much for any help. It's very important to me be able of
>> send a long message in chunks
>
> --
> You received this message because you are subscribed to the Google Groups
> "graylog2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Logs from Cisco ASA with bad "source" field
Roberto, ah, okay. Sorry, I didn't know that you have other machines reporting via Syslog. Then you should create the Syslog input again. Make sure that the Syslog and Raw input are not listening on the same port! So you either have to change the port on your Cisco ASA or on your windows machines. Regarding syslog-ng: You can install syslog-ng and forward the Cisco ASA messages via that one. But then you have to pre-process the messages in syslog-ng. Otherwise the same messages would arrive in Graylog. Regards, Bernd On 2 March 2015 at 16:47, wrote: > Bernd, I've created a Raw INPUT as you said but after that all the sources > from Windows servers are bad. > > So maybe I can correct de Cisco servers logs but I buy a new problem with my > Windows servers. > > Is there any universal solution ? Maybe like Alejandro says, installing just > a syslog-ng for cisco servers and forward the logs after that to graylog?? > > Thanks again, > > Roberto > > El lunes, 2 de marzo de 2015, 7:58:30 (UTC-3), Bernd Ahlers escribió: >> >> Roberto, >> >> you replace the Syslog input with a Raw input. The extractors are >> applied to the Raw input to parse the logs then. >> In your setup, remove the Syslog input and start a Raw input on the >> same port. Then add the extractors as described in the blog post I >> sent you earlier. >> >> Regards, >> Bernd >> >> On 27 February 2015 at 20:17, wrote: >> > Dear Bernd, thanks for your helpful respondebut now I have a new >> > question. >> > >> > I have a Graylog2 server with just one INPUT "Syslog UDP" listening on >> > port >> > UDP/10514, and the tutorial said I have to create another INPUT "Raw" >> > suppose listening on port UDP/. >> > >> > How can I connect the raw input with the syslog input ??? I got lost... >> > >> > Thanks in advance, >> > >> > Roberto >> > >> > El viernes, 27 de febrero de 2015, 13:57:08 (UTC-3), Bernd Ahlers >> > escribió: >> >> >> >> Roberto, >> >> >> >> the Cisco ASA does not send valid Syslog, unfortunately. You have to >> >> create a "Raw" input and create extractors. >> >> >> >> There is a blog post about this here: >> >> http://spottedhyena.co.uk/2015/01/graylog2-cisco-asa-cisco-catalyst/ >> >> >> >> Hope that helps! >> >> >> >> Regards, >> >> Bernd >> >> >> >> On 27 February 2015 at 15:57, wrote: >> >> > Dear, I have a Graylog2 version 0.20.6 as our syslog server of our >> >> > company. >> >> > >> >> > I defined an INPUT "Syslog UDP" running on port UDP/10514, and after >> >> > that we >> >> > point several Windows and Linux servers to the Graylog2 with no >> >> > problems. >> >> > >> >> > But in the case of the Cisco ASA firewalls, we have a problem because >> >> > the >> >> > source sometimes matches something like: >> >> > >> >> > :%ASA-session-6-302013: >> >> > >> >> > In the Cisco ASA's I setup: >> >> > >> >> > logging enable >> >> > logging emblem >> >> > logging trap informational >> >> > logging history debugging >> >> > logging asdm debugging >> >> > logging device-id hostname >> >> > logging host inside_Frontend 10.1.1.1 format emblem >> >> > >> >> > I want to have the original hostname in the "source" field, so what >> >> > can >> >> > I >> >> > do??? >> >> > >> >> > Regards, >> >> > >> >> > Roberto >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "graylog2" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> > send >> >> > an >> >> > email to graylog2+u...@googlegroups.com. >> >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> >> >> -- >> >> Developer >> >> >> >> Tel.: +49 (0)40 609 452 077 >> >> Fax.: +49 (0)40 609 452 078 >> >> >> >>
Re: [graylog2] Graylog 1.0 UDP process buffer performance
Thanks for the feedback! :) Bernd On 2 March 2015 at 11:26, wrote: > I installed unbound locally and used this, and it seems to have resolved the > issue. It's odd that the old server didn't show this behavior, but I'm happy > enough that it's resolved anyway. :) > > Regards > Johan > > On Friday, February 27, 2015 at 2:02:08 PM UTC+1, Bernd Ahlers wrote: >> >> Johan, Henrik, >> >> I tried to track this problem down.The problem is that the JVM does >> not cache reverse DNS lookups. The available JVM DNS cache settings >> like "networkaddress.cache.ttl" only affect forward DNS lookups. >> >> The code for doing the reverse lookups in Graylog did not change in a >> long time, so this problem is not new in 1.0. >> >> I my test setup enabling "force_rdns" for a syslog input reduced the >> throughput from around 7000 msg/s to 300 msg/s. This was without a >> local DNS cache. Once I installed a DNS cache on the Graylog server, >> the throughput went up to around 3000 msg/s. >> >> We will investigate if there is a sane way to cache the reverse >> lookups ourselves. In the meantime I suggest to test with a DNS cache >> installed on the Graylog server nodes to see if that helps or to >> disable the "force_rdns" setting. >> >> Regards, >> Bernd >> >> On 25 February 2015 at 18:00, Bernd Ahlers wrote: >> > Johan, Henrik, >> > >> > thanks for the details. I created an issue on GitHub and will >> > investigate. >> > >> > https://github.com/Graylog2/graylog2-server/issues/999 >> > >> > Regards, >> > Bernd >> > >> > On 25 February 2015 at 17:48, Henrik Johansen wrote: >> >> Bernd, >> >> >> >> Correct - that issue started after 0.92.x. >> >> >> >> We are still seeing evaluated CPU utilisation but we are attributing >> >> that >> >> to the fact that 0.92 was loosing messages in our setup. >> >> >> >> >> >>> On 25 Feb 2015, at 17:37, Bernd Ahlers wrote: >> >>> >> >>> Henrik, >> >>> >> >>> uh, okay. I suppose it worked for you in 0.92 as well? >> >>> >> >>> I will create an issue on GitHub for that. >> >>> >> >>> Bernd >> >>> >> >>> On 25 February 2015 at 17:14, Henrik Johansen wrote: >> >>>> Bernd, >> >>>> >> >>>> We saw the exact same issue - here is a graph over the CPU idle >> >>>> percentage across a few of the cluster nodes during the upgrade : >> >>>> >> >>>> http://5.9.37.177/graylog_cluster_cpu_idle.png >> >>>> >> >>>> We went from ~20% CPU utilisation to ~100% CPU utilisation across >> >>>> ~200 cores and things only settled down after disabling force_rdns. >> >>>> >> >>>> >> >>>> On 25 Feb 2015, at 11:55, Bernd Ahlers wrote: >> >>>> >> >>>> Johan, >> >>>> >> >>>> the only thing that changed from 0.92 to 1.0 is that the DNS lookup >> >>>> is >> >>>> now done when the messages are read from the journal and not in the >> >>>> input path where the messages are received. Otherwise, nothing has >> >>>> changed in that regard. >> >>>> >> >>>> We do not do any manual caching of the DNS lookups, but the JVM >> >>>> caches >> >>>> them by default. Check >> >>>> >> >>>> http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html >> >>>> for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. >> >>>> >> >>>> Regards, >> >>>> Bernd >> >>>> >> >>>> On 25 February 2015 at 08:56, wrote: >> >>>> >> >>>> This is strange, I went through all of the settings for my reply, and >> >>>> we are >> >>>> indeed using rdns, and it seems to be the culprit. The strangeness is >> >>>> that >> >>>> it works fine on the old servers even though they're on the same >> >>>> networks, >> >>>> and using the same DNS's and resolver settings. >> >>>> Did something regarding reverse DNS change
Re: [graylog2] Re: Problem generating/loading chunked Gelf message in graylog2
You can find an example in the gelf-php project.
https://github.com/bzikarsky/gelf-php/blob/master/src/Gelf/Transport/UdpTransport.php#L106
Regards,
Bernd
On 2 March 2015 at 23:00, Jesús Alberto Vidal Cortés
wrote:
> Thanks Bernd, but we want to send log to graylog2 without modifying PHP
> configuration or application. Could you write a very simple sample of
> chunked message for graylog2 (in the official documentation there isn't any
> sample of chunked message, personally I think it is not sufficiently
> explained)
>
> Thank you again.
> Regards
> Alberto
>
> On Monday, March 2, 2015 at 1:54:53 PM UTC+1, Bernd Ahlers wrote:
>>
>> Hey,
>>
>> if you want to send GELF messages from your PHP application, you might
>> want to look at https://github.com/bzikarsky/gelf-php/.
>> This is a ready to use PHP GELF library which also supports chunking.
>>
>> Hope that helps!
>>
>> Regards,
>> Bernd
>>
>> On 1 March 2015 at 19:31, Jesús Alberto Vidal Cortés
>> wrote:
>> > Can anyone write a detailed sample of a a chunked message?
>> >
>> > Thank you very much
>> >
>> >
>> > On Friday, February 27, 2015 at 6:32:46 PM UTC+1, Jesús Alberto Vidal
>> > Cortés
>> > wrote:
>> >>
>> >> Hi, I'm trying to process with gawk a PHP log for loading it graylog2
>> >> (I
>> >> have many log lines really big). I'm not able of send the correct
>> >> information to graylog2 input UDP 12200
>> >>
>> >> If I want to send the next log (is gelf formated) entry to graylog2
>> >> using
>> >> two chunks how could I do it? What information must have exactly each
>> >> chunk?
>> >>
>> >> {\n \"version\": \"1.1\",\n \"host\":\"phcaeproma01\",\n
>> >> \"short_message\":\"Chunked message\",\n \"timestamp\": 123455134,\n
>> >> \"level\":1,\n \"_remote_addr\":\"10.1.104.57\",\n
>> >> \"_idf\":\"987297342\",\n \"_process\":\"Process\",\n
>> >> \"_uid\":\"9798742.938292\",\n \"_idcert\":\"9386101233\" \n}
>> >>
>> >> I'm able of loading this log line without using chunks (it's a simple
>> >> log
>> >> line sample) I'm trying to send the next two chunks to graylog2:
>> >>
>> >> 1.
>> >> \x1e\x0f000102{\n \"version\": \"1.1\",\n
>> >> \"host\":\"phcaeproma01\",\n \"short_message\":\"%s\",\n
>> >> \"timestamp\":
>> >> %d,\n \"level\":%d,\n \"_remote_addr\":\"%s\",\n \"_idf\":\"%s\",\n
>> >> \"_process\":\"%s\",\n
>> >>
>> >> 2.
>> >> \x1e\x0f000112\"_uid\":\"%s\",\n \"_idcert\":\"%s\" \n}
>> >>
>> >> and I obtain the next trace in graylog2 server log
>> >>
>> >> 2015-02-26 16:59:05,389 DEBUG:
>> >> org.graylog2.plugin.inputs.transports.NettyTransport - More chunks
>> >> necessary
>> >> to complete this message
>> >> 2015-02-26 16:59:05,390 DEBUG:
>> >> org.graylog2.inputs.codecs.GelfChunkAggregator - Dumping GELF chunk map
>> >> [chunks for 1 messages]:
>> >> Message <3030303030303031> Chunks:
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> &g
Re: [graylog2] Inputs gone after updating to 1.0.1 from the latest 0.9x
Arie,
please make sure you copy the node id file over to the new installation.
The inputs are assigned to Graylog server nodes based on the node id. If
that changes, your node does not have any inputs.
Bernd
Arie [Wed, Mar 18, 2015 at 05:21:25AM -0700] wrote:
>Hi all,
>
>some help needed. After updating to 1.0.1 all my inputs (2) and extractors
>are gone.
>Before the updateI created a contend pack, is there anyone that can help
>rewriting it
>to get my inpus back?
>
>below her is the pack.
>
>{
> "id" : null,
> "name" : "Nagios bundle",
> "description" : "Backup",
> "category" : "Monitoring",
> "inputs" : [ {
>"title" : "nagiosserver",
>"configuration" : {
> "port" : 8100,
> "allow_override_date" : true,
> "bind_address" : "0.0.0.0",
> "recv_buffer_size" : 1048576
>},
>"type" : "org.graylog2.inputs.syslog.tcp.SyslogTCPInput",
>"global" : false,
>"extractors" : [ {
> "title" : "extracthostname",
> "type" : "REGEX",
> "configuration" : {
>"regex_value" : "([a-zA-Z0-9\\-.]+)([a-z\\.]?)*;"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "hostname",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>}, {
> "title" : "service_message",
> "type" : "SPLIT_AND_INDEX",
> "configuration" : {
>"index" : 2,
>"split_by" : ";"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "service_message",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>}, {
> "title" : "alert_status",
> "type" : "SPLIT_AND_INDEX",
> "configuration" : {
>"index" : 3,
>"split_by" : ";"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "alert_status",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>}, {
> "title" : "error_message",
> "type" : "SPLIT_AND_INDEX",
> "configuration" : {
>"index" : 6,
>"split_by" : ";"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "error_message",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>}, {
> "title" : "CBS_partner",
> "type" : "REGEX",
> "configuration" : {
>"regex_value" : "\\s([1-9][0-9]0_[A-Z][A-Z]*.\\b)"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "CBS_Partner",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>}, {
> "title" : "RIS_Partner",
> "type" : "REGEX",
> "configuration" : {
>"regex_value" : "\\s(SRK_[A-Z][A-Z]*.\\b)"
> },
> "converters" : [ ],
> "order" : 0,
> "cursor_strategy" : "COPY",
> "target_field" : "RIS_Partner",
> "source_field" : "message",
> "condition_type" : "NONE",
> "condition_value" : ""
>} ],
>"static_fields" : { }
> }, {
>"title" : "ohdnetwerk",
>"configuration" : {
> "port" : 8000,
> "bind_address" : "0.0.0.0",
> "recv_buffer_size" : 1048576
>},
>"type" : "org.graylog2.inputs.gelf.udp.GELFUDPInput",
>"global" : false,
>"extractors" : [ ],
>"static_fields" : { }
> } ],
> "streams" : [ {
>"id" : "54ae9b0724ac1c3ac18cf641",
>"title" : "Java Service Error",
>"description" : "Java Service Error Meldingen",
>"disabled" : false,
>"outputs" : [ ],
>"stream_rules" : [ {
> "type" : "EXACT",
> "field" : "service_message",
> "value" : "proc_JAVA",
> "inverted" : false
>} ]
> }, {
>"id" : "54b7b9cd24acf433218a83d7",
>"title" : "CBS Parter 900_SRK heeft status Fail",
>"description" : "Partner 900_SRK probleem op het CBS systeem",
>"disabled" : false,
>"outputs" : [ ],
>"stream_rules" : [ {
> "type" : "REGEX",
> "field" : "message",
> "value" : "(?=.*cbs-prod).*ALERT.*False.*900_SRK",
> "inverted" : false
>} ]
> }, {
>"id" : "54b7b3f524acf433218a7d80",
>"title" : "RIS Parter SRK_CBS heeft status Fail",
>"description" : "Parter SRK_CBS probleem op het RIS systeem",
>"disabled" : false,
>"outputs" : [ ],
>"stream_rules" : [ {
> "type" : "REGEX",
> "field" : "message",
> "value" : "(?=.*ris-prod).*ALERT.*False.*SRK_CBS",
> "inverted" : false
>} ]
> }, {
>"id" : "549b001f24ac266f4e59c913",
>"title" : "Hosts Down",
>"description" : "Hosts die down gemeld worden in Nagios",
>"disabled" : false,
>"outputs" : [ ],
>"stream_rules" : [ {
> "type" : "EXACT",
> "field" : "service_message",
> "value" : "DOWN",
>
Re: [graylog2] Re: collector questions
Arie,
thanks for the report. I will try to reproduce the service installation
error and come back to you.
Bernd
Arie [Wed, May 27, 2015 at 07:02:20AM -0700] wrote:
>Okay it is running and sending data to graylog from windows,
>Now it is only not installing as a service, having the following error.
>
>C:\collector\bin>graylog-collector-service.bat install GA
>Installing service for Graylog Collector
>
>Service name: "GA"
>JAVA_HOME:"C:\Program Files\Java\jre7\"
>ARCH: "x86"
>
>WARNING: JAVA_HOME points to a JRE and not JDK installation; a client (not
>a server) JVM will be used...
>[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
>C:\collector\bin\\windows\graylog-collector-service-x86.exe
>[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
>[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
>exit value: 1 (Failed to parse command line arguments)
>ERROR: Failed to install service: GA
>
>C:\collector\bin>
>
>
>On Wednesday, May 27, 2015 at 3:30:20 PM UTC+2, Arie wrote:
>>
>> Sorry, I see the typo in my config :-( It is running now
>>
>>
>> On Wednesday, May 27, 2015 at 2:29:31 PM UTC+2, Arie wrote:
>>>
>>> I am playing around with the collector.
>>>
>>> From a linux machine we are getting data into our test machine, Al-tho
>>> data is flat/ one line message.
>>>
>>> Within windows(2003) we have the following error:
>>>
>>> C:\collector\bin>graylog-collector.bat run -f
>>> c:\collector\config\collector.conf
>>> 2015-05-27T13:20:06.846+0200 INFO [main] cli.commands.Run - Starting
>>> Collector v0.2.1 (commit 93f4b8e)
>>> 2015-05-27T13:20:08.940+0200 INFO [main] collector.utils.CollectorId -
>>> Collector ID: dd6c1e19-19b5-422e-b06f-14799d5f7b14
>>> 2015-05-27T13:20:08.987+0200 ERROR [main] cli.commands.Run -
>>> Configuration Error: [local-syslog] No configuration setting found for key
>>> 'type'
>>> 2015-05-27T13:20:09.002+0200 INFO [main] cli.commands.Run - Exit
>>> 2015-05-27T13:20:09.002+0200 INFO [Thread-1] cli.commands.Run -
>>> Stopping...
>>>
>>> in the config on windows (2003):
>>>
>>> inputs {
>>> local-syslog {
>>> win-application {
>>> type = "windows-eventlog"
>>> source-name = "Application"
>>> poll-interval = 1s
>>> }
>>> }
>>> }
>>>
>>>
>>>
>>> Is this just where we are now, and are there improvements coming up/
>>>
>>>
>>> nice work.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"graylog2" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to graylog2+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: collector questions
Arie,
the following worked for me on Windows 7.
C:\collector\bin>graylog-collector-service.bat install GC
Installing service for Graylog Collector
Service name: "GC"
JAVA_HOME:"C:\Program Files\Java\jre7\"
ARCH: "x86"
WARNING: JAVA_HOME points to a JRE and not JDK installation; a client (not
a server) JVM will be used...
Service 'GC' has been installed
So I guess that there is an issue with Windows 2003 and the service
script. I will try to find a Windows 2003 VM to reproduce it.
I also created an issue on GitHub for it.
https://github.com/Graylog2/collector/issues/15
Regards,
Bernd
Arie [Wed, May 27, 2015 at 07:02:20AM -0700] wrote:
>Okay it is running and sending data to graylog from windows,
>Now it is only not installing as a service, having the following error.
>
>C:\collector\bin>graylog-collector-service.bat install GA
>Installing service for Graylog Collector
>
>Service name: "GA"
>JAVA_HOME:"C:\Program Files\Java\jre7\"
>ARCH: "x86"
>
>WARNING: JAVA_HOME points to a JRE and not JDK installation; a client (not
>a server) JVM will be used...
>[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
>C:\collector\bin\\windows\graylog-collector-service-x86.exe
>[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
>[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
>exit value: 1 (Failed to parse command line arguments)
>ERROR: Failed to install service: GA
>
>C:\collector\bin>
>
>
>On Wednesday, May 27, 2015 at 3:30:20 PM UTC+2, Arie wrote:
>>
>> Sorry, I see the typo in my config :-( It is running now
>>
>>
>> On Wednesday, May 27, 2015 at 2:29:31 PM UTC+2, Arie wrote:
>>>
>>> I am playing around with the collector.
>>>
>>> From a linux machine we are getting data into our test machine, Al-tho
>>> data is flat/ one line message.
>>>
>>> Within windows(2003) we have the following error:
>>>
>>> C:\collector\bin>graylog-collector.bat run -f
>>> c:\collector\config\collector.conf
>>> 2015-05-27T13:20:06.846+0200 INFO [main] cli.commands.Run - Starting
>>> Collector v0.2.1 (commit 93f4b8e)
>>> 2015-05-27T13:20:08.940+0200 INFO [main] collector.utils.CollectorId -
>>> Collector ID: dd6c1e19-19b5-422e-b06f-14799d5f7b14
>>> 2015-05-27T13:20:08.987+0200 ERROR [main] cli.commands.Run -
>>> Configuration Error: [local-syslog] No configuration setting found for key
>>> 'type'
>>> 2015-05-27T13:20:09.002+0200 INFO [main] cli.commands.Run - Exit
>>> 2015-05-27T13:20:09.002+0200 INFO [Thread-1] cli.commands.Run -
>>> Stopping...
>>>
>>> in the config on windows (2003):
>>>
>>> inputs {
>>> local-syslog {
>>> win-application {
>>> type = "windows-eventlog"
>>> source-name = "Application"
>>> poll-interval = 1s
>>> }
>>> }
>>> }
>>>
>>>
>>>
>>> Is this just where we are now, and are there improvements coming up/
>>>
>>>
>>> nice work.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"graylog2" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to graylog2+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: collector questions
Arie,
can you please check if this script works for you?
https://gist.github.com/bernd/d26366422d42154534db
Thanks!
Bernd
Arie [Wed, May 27, 2015 at 07:02:20AM -0700] wrote:
>Okay it is running and sending data to graylog from windows,
>Now it is only not installing as a service, having the following error.
>
>C:\collector\bin>graylog-collector-service.bat install GA
>Installing service for Graylog Collector
>
>Service name: "GA"
>JAVA_HOME:"C:\Program Files\Java\jre7\"
>ARCH: "x86"
>
>WARNING: JAVA_HOME points to a JRE and not JDK installation; a client (not
>a server) JVM will be used...
>[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
>C:\collector\bin\\windows\graylog-collector-service-x86.exe
>[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
>[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
>exit value: 1 (Failed to parse command line arguments)
>ERROR: Failed to install service: GA
>
>C:\collector\bin>
>
>
>On Wednesday, May 27, 2015 at 3:30:20 PM UTC+2, Arie wrote:
>>
>> Sorry, I see the typo in my config :-( It is running now
>>
>>
>> On Wednesday, May 27, 2015 at 2:29:31 PM UTC+2, Arie wrote:
>>>
>>> I am playing around with the collector.
>>>
>>> From a linux machine we are getting data into our test machine, Al-tho
>>> data is flat/ one line message.
>>>
>>> Within windows(2003) we have the following error:
>>>
>>> C:\collector\bin>graylog-collector.bat run -f
>>> c:\collector\config\collector.conf
>>> 2015-05-27T13:20:06.846+0200 INFO [main] cli.commands.Run - Starting
>>> Collector v0.2.1 (commit 93f4b8e)
>>> 2015-05-27T13:20:08.940+0200 INFO [main] collector.utils.CollectorId -
>>> Collector ID: dd6c1e19-19b5-422e-b06f-14799d5f7b14
>>> 2015-05-27T13:20:08.987+0200 ERROR [main] cli.commands.Run -
>>> Configuration Error: [local-syslog] No configuration setting found for key
>>> 'type'
>>> 2015-05-27T13:20:09.002+0200 INFO [main] cli.commands.Run - Exit
>>> 2015-05-27T13:20:09.002+0200 INFO [Thread-1] cli.commands.Run -
>>> Stopping...
>>>
>>> in the config on windows (2003):
>>>
>>> inputs {
>>> local-syslog {
>>> win-application {
>>> type = "windows-eventlog"
>>> source-name = "Application"
>>> poll-interval = 1s
>>> }
>>> }
>>> }
>>>
>>>
>>>
>>> Is this just where we are now, and are there improvements coming up/
>>>
>>>
>>> nice work.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"graylog2" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to graylog2+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: collector questions
Arie,
thank you for the update!
Bernd
Arie [Wed, May 27, 2015 at 02:14:32PM -0700] wrote:
>It appears to go wrong at this line:
>
>"%PROCRUN%" //IS//%SERVICE_NAME% .. etc.
>
>No errors before.
>
>
>
>Op woensdag 27 mei 2015 22:25:02 UTC+2 schreef Bernd Ahlers:
>>
>> Arie,
>>
>> can you please check if this script works for you?
>>
>> https://gist.github.com/bernd/d26366422d42154534db
>>
>> Thanks!
>>
>> Bernd
>>
>> Arie [Wed, May 27, 2015 at 07:02:20AM -0700] wrote:
>> >Okay it is running and sending data to graylog from windows,
>> >Now it is only not installing as a service, having the following error.
>> >
>> >C:\collector\bin>graylog-collector-service.bat install GA
>> >Installing service for Graylog Collector
>> >
>> >Service name: "GA"
>> >JAVA_HOME:"C:\Program Files\Java\jre7\"
>> >ARCH: "x86"
>> >
>> >WARNING: JAVA_HOME points to a JRE and not JDK installation; a client
>> (not
>> >a server) JVM will be used...
>> >[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
>> >C:\collector\bin\\windows\graylog-collector-service-x86.exe
>> >[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
>> >[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
>> >exit value: 1 (Failed to parse command line arguments)
>> >ERROR: Failed to install service: GA
>> >
>> >C:\collector\bin>
>> >
>> >
>> >On Wednesday, May 27, 2015 at 3:30:20 PM UTC+2, Arie wrote:
>> >>
>> >> Sorry, I see the typo in my config :-( It is running now
>> >>
>> >>
>> >> On Wednesday, May 27, 2015 at 2:29:31 PM UTC+2, Arie wrote:
>> >>>
>> >>> I am playing around with the collector.
>> >>>
>> >>> From a linux machine we are getting data into our test machine, Al-tho
>> >>> data is flat/ one line message.
>> >>>
>> >>> Within windows(2003) we have the following error:
>> >>>
>> >>> C:\collector\bin>graylog-collector.bat run -f
>> >>> c:\collector\config\collector.conf
>> >>> 2015-05-27T13:20:06.846+0200 INFO [main] cli.commands.Run - Starting
>> >>> Collector v0.2.1 (commit 93f4b8e)
>> >>> 2015-05-27T13:20:08.940+0200 INFO [main] collector.utils.CollectorId
>> -
>> >>> Collector ID: dd6c1e19-19b5-422e-b06f-14799d5f7b14
>> >>> 2015-05-27T13:20:08.987+0200 ERROR [main] cli.commands.Run -
>> >>> Configuration Error: [local-syslog] No configuration setting found for
>> key
>> >>> 'type'
>> >>> 2015-05-27T13:20:09.002+0200 INFO [main] cli.commands.Run - Exit
>> >>> 2015-05-27T13:20:09.002+0200 INFO [Thread-1] cli.commands.Run -
>> >>> Stopping...
>> >>>
>> >>> in the config on windows (2003):
>> >>>
>> >>> inputs {
>> >>> local-syslog {
>> >>> win-application {
>> >>> type = "windows-eventlog"
>> >>> source-name = "Application"
>> >>> poll-interval = 1s
>> >>> }
>> >>> }
>> >>> }
>> >>>
>> >>>
>> >>>
>> >>> Is this just where we are now, and are there improvements coming up/
>> >>>
>> >>>
>> >>> nice work.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >
>> >--
>> >You received this message because you are subscribed to the Google Groups
>> "graylog2" group.
>> >To unsubscribe from this group and stop receiving emails from it, send an
>> email to graylog2+u...@googlegroups.com .
>> >For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> Developer
>>
>> Tel.: +49 (0)40 609 452 077
>> Fax.: +49 (0)40 609 452 078
>>
>> TORCH GmbH - A Graylog company
>> Steckelhörn 11
>> 20457 Hamburg
>> Germany
>>
>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
>> Geschäftsführer: Lennart Koopmann (CEO)
>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"graylog2" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to graylog2+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog 1.1.0-beta.2 collector issue in webinterface
Arie, thanks for the report. Do you still have that problem with beta.3? Bernd Arie [Thu, May 28, 2015 at 06:22:49AM -0700] wrote: >Hi All, > >When we look @ System > Collectors and select "show messages", >no messages are show in the UI. > >Messages are visible with a normal search. > > >Running on centos-6.6 / elastic 1.5.2 / JRE 1.8 > >hth,, > >Arie > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: collector questions
Arie,
can you put an ECHO in front of the "%PROCRUN //IS//%SERVICE_NAME%"
line. That should print the command as it would be executed. Then try to
fiddle with it until it works. That would be awesome!
Bernd
Arie [Wed, May 27, 2015 at 02:14:32PM -0700] wrote:
>It appears to go wrong at this line:
>
>"%PROCRUN%" //IS//%SERVICE_NAME% .. etc.
>
>No errors before.
>
>
>
>Op woensdag 27 mei 2015 22:25:02 UTC+2 schreef Bernd Ahlers:
>>
>> Arie,
>>
>> can you please check if this script works for you?
>>
>> https://gist.github.com/bernd/d26366422d42154534db
>>
>> Thanks!
>>
>> Bernd
>>
>> Arie [Wed, May 27, 2015 at 07:02:20AM -0700] wrote:
>> >Okay it is running and sending data to graylog from windows,
>> >Now it is only not installing as a service, having the following error.
>> >
>> >C:\collector\bin>graylog-collector-service.bat install GA
>> >Installing service for Graylog Collector
>> >
>> >Service name: "GA"
>> >JAVA_HOME:"C:\Program Files\Java\jre7\"
>> >ARCH: "x86"
>> >
>> >WARNING: JAVA_HOME points to a JRE and not JDK installation; a client
>> (not
>> >a server) JVM will be used...
>> >[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
>> >C:\collector\bin\\windows\graylog-collector-service-x86.exe
>> >[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
>> >[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
>> >exit value: 1 (Failed to parse command line arguments)
>> >ERROR: Failed to install service: GA
>> >
>> >C:\collector\bin>
>> >
>> >
>> >On Wednesday, May 27, 2015 at 3:30:20 PM UTC+2, Arie wrote:
>> >>
>> >> Sorry, I see the typo in my config :-( It is running now
>> >>
>> >>
>> >> On Wednesday, May 27, 2015 at 2:29:31 PM UTC+2, Arie wrote:
>> >>>
>> >>> I am playing around with the collector.
>> >>>
>> >>> From a linux machine we are getting data into our test machine, Al-tho
>> >>> data is flat/ one line message.
>> >>>
>> >>> Within windows(2003) we have the following error:
>> >>>
>> >>> C:\collector\bin>graylog-collector.bat run -f
>> >>> c:\collector\config\collector.conf
>> >>> 2015-05-27T13:20:06.846+0200 INFO [main] cli.commands.Run - Starting
>> >>> Collector v0.2.1 (commit 93f4b8e)
>> >>> 2015-05-27T13:20:08.940+0200 INFO [main] collector.utils.CollectorId
>> -
>> >>> Collector ID: dd6c1e19-19b5-422e-b06f-14799d5f7b14
>> >>> 2015-05-27T13:20:08.987+0200 ERROR [main] cli.commands.Run -
>> >>> Configuration Error: [local-syslog] No configuration setting found for
>> key
>> >>> 'type'
>> >>> 2015-05-27T13:20:09.002+0200 INFO [main] cli.commands.Run - Exit
>> >>> 2015-05-27T13:20:09.002+0200 INFO [Thread-1] cli.commands.Run -
>> >>> Stopping...
>> >>>
>> >>> in the config on windows (2003):
>> >>>
>> >>> inputs {
>> >>> local-syslog {
>> >>> win-application {
>> >>> type = "windows-eventlog"
>> >>> source-name = "Application"
>> >>> poll-interval = 1s
>> >>> }
>> >>> }
>> >>> }
>> >>>
>> >>>
>> >>>
>> >>> Is this just where we are now, and are there improvements coming up/
>> >>>
>> >>>
>> >>> nice work.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >
>> >--
>> >You received this message because you are subscribed to the Google Groups
>> "graylog2" group.
>> >To unsubscribe from this group and stop receiving emails from it, send an
>> email to graylog2+u...@googlegroups.com .
>> >For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> Developer
>>
>> Tel.: +49 (0)40 609 452 077
>> Fax.: +49 (0)40 609 452 078
>>
>> TORCH GmbH - A Graylog company
>> Steckelhörn 11
>> 20457 Hamburg
>> Germany
>>
>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
>> Geschäftsführer: Lennart Koopmann (CEO)
>>
>
>--
>You received this message because you are subscribed to the Google Groups
>"graylog2" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to graylog2+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
--
Developer
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078
TORCH GmbH - A Graylog company
Steckelhörn 11
20457 Hamburg
Germany
Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog 1.1.0-beta.2 collector issue in webinterface
Arie, thanks for he report. There is an issue and a pull request to fix the issue on GitHub. https://github.com/Graylog2/graylog2-web-interface/issues/1334 https://github.com/Graylog2/graylog2-server/pull/1190 This will be fixed in the next beta or rc. Regards, Bernd Arie [Thu, May 28, 2015 at 07:12:30AM -0700] wrote: >Hi Bernd, > >Just installed and tried it, the error is still there. > >Tested it with a windows and linux collector, and in both cases, no results. > >Arie. > >On Thursday, May 28, 2015 at 3:58:56 PM UTC+2, Bernd Ahlers wrote: >> >> Arie, >> >> thanks for the report. Do you still have that problem with beta.3? >> >> Bernd >> >> Arie [Thu, May 28, 2015 at 06:22:49AM -0700] wrote: >> >Hi All, >> > >> >When we look @ System > Collectors and select "show messages", >> >no messages are show in the UI. >> > >> >Messages are visible with a normal search. >> > >> > >> >Running on centos-6.6 / elastic 1.5.2 / JRE 1.8 >> > >> >hth,, >> > >> >Arie >> > >> >-- >> >You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> >To unsubscribe from this group and stop receiving emails from it, send an >> email to graylog2+u...@googlegroups.com . >> >For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> Developer >> >> Tel.: +49 (0)40 609 452 077 >> Fax.: +49 (0)40 609 452 078 >> >> TORCH GmbH - A Graylog company >> Steckelhörn 11 >> 20457 Hamburg >> Germany >> >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >> Geschäftsführer: Lennart Koopmann (CEO) >> > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
