[graylog2] Re: HA setup with redundant paths: avoid duplicates
This is a good question. Redundant log entries are a problem for us as well. I'm looking for a way to get rid of them. I hope someone answers... On Tuesday, March 15, 2016 at 11:32:20 AM UTC-4, tok...@gmx.net wrote: > > Hi! > > We are currently planning a large scale Graylog setup, consisting of > syslog-based shipping to dedicated Logstash forwarders (for preprocessing) > and then transferring into Graylog. We now have some issues regarding the > overall architecture for which I would appreciate your support: > > Due to high availability requirements, each individual component is > required to be redundant. Bottom up, this seems to be achievable with > Elasticsearch clustering, a MongoDB replica set and multiple Graylog nodes. > However, if we implement redundant syslog shipping, i.e. each log source > sends its events to two distinct forwarders (via different network paths) > and then into Graylog, we most likely will end up with duplicate log > entries. What is an approach to avoid this? Is it possible to solve this in > the message queue component? > > Thank you in advance for feedback. > > Best regards > tokred > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/2d805358-7ea2-4e20-b4fd-3666f5d183db%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Enable TLS on Graylog 'GELF TCP' input
Thanks Jochen! This helps. I think we need to upgrade our Gelf Client to 1.3.1 to do this. Things look pretty straight forward after that. Best, Skip On Mon, Apr 4, 2016 at 9:09 AM, Jochen Schalanda wrote: > Hi Skip, > > you can configure the GELF client to enable TLS in general and client-side > TLS validation using the GelfConfiguration class. > > See the following methods for details: > > >- > > https://static.javadoc.io/org.graylog2/gelfclient/1.3.1/org/graylog2/gelfclient/GelfConfiguration.html#enableTls-- >- > > https://static.javadoc.io/org.graylog2/gelfclient/1.3.1/org/graylog2/gelfclient/GelfConfiguration.html#enableTlsCertVerification-- >- > > https://static.javadoc.io/org.graylog2/gelfclient/1.3.1/org/graylog2/gelfclient/GelfConfiguration.html#tlsTrustCertChainFile-java.io.File- > > > Cheers, > Jochen > > On Thursday, 31 March 2016 22:00:35 UTC+2, Skip Cole wrote: >> >> Hi, >> >> I'm trying to enable TLS communication with a Graylog Input. >> >> We have a custom piece of software that runs on the machines we are >> auditing (clients) which sends log entries to the Graylog input. Real >> straightforward: >> >> GelfTransport transport = GelfTransports.create(config); >> transport.send(gelfMessage); >> >> >> I am looking for how I set the clients up to use TLS. I believe it is in >> the GelfConfiguration, but if anyone can point me to some documentation, or >> pointers, on how to do that, it will be greatly appreciated. >> >> Thanks, >> Skip >> >> >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/graylog2/1JPmz2D-R6E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/c107cd3e-a0f7-44b5-b4b1-f56831158921%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/c107cd3e-a0f7-44b5-b4b1-f56831158921%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAHYTHi6WCz5APydHDtO5DHATjATuTyKdvKZoNfYr5p5wRLtO2Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Enable TLS on Graylog 'GELF TCP' input
Hi, I'm trying to enable TLS communication with a Graylog Input. We have a custom piece of software that runs on the machines we are auditing (clients) which sends log entries to the Graylog input. Real straightforward: GelfTransport transport = GelfTransports.create(config); transport.send(gelfMessage); I am looking for how I set the clients up to use TLS. I believe it is in the GelfConfiguration, but if anyone can point me to some documentation, or pointers, on how to do that, it will be greatly appreciated. Thanks, Skip -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/6188ffc1-b7dd-47af-8bd0-1b3091b4abbe%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Automating Graylog Dashboard Upgrades via Chef and the Graylog REST api
Dear Graylog Rock Stars, We are using Chef to automate the installation of our Graylog Server. We automate the creation of inputs, dashboards and widgets via the uploading of content packs via the REST api. (It is, quite honestly, super cool.) The question has come up "How do we automate upgrades?" So if one of our dashboards changes, or new ones get added, how do we use Chef to handle the upgrade of our content? One thing that occurred to us was that, if we knew the ids of the dashboards that we had added, we could modify them directly via the REST api. But it does not seem possible, with the current rest API, to upload a dashboard with a specific ID. When we try we get the message, "Unable to map property id. Known properties include: title, description". It seems like Graylog is always assigning the id. This makes things a bit more difficult, since to modify an existing dashboard, we will have to get its id first. Any thoughts on how to a.) Automate updates of Graylog content (inputs, dashboard, etc.) via Chef, or b.) set the specific ids for objects, such as dashboards, in Graylog? I suppose it might be possible to go straight into the Mongo database and change things. We haven't investigated that path. All thoughts are welcome and appreciated. Best, Skip -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f933f25f-4f92-4e27-9f52-e2f989c3cb19%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Timestamp in graylog
:-)
Thanks for this information!
You have brightened my day.
Sincerely,
Skip
On Thu, Jan 28, 2016 at 8:46 AM, Jochen Schalanda
wrote:
> Hi,
>
> the value of the timestamp field in the actual GELF message
> (1453388845999.000) is invalid or rather translates to a date very far in
> the future (48026-01-22T03:53:19.000Z to be exact).
>
> The timestamp field in a GELF message has to contain the seconds (!) since
> UNIX epoch, not milliseconds since UNIX epoch; see
> https://www.graylog.org/resources/gelf/. I think it's simply a matter of
> an incorrect conversion in the library being used (1453388845.999 would
> translate to 2016-01-21T15:07:25.999Z which looks much more sane).
>
>
> Cheers,
> Jochen
>
> On Thursday, 28 January 2016 12:46:30 UTC+1, Skip Cole wrote:
>>
>> Hi Jan,
>>
>> Thanks for the reply. The thing is, we are passing in the timestamp. Here
>> is a 'toString' of the GelfMessage:
>>
>> GelfMessage{version="1.1" timestamp="1453388845999.000"
>> short_message="{timestamp=2016-01-21T10:07:25.999-05:00, level=DEBUG,
>> action=HTTP_REQUEST, user=felixfrankfurter}", level="ALERT(1)"}
>>
>> I notice that the timestamp in the short message is not in the right
>> format, so I'm going to change that, and see if that helps. But I thought
>> that the timestamp set at the higher level would be sufficient.
>>
>> I'll try this change out, and let you know how it goes.
>>
>> Thanks,
>> Skip
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/graylog2/o6inV7A4PBw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/b1b7bd24-fd0e-48cc-b1f9-aa65135d518c%40googlegroups.com
> <https://groups.google.com/d/msgid/graylog2/b1b7bd24-fd0e-48cc-b1f9-aa65135d518c%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAHYTHi4cin8sHSKAB2A%2BYRkJ%2BYGbi2OBX%2Bo_Y%2BR%2B6k8j0ZM84w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Timestamp in graylog
Hi Jan,
Thanks for the reply. The thing is, we are passing in the timestamp. Here
is a 'toString' of the GelfMessage:
GelfMessage{version="1.1" timestamp="1453388845999.000"
short_message="{timestamp=2016-01-21T10:07:25.999-05:00, level=DEBUG,
action=HTTP_REQUEST, user=felixfrankfurter}", level="ALERT(1)"}
I notice that the timestamp in the short message is not in the right
format, so I'm going to change that, and see if that helps. But I thought
that the timestamp set at the higher level would be sufficient.
I'll try this change out, and let you know how it goes.
Thanks,
Skip
On Thu, Jan 28, 2016 at 6:44 AM, Skip Cole wrote:
> Hi Jan,
>
> Thanks for the reply. The thing is, we are passing in the timestamp. Here
> is a 'toString' of the GelfMessage:
>
> GelfMessage{version="1.1" timestamp="1453388845999.000"
> short_message="{timestamp=2016-01-21T10:07:25.999-05:00, level=DEBUG,
> action=HTTP_REQUEST, user=hso_admin}", level="ALERT(1)"}
>
> I notice that the timestamp in the short message is not in the right
> format, so I'm going to change that, and see if that helps. But I thought
> that the timestamp set at the higher level would be sufficient.
>
> I'll try this change out, and let you know how it goes.
>
> Thanks,
> Skip
>
> On Thu, Jan 28, 2016 at 4:33 AM, Jan Doberstein wrote:
>
>> Hej Skip,
>>
>> just an idea - based on that what you have written.
>>
>> If you change the timestamp in gelf, graylog is not able to parse this.
>> (Reference: https://www.graylog.org/resources/gelf/ )
>>
>> *timestamp* number
>>
>> Seconds since UNIX epoch with optional decimal places for milliseconds;
>> SHOULD be set by client library. Will be set to NOW by server if absent.
>>
>> I guess that this will explain your findings ...
>>
>> regards
>> Jan
>>
>> 2016-01-27 17:26 GMT+01:00 Skip Cole :
>>
>>> Dear Wonderfull People,
>>>
>>> We send gelf messages to graylog to record our usage events. I have
>>> manipulated the gelf message to have the timestamp we want, but the
>>> messages are all showing up in graylog at the moment they were received. (I
>>> dump in 100 messages of events that took place over a 2 week period, and
>>> they all show up in the graphs at the moment I uploaded them.)
>>>
>>> I have been banging my head on this, and I bet there is a simple way
>>> around it. Any ideas?
>>>
>>> Thanks,
>>> Skip
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Graylog Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to graylog2+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/graylog2/0f677733-078f-4ae1-82d2-423ee7d3b62d%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> | Voice: +49 173 7100308 | Text: j...@jalogisch.de
>> | http:// <http://about.me/jandoberstein>jalogis.ch/bio
>> |---
>> | send from my extraordinary device
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Graylog Users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/graylog2/o6inV7A4PBw/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/graylog2/CAGm-bLYnRSnAx1ZfRq_%2BJBztQZNrH8nLrEKeCQwbizFwUrwzHg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/graylog2/CAGm-bLYnRSnAx1ZfRq_%2BJBztQZNrH8nLrEKeCQwbizFwUrwzHg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAHYTHi6opighYqS2rCVukZ6DeAsmp-XpELkM%3Dihtpk3oU5fDuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Timestamp in graylog
Hi Jan,
Thanks for the reply. The thing is, we are passing in the timestamp. Here
is a 'toString' of the GelfMessage:
GelfMessage{version="1.1" timestamp="1453388845999.000"
short_message="{timestamp=2016-01-21T10:07:25.999-05:00, level=DEBUG,
action=HTTP_REQUEST, user=hso_admin}", level="ALERT(1)"}
I notice that the timestamp in the short message is not in the right
format, so I'm going to change that, and see if that helps. But I thought
that the timestamp set at the higher level would be sufficient.
I'll try this change out, and let you know how it goes.
Thanks,
Skip
On Thu, Jan 28, 2016 at 4:33 AM, Jan Doberstein wrote:
> Hej Skip,
>
> just an idea - based on that what you have written.
>
> If you change the timestamp in gelf, graylog is not able to parse this.
> (Reference: https://www.graylog.org/resources/gelf/ )
>
> *timestamp* number
>
> Seconds since UNIX epoch with optional decimal places for milliseconds;
> SHOULD be set by client library. Will be set to NOW by server if absent.
>
> I guess that this will explain your findings ...
>
> regards
> Jan
>
> 2016-01-27 17:26 GMT+01:00 Skip Cole :
>
>> Dear Wonderfull People,
>>
>> We send gelf messages to graylog to record our usage events. I have
>> manipulated the gelf message to have the timestamp we want, but the
>> messages are all showing up in graylog at the moment they were received. (I
>> dump in 100 messages of events that took place over a 2 week period, and
>> they all show up in the graphs at the moment I uploaded them.)
>>
>> I have been banging my head on this, and I bet there is a simple way
>> around it. Any ideas?
>>
>> Thanks,
>> Skip
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/graylog2/0f677733-078f-4ae1-82d2-423ee7d3b62d%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> | Voice: +49 173 7100308 | Text: j...@jalogisch.de
> | http:// <http://about.me/jandoberstein>jalogis.ch/bio
> |---
> | send from my extraordinary device
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/graylog2/o6inV7A4PBw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/CAGm-bLYnRSnAx1ZfRq_%2BJBztQZNrH8nLrEKeCQwbizFwUrwzHg%40mail.gmail.com
> <https://groups.google.com/d/msgid/graylog2/CAGm-bLYnRSnAx1ZfRq_%2BJBztQZNrH8nLrEKeCQwbizFwUrwzHg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAHYTHi4KXea4uQAJndZ6HJhqEAqr3OrvbBRQEet0fjcHBThNQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Timestamp in graylog
Dear Wonderfull People, We send gelf messages to graylog to record our usage events. I have manipulated the gelf message to have the timestamp we want, but the messages are all showing up in graylog at the moment they were received. (I dump in 100 messages of events that took place over a 2 week period, and they all show up in the graphs at the moment I uploaded them.) I have been banging my head on this, and I bet there is a simple way around it. Any ideas? Thanks, Skip -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/0f677733-078f-4ae1-82d2-423ee7d3b62d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Dashboard config export
Thank you! I'm looking into how to use these now. For anyone else looking at this thread, I found the documentation on content packs here: http://docs.graylog.org/en/1.3/pages/sending_data.html#content-packs And a marketplace of Content Packs here: https://marketplace.graylog.org/addons?kind=content_pack Best, Skip On Friday, January 8, 2016 at 5:20:50 AM UTC-5, Jochen Schalanda wrote: > > Hi Skip, > > you can export stream and dashboard configuration of a Graylog instance > using content packs and import those into other Graylog instances. > > > Cheers, > Jochen > > On Thursday, 7 January 2016 18:34:51 UTC+1, Skip Cole wrote: >> >> Hi, >> >> Has any progress been made on this? >> I looked through the issues and didn't see any matches, but maybe I >> missed something. >> >> We would like to completely automate (with Chef) our deployment. Does >> someone always have to go in by hand and create Dashboards and Streams? >> >> I can look into using your REST Api. You mention it for streams, does it >> also work for Dashboards? >> >> Thanks, >> Skip >> >> >> On Wednesday, February 12, 2014 at 5:19:06 AM UTC-5, Kay Röpke wrote: >>> >>> No, we don't have that feature yet. >>> Could you please create a github issue for this? >>> >>> To work around this problem you could use the REST API to create the >>> streams automatically, if you have lots of them. >>> Otherwise I'm afraid this will be a manual process :( >>> >>> Best, >>> Kay >>> >>> On Wednesday, February 12, 2014 11:03:18 AM UTC+1, Jean-Luc Bassereau >>> wrote: >>>> >>>> Hello, >>>> >>>> In our IT architecture, we have Dev servers and Prod servers. We are >>>> testing settings on Dev servers and we apply these settings on Prod >>>> servers >>>> when these settings seem good. >>>> Is it possible to export Dashboard and/or Stream settings from a >>>> Graylog2 instance and then import it into another one ? >>>> >>>> -- >>>> Regards, >>>> Jean-Luc Bassereau >>>> >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/37da1bfe-ccad-4ef0-bcc0-361c89749101%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
