[graylog2] prioritizing realtime inflow over journal catchup ?
Thinking about failure/recovery mode ... Let's say we've had an incident, our search system flopped over and died or something, and graylog is saturated with messages; graylog is busy flushing out as fast as possible in order to catch up. In this mode, for the duration of this "catching up" phase, we're blacked out on realtime flow of log events. I'm wondering how we reduce the time to recover on this. My customers want to use this system for a variety of purposes but mainly the data must be fresh and hot. One way I'm thinking about doing this, and I admit I don't particularly like this idea, is to spin up a bunch of new graylog instances and cut the realtime traffic into it - at least this way we can bypass the saturated graylogs and get fresher data into the indexes. The penalty with this, of course, is that it requires more nodes to be added, more connections, it feels like a fair amount of change to incur to recover. Maybe that's the best we can do. I'm wondering if anyone out there either has established and trusted runbooks (or fail safe implementations) for responding to this scenario and would be willing to share advice. I'm also wondering if this is a feature area the Graylog dev team has been thinking about, and would love to get your thoughts. Hope you're having a great Monday. Dave -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAJqE1Kdx5kWyjZLim_Pcw5r7GLFdmwgffrZ%3D8gyiOwnFsxsixw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Streams and Inputs
Hi Felipe, by default all messages from all inputs will be checked and tagged for all streams for which they match the stream rules. If you only want to route messages of a specific input into a stream, you can check the "internal" field gl2_source_input in your stream rules which will contain the ID of the input the message was received with. Cheers, Jochen On Monday, 14 September 2015 16:57:20 UTC+2, Felipe Santos wrote: > > I would like to know If I create a stream, it will process all messages > from all inputs, or is there anyway to select which input that stream will > process? > > Thanks > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/102a7dd8-bc55-44e9-8165-cd846ab3a267%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: System information unavailable on a three node Graylog cluster
Thanks Jochen, increasing the timeout to 30 seconds solved the problem. Now the System/Overview page displays correctly on the Graylog WebUI. I will investigate if any particular errors appear on the server.log of the nodes when that page is requested (it currently takes some 10 seconds to load... I have no idea if this is a reasonable time, considering our setup). Ciao Lorenzo Il 14/09/2015 16:51, Jochen Schalanda ha scritto: Hi Lorenzo, the error message you've seen ("[…] We expected HTTP 200, but got a HTTP -1.") is usually a sign of a request timeout. By default the request timeout for HTTP requests from the Graylog web interface to a Graylog server node is 5 seconds and can be customized in the configuration file of your Graylog web interface, see https://github.com/Graylog2/graylog2-web-interface/blob/1.2.0/misc/graylog-web-interface.conf.example#L31-L32 for details. Of course it would also be interesting to find out why the request worked in the past and is running into a timeout right now. You should find some hints about this in your Graylog server node's log messages. Cheers, Jochen On Monday, 14 September 2015 16:39:21 UTC+2, Lorenzo Marotta wrote: Hello everyone, i've been running Graylog (currently, version 1.1.6) on a three node cluster for several months, and recently (i suspect, from last week) the "System/Overview" page on the Web Interface has stopped running. The error is *Reason:* Could not fetch system information. We expected HTTP 200, but got a HTTP -1. And the stack trace is ---cut--- * org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#execute (/ApiClientImpl.java:451/) * org.graylog2.restclient.models.ClusterService#getNumberOfSystemMessages (/ClusterService.java:128/) * controllers.SystemController#index (/SystemController.java:69/) * Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply (/routes_routing.scala:1931/) * Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply (/routes_routing.scala:1931/) * play.core.Router$HandlerInvokerFactory$$anon$4#resultCall (/Router.scala:264/) * play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#invocation (/Router.scala:255/) * play.core.j.JavaAction$$anon$1#call (/JavaAction.scala:55/) * play.GlobalSettings$1#call (/GlobalSettings.java:67/) * play.mvc.Security$AuthenticatedAction#call (/Security.java:44/) * play.core.j.JavaAction$$anonfun$11#apply (/JavaAction.scala:82/) * play.core.j.JavaAction$$anonfun$11#apply (/JavaAction.scala:82/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (/Future.scala:24/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#run (/Future.scala:24/) * play.core.j.HttpExecutionContext$$anon$2#run (/HttpExecutionContext.scala:40/) * play.api.libs.iteratee.Execution$trampoline$#execute (/Execution.scala:46/) * play.core.j.HttpExecutionContext#execute (/HttpExecutionContext.scala:32/) * scala.concurrent.impl.Future$#apply (/Future.scala:31/) * scala.concurrent.Future$#apply (/Future.scala:485/) * play.core.j.JavaAction$class#apply (/JavaAction.scala:82/) * play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#apply (/Router.scala:252/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (/Action.scala:130/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (/Action.scala:130/) * play.utils.Threads$#withContextClassLoader (/Threads.scala:21/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (/Action.scala:129/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (/Action.scala:128/) * scala.Option#map (/Option.scala:145/) * play.api.mvc.Action$$anonfun$apply$1#apply (/Action.scala:128/) * play.api.mvc.Action$$anonfun$apply$1#apply (/Action.scala:121/) * play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (/Iteratee.scala:483/) * play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (/Iteratee.scala:483/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (/Iteratee.scala:519/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (/Iteratee.scala:519/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (/Iteratee.scala:496/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (/Iteratee.scala:496/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (/Future.scala:24/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#run
[graylog2] Streams and Inputs
I would like to know If I create a stream, it will process all messages from all inputs, or is there anyway to select which input that stream will process? Thanks -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ad1469f8-22a8-43a8-af23-667bd8c76572%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: System information unavailable on a three node Graylog cluster
Hi Lorenzo, the error message you've seen ("[…] We expected HTTP 200, but got a HTTP -1.") is usually a sign of a request timeout. By default the request timeout for HTTP requests from the Graylog web interface to a Graylog server node is 5 seconds and can be customized in the configuration file of your Graylog web interface, see https://github.com/Graylog2/graylog2-web-interface/blob/1.2.0/misc/graylog-web-interface.conf.example#L31-L32 for details. Of course it would also be interesting to find out why the request worked in the past and is running into a timeout right now. You should find some hints about this in your Graylog server node's log messages. Cheers, Jochen On Monday, 14 September 2015 16:39:21 UTC+2, Lorenzo Marotta wrote: > > Hello everyone, > > i've been running Graylog (currently, version 1.1.6) on a three node > cluster for several months, and recently (i suspect, from last week) the > "System/Overview" page on the Web Interface has stopped running. > > The error is > > *Reason:* Could not fetch system information. We expected HTTP 200, but > got a HTTP -1. > > > And the stack trace is > > ---cut--- > >- org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#execute ( >*ApiClientImpl.java:451*) >- org.graylog2.restclient.models.ClusterService#getNumberOfSystemMessages >(*ClusterService.java:128*) >- controllers.SystemController#index (*SystemController.java:69*) >- > Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply >(*routes_routing.scala:1931*) >- > Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply >(*routes_routing.scala:1931*) >- play.core.Router$HandlerInvokerFactory$$anon$4#resultCall ( >*Router.scala:264*) >- > play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#invocation > >(*Router.scala:255*) >- play.core.j.JavaAction$$anon$1#call (*JavaAction.scala:55*) >- play.GlobalSettings$1#call (*GlobalSettings.java:67*) >- play.mvc.Security$AuthenticatedAction#call (*Security.java:44*) >- play.core.j.JavaAction$$anonfun$11#apply (*JavaAction.scala:82*) >- play.core.j.JavaAction$$anonfun$11#apply (*JavaAction.scala:82*) >- scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 >(*Future.scala:24*) >- scala.concurrent.impl.Future$PromiseCompletingRunnable#run ( >*Future.scala:24*) >- play.core.j.HttpExecutionContext$$anon$2#run ( >*HttpExecutionContext.scala:40*) >- play.api.libs.iteratee.Execution$trampoline$#execute ( >*Execution.scala:46*) >- play.core.j.HttpExecutionContext#execute ( >*HttpExecutionContext.scala:32*) >- scala.concurrent.impl.Future$#apply (*Future.scala:31*) >- scala.concurrent.Future$#apply (*Future.scala:485*) >- play.core.j.JavaAction$class#apply (*JavaAction.scala:82*) >- > play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#apply > >(*Router.scala:252*) >- > play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply >(*Action.scala:130*) >- > play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply >(*Action.scala:130*) >- play.utils.Threads$#withContextClassLoader (*Threads.scala:21*) >- play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply ( >*Action.scala:129*) >- play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply ( >*Action.scala:128*) >- scala.Option#map (*Option.scala:145*) >- play.api.mvc.Action$$anonfun$apply$1#apply (*Action.scala:128*) >- play.api.mvc.Action$$anonfun$apply$1#apply (*Action.scala:121*) >- play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply ( >*Iteratee.scala:483*) >- play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply ( >*Iteratee.scala:483*) >- play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply ( >*Iteratee.scala:519*) >- play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply ( >*Iteratee.scala:519*) >- > play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply >(*Iteratee.scala:496*) >- > play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply >(*Iteratee.scala:496*) >- scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 >(*Future.scala:24*) >- scala.concurrent.impl.Future$PromiseCompletingRunnable#run ( >*Future.scala:24*) >- akka.dispatch.TaskInvocation#run (*AbstractDispatcher.scala:41*) >- akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask#exec ( >*AbstractDispatcher.scala:393*) >- scala.concurrent.forkjoin.ForkJoinTask#doExec ( >*ForkJoinTask.java:260*) >- scala.concurrent.forkjoin.ForkJoinPool$WorkQueue#runTask ( >*ForkJoinPool.java:1339*) >- scala.concurrent.forkjoin.ForkJoinPool#runWorker ( >*ForkJoinPool.java:1979*) >-
[graylog2] System information unavailable on a three node Graylog cluster
Hello everyone, i've been running Graylog (currently, version 1.1.6) on a three node cluster for several months, and recently (i suspect, from last week) the "System/Overview" page on the Web Interface has stopped running. The error is *Reason:* Could not fetch system information. We expected HTTP 200, but got a HTTP -1. And the stack trace is ---cut--- * org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#execute (/ApiClientImpl.java:451/) * org.graylog2.restclient.models.ClusterService#getNumberOfSystemMessages (/ClusterService.java:128/) * controllers.SystemController#index (/SystemController.java:69/) * Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply (/routes_routing.scala:1931/) * Routes$$anonfun$routes$1$$anonfun$applyOrElse$41$$anonfun$apply$561#apply (/routes_routing.scala:1931/) * play.core.Router$HandlerInvokerFactory$$anon$4#resultCall (/Router.scala:264/) * play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#invocation (/Router.scala:255/) * play.core.j.JavaAction$$anon$1#call (/JavaAction.scala:55/) * play.GlobalSettings$1#call (/GlobalSettings.java:67/) * play.mvc.Security$AuthenticatedAction#call (/Security.java:44/) * play.core.j.JavaAction$$anonfun$11#apply (/JavaAction.scala:82/) * play.core.j.JavaAction$$anonfun$11#apply (/JavaAction.scala:82/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (/Future.scala:24/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#run (/Future.scala:24/) * play.core.j.HttpExecutionContext$$anon$2#run (/HttpExecutionContext.scala:40/) * play.api.libs.iteratee.Execution$trampoline$#execute (/Execution.scala:46/) * play.core.j.HttpExecutionContext#execute (/HttpExecutionContext.scala:32/) * scala.concurrent.impl.Future$#apply (/Future.scala:31/) * scala.concurrent.Future$#apply (/Future.scala:485/) * play.core.j.JavaAction$class#apply (/JavaAction.scala:82/) * play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#apply (/Router.scala:252/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (/Action.scala:130/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (/Action.scala:130/) * play.utils.Threads$#withContextClassLoader (/Threads.scala:21/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (/Action.scala:129/) * play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (/Action.scala:128/) * scala.Option#map (/Option.scala:145/) * play.api.mvc.Action$$anonfun$apply$1#apply (/Action.scala:128/) * play.api.mvc.Action$$anonfun$apply$1#apply (/Action.scala:121/) * play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (/Iteratee.scala:483/) * play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (/Iteratee.scala:483/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (/Iteratee.scala:519/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (/Iteratee.scala:519/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (/Iteratee.scala:496/) * play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (/Iteratee.scala:496/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (/Future.scala:24/) * scala.concurrent.impl.Future$PromiseCompletingRunnable#run (/Future.scala:24/) * akka.dispatch.TaskInvocation#run (/AbstractDispatcher.scala:41/) * akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask#exec (/AbstractDispatcher.scala:393/) * scala.concurrent.forkjoin.ForkJoinTask#doExec (/ForkJoinTask.java:260/) * scala.concurrent.forkjoin.ForkJoinPool$WorkQueue#runTask (/ForkJoinPool.java:1339/) * scala.concurrent.forkjoin.ForkJoinPool#runWorker (/ForkJoinPool.java:1979/) * scala.concurrent.forkjoin.ForkJoinWorkerThread#run (/ForkJoinWorkerThread.java:107/) ---cut--- MongoDB runs as a replica set of 3 nodes, on the 3 servers which also run graylog-server and graylog-web. Apart from the unavailability of the system information, everything seems to be OK. I tried checking the configurationm files, and everything seems to be OK; i've just reinstalled graylog-server and graylog-web on all three nodes, but to no avail. Has everyone here experienced a similar problem? Could it be related with a corruption of the configuration data in Mongodb? Just an idea... The error appear on the web interface of all 3 nodes. Long live and prosper, Lorenzo -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55F6DBB8.3060608%40gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Kinesis as Input
Thanks Bernd, https://graylog.ideas.aha.io/ideas/GL2E-I-447 On Wednesday, August 19, 2015 at 12:14:30 PM UTC+2, Bernd Ahlers wrote: > > Hey Zulfikar, > > Graylog cannot consume Kinesis streams yet. You could open a new feature > request in our ideas portal for this. > > https://www.graylog.org/product-ideas/ > > Regards, > Bernd > > Zulfikar Dharmawan [Wed, Aug 05, 2015 at 07:18:56AM -0700] wrote: > >Hi all, > > > >Just starting my journey with Graylog. We're planning to deploy Graylog > in > >AWS EC2. Is taking input from Kinesis stream in the pipeline? > >Thanks. > > > >Regards, > > > >Zul > > > >-- > >You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > >To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+u...@googlegroups.com . > >To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/57c220cb-62e0-4d5c-af1c-e61c64bad774%40googlegroups.com. > > > >For more options, visit https://groups.google.com/d/optout. > > > -- > Developer > > Tel.: +49 (0)40 609 452 077 > Fax.: +49 (0)40 609 452 078 > > TORCH GmbH - A Graylog company > Steckelhörn 11 > 20457 Hamburg > Germany > > Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 > Geschäftsführer: Lennart Koopmann (CEO) > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/df7fd668-041e-4a94-a4f2-c1a0205c103a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Unable to get the graylog web interface login page.
Hi Jochen, I have not made any change to the code, simply compiled it using typesafe activator and the made a jar from the compiled code. I also compared my jar(graylog-web-interface.graylog-web-interface-1.1.6) with the official jar(graylog-web-interface.graylog-web-interface-1.1.6) and my jar was missing the routes file so I copied the official routes file into my custom compiled code and the bundled it to make a jar. Can this be the reason for the issue.??? Can you please suggest me what could be the possible reason for this issue, so that I can go ahead to solve this. As off now I have no idea where to look for the cause of this error On Monday, 14 September 2015 18:09:34 UTC+5:30, Jochen Schalanda wrote: > > Hi Anant, > > I can't reproduce your problem with the official Graylog 1.1.6 web > interface, so I guess it's because of some changes you've made to the code > of your custom compiled version. > > > Cheers, > Jochen > > On Monday, 14 September 2015 14:10:29 UTC+2, Anant Sawant wrote: >> >> Hi, >> >> I am running Graylog 1.1.6 server component and Graylog web component >> 1.1.6 which I have compiled. >> I am running this on ubuntu 14.04.1. For this I have installed >> Elasticsearch 1.7.1, mongodb version v3.0.6 and Java 1.8.0_60. The Graylog >> 1.1.6 server component, Graylog web component 1.1.6, Mongod and >> Elasticsearch are on the same machine. For configuration I have referred >> http://docs.graylog.org/en/1.2/pages/installation/manual_setup.html#configuring-the-web-interface. >> >> As per this document Graylog 1.1.6 server component and Graylog web >> component 1.1.6 both are running well/as expected as I can see the expected >> result on the console, also the logs shows no errors. Following are the >> logs that I got on the console for server and web component respectively. >> >> ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service elasticsearch status >> * elasticsearch is running >> ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service mongod status >> mongod start/running, process 758 >> ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo java -jar graylog.jar server >> 2015-09-14 15:29:32,036 INFO : org.graylog2.bootstrap.CmdLineTool - >> Loaded plugins: [Anonymous Usage Statistics 1.1.1 >> [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] >> 2015-09-14 15:29:32,325 INFO : org.graylog2.bootstrap.CmdLineTool - >> Running with JVM arguments: >> 2015-09-14 15:29:35,643 INFO : >> org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. >> Falling back to JMX implementations. >> 2015-09-14 15:29:43,437 INFO : >> org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled. >> 2015-09-14 15:29:45,351 INFO : kafka.log.LogManager - Found clean >> shutdown file. Skipping recovery for all logs in data directory >> '/opt/graylog-server-1.1.6/data/journal' >> 2015-09-14 15:29:45,357 INFO : kafka.log.LogManager - Loading log >> 'messagejournal-0' >> 2015-09-14 15:29:45,760 INFO : org.graylog2.shared.journal.KafkaJournal - >> Initialized Kafka based journal at data/journal >> 2015-09-14 15:29:45,869 INFO : >> org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl >> with ring size <65536> and wait strategy , running 2 >> parallel message handlers. >> 2015-09-14 15:29:47,182 INFO : org.graylog2.plugin.system.NodeId - Node >> ID: 996299ed-68ea-4a64-a1e6-74f6cb5cefc9 >> 2015-09-14 15:29:48,193 INFO : org.elasticsearch.node - [graylog2-server] >> version[1.6.2], pid[1629], build[6220391/2015-07-29T09:24:47Z] >> 2015-09-14 15:29:48,194 INFO : org.elasticsearch.node - [graylog2-server] >> initializing ... >> 2015-09-14 15:29:48,668 INFO : org.elasticsearch.plugins - >> [graylog2-server] loaded [graylog2-monitor], sites [] >> 2015-09-14 15:29:57,310 INFO : org.elasticsearch.node - [graylog2-server] >> initialized >> 2015-09-14 15:29:57,331 INFO : org.graylog2.shared.buffers.ProcessBuffer >> - Initialized ProcessBuffer with ring size <65536> and wait strategy >> . >> 2015-09-14 15:30:04,824 INFO : >> org.graylog2.bindings.providers.RulesEngineProvider - No static rules file >> loaded. >> 2015-09-14 15:30:05,033 INFO : org.graylog2.buffers.OutputBuffer - >> Initialized OutputBuffer with ring size <65536> and wait strategy >> . >> 2015-09-14 15:30:06,414 INFO : >> org.hibernate.validator.internal.util.Version - HV01: Hibernate >> Validator 5.1.3.Final >> 2015-09-14 15:30:08,527 INFO : org.graylog2.bootstrap.ServerBootstrap - >> Graylog server 1.1.6 (${git.commit.id.abbrev}) starting up. (JRE: Oracle >> Corporation 1.8.0_60 on Linux 3.13.0-32-generic) >> 2015-09-14 15:30:08,745 INFO : >> org.graylog2.shared.initializers.PeriodicalsService - Starting 21 >> periodicals ... >> 2015-09-14 15:30:08,752 INFO : org.graylog2.periodical.Periodicals - >> Starting [org.graylog2.periodical.ThroughputCounterManagerThread] >> periodical in [0s], polling every [1s]. >> 2015-09-14 15:30:08,762 INFO : org.graylog2.periodical.P
[graylog2] Re: Unable to get the graylog web interface login page.
Hi Anant, I can't reproduce your problem with the official Graylog 1.1.6 web interface, so I guess it's because of some changes you've made to the code of your custom compiled version. Cheers, Jochen On Monday, 14 September 2015 14:10:29 UTC+2, Anant Sawant wrote: > > Hi, > > I am running Graylog 1.1.6 server component and Graylog web component > 1.1.6 which I have compiled. > I am running this on ubuntu 14.04.1. For this I have installed > Elasticsearch 1.7.1, mongodb version v3.0.6 and Java 1.8.0_60. The Graylog > 1.1.6 server component, Graylog web component 1.1.6, Mongod and > Elasticsearch are on the same machine. For configuration I have referred > http://docs.graylog.org/en/1.2/pages/installation/manual_setup.html#configuring-the-web-interface. > > As per this document Graylog 1.1.6 server component and Graylog web > component 1.1.6 both are running well/as expected as I can see the expected > result on the console, also the logs shows no errors. Following are the > logs that I got on the console for server and web component respectively. > > ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service elasticsearch status > * elasticsearch is running > ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service mongod status > mongod start/running, process 758 > ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo java -jar graylog.jar server > 2015-09-14 15:29:32,036 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded > plugins: [Anonymous Usage Statistics 1.1.1 > [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] > 2015-09-14 15:29:32,325 INFO : org.graylog2.bootstrap.CmdLineTool - > Running with JVM arguments: > 2015-09-14 15:29:35,643 INFO : > org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. > Falling back to JMX implementations. > 2015-09-14 15:29:43,437 INFO : org.graylog2.shared.buffers.InputBufferImpl > - Message journal is enabled. > 2015-09-14 15:29:45,351 INFO : kafka.log.LogManager - Found clean shutdown > file. Skipping recovery for all logs in data directory > '/opt/graylog-server-1.1.6/data/journal' > 2015-09-14 15:29:45,357 INFO : kafka.log.LogManager - Loading log > 'messagejournal-0' > 2015-09-14 15:29:45,760 INFO : org.graylog2.shared.journal.KafkaJournal - > Initialized Kafka based journal at data/journal > 2015-09-14 15:29:45,869 INFO : org.graylog2.shared.buffers.InputBufferImpl > - Initialized InputBufferImpl with ring size <65536> and wait strategy > , running 2 parallel message handlers. > 2015-09-14 15:29:47,182 INFO : org.graylog2.plugin.system.NodeId - Node > ID: 996299ed-68ea-4a64-a1e6-74f6cb5cefc9 > 2015-09-14 15:29:48,193 INFO : org.elasticsearch.node - [graylog2-server] > version[1.6.2], pid[1629], build[6220391/2015-07-29T09:24:47Z] > 2015-09-14 15:29:48,194 INFO : org.elasticsearch.node - [graylog2-server] > initializing ... > 2015-09-14 15:29:48,668 INFO : org.elasticsearch.plugins - > [graylog2-server] loaded [graylog2-monitor], sites [] > 2015-09-14 15:29:57,310 INFO : org.elasticsearch.node - [graylog2-server] > initialized > 2015-09-14 15:29:57,331 INFO : org.graylog2.shared.buffers.ProcessBuffer - > Initialized ProcessBuffer with ring size <65536> and wait strategy > . > 2015-09-14 15:30:04,824 INFO : > org.graylog2.bindings.providers.RulesEngineProvider - No static rules file > loaded. > 2015-09-14 15:30:05,033 INFO : org.graylog2.buffers.OutputBuffer - > Initialized OutputBuffer with ring size <65536> and wait strategy > . > 2015-09-14 15:30:06,414 INFO : > org.hibernate.validator.internal.util.Version - HV01: Hibernate > Validator 5.1.3.Final > 2015-09-14 15:30:08,527 INFO : org.graylog2.bootstrap.ServerBootstrap - > Graylog server 1.1.6 (${git.commit.id.abbrev}) starting up. (JRE: Oracle > Corporation 1.8.0_60 on Linux 3.13.0-32-generic) > 2015-09-14 15:30:08,745 INFO : > org.graylog2.shared.initializers.PeriodicalsService - Starting 21 > periodicals ... > 2015-09-14 15:30:08,752 INFO : org.graylog2.periodical.Periodicals - > Starting [org.graylog2.periodical.ThroughputCounterManagerThread] > periodical in [0s], polling every [1s]. > 2015-09-14 15:30:08,762 INFO : org.graylog2.periodical.Periodicals - > Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], > polling every [1s]. > 2015-09-14 15:30:08,768 INFO : org.elasticsearch.node - [graylog2-server] > starting ... > 2015-09-14 15:30:08,790 INFO : org.graylog2.periodical.Periodicals - > Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], > polling every [60s]. > 2015-09-14 15:30:08,808 INFO : org.graylog2.periodical.Periodicals - > Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] > periodical in [0s], polling every [1s]. > 2015-09-14 15:30:08,812 INFO : org.graylog2.periodical.Periodicals - > Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in > [0s], polling every [20s]. > 2015-09-14 15:30:08,813 INFO : org.graylog2.periodical.Periodicals - > St
[graylog2] Unable to get the graylog web interface login page.
Hi, I am running Graylog 1.1.6 server component and Graylog web component 1.1.6 which I have compiled. I am running this on ubuntu 14.04.1. For this I have installed Elasticsearch 1.7.1, mongodb version v3.0.6 and Java 1.8.0_60. The Graylog 1.1.6 server component, Graylog web component 1.1.6, Mongod and Elasticsearch are on the same machine. For configuration I have referred http://docs.graylog.org/en/1.2/pages/installation/manual_setup.html#configuring-the-web-interface. As per this document Graylog 1.1.6 server component and Graylog web component 1.1.6 both are running well/as expected as I can see the expected result on the console, also the logs shows no errors. Following are the logs that I got on the console for server and web component respectively. ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service elasticsearch status * elasticsearch is running ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo service mongod status mongod start/running, process 758 ubuntu@ubuntu:/opt/graylog-server-1.1.6$ sudo java -jar graylog.jar server 2015-09-14 15:29:32,036 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugins: [Anonymous Usage Statistics 1.1.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] 2015-09-14 15:29:32,325 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: 2015-09-14 15:29:35,643 INFO : org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR. Falling back to JMX implementations. 2015-09-14 15:29:43,437 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled. 2015-09-14 15:29:45,351 INFO : kafka.log.LogManager - Found clean shutdown file. Skipping recovery for all logs in data directory '/opt/graylog-server-1.1.6/data/journal' 2015-09-14 15:29:45,357 INFO : kafka.log.LogManager - Loading log 'messagejournal-0' 2015-09-14 15:29:45,760 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at data/journal 2015-09-14 15:29:45,869 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers. 2015-09-14 15:29:47,182 INFO : org.graylog2.plugin.system.NodeId - Node ID: 996299ed-68ea-4a64-a1e6-74f6cb5cefc9 2015-09-14 15:29:48,193 INFO : org.elasticsearch.node - [graylog2-server] version[1.6.2], pid[1629], build[6220391/2015-07-29T09:24:47Z] 2015-09-14 15:29:48,194 INFO : org.elasticsearch.node - [graylog2-server] initializing ... 2015-09-14 15:29:48,668 INFO : org.elasticsearch.plugins - [graylog2-server] loaded [graylog2-monitor], sites [] 2015-09-14 15:29:57,310 INFO : org.elasticsearch.node - [graylog2-server] initialized 2015-09-14 15:29:57,331 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy . 2015-09-14 15:30:04,824 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded. 2015-09-14 15:30:05,033 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy . 2015-09-14 15:30:06,414 INFO : org.hibernate.validator.internal.util.Version - HV01: Hibernate Validator 5.1.3.Final 2015-09-14 15:30:08,527 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 1.1.6 (${git.commit.id.abbrev}) starting up. (JRE: Oracle Corporation 1.8.0_60 on Linux 3.13.0-32-generic) 2015-09-14 15:30:08,745 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 21 periodicals ... 2015-09-14 15:30:08,752 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCounterManagerThread] periodical in [0s], polling every [1s]. 2015-09-14 15:30:08,762 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. 2015-09-14 15:30:08,768 INFO : org.elasticsearch.node - [graylog2-server] starting ... 2015-09-14 15:30:08,790 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s]. 2015-09-14 15:30:08,808 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2015-09-14 15:30:08,812 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s]. 2015-09-14 15:30:08,813 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever. 2015-09-14 15:30:08,848 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.DeadLetterThread] periodical, running forever. 2015-09-14 15:30:08,857 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2015-09-14 15:30:08,873 INFO : org.g
[graylog2] Re: Search Issue ...
Hi again ! Forgot to escape the ':' character source_file:"C\:\\Program Files\\IBM\WebSphere\\AppServer8\\profiles\\AppSrv01\\logs\\MyServer\\SystemOut.log" but nothing wil be found. Do i overlook something? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/df6e02ef-c71f-411e-b840-8ed2ac2e4eb5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Search Issue ...
Hi Jochen ! Thank you for the info but i tried it already with this search string source_file:"C:\\Program Files\\IBM\WebSphere\\AppServer8\\profiles\\AppSrv01\\logs\\MyServer\\SystemOut.log" but nothing wil be found. This is the elastic search query: { "from": 0, "size": 100, "query": { "query_string": { "query": "source_file:\"C:Program FilesIBM\\WebSphereAppServer8profilesAppSrv01logsMyServerSystemOut.log\"", "allow_leading_wildcard": false } }, "post_filter": { "bool": { "must": { "range": { "timestamp": { "from": "2015-09-14 00:58:24.644", "to": "2015-09-14 08:58:24.644", "include_lower": true, "include_upper": true } } } } }, "sort": [ { "timestamp": { "order": "desc" } } ] } -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/dfcfc144-782a-4c0b-b44a-99221d5b3841%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Search Issue ...
Hi Claus, certain characters have to be escaped in the Lucene query syntax (which is being used by Graylog and Elasticsearch), see http://docs.graylog.org/en/1.1/pages/queries.html#escaping for details. Cheers, Jochen On Tuesday, 8 September 2015 10:31:14 UTC+2, Claus Koell wrote: > > Hi ! > > We are using graylog 1.1.6 and we have troubles with some search strings. > We are using a collector to reading files > from a windows system. We can see a field named 'source_file' in these > messages > > Sample Value: C:\Program > Files\IBM\WebSphere\AppServer8\profiles\AppSrv01\logs\MyServer\SystemOut.log > > If we try to search for all logs from a specific source_file it does not > work. > > This is the elastic search query: > > { > "from": 0, > "size": 100, > "query": { > "query_string": { > "query": "source_file:\"C:\\Program > Files\\IBM\\WebSphere\\AppServer8\\profiles\\AppSrv01\\logs\\MyServer\\SystemOut.log\"", > "allow_leading_wildcard": false > } > }, > "post_filter": { > "bool": { > "must": { > "range": { > "timestamp": { > "from": "2015-09-08 00:28:10.547", > "to": "2015-09-08 08:28:10.547", > "include_lower": true, > "include_upper": true > } > } > } > } > }, > "sort": [ > { > "timestamp": { > "order": "desc" > } > } > ] > } > > Maybe the backslashes make the trouble ? > > thanks for help ! > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5437d888-0c91-4726-8101-3f3f3ef0feaf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: "include"-statement in Graylog Collector config
Hi Fabian, as a matter of fact, including other configuration files using an "include" statement is supported in the configuration file used by the Graylog Collector, see https://github.com/typesafehub/config/blob/v1.2.1/HOCON.md#includes for details. Centralized management for the Graylog Collector instances in a cluster is on our roadmap (and was one of the reasons to start this project in the first place), but probably won't happen in Graylog 1.x. Cheers, Jochen On Monday, 14 September 2015 10:04:10 UTC+2, Fabian Danner wrote: > > Is there anything like an "include" statement to include different config > files for Graylog Collector. > I want to centralize the different configuration files of all windows > servers on a network share, so that multiple servers could be configured in > on file only? > Is something like this possible? > How about a centralized configuration solution forGraylog Collector? > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/12a28895-7b24-4a59-84ac-89b5a6fafb8b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Search Issue ...
Can somebody have a look at this please ? thanks ! > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/e09d66cc-7a40-4f50-8854-f442365dab7c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] "include"-statement in Graylog Collector config
Is there anything like an "include" statement to include different config files for Graylog Collector. I want to centralize the different configuration files of all windows servers on a network share, so that multiple servers could be configured in on file only? Is something like this possible? How about a centralized configuration solution forGraylog Collector? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7e20e807-fd9f-422a-be6a-5b9d8e8b4af7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.