[graylog2] Optimize graylog for high performance

Hello, I want to optimize my current Graylog for high performance, with the input rate about 4000 - 5000 m/s. I has changed some configuration but the server still load too high. Below is my updated configuration processbuffer_processors = 30 outputbuffer_processors = 8 When input about 4-5k

[graylog2] Re: Graylog Time on Turkey

joda-time-2.9.5.jar version has just released. We copied this jar file to our graylog2 server and restarted the services. But still graylog server time is wrong. # cp /home/aziz/joda-time-2.9.5.jar /usr/share/elasticsearch/lib/ # rm -rf /usr/share/elasticsearch/lib/joda-time-2.9.4.jar # service

[graylog2] Re: Unable to get Geo-Location to work

What stumps me more, is I have just added another host, in the Eventlog it has a Destination Address IP, this is being updated and a _geolocation field added. Any idea why it would work on *some* fields and not others? On Friday, November 4, 2016 at 8:55:03 AM UTC+13, Werner van der Merwe

[graylog2] Re: Show count/sum of received messages (of a certain query) in chart

Do you know if it is planned to implement such a function? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this

[graylog2] Re: problems connecting to the Graylog server

i'm seeing the same problem. On Thursday, November 3, 2016 at 12:58:27 PM UTC-5, Andrew Morgan wrote: > > I had set it because I am behind a http proxy, is it used for the proxy to > communicate with graylog box? I set it to the internal ip address which the > proxy will use to get to it > > >

[graylog2] Re: Error: There was no master Graylog server node detected in the cluster.

Usually when I see this set of messages and both Graylog servers are in fact up, the time is wrong on one or both of the servers. Make sure that your ntpd.conf on both servers is pointing at the same ntp server, and to see whether it's actually sync'ing, stop ntpd and run ntpdate against the

[graylog2] Unable to get Geo-Location to work

Slightly stumped, followed the doco and some threads online of others struggling. Here's what I've done so far: Download the MaxMind binary from https://dev.maxmind.com/geoip/geoip2/geolite2/ In the "Message Processors Configuration", ensure GeoIP resolver is the last step. Under Plugins,

[graylog2] Re: capture log file legacy (old) Windows

Leandro, You can send plain text files to graylog using the Graylog Collector Sidecar (http://docs.graylog.org/en/2.1/pages/collector_sidecar.html). You would have to deploy the graylog collector sidecar agent to the server, configure it w/ a tag to identify it to graylog, and then in the

[graylog2] capture log file legacy (old) Windows

I would like to capture log file legacy (old) in the windows directory and display them in Graylog. How could I do? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an

[graylog2] Re: Show count/sum of received messages (of a certain query) in chart

Ah, sorry, I misunderstood you. Yes, graylog doesn't appear to have that type of statistical info in it yet. On Thursday, November 3, 2016 at 2:59:05 AM UTC-4, Hans Maulwurf wrote: > > thanks for your answer, but this is not waht i need. I try to describe it: > it's 10:00am and graylog

[graylog2] Re: 'API Browser' button does not work with no DNS and https enabled in cluster

Thanks, After about 3 min running it showed APIs though with exception - is it OK? 2016-11-03T18:03:21.358Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource java.lang.IllegalStateException: zip file closed at java.util.zip.ZipFile.ensureOpen(ZipFile.java:669)

[graylog2] Re: problems connecting to the Graylog server

I had set it because I am behind a http proxy, is it used for the proxy to communicate with graylog box? I set it to the internal ip address which the proxy will use to get to it rest_listen_uri = http://10.1.80.18:9000/api/ rest_transport_uri = http://10.1.80.18:12900/ web_listen_uri =

[graylog2] Re: Graylog Time on Turkey

Hi Jochen, Is there any workaround about this situation that you know before any updates? Can we change joda-time by ourselves? Thanks, Aziz On Thursday, November 3, 2016 at 8:18:14 PM UTC+3, Jochen Schalanda wrote: > > Hi Aziz, > > Graylog is using Joda-Time internally, which is using tzdata

[graylog2] Re: Graylog Time on Turkey

Thank you Sébastien :) On Thursday, November 3, 2016 at 8:11:08 PM UTC+3, Sébastien cieloch wrote: > > Hum... No more ideas from my part, I am not using exactly the same > environnement... > Just be sure that your VM have the good timzome configured and restart > your services when you make any

[graylog2] Re: problems connecting to the Graylog server

Well I did set because my server is running behind a HTTP proxy, I would think it is used for the httpproxy box to send communication tot eh graylog instance?. I had it with the internal ip address of the graylog server box and external and commented it out but nothing works.I commented it out

Re: [graylog2] Re: new graylog instance having issues

Here is the output from the dev console: Download the React DevTools for a better development experience: https://fb.me/react-devtools http://10.12.183.70:12900/system/sessions Failed to load resource: net::ERR_EMPTY_RESPONSE http://10.12.183.70:12900/system/cluster/node Failed to load resource:

[graylog2] Re: Graylog Time on Turkey

Hi Aziz, Graylog is using Joda-Time internally, which is using tzdata 2016d (in case of Joda-Time 2.9.4 used in Graylog 2.1.x, http://www.joda.org/joda-time/upgradeto294.html). The change for Turkey's timezone has been introduced in tzdata 2016g

[graylog2] Re: Graylog Time on Turkey

Hum... No more ideas from my part, I am not using exactly the same environnement... Just be sure that your VM have the good timzome configured and restart your services when you make any change on graylog, mongoDB or Elasticsearch. But i'm pretty sure you know that :) Good luck Le jeudi 3

[graylog2] Re: Graylog Time on Turkey

Hi Jochen, # date -R Thu, 03 Nov 2016 20:04:51 +0300 # grep ZONE /etc/sysconfig/clock ZONE="Europe/Istanbul" On Thursday, November 3, 2016 at 7:43:56 PM UTC+3, Jochen Schalanda wrote: > > Hi Aziz, > > which timezone have you configured on the virtual machine? > > What's the output of "*date

Re: [graylog2] Re: new graylog instance having issues

I installed graylog using rpm repos using the following sites as reference: http://www.tutorialspoint.com/articles/how-to-install-graylog-1-3graylog2-on-centos-7rhel-7 https://techknight.eu/2015/11/12/setup-configure-graylog2-centos-7/

[graylog2] Re: Graylog Time on Turkey

Hi Aziz, which timezone have you configured on the virtual machine? What's the output of "*date -R*" on that virtual machine? Cheers, Jochen On Thursday, 3 November 2016 17:39:38 UTC+1, Aziz Öğütlü wrote: > > Thank you for your answer Sébastien. > > We know where the graylog conf file is

[graylog2] Re: problems connecting to the Graylog server

Hi Andrew, On Thursday, 3 November 2016 15:24:11 UTC+1, Andrew Morgan wrote: > > Thank you for your response, honestly I tried a many different ways based > on research and I read that it should be the public ip of your instance, so > I set it to that. > Okay, but what do you think the setting

[graylog2] Re: Graylog Time on Turkey

Thank you for your answer Sébastien. We know where the graylog conf file is (/etc/graylog/server/server.conf). And when we search "graylog2 timezone configuration" on-line, we only find root_timezone = UTC parameter. We tried time settings with changing this parameter to Etc/GMT-3, but Graylog

[graylog2] Re: new graylog instance having issues

Hi Andrew, how exactly did you install and configure Graylog? What messages are in the logs of your Graylog node(s)? What messages are in the developer console (JavaScript console) of your web browser? Cheers, Jochen On Thursday, 3 November 2016 15:51:14 UTC+1, Andrew Meyer wrote: > > I just

[graylog2] Re: Blank Page when i want to reach the Web Interface

Hi, On Thursday, 3 November 2016 15:50:11 UTC+1, KhaM wrote: > > Indeed, I wanted to run the web browser and Graylog on the same machine > for the test, but apparently that does not work. > Running in a virtual machine is not running on the same machine. That's not how networking works!

[graylog2] Re: Graylog Time on Turkey

Ok got it ! And do you know where is you equivalent for /opt/graylog/conf ? Have you tried to find the configuration file and check what's on ? I installed Graylog from the OVA for my part... If you're running a VM, disable the VMware Time Sync if it's enabled( and if you're running this

[graylog2] Re: Alert conditions on a multi-node setup

Great! Thanks Jochen! On Thursday, November 3, 2016 at 4:00:13 PM UTC, Jochen Schalanda wrote: > > Hi Rui, > > On Thursday, 3 November 2016 16:45:00 UTC+1, Rui Goncalves wrote: >> >> * It's a periodical query to elasticsearch? >> > > Yes, the Graylog master node regularly runs Elasticsearch

[graylog2] Re: Alert conditions on a multi-node setup

Hi Rui, On Thursday, 3 November 2016 16:45:00 UTC+1, Rui Goncalves wrote: > > * It's a periodical query to elasticsearch? > Yes, the Graylog master node regularly runs Elasticsearch queries, also see the alert_check_interval

[graylog2] Re: threw an exception during a multi-realm authentication attempt: java.lang.IllegalArgumentException: Odd number of characters.

Hi Dylan, recheck your root_password_sha2 setting. The password SHA-256 hash seems to be invalid. It has to be exactly 64 characters long (the hexadecimal representation of a 256-bit value). Cheers, Jochen On Thursday, 3 November 2016 16:32:18 UTC+1, Dylan Humphreys wrote: > > Hi Everyone, >

[graylog2] Alert conditions on a multi-node setup

Hi. How does graylog evaluates alert conditions. * It's a periodical query to elasticsearch? * Or alert conditions are evaluated on the fly on the processing phase? If so, how does it work in a multi-node setup? Thanks, Rui -- You received this message because you are subscribed to the

[graylog2] Re: Duplicate messages being seen from Winlogbeat input

Looks like I spoke too soon as the duplicates are back. I did notice that the registry file gets recreated back on the root of the C drive, so I guess the graylog-collector executable is doing that via whatever command it uses to start Winlogbeat. That's still an issue as the process doesn't

[graylog2] threw an exception during a multi-realm authentication attempt: java.lang.IllegalArgumentException: Odd number of characters.

Hi Everyone, graylog 2.1.1-2 on debian 8, elasticsearch 2.4.1 and mongo 2.6.12 manual install. I have set password_secret = qBbUBsZDwL9... and root_password_sha2 = 240be518fa... Using the commands mentioned in server.conf, however whenever I try and login using the web interface I get this

[graylog2] Re: Duplicate messages being seen from Winlogbeat input

I carried out some additional testing by downloading the same version of the Winlogbeat application (1.2.3) from the Elastic website and running it via the CLI. I noticed that the duplicates were gone, so I tried again running the Graylog version manually and again only saw single events.

[graylog2] Re: Graylog Time on Turkey

Thank you for your answer But we installed graylog2 from rpm tarball, and there is no graylog-ctl. This page says, there is no graylog-ctl in tarball -> http://docs.graylog.org/en/2.0/pages/configuration/graylog_ctl.html#graylog-ctl We install graylog2 server like below on centos server. #

[graylog2] Check activity on a define time range only

Hello everyone, I actually have a simple need, but it seems to be more difficult to get in place... That's why I need you :D I'm sending my Windows serveur logs via NXLog (GELF Format) and my linux based OS logs via Rsylog. The goal is, in few words, to check if there is log-on and log-off

[graylog2] Re: new graylog instance having issues

Error messageBad requestOriginal RequestGET http://10.12.183.70:12900/system/sessionsStatus codeundefinedFull error messageError: Request has been terminated Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc. On

[graylog2] Re: new graylog instance having issues

Running Graylog2 2.0.3. Installed from RPM. On Thursday, November 3, 2016 at 9:51:14 AM UTC-5, Andrew Meyer wrote: > > I just spun up an additional graylog instance running on RHEL 6.6. > However I have seen this on RHEL/Centos 7 as well but sometimes the web > interface ceases to work. I

[graylog2] Re: Graylog Time on Turkey

Hello Mister Aziz ! Try to use the command according to this documentation http://www.joda.org/joda-time/timezones.html : sudo graylog-ctl set-timezone Etc/GMT-3 Don't forget to reconfigure the graylog suite : sudo graylog-ctl reconfigure And it should work ! Le jeudi 3 novembre 2016

[graylog2] new graylog instance having issues

I just spun up an additional graylog instance running on RHEL 6.6. However I have seen this on RHEL/Centos 7 as well but sometimes the web interface ceases to work. I can't log into it and it throw a message stating its not running because something failed. The logs don't say much. -- You

[graylog2] Re: Blank Page when i want to reach the Web Interface

Hi Jochen, Indeed, I wanted to run the web browser and Graylog on the same machine for the test, but apparently that does not work. I would know if i use my physical machine as client and my virtual machine as server, does this method work? Thanks Jochen. -- You received this message

[graylog2] Graylog Time on Turkey

Hi, Turkey switched from EET/EEST (+02/+03) to permanent +03. Our Graylog2 server (v .0.3 (f07c170)) time settings are wrong. We tried to use Tzupdater tool (http://www.oracle.com/technetwork/java/javase/documentation/tzupdater-readme-136440.html), but the graylog server time is still wrong.

[graylog2] Re: problems connecting to the Graylog server

Thank you for your response, honestly I tried a many different ways based on research and I read that it should be the public ip of your instance, so I set it to that.Can you please tell me what it should be . Thank you On Thursday, November 3, 2016 at 4:49:57 AM UTC-4, Jochen Schalanda

[graylog2] Re: Blank Page when i want to reach the Web Interface

Hi, you're binding the Graylog web interface and the Graylog REST API to the local loopback interface (127.0.0.1) of your virtual machine inside VirtualBox. Unless you're running your web browser and Graylog on the same machine, that won't work. FWIW, you could also use our official virtual

[graylog2] Blank Page when i want to reach the Web Interface

Hi everyone ! I try to set up graylog for my project school and i meet a problem during the connection on Graylog Web interface. ( I tape this URL: 127.0.0.1:9000) the page is blank. When I look the log, i do not notice error. 2016-11-03T12:28:24.736+01:00 INFO [Server] SIGNAL received.

[graylog2] Re: Graylog Time is Wrong

Thanks -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit

[graylog2] Re: Very low message throughput after upgrading from GL 1.3.4/ES 1.7.1 to GL2.1.1/ES 2.3.5 + Error messages

Hi bob, appreciate your answer! The VM specs don´t change while upgrading and judging from the CPU utilization I can´t really see a difference between ES 2.3.5 and ES 1.7.1. My inputs come from various sources, logstash, nxlog, raw udp, etc. Inputs don´t change when upgrading though and I get

[graylog2] Re: Graylog Time is Wrong

Hi, you can configure the timezone of the admin user with the root_timezone setting. Cheers, Jochen On Thursday, 3 November 2016 13:12:13 UTC+1, Schwään wrote: > > User *admin*:2016-11-03 12:11:25 +00:00Your

[graylog2] Re: How to prevent insert duplicated data?

Thanks for your help, Jochen. as I'm new to ElasticSearch and Graylog, deleting data without understanding is not safe, I guess. but... seems like there's no solution for it... I'll try it. :) Thanks. On Thursday, November 3, 2016 at 8:22:26 PM UTC+9, Jochen Schalanda wrote: > > Hi Eugene, > >

[graylog2] Re: Very low message throughput after upgrading from GL 1.3.4/ES 1.7.1 to GL2.1.1/ES 2.3.5 + Error messages

With 4 cpu's on your ES server your bulk threads will max out at 4, with a 1000 message output batch size that would set your max to essentially 4k msg/s. I would increase the output batch size and add cpu on the ES vm. Also you didn't mention what your input config is in graylog, are you sure

[graylog2] Graylog Time is Wrong

User *admin*:2016-11-03 12:11:25 +00:00Your web browser:2016-11-03 13:11:25 +01:00Graylog server:2016-11-03 13:11:25 +01:00 Hello my logs come with false date because user admi is 1 hour wrong what can i do? -- You received this message because you are subscribed to the Google Groups "Graylog

[graylog2] Re: Backup indices under /var/lib/elasticsearch/graylog2/nodes/0/indices

Hi Roberto, the only way to get consistent backups of your indices (without shutting down the whole ES cluster) is using the ES snapshot/restore API: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-snapshots.html Cheers, Jochen On Monday, 31 October 2016 14:37:29 UTC+1,

[graylog2] Re: How to prevent insert duplicated data?

Hi Eugene, once you've identified the duplicate documents in Elasticsearch, you can remove these using the Elasticsearch Delete API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-delete.html) or the Delete By Query plugin

Re: [graylog2] Re: Can't Show received messages from NetFlow input

Hi, On Thursday, 3 November 2016 11:31:13 UTC+1, Shamim Reza Sohag wrote: > > I have faced similar kind of problem while i used NetFlow V9. > But when i used NetFlow V5 everything was OK and the flow was showing > up. > The Graylog Netflow plugin currently doesn't support NetFlow v9, see

[graylog2] Re: How to prevent insert duplicated data?

Hello, Jochen. I've checked that link but unfortunately, I'm new to ElasticSearch. and It seems like that link only shows 'how to check duplicate data'. does it mean that I insert data first and check duplicate, and than delete duplicated data? I'm confusing :( I'm really sorry but would you

[graylog2] Re: Very low message throughput after upgrading from GL 1.3.4/ES 1.7.1 to GL2.1.1/ES 2.3.5 + Error messages

I´d like to add 2 more Graphs we have from our ES host (upgrade took place on wednesday and was reverted thursday): http://imgur.com/euZOPMq -> TCP MIB, where you can clearly see a lot less established session with the upgraded setup and http://imgur.com/qPnR69Y -> Traffic took a serious drop

Re: [graylog2] Re: Can't Show received messages from NetFlow input

I have faced similar kind of problem while i used NetFlow V9. But when i used NetFlow V5 everything was OK and the flow was showing up. On Monday, August 15, 2016 at 3:25:09 PM UTC+6, Arief Hydayat wrote: > > Hi everyone, > > anyone can help me on these? I tried to put the time range but

[graylog2] Re: How to prevent insert duplicated data?

Hi Eugene, Graylog itself doesn't filter out "duplicate" data. You could check your Elasticsearch indices using the approach laid out in https://qbox.io/blog/minimizing-document-duplication-in-elasticsearch after ingesting the missing data. Cheers, Jochen On Thursday, 3 November 2016

Re: [graylog2] we are not able to install agent due to conflict of previous version

Hi, you don't seem to have uninstalled the old package properly, at least that's what the error message is telling you. Additionally, there might be a conflict in your machine architecture (x86_64 in the old package, i386 in the new package) which you should sort out. Cheers, Jochen On

[graylog2] Re: Grayling rpm build wiki

Hi Rayees, you can find the Graylog package specifications at https://github.com/Graylog2/fpm-recipes Cheers, Jochen On Thursday, 3 November 2016 04:36:02 UTC+1, Rayees Namathponnan wrote: > > Hi All, > > Someone please provide the wiki to build rpm for graylog. > > Regards, > Rayees > --

[graylog2] Re: 'API Browser' button does not work with no DNS and https enabled in cluster

Hi Evgueni, Graylog tries to be smart about the IP address (or network interface) being used if rest_listen_uri is 0.0.0.0. In your case, this seems to fail, but you can simply specify the correct address with the rest_transport_uri

[graylog2] Re: [Dashboard] How to create Top of city name widget using geolocation fields ?

Hi, that's currently not possible with the map widget plugin. Feel free to subscribe to https://github.com/Graylog2/graylog-plugin-map-widget/issues/10 to stay updated about the feature request. Cheers, Jochen On Wednesday, 2 November 2016 23:33:17 UTC+1, markham extra wrote: > > Hi, > i

[graylog2] Re: problems connecting to the Graylog server

Hi Andrew, please describe in your own words, what the rest_transport_uri is doing. Cheers, Jochen On Wednesday, 2 November 2016 22:41:08 UTC+1, Andrew Morgan wrote: > > Hello All, > > I am getting this error and cannot connect to graylog, I get no errors in > the logs: > > > rest_listen_uri

[graylog2] How to prevent insert duplicated data?

Hello all, I have a problem. please help :( I'm running 3 node of Elasticsearch. 1 master + data, and 2 data node. log shipper is fluentd. Unfortunately, today, I accidentally restarted graylog server. this caused lost of some data, which sent from another servers. so I must re-insert data,

[graylog2] Re: Error: There was no master Graylog server node detected in the cluster.

Hi, could you give me some advice, i do not have solution to this problem. Thanks very much. Dňa štvrtok, 20. októbra 2016 12:25:29 UTC+2 Ľubo napísal(-a): > > Hi all, > > We have two nodes in Graylog cluster 2.1.0. > > There is Error: There was no master Graylog server node detected in the >

[graylog2] Re: Show count/sum of received messages (of a certain query) in chart

thanks for your answer, but this is not waht i need. I try to describe it: it's 10:00am and graylog receives 1 message -> in the chart should be a dot at time 10:00am and at y-axis at 1 it's 11:00am and graylog receives 3 messages -> in the chart should be a dot at time 11:00am and at the y-axis

Re: [graylog2] we are not able to install agent due to conflict of previous version

Hi Team, already I uninstalled the older sidecar collector. But still I am facing the same issue. [root@dev2wlc-wls-1 tmp]# rpm -ivh collector-sidecar-0.1.0-1.i386.rpm Preparing...### [100%] file /usr/bin/filebeat from install of