[graylog2] Re: Nodes with too long GC pauses
Hi Nitzan, On Thursday, 16 February 2017 14:20:33 UTC+1, Nitzan Haimovich wrote: > > Where do I configure the threshold for this? > You can configure this with the gc_warning_threshold setting: https://github.com/Graylog2/graylog2-server/ blob/2.2.0/misc/graylog.conf#L527-L529 But be aware that you're merely covering up the problem and not solving anything with increasing this threshold. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f2d21cb2-19fe-4047-a3d3-8a4f0540%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Nodes with too long GC pauses
I reverted, no change. It doesn't affect the GC long pauses. Where do I configure the threshold for this? Thanks On Monday, February 6, 2017 at 11:44:26 AM UTC+2, Jochen Schalanda wrote: > > Hi Nitzan, > > you've configure a very high number of processbuffer_processors and > outputbuffer_processors in your Graylog configuration which usually > counterproductive. > > Please revert to the default values and only increase these values very > conservatively. > > Cheers, > Jochen > > On Sunday, 5 February 2017 07:55:25 UTC+1, Nitzan Haimovich wrote: >> >> Hi Jochen, >> >> Thanks for you help. I attached one file of the configuration - The >> configuration is the same in all 3 Graylog machines in our cluster (except >> is_master parameter and the IP parameter). >> I also attached one file of the logs, again - same logs are all around >> the cluster (except the master periodical duties). >> >> Our cluster: >> 3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem. >> With Disks of 2.5TB. >> >> Let me know if you need anything else! >> >> Nitzan >> >> On Thursday, February 2, 2017 at 6:38:57 PM UTC+2, Jochen Schalanda wrote: >>> >>> Hi Nitzan, >>> >>> please post the configuration and logs of all Graylog nodes and a >>> description of your hardware. >>> >>> Cheers, >>> Jochen >>> >>> On Thursday, 2 February 2017 17:18:12 UTC+1, Nitzan Haimovich wrote: Hi all, I'm getting this message (*Nodes with too long GC pauses*) on my Graylog cluster. I saw many people were posting about it but not a single thread with solutions for how to solve/fix/approach it. I would be glad for any help. My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory (heap size is configured to be : Xms - 1GB , Xmx - 8GB). If you need any more details please let me know. Thanks!! Nitzan >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d7483bff-11c0-4044-a16a-ebdf366452e8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Nodes with too long GC pauses
Hi Nitzan, you've configure a very high number of processbuffer_processors and outputbuffer_processors in your Graylog configuration which usually counterproductive. Please revert to the default values and only increase these values very conservatively. Cheers, Jochen On Sunday, 5 February 2017 07:55:25 UTC+1, Nitzan Haimovich wrote: > > Hi Jochen, > > Thanks for you help. I attached one file of the configuration - The > configuration is the same in all 3 Graylog machines in our cluster (except > is_master parameter and the IP parameter). > I also attached one file of the logs, again - same logs are all around the > cluster (except the master periodical duties). > > Our cluster: > 3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem. > With Disks of 2.5TB. > > Let me know if you need anything else! > > Nitzan > > On Thursday, February 2, 2017 at 6:38:57 PM UTC+2, Jochen Schalanda wrote: >> >> Hi Nitzan, >> >> please post the configuration and logs of all Graylog nodes and a >> description of your hardware. >> >> Cheers, >> Jochen >> >> On Thursday, 2 February 2017 17:18:12 UTC+1, Nitzan Haimovich wrote: >>> >>> Hi all, >>> >>> I'm getting this message (*Nodes with too long GC pauses*) on my >>> Graylog cluster. I saw many people were posting about it but not a single >>> thread with solutions for how to solve/fix/approach it. >>> I would be glad for any help. >>> >>> My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory >>> (heap size is configured to be : Xms - 1GB , Xmx - 8GB). >>> If you need any more details please let me know. >>> >>> Thanks!! >>> >>> Nitzan >>> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/276bc8b7-10be-4388-9a5b-c2ae2cef7f99%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Nodes with too long GC pauses
Hi Jochen, Thanks for you help. I attached one file of the configuration - The configuration is the same in all 3 Graylog machines in our cluster (except is_master parameter and the IP parameter). I also attached one file of the logs, again - same logs are all around the cluster (except the master periodical duties). Our cluster: 3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem. With Disks of 2.5TB. Let me know if you need anything else! Nitzan On Thursday, February 2, 2017 at 6:38:57 PM UTC+2, Jochen Schalanda wrote: > > Hi Nitzan, > > please post the configuration and logs of all Graylog nodes and a > description of your hardware. > > Cheers, > Jochen > > On Thursday, 2 February 2017 17:18:12 UTC+1, Nitzan Haimovich wrote: >> >> Hi all, >> >> I'm getting this message (*Nodes with too long GC pauses*) on my Graylog >> cluster. I saw many people were posting about it but not a single thread >> with solutions for how to solve/fix/approach it. >> I would be glad for any help. >> >> My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory >> (heap size is configured to be : Xms - 1GB , Xmx - 8GB). >> If you need any more details please let me know. >> >> Thanks!! >> >> Nitzan >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/942c1187-99bf-492a-b226-e91d89f99392%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. # GRAYLOG CONFIGURATION FILE # If you are running more than one instances of Graylog server you have to select one of these # instances as master. The master will perform some periodical tasks that non-masters won't perform. is_master = true # The auto-generated node ID will be stored in this file and read after restarts. It is a good idea # to use an absolute file path here if you are starting Graylog server from init scripts or similar. node_id_file = /etc/graylog/server/node-id # You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters. # Generate one by using for example: pwgen -N 1 -s 96 password_secret = $SECRET # You MUST specify a hash password for the root user (which you only need to initially set up the # system and in case you lose connectivity to your authentication backend) # This password cannot be changed using the API or via the web interface. If you need to change it, # modify it in this file. # Create one by using for example: echo -n yourpassword | shasum -a 256 # and put the resulting hash value into the following line root_password_sha2 = $PASS_SHA # The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones. # Default is UTC root_timezone = Asia/Jerusalem # Set plugin directory here (relative or absolute) plugin_dir = /usr/share/graylog-server/plugin # REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster. # When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors. rest_listen_uri = http://0.0.0.0:12900/ # REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri # is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used. # If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on # this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri) # You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting # the scheme, host name or URI. # This must not contain a wildcard address (0.0.0.0). rest_transport_uri = http://$IP:12900/ # Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly. # If these are disabled, modern browsers will not be able to retrieve resources from the server. # This is enabled by default. Uncomment the next line to disable it. rest_enable_cors = true # Enable HTTPS support for the REST API. This secures the communication with the REST API with # TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the # next line to enable it. #rest_enable_tls = true # The X.509 certificate chain file in PEM format to use for securing the REST API. #rest_tls_cert_file = /etc/graylog/server/domain.crt # The PKCS#8 private key file in PEM format to use for securing the REST API. #rest_tls_key_file = /etc/graylog/server/domain.key # Web interface listen URI. # Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement #
[graylog2] Re: Nodes with too long GC pauses
Hi Nitzan, please post the configuration and logs of all Graylog nodes and a description of your hardware. Cheers, Jochen On Thursday, 2 February 2017 17:18:12 UTC+1, Nitzan Haimovich wrote: > > Hi all, > > I'm getting this message (*Nodes with too long GC pauses*) on my Graylog > cluster. I saw many people were posting about it but not a single thread > with solutions for how to solve/fix/approach it. > I would be glad for any help. > > My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory > (heap size is configured to be : Xms - 1GB , Xmx - 8GB). > If you need any more details please let me know. > > Thanks!! > > Nitzan > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/67436c8a-659b-4dce-b24a-3fec658852ec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
