[graylog2] Graylog Stream Messages Disappearing

2016-09-21 Thread Kenneth Gyan 
I have setup host devices consisting of routers and switches and the logs 
are been collected in the input with no issues. I have setup a stream to 
capture syslog messages with level 0-4. Whenever this condition is met, and 
it captures the syslog message in the stream, after some time (about a 
couple of hours), the message/s in the stream just disappears and I am 
trying to figure out why this is happening. Could it be a stream retention 
time that needs to be set? Any assistance on this will be helpful. Thank 
you. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e24b85ff-0919-4bc4-8591-b5ee6f7b4237%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog configuration of filebeat and graylog collector sidecar

2016-09-21 Thread Werner van der Merwe 
Hi Kunal,

Kindly paste your configs, from what I can make out in the screenshot, your 
newline identifier is not set correctly. The %{host} is more than likely 
from incorrectly parsing the logs.

If you're willing to try NXLog, they have snipets for the config in their 
doco:
https://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928#processing_parsers_combined_log_format_example

What might help, NXLog (or beats) is the application that ships logs to 
Graylog. Sidecar is an extension of Graylog allowing you to centralise, 
manage and distribute profiles to enable easier collection of logs.
Thus, if you use sidecar, you don't have to worry about the config of NXLog 
(or beats), as that will be supplied by Sidecar.

Sidecar on the client side, you select snippets as elements in the 'tags' 
array. But adding a tag in that array assumes you've created a 
configuration in Graylog and assigned a tag with similar name to the config 
element.

On your client, you are calling the apache tag, which is correct. Just 
ensure you have a configuration matching that tag.
In Graylog, browse System -> Collectors, then click the "Manage Collectors" 
button.
This will present you with your different configurations, ensure one of 
them at least has the apache tag allocated to it.

If it does, you only need to worry about the configuration within that 
entry. From what I see I am expecting the parser is not correctly 
configured.


On Thursday, September 22, 2016 at 8:27:34 AM UTC+12, Kunal Patil wrote:
>
> Hello
>
> I have read the document previous issue has been resolved 
> i m getting data but some data  come under %{host} source filed
> I have configured apache logs as shown in documentation
>
> please refer attached screenshot 
>
>
>
>  
>
>
>
>
>
>
> *REGARDS:KUNAL VIKAS PATIL9860265594*
>
> On Thu, Sep 22, 2016 at 1:20 AM, Marius Sturm  > wrote:
>
>> Kunal,
>> please read the Sidecar documentation first. You have to create a 
>> configuration in the Graylog web interface and tag it with the same tag 
>> like you started the Sidecar instance. There is a step-by-step guide even 
>> with screenshots here: 
>> http://docs.graylog.org/en/2.1/pages/collector_sidecar.html#step-by-step-guide
>>
>> Cheers,
>> Marius
>>
>>
>> On 21 September 2016 at 20:52, Kunal Patil > > wrote:
>>
>>> hello 
>>> Thanks for the quick reply and solution as you guys suggested i m trying 
>>> to implement filebeat with help of documentation but i m getting below 
>>> error on web gui please check ad revert 
>>>  
>>> Sidecar
>>> Tags:apacheIP:
>>> CPU Idle:99.47%Load:0.06Volumes > 75%:
>>> --
>>>  *Status*: No configuration found for configured tags!
>>> Backends
>>>  *Filebeat*: Collector exits immediately, this should not happen! 
>>> Please check your collector configuration!
>>>
>>>
>>>
>>>
>>>
>>>
>>> *REGARDS:KUNAL VIKAS PATIL9860265594*
>>>
>>> On Wed, Sep 21, 2016 at 9:22 PM, Jochen Schalanda >> > wrote:
>>>
 Hi Kunal,

 nxlog and Filebeat are two different log shippers, each with its own 
 advantages and disadvantages, which are supported by the Graylog Collector 
 Sidecar.

 Both, nxlog and Filebeat, do support multiline messages:

- 

 https://www.elastic.co/guide/en/beats/filebeat/1.3/multiline-examples.html
- 
https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_multiline

 It's up to you which log shipper you want to use in the end and how you 
 configure it.

 Cheers,
 Jochen

 On Wednesday, 21 September 2016 17:43:44 UTC+2, Kunal Patil wrote:
>
> I m little confused here 
> After reading document
> In document u guys have given steps  for beat and nx log configuration 
> Can u brief more about that 
> My doubt is
> If i have beat to send data to graylog why i want nxlog 
> And if  nxlog is required then what is role of beat
>
 -- 
 You received this message because you are subscribed to a topic in the 
 Google Groups "Graylog Users" group.
 To unsubscribe from this topic, visit 
 https://groups.google.com/d/topic/graylog2/QVxdxyLWmww/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to 
 graylog2+u...@googlegroups.com .
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/graylog2/42f77a7e-b46f-4df6-9d2b-3366af1415da%40googlegroups.com
  
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Graylog Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to

[graylog2] Problem using sidecar with Win2003

2016-09-21 Thread Werner van der Merwe 
Due to some legacy software still in process of being migrated, we have a 
few Windows Server 2003 (i386) boxes about.

Installing sidecar goes without problem, but I am unable to start sidecar:

C:\Program Files\graylog\collector-sidecar>graylog-collector-sidecar.exe 
-service 
install
panic: Failed to find GetTickCount64 procedure in kernel32: The specified 
procedure could not be found.


goroutine 1 [running]:
panic(0x7a8e20, 0x10eb1640)
/opt/go/src/runtime/panic.go:481 +0x326
syscall.(*DLL).MustFindProc(0x10e68540, 0x831430, 0xe, 0x10e68570)
/opt/go/src/syscall/dll_windows.go:110 +0x6c
github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar
.init()
/go/src/github.com/Graylog2/collector-sidecar/vendor/github.com/
cloudfoundry/gosigar/sigar_windows.go:21 +0x153
github.com/Graylog2/collector-sidecar/common.init()
/go/src/github.com/Graylog2/collector-sidecar/common/sigar.go:162 +
0x93
github.com/Graylog2/collector-sidecar/api/graylog.init()
/go/src/github.com/Graylog2/collector-sidecar/api/graylog/responses.
go:54 +0x3e
github.com/Graylog2/collector-sidecar/backends.init()
/go/src/github.com/Graylog2/collector-sidecar/backends/registry.go:
92 +0x48
main.init()
/go/src/github.com/Graylog2/collector-sidecar/main.go:146 +0x66

I've trawled the net and found a few references to similar issues in other 
software, but have not been able to identify possible resolutions for the 
missing uptime procedure yet.

 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1693be9c-f55b-45c0-bcfa-f6dc86128819%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog configuration of filebeat and graylog collector sidecar

2016-09-21 Thread Kunal Patil 
Hello

I have read the document previous issue has been resolved
i m getting data but some data  come under %{host} source filed
I have configured apache logs as shown in documentation

please refer attached screenshot










*REGARDS:KUNAL VIKAS PATIL9860265594*

On Thu, Sep 22, 2016 at 1:20 AM, Marius Sturm  wrote:

> Kunal,
> please read the Sidecar documentation first. You have to create a
> configuration in the Graylog web interface and tag it with the same tag
> like you started the Sidecar instance. There is a step-by-step guide even
> with screenshots here: http://docs.graylog.org/en/2.
> 1/pages/collector_sidecar.html#step-by-step-guide
>
> Cheers,
> Marius
>
>
> On 21 September 2016 at 20:52, Kunal Patil 
> wrote:
>
>> hello
>> Thanks for the quick reply and solution as you guys suggested i m trying
>> to implement filebeat with help of documentation but i m getting below
>> error on web gui please check ad revert
>>
>> Sidecar
>> Tags:apacheIP:
>> CPU Idle:99.47%Load:0.06Volumes > 75%:
>> --
>>  *Status*: No configuration found for configured tags!
>> Backends
>>  *Filebeat*: Collector exits immediately, this should not happen! Please
>> check your collector configuration!
>>
>>
>>
>>
>>
>>
>> *REGARDS:KUNAL VIKAS PATIL9860265594*
>>
>> On Wed, Sep 21, 2016 at 9:22 PM, Jochen Schalanda 
>> wrote:
>>
>>> Hi Kunal,
>>>
>>> nxlog and Filebeat are two different log shippers, each with its own
>>> advantages and disadvantages, which are supported by the Graylog Collector
>>> Sidecar.
>>>
>>> Both, nxlog and Filebeat, do support multiline messages:
>>>
>>>- https://www.elastic.co/guide/en/beats/filebeat/1.3/multiline
>>>-examples.html
>>>- https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#x
>>>m_multiline
>>>
>>> It's up to you which log shipper you want to use in the end and how you
>>> configure it.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Wednesday, 21 September 2016 17:43:44 UTC+2, Kunal Patil wrote:

 I m little confused here
 After reading document
 In document u guys have given steps  for beat and nx log configuration
 Can u brief more about that
 My doubt is
 If i have beat to send data to graylog why i want nxlog
 And if  nxlog is required then what is role of beat

>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Graylog Users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>> pic/graylog2/QVxdxyLWmww/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> graylog2+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/graylog2/42f77a7e-b46f-4df6-9d2b-3366af1415da%40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/graylog2/CAJa2o%3D85b_XKO2sgzBvDJ5YjoBX-o3RFJjZ%3D%3DJOR
>> jw%3D2%3DktESA%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Developer
>
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
>
> https://www.graylog.com 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/graylog2/QVxdxyLWmww/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/CAMqbBbJfWA08j_rVraiJpHOA9cpHM4Gwvk0tyZ9Eu3e0
> krRLiQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to

Re: [graylog2] Re: Graylog configuration of filebeat and graylog collector sidecar

2016-09-21 Thread Marius Sturm 
Kunal,
please read the Sidecar documentation first. You have to create a
configuration in the Graylog web interface and tag it with the same tag
like you started the Sidecar instance. There is a step-by-step guide even
with screenshots here:
http://docs.graylog.org/en/2.1/pages/collector_sidecar.html#step-by-step-guide

Cheers,
Marius


On 21 September 2016 at 20:52, Kunal Patil  wrote:

> hello
> Thanks for the quick reply and solution as you guys suggested i m trying
> to implement filebeat with help of documentation but i m getting below
> error on web gui please check ad revert
>
> Sidecar
> Tags:apacheIP:
> CPU Idle:99.47%Load:0.06Volumes > 75%:
> --
>  *Status*: No configuration found for configured tags!
> Backends
>  *Filebeat*: Collector exits immediately, this should not happen! Please
> check your collector configuration!
>
>
>
>
>
>
> *REGARDS:KUNAL VIKAS PATIL9860265594*
>
> On Wed, Sep 21, 2016 at 9:22 PM, Jochen Schalanda 
> wrote:
>
>> Hi Kunal,
>>
>> nxlog and Filebeat are two different log shippers, each with its own
>> advantages and disadvantages, which are supported by the Graylog Collector
>> Sidecar.
>>
>> Both, nxlog and Filebeat, do support multiline messages:
>>
>>- https://www.elastic.co/guide/en/beats/filebeat/1.3/multiline
>>-examples.html
>>- https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#
>>xm_multiline
>>
>> It's up to you which log shipper you want to use in the end and how you
>> configure it.
>>
>> Cheers,
>> Jochen
>>
>> On Wednesday, 21 September 2016 17:43:44 UTC+2, Kunal Patil wrote:
>>>
>>> I m little confused here
>>> After reading document
>>> In document u guys have given steps  for beat and nx log configuration
>>> Can u brief more about that
>>> My doubt is
>>> If i have beat to send data to graylog why i want nxlog
>>> And if  nxlog is required then what is role of beat
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Graylog Users" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/graylog2/QVxdxyLWmww/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> graylog2+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/graylog2/42f77a7e-b46f-4df6-9d2b-3366af1415da%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/CAJa2o%3D85b_XKO2sgzBvDJ5YjoBX-o3RFJjZ%3D%
> 3DJORjw%3D2%3DktESA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbJfWA08j_rVraiJpHOA9cpHM4Gwvk0tyZ9Eu3e0krRLiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Broken Streams?

2016-09-21 Thread Nathan Mace 
Recently upgraded to 2.1 and just noticed this behavior.

I have a stream that matches against two rules:

EventID = 4625
AND
TargetUserName NOT EXACTLY "XX"

If a log matches both of those, send an email.  The emails are not being 
sent.  Looking into it, if I force a failed login attempt it generates a 
message that should match the stream.  I go manually find the message and 
in the details off to the side it does say it was routed into the stream. 
 Additionally, if I copy the message ID and load it into the stream it 
gives two green lines and says it should match.  Also, I can click on the 
title of the stream that takes me to the search screen with the rules of 
the stream applied, and the message shows up there as well.  I tried 
deleting and re-creating the stream, that did not help either.

Sending a test email from the stream is successful.

Any ideas?  These are Windows event logs, but I don't think that matters. 
 Thanks.

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a5f172c1-b6ed-471e-9625-6d8ea33e2d21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog configuration of filebeat and graylog collector sidecar

2016-09-21 Thread Jochen Schalanda 
Hi Kunal,

On Wednesday, 21 September 2016 17:00:48 UTC+2, Kunal Patil wrote:
>
> I m unable to fetch multiline logs
> Please help with that
>
 
nxlog supports parsing multiline messages, 
see https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_multiline 
for details.

Cheers,
Jochen

>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b7e543fe-27d0-412f-b4a8-8f8e0388bc8d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Issues after upgrading to 2.1

2016-09-21 Thread Jochen Schalanda 
Hi Chris,

what's the output of the following curl commands if you run it on the 
machine your web browser is running on?

curl -v -X GET http://10.18.16.15:9000/
curl -v -X GET -H 'Accept: application/json' http://10.18.16.15:9000/api/


The long GC pauses (over 1 minute is really bad and unusual) are also 
strange. Try giving Graylog more memory (currently it's 1 GB according to 
the logs, try 2 GB), see 
http://docs.graylog.org/en/2.1/pages/faq.html#raise-the-java-heap.


Cheers,
Jochen

On Wednesday, 21 September 2016 15:59:35 UTC+2, Chris Call wrote:
>
> I had a great experience with Graylog before version 2.0 but I started 
> fresh with a 2.0 install and had issues (streams kept stopping) and then 
> upgraded to 2.0.1 I believe and now finally to 2.1.
>
> Right now, I can't open the web interface when I browse to 
> 10.18.16.15:9000 (IP address of the host).  I get nothing in the browser. 
>  Chrome gives me this "The 10.18.16.15 page isn’t working.  10.18.16.15 
> didn’t send any data." and no errors or messages in the developer java 
> console.
>
> This is a single system running everything for Graylog and here are my 
> config files and output:
>
> etc/graylog/server/server.conf:
> is_master = true
> node_id_file = /etc/graylog/server/node-id
> password_secret = 
> root_password_sha2 = 
> root_email = 
> plugin_dir = plugin
> rest_listen_uri = http://10.18.16.15:9000/api/
> web_enable = true
> web_listen_uri = http://10.18.16.15:9000/
> rotation_strategy = count
> elasticsearch_max_docs_per_index = 2000
> elasticsearch_max_number_of_indices = 20
> retention_strategy = delete
> elasticsearch_shards = 4
> elasticsearch_replicas = 0
> elasticsearch_index_prefix = graylog
> allow_leading_wildcard_searches = false
> allow_highlighting = false
> elasticsearch_analyzer = standard
> output_batch_size = 500
> output_flush_interval = 1
> output_fault_count_threshold = 5
> output_fault_penalty_seconds = 30
> processbuffer_processors = 5
> outputbuffer_processors = 3
> processor_wait_strategy = blocking
> ring_size = 65536
> inputbuffer_ring_size = 65536
> inputbuffer_processors = 2
> inputbuffer_wait_strategy = blocking
> message_journal_enabled = true
> message_journal_dir = /var/lib/graylog-server/journal
> lb_recognition_period_seconds = 3
> mongodb_uri = mongodb://localhost/graylog
> mongodb_max_connections = 1000
> mongodb_threads_allowed_to_block_multiplier = 5
> content_packs_loader_enabled = false
> content_packs_auto_load = grok-patterns.json
> proxied_requests_thread_pool_size = 32
>
>
> /etc/elasticsearch/elasticsearch.yml 
> #  Elasticsearch Configuration 
> =
>  cluster.name: graylog
>
> result of " curl -XGET 'http://localhost:9200/_cluster/health?pretty=true
> ' ":
> {
>   "cluster_name" : "graylog",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 20,
>   "active_shards" : 20,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0,
>   "delayed_unassigned_shards" : 0,
>   "number_of_pending_tasks" : 0,
>   "number_of_in_flight_fetch" : 0,
>   "task_max_waiting_in_queue_millis" : 0,
>   "active_shards_percent_as_number" : 100.0
> }
>
>  /var/log/graylog-server/server.log:
> 2016-09-21T09:41:42.573-04:00 WARN  [PluginLoader] Plugin directory 
> /plugin does not exist, not loading plugins.
> 2016-09-21T09:41:43.030-04:00 INFO  [CmdLineTool] Running with JVM 
> arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB 
> -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled 
> -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
> -XX:-OmitStackTraceInFastThrow 
> -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml 
> -Djava.library.path=/usr/share/graylog-server/lib/sigar 
> -Dgraylog2.installation_source=deb
> 2016-09-21T09:41:47.611-04:00 INFO  [InputBufferImpl] Message journal is 
> enabled.
> 2016-09-21T09:41:47.678-04:00 INFO  [NodeId] Node ID: 
> ecdff2ab-d0a2-4ddb-975e-d2379fb3625d
> 2016-09-21T09:41:48.054-04:00 INFO  [LogManager] Loading logs.
> 2016-09-21T09:41:48.184-04:00 WARN  [Log] Found a corrupted index file, 
> /var/lib/graylog-server/journal/messagejournal-0/000167286888.index, 
> deleting and rebuilding index...
> 2016-09-21T09:41:49.740-04:00 INFO  [LogManager] Logs loading complete.
> 2016-09-21T09:41:49.740-04:00 INFO  [KafkaJournal] Initialized Kafka based 
> journal at /var/lib/graylog-server/journal
> 2016-09-21T09:41:49.779-04:00 INFO  [InputBufferImpl] Initialized 
> InputBufferImpl with ring size <65536> and wait strategy 
> , running 2 parallel message handlers.
> 2016-09-21T09:41:49.825-04:00 INFO  [cluster] Cluster created with 
> settings {hosts=[localhost:27017], mode=SINGLE, 
> requiredClusterType=UNKNOWN, serverSelectionTimeout='3 ms', 
> maxWaitQueueSize=5000}
> 2016-09-21T09:41:49.946-04:00 INFO  [cluster] No server chosen by 
>

Re: [graylog2] Re: Weird Stream behaviour

2016-09-21 Thread Tony 
Thank you Jochen to Highlight me :-) Now it works.

Thanks

Tony

2016-09-21 11:06 GMT+01:00 Jochen Schalanda :

> Hi Tony,
>
> your last post is missing the important part: Are the stream rules
> evaluated with logical AND (all rules have to match) or logical OR (only
> one rule has to match).
>
> Additionally, your second rule, "message field must match exactly WARN"
> is wrong, as the message field clearly does not only contain the word
> "WARN". You can either use a regular expression to match the message
> field or extract that word into a separate field.
>
> Cheers,
> Jochen
>
> On Wednesday, 21 September 2016 00:06:53 UTC+2, Tony wrote:
>>
>> Hi Jochen,
>> thank you for your answer and help. In the first screenshot I capture
>> from the field debug_level the word INFO and it works.
>> The second is supposed to capture the word WARN from the field message
>> and doesn't work. The third screenshot is the message line.
>>
>> Thanks
>>
>> Tony
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Graylog Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/graylog2/61L7jHB8jok/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/213a75cf-28d3-43cb-90b9-7b5225080307%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CACjATf-WotDW_qiqs-72xZHQ5FzQ9XUzXYz7UO02k8DXkDECKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.1.1 Web Interface Problem

2016-09-21 Thread 'Chris' via Graylog Users 
Hi Jochen,

You are correct, I was indeed missing the */api/ from the URL.

I can now log in and I will start looking at using some beats forwarders to 
get some data into GrayLog/ElasticSearch.

Cheers,

Chris. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/0ca00c18-7f99-4b77-80a3-e82192b3ded1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Debian Jessie Graylog 2.1 Apache 2.4 - Cannot access web interface

2016-09-21 Thread Wesley Pallete de Sousa 
My server.conf looks like this now

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = * SECRET HASH *
root_username = * SECRET USER *
root_password_sha2 = * SECRET AGAIN *
#root_email = ""
root_timezone = America/Sao_Paulo
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://hom.example:9000/api/
#rest_transport_uri = http://hom.example:12900
#rest_enable_cors = false
#rest_enable_gzip = false
#rest_enable_tls = true
#rest_tls_cert_file = /path/to/graylog.crt
#rest_tls_key_file = /path/to/graylog.key
#rest_tls_key_password = secret
#rest_max_header_size = 8192
#rest_max_initial_line_length = 4096
#rest_thread_pool_size = 16
#trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
web_enable = true
web_listen_uri = http://hom.example:9000/
web_endpoint_uri = http://hom.example:9000/api/
web_enable_cors = true
#web_enable_gzip = false
#web_enable_tls = true
#web_tls_cert_file = /path/to/graylog-web.crt
#web_tls_key_file = /path/to/graylog-web.key
#web_tls_key_password = secret
#web_max_header_size = 8192
#web_max_initial_line_length = 4096
#web_thread_pool_size = 16
elasticsearch_config_file = /etc/elasticsearch/elasticsearch.yml
rotation_strategy = count
elasticsearch_max_docs_per_index = 2000
#elasticsearch_max_size_per_index = 1073741824
#elasticsearch_max_time_per_index = 1d
#elasticsearch_disable_version_check = true
#no_retention = false
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
#elasticsearch_template_name = graylog-internal
allow_leading_wildcard_searches = false
#elasticsearch_cluster_name = graylog
#elasticsearch_node_name_prefix = graylog-
#elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
#elasticsearch_node_master = false
#elasticsearch_node_data = false
#elasticsearch_transport_tcp_port = 9350
#elasticsearch_http_enabled = false
#elasticsearch_cluster_discovery_timeout = 5000
#elasticsearch_network_host =
#elasticsearch_network_bind_host =
#elasticsearch_network_publish_host =
#elasticsearch_discovery_initial_state_timeout = 3s
elasticsearch_analyzer = standard
#elasticsearch_request_timeout = 1m
#index_ranges_cleanup_interval = 1h
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
#outputbuffer_processor_keep_alive_time = 5000
#outputbuffer_processor_threads_core_pool_size = 3
#outputbuffer_processor_threads_max_pool_size = 30
#udp_recvbuffer_sizes = 1048576
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
#message_journal_max_age = 12h
#message_journal_max_size = 5gb
#message_journal_flush_age = 1m
#message_journal_flush_interval = 100
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb
#async_eventbus_processors = 2
lb_recognition_period_seconds = 3
#lb_throttle_threshold_percentage = 95
#stream_processing_timeout = 2000
#stream_processing_max_faults = 3
#alert_check_interval = 60
#output_module_timeout = 1
#stale_master_timeout = 2000
#shutdown_timeout = 3
mongodb_uri = mongodb://SECRET_USER:SECRET_PASS@127.0.0.1:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
#rules_file = /etc/graylog/server/rules.drl
#transport_email_enabled = false
#transport_email_hostname = mail.example.com
#transport_email_port = 587
#transport_email_use_auth = true
#transport_email_use_tls = true
#transport_email_use_ssl = true
#transport_email_auth_username = y...@example.com
#transport_email_auth_password = secret
#transport_email_subject_prefix = [graylog]
#transport_email_from_email = gray...@example.com
#transport_email_web_interface_url = https://graylog.example.com
#http_connect_timeout = 5s
#http_read_timeout = 10s
#http_write_timeout = 10s
#http_proxy_uri =
#disable_index_optimization = true
#index_optimization_max_num_segments = 1
#gc_warning_threshold = 1s
#ldap_connection_timeout = 2000
#disable_sigar = false
#dashboard_widget_default_cache_time = 10s
#content_packs_loader_enabled = true
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9df1fc58-1ba3-4df1-8368-bc5f3165b0a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.1.1 Web Interface Problem

2016-09-21 Thread Jochen Schalanda 
Hi Chris,

according to the details you've shared previously, the correct value for 
web_endpoint_uri would be http://MY_AWS_RT53_DNS:12900/api/.

Cheers,
Jochen

On Wednesday, 21 September 2016 12:37:01 UTC+2, Chris wrote:
>
> Thanks Jochen,
>
> I have updated the web_endpoint_uri to:
>
> web_endpoint_uri = http://MY_AWS_RT53_DNS:12900/
>
>
> Then when I go to the browser it allows my enter my log in details and 
> gives this error:
>  
> Error - the server returned: 404 - cannot POST 
> http://MY_AWS_RT53_DNS:12900/system/sessions 
> (404)
>
> I tried this on the aws servers cli expecting a response - curl http://
> 172.31.29.124:12900/system/cluster/node:
>
> 
> 
>   
> 
> 
> 
> Graylog Web Interface
> 
>
>   
>   
> 
> 
>
> 
>
>  src="/assets/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.0dd5605a30a0f21974ba.js">
>
>  src="/assets/plugin/org.graylog.plugins.map.MapWidgetPlugin/plugin.org.graylog.plugins.map.MapWidgetPlugin.7bf4cb9247ec32aa855e.js">
>
>  src="/assets/plugin/org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin/plugin.org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin.29f87bd4971d7f01c535.js">
>
>  src="/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.bd2ae365ad1b86102f68.js">
>
> 
>
>   
> 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7db23861-46e9-4b79-b062-95bf49f13dc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: send owncloud/nextcloud logs to graylog

2016-09-21 Thread Jochen Schalanda 
Hi Stefan,

On Wednesday, 21 September 2016 12:08:02 UTC+2, Stefan Krüger wrote:
>
> will graylog create his own log shipper in the near feature?
> I think some people want to have on solution from on source  
>

Been there, done that . In other 
words: no.

The Graylog Collector Sidecar 
 is the way to 
go.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/366084b1-d76a-42f1-ba37-232331f1252a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog configuration of filebeat and graylog collector sidecar

2016-09-21 Thread Jochen Schalanda 
Hi Kunal,

we can't help you without more details about what you did, what you 
expected it to do, and what it actually did.

If you think that there are steps missing in the documentation, feel free 
to open an issue at https://github.com/Graylog2/documentation/issues.

And last but not least, there is professional support for Graylog if you 
need help: https://www.graylog.org/professional-support

Cheers,
Jochen

On Wednesday, 21 September 2016 11:54:12 UTC+2, Kunal Patil wrote:
>
> Documentation is incomplete
>  please tell me steps for achieving desired output with sidecar collector
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/027eea8d-61d2-48e0-b237-05cf589be2f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Compress collected data or move to a new HDD?

2016-09-21 Thread Jochen Schalanda 
Hi,

Phil already mentioned the part of the Graylog documentation explaining how 
to expand the disk space in the virtual machine appliances: 
http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html#extend-ova-disk

You can also activate a better compression in Elasticsearch at the expense 
of CPU usage:

   - 
   
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index-modules.html#_static_index_settings
   - https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0
   - 
   https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch
   

And of course there's always the possibility to tune your index retention 
settings to simply keep less logs on the System / Indices page of the 
Graylog web interface.


Cheers,
Jochen

On Wednesday, 21 September 2016 04:43:24 UTC+2, 8bits...@gmail.com wrote:
>
> I have Elasticsearch data and it's logs written to a 2nd HDD than where 
> the OS is.  This HDD, 100GB, is constantly getting maxed out with ES's logs 
> which I manually delete, but I see the indices are slowly creeping up in 
> size too.  Is there a compression option that I am missing?  Or how would I 
> move data to a 3rd HDD, bigger in size of course, without losing anything 
> collected this far?  Would it be as simple as stopping Graylog, copying the 
> folders over, define new path, and restart Graylog?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7757d463-f338-4043-af45-18a2ae05f7f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog 2.1.1 Web Interface Problem

2016-09-21 Thread 'Chris' via Graylog Users 
Today, I tried to install graylog 2.1.1 in new Amazon Instance to test 
features of new graylog. After I installed elastic search 2.4.0, mongodb 
3.2.9 and graylog 2.1.1, I configured elasticsearch.yml and graylog config 
as below. Then, even though graylog server is up, running and elastic 
search added graylog node in logs, I encountered weird problem. Then I 
typed the graylog server ip ( <"my amazon instance public ip">:9000 ) in 
chrome and safari. However, when I entered my credentials ( admin/graylog 
password) and clicked signin, nothing was fired. Then 15-30 seconds later, 
graylog web interface gave an error as below:


We are experiencing problems connecting to the Graylog server running on 
*http://172.31.29.124:12900/api/*. Please verify that the server is healthy 
and working correctly..


My graylog config looks like this (/etc/graylog/server/server.conf):

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = SECRECT
root_password_sha2 = SECRET
root_timezone = GMT
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://0.0.0.0:12900/api/
external_rest_uri: http://MY_AWS_RT53_DNS/api
web_listen_uri = http://0.0.0.0:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 2000
rotation_strategy = count
elasticsearch_max_docs_per_index = 2000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 172.31.29.124:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

My elasticsearch yml file looks like this 
(/etc/elasticsearch/elasticsearch.yml):

cluster.name: graylog
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
script.inline: false
script.indexed: false
script.file: false
network.host: 172.31.29.124
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["172.31.29.124:9300"]

This is the graylog server log (/var/log/graylog-server/server.log):

2016-09-21T08:45:59.563Z INFO  [CmdLineTool] Loaded plugin: Elastic Beats 
Input 1.1.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2016-09-21T08:45:59.564Z INFO  [CmdLineTool] Loaded plugin: Collector 1.1.1 
[org.graylog.plugins.collector.CollectorPlugin]
2016-09-21T08:45:59.565Z INFO  [CmdLineTool] Loaded plugin: Enterprise 
Integration Plugin 1.1.1 
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-09-21T08:45:59.565Z INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 
1.1.1 [org.graylog.plugins.map.MapWidgetPlugin]
2016-09-21T08:45:59.565Z INFO  [CmdLineTool] Loaded plugin: Pipeline 
Processor Plugin 1.1.1 
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-09-21T08:45:59.566Z INFO  [CmdLineTool] Loaded plugin: Anonymous Usage 
Statistics 2.1.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-09-21T08:45:59.676Z INFO  [CmdLineTool] Running with JVM arguments: 
-Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC 
-XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
-XX:-OmitStackTraceInFastThrow 
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml 
-Djava.library.path=/usr/share/graylog-server/lib/sigar 
-Dgraylog2.installation_source=rpm
2016-09-21T08:46:01.979Z INFO  [InputBufferImpl] Message journal is enabled.
2016-09-21T08:46:02.003Z INFO  [NodeId] Node ID: 
a5e73742-5454-49d7-a089-eb3beb6443b8
2016-09-21T08:46:02.202Z INFO  [LogManager] Loading logs.
2016-09-21T08:46:02.257Z INFO  [LogManager] Logs loading complete.
2016-09-21T08:46:02.257Z INFO  [KafkaJournal] Initialized Kafka based 
journal at /var/lib/graylog-server/journal
2016-09-21T08:46:02.274Z INFO  [InputBufferImpl] Initialized 
InputBufferImpl with ring size <65536> and wait strategy 
, running 2 parallel message handlers.
2016-09-21T08:46:02.300Z INFO  [cluster] Cluster created with settings 
{hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, 
serverSelectionTimeout='3 ms', maxWaitQueueSize=5000}
2016-09-21T08:46:02.355Z INFO

[graylog2] Re: Compress collected data or move to a new HDD?

2016-09-21 Thread Phil Sumner 
Can't talk about compression, but moving data to a new disk is referenced 
in the manual:

http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html#extend-ova-disk



On Wednesday, 21 September 2016 03:43:24 UTC+1, 8bits...@gmail.com wrote:
>
> I have Elasticsearch data and it's logs written to a 2nd HDD than where 
> the OS is.  This HDD, 100GB, is constantly getting maxed out with ES's logs 
> which I manually delete, but I see the indices are slowly creeping up in 
> size too.  Is there a compression option that I am missing?  Or how would I 
> move data to a 3rd HDD, bigger in size of course, without losing anything 
> collected this far?  Would it be as simple as stopping Graylog, copying the 
> folders over, define new path, and restart Graylog?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/324c767d-2361-4434-ae8b-91409361efd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.