Re: [graylog2] Starting graylog2-server redirecting stderr to stdout

[Marcus Franke]

On Thu, Aug 18, 2016 at 02:55:10PM +0200, Charmant Sampi wrote:
> I install graylog2 on ubuntu 14.04 amd64 Desktop
> i follow this tuto
> https://www.digitalocean.com/community/tutorials/how-to-install-graylog2-and-centralize-logs-on-ubuntu-14-04
> 
> But at the step where to start graylog2 I see this message
> Starting graylog2-server
> redirecting stderr to stdout
> Help me!

This is just a start message. It tells you that all messages that would be
logged to the stderr channel are instead logged to the stdout channel. This is
a common pattern.

Usually the logfile would be found in /var/log/graylog-server/server.log

But that blog post is kind of old, the release described is very old. You should
consider to install the current 2.0.3 stable release of graylog with the
latest 2.3.5 release of elasticsearch.


Regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/20160818131617.GF5168%40arch.iathh.local.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Syslog severity mapper decorator

[Marcus Franke]

On Thu, Aug 18, 2016 at 12:02:12PM +0200, Jan Doberstein wrote:
> Hej Marcus,
> 
> Do you have a slightly more elaborate example? As I wrote, in the search  
> results all logevents still show up with level: 6 and no names. 
> 
> The decorator do not manipulate the data, it changes only the view in the
> search.

That whats what I hoped for, but to me it looks like nothing has changed at
all. Everything is like it was with 2.0 and/or 2.1beta2. I must be kind of
too blind to see ;)

>From my understanding I could still search for something like:
level:<4 AND message:foo

But I would expect to see in the search window of my message a decorated view
of my messages with ERROR, WARNING, FATAL instead of the kind of raw numbers.

I think I completely misunderstood what decorators meant in the release notes.
Or something is broken with my installation. Ill post a screenshot of one of
the messages.


Regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/20160818130028.GE5168%40arch.iathh.local.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Syslog severity mapper decorator

[Marcus Franke]

Hi Jan,

> Just give it a try and you will notice how it work

Do you have a slightly more elaborate example? As I wrote, in the search
results all logevents still show up with level: 6 and no names.

Generating the quick values for level show (in my case) just 5 and 6, no named
values like NOTICE or INFO.

Maybe it's because of my input? Im using the SystemdJournal2Gelf tool from the
marketplace to send sample data into my development cluster running the
lastest beta release.



Puzzled regards ;)

Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/20160818094423.GC5168%40arch.iathh.local.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Syslog severity mapper decorator

[Marcus Franke]

Hi all,

in the graylog 2.1-beta3 announcement there is this announcement about the
syslog severity mapper decorator, but I am unable to find a place where to
configure this.

In the mongodb I see an empty collection named decorators with zero entries:

> db.decorators.count()
0

And when I send a search all my messages still have fields like level: 6,
somehow I would have expected this to become level: INFO.

Will this decorator only work on syslog inputs or for all different inputs,
especially the GELF ones.


Kind regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/20160817085156.GG12732%40arch.iathh.local.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: about graylog bigger setup, Mongodb replicaset

[Marcus Franke]

Am Montag, 8. August 2016 11:12:03 UTC+2 schrieb sangh:
>
> For those who install a bigger setup for Graylog
> i have two server and i want to deploy a replicaset for each graylog 
> server. how ever i keep getting this error :
>
> rs.initiate({_id: "gray",members: [{ _id: 0, host : " 
> Graylog2.localdomain:27017" },{ _id: 1, host : " 
> Graylog1.localdomain:27017" }]})
> {
> "ok" : 0,
> "errmsg" : "No host described in new configuration 1 for replica 
> set gray maps to this node",
> "code" : 93
>
> Anyone knows how to fix this ?? 
>

Do you have three nodes for the replica set?

I guess your members in the above example are the two other nodes? You have 
to put all three nodes in the config block.

rs.initiate(
  {
_id: "graylog"
version: 1,
members: [
  { _id: 0: host: "localhost:27017" }
  { _id: 1, host: "graylog1.example.com:27017" },
  { _id: 2, host: "graylog2.example.com:27017" }
]
  }
)

The error message, as I understand it, is pointing out, that you are trying 
to initialse a replicaset on "localhost" and
you're trying to add the two others into the replSet. This won't work, the 
machine you're connected to in the shell
must be part of the members.

Have a look at the mongodb docs:
https://docs.mongodb.com/manual/reference/method/rs.initiate/#example

The config block above is taken from that page.


Regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/346fa04a-b818-450d-ab51-0c0c4d96b2a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: How to use API token in curl command to call REST API

[Marcus Franke]

>
>
> Problem:
>
> I don't like to use username/password for authentication. Instead I want 
> to use the API access token I generated for this user.
>
> What is parameter/header I have to set?
>
>
Hi Clifford,

did you solve this? We are stuggling with the same issue.


Regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/1212c755-0067-4b7b-b22d-7ef59a273ea6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Elasticsearch cluster unhealthy (RED)

[Marcus Franke]

Hi Arief,

you should have a look into your elaticsearch logfiles like Jochen wrote.
ES will tell you about what's going on in case something is happening.

Then, there are the various api's you could use to find out, why your
cluster is yellow. Probably the different localhost:9200/_cat/ endpoints
like
nodes, indices and shards should help you to find the one that isn't settled
and will help you to find out why.

https://www.elastic.co/guide/en/elasticsearch/reference/1.4/cat.html

You can further install plugins into your ES nodes like kopf, the german
translation of head, which is another plugin that would give you visual
overview
into your cluster.

https://github.com/lmenezes/elasticsearch-kopf


Have a look into these,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVc9ojxoEcJa9Yseg_Xrp0kVW_NZ%3DBygDavTmX-XvfZtBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Dashboard permissions, users and roles

[Marcus Franke]

Hi Maren,

I second that, graylog is great, but its dashboard capabilities need some
more love. More different widget types would be cool.
And the roles/permissions aren't very granular. The pure reader role is
just allowed to login, you need at least a view dashboard
role and even that one is so limited one can use it for large dashboard
screens in the canteen. At least we do.

My guess, why nobody is allowed to use the search is not to overload the
underlying elasticsearch with wild user queries like:
"SELECT * FROM * INTO OUTFILE full_export.json" ;)

Regards,
Marcus

Maren Dewell <time2t...@online-gadget.com> schrieb am Di., 26. Juli 2016 um
17:56 Uhr:

>
>
> On Tuesday, July 26, 2016 at 5:00:08 PM UTC+2, Marcus Franke wrote:
>
>> am I wrong with the observation, that a user has to have the admin role
>> to access the search panel and perform searches?
>>
>
> Having the same issue. Created a user and wanted to show only the search
> for a predefined set of messages (using streams). Found the following
> entry, but it did not help me:
>
> https://github.com/Graylog2/graylog2-web-interface/issues/620#issuecomment-234900616
>
> Background:
> We are currently investigating in replacing Kibana in favor of Graylog,
> because of alerts and user permissions. We want to use Graylog as central
> logging server (like we are doing with Kibana). We have several teams that
> work on several, independent projects. Only users from a specific project
> need to see log messages of that specific project. So we need a user
> permission to see only a predefined search (read only, by project name). It
> would be great if this is possible, besides this, Graylog is great!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/4d346959-b47d-4391-a66f-10e569f76700%40googlegroups.com
> <https://groups.google.com/d/msgid/graylog2/4d346959-b47d-4391-a66f-10e569f76700%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVerZbUHHAeGLBVZeMhU%2BWJG9TXydCA5bMW_Ee9k2dTeCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Increase OUTPUT speed

[Marcus Franke]

Hi Roberto,

If you have performance issues with the processing, then you have two
choices:

Have a look into your server.conf settings.
There are several options to possibly speed up the output performance
output_batch_size defaults to 25? If you have thousands of messages in the
incoming then the batches are too small
outputbuffer_processors .. if you have plenty of idle cpu power raise the
value

If you still have problems with the output buffer filling up you should
have a look at tuning your es cluster for more indexing speed.

Then there is the obvious option, do you have enough RAM in your server?
Have you raised Xms and Xmx settings for the jvm graylog-server is running
in? On RHEL based distributions this can be found in
/etc/sysconfig/graylog-server.

For example, I have set them to 24G for my setup, be aware of the java
compressed pointers and thus you should not raise above 31G.


regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVcjSWLi1mKs%3DD9CJj5C41n%2BNXqaL2C2F_dfAbV_v7P%3DKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Dashboard permissions, users and roles

[Marcus Franke]

Hello,

am I wrong with the observation, that a user has to have the admin role to
access the search panel and perform searches?

Background: I created a role for group of non admin users to work with some
dashboards, but these users cannot access the "Replay Search" button. I
granted the "allow editing" permission to the role. Now these users are
able to delete widgets and reorder the dashboard.

How should my users change the underlying search for a widget or even
create new ones, when they have the editing rights for a dashboard?

Am I missing something? Are there more permissions I missed to find?


kind regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVfR76HisNWPUdMtyDcfvZnC9myRX8GWqCHDAArCMXsT1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] graylog Bigger production setup

[Marcus Franke]

Hi,

did you try to connect to the elasticsearch by _hand_, is your es setup
reachable from the graylog server?

you can test with a plain telnet on both graylog servers:

telnet es_node1 9300
telnet es_node2 9300
telnet es_node3 9300

As your servers are located in different subnets, I assume you checked the
routing between the two networks?
Have you checked if there are firewalls in place, blocking the traffic?

Btw. my graylog servers elasticsearch.yml contains for the key
discovery.zen.ping.unicast.hosts all es nodes, not just one.

sangh  schrieb am Fr., 22. Juli 2016 um 09:42 Uhr:

> Hi
> I have two graylog server ( with mangodb) [192.168.80.125,
> 192.168.80.126], a cluster of three elasticsearch node [ 192.168.40.125,
> 192.168.40.126, 192.168.40.127].
>
>
> curl -XGET 'http://192.168.40.125:9200/_cluster/health?pretty=true'
> {
>   "cluster_name" : "graylog",
>   "status" : "green",
>   "timed_out" : false,
>   "number_of_nodes" : 3,
>   "number_of_data_nodes" : 3,
>   "active_primary_shards" : 0,
>   "active_shards" : 0,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 0,
>   "delayed_unassigned_shards" : 0,
>   "number_of_pending_tasks" : 0,
>   "number_of_in_flight_fetch" : 0,
>   "task_max_waiting_in_queue_millis" : 0,
>   "active_shards_percent_as_number" : 100.0
>
> In the graylog config file i set :
> - elasticsearch_cluster_name = graylog
> - elasticsearch_discovery_zen_ping_unicast_hosts = 192.168.40.125:9300
> - elasticsearch_network_host = 192.168.80.125:q
>
>
>
> i get this warning when startibg graylog
>
> [GeoIpResolverEngine] GeoIP database file does not exist:
> /tmp/GeoLite2-City.mmdb
>
> WARN  [DeadEventLoggingListener] Received unhandled event of type
>  from event bus
> 
>
> [discovery] [graylog-d3e992d3-0786-47e4-8aff-9c7f1a46cf8d] waited for 3s
> and no initial state was set by the discovery
>
> [IndexerSetupService] Could not connect to Elasticsearch
>
>  INFO  [zen] [graylog-d3e992d3-0786-47e4-8aff-9c7f1a46cf8d] failed to send
> join request to master
> [{elasticnode-3}{0wVrxpdAT0yr9GpYECzWFw}{192.168.40.127}{192.168.40.127:9300}],
> reason 
> [RemoteTransportException[[elasticnode-3][192.168.40.127:9300][internal:discovery/zen/join]];
> nested:
> ConnectTransportException[[graylog-d3e992d3-0786-47e4-8aff-9c7f1a46cf8d][
> 192.168.80.125:9350] connect_timeout[30s]]; nested:
> NotSerializableExceptionWrapper[connect_timeout_exception: connection timed
> out: /192.168.80.125:9350]; ]
> 2016-07-22T11:13:42.331+02:00 INFO  [IndexerClusterCheckerThread] Indexer
> not fully initialized yet. Skipping periodic cluster check.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/1f544163-bf89-4f23-8046-4cadd604e08f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVc0e_u0ZQN4gpYGseRtErd03gPYrGfw8haZ8i3RJTA%3D%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Elasticsearch cluster unhealthy (RED)

[Marcus Franke]

Hi,

there are some REST API endpoints in elasticsearch you can check:

General Overview:
curl 'http://localhost:9200/_cluster/health?pretty=true'

Overview over your indices:
curl http://localhost:9200/_cat/indices

This will list you the index that is red, I guess not enough diskspace and
thus
unallocated shards. I had the same problem.

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/cat-indices.html

My problem was the newly created deflector index could not be allocated, I
deleted
some older indices from the graylog webinterface and curl'ed the
unallocated
index away:

curl -XDELETE http://localhost:9200/graylog2_1234/

that particular index was created again, as my current deflector was _full_
and everything was fine again. Now I have a tighter look on the diskspace
of my ES nodes.


Greetings,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVeMLYnSRkOdtijRvU_hXzAppajM2xjtjxwDWa%3Di6w-EXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Dasboard time range problem

[Marcus Franke]

Marcus Franke <marcus.fra...@gmail.com> schrieb am Di., 28. Juni 2016 um
16:28 Uhr:

> Hi,
>
> sure, I guess thats the root_timezone setting from the server.conf and
> thats Europe/Berlin, too
> In the system -> overview its +0200 for my user, my browser and the
> graylog server.
>
> If I switch my users profile from Europe/Berlin to UTC the dasboard will
> loose another two hours,
>
> Marcus
>

Answering to myself:

I updated my graylog installation from 2.0.0 to latest 2.0.3 and the
problem still exists.
One last thing, I calculate the timestamps using an extractor with a date
converter.

The filter reads from a field that contains an iso8601 formated timestamp
like:
2016-06-15T23:45:02.350+0200

The generated timestamp looks like:
2016-06-15T21:45:02.350Z

The config of the date converter is:
Format string: -MM-dd'T'HH:mm:ss.SSSZ
Time Zone: UTC

Before the extractor Time Zone was at Europe/Berlin too. I just deleted my
indexes
and restarted the ingestion of the logs a second time, but sadly I see no
difference. The
dashboard moves again backwards those two hours.

Greetings,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFRuYVd%3DaRveT2N88rbnecGg%3DW_uonWmCEBUm1Fd8suyd7yDcQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Dasboard time range problem

[Marcus Franke]

Hi,

Ive got a (little) dashboard problem, that drives me crazy. I want the 
histogram showing the message flow of a day.
I do a general search for all messages of a certain day using the time 
picker from 2016-06-15 00:00:00 to 2016-06-16 00:00:00.
My timezone is Europe/Berlin, so plus 2 hours at the moment. In the browser 
URL I can see "from=2016-06-14T22%3A00"


 











The histogram shows 24 bars from 00 to 23, everything is fine until I add 
the histogram to a dashboard.

Now the histogram has that particular time range from 2016-05-14 
22:00:00:00 to 2016-06-15 22:00:00:











What am I doing wrong, that this -2 hours shift happens? Btw. I have 
installed Graylog 2.0.0 (2dc6c03).



With kind regards,
Marcus

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a89a330d-3e19-4ad7-8254-239c81193dc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.