[graylog2] How to use custom field value in field chart
Hi, I'd like to generate a chart using a custom field value, that is inside the messages sent to Graylog2, but using Graylog Web Interface I see no way to do this. Is there a way to do this? Is it possible retrieve the custom field value searching 'fieldhistogram' by using the Graylog2 REST API? Regards, Reginaldo Russinholi -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [graylog2] How to use custom field value in field chart
Hey Reginaldo, select a field you want to chart from the sidebar, hit the little cog and press "Generate chart". Note that this is of course only possible for numeric values. You can also get the same data via the REST APIs, that is true. :) Let me know if you need help with anything. Thanks, Lennart On Wed, Feb 19, 2014 at 9:01 PM, Reginaldo Russinholi wrote: > Hi, > > I'd like to generate a chart using a custom field value, that is inside the > messages sent to Graylog2, but using Graylog Web Interface I see no way to > do this. > > Is there a way to do this? Is it possible retrieve the custom field value > searching 'fieldhistogram' by using the Graylog2 REST API? > > Regards, > > Reginaldo Russinholi > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [graylog2] How to use custom field value in field chart
Hi Lennart, I did that, but the generate chart only display the mean, count, total, maximum or minimum value of my custom field inside the defined interval (minute, hour and etc.) and I wanted to generate a chart with the absolute field value. Of course, if I have 1 message per minute the mean will be equal to the absolute value, but what if I have a lot of messages during a minute and I want a chart to see how the values are varying from a message to other. About the REST API, I tried to use "/search/universal/relative/fieldhistogram" but I couldn't find a way to get the value of my custom field, it seems that this method only accept the values 'mean','count', 'total' in the field parameter. Is there any documentation about that I can learn from? Thanks, Reginaldo Russinholi Em quinta-feira, 20 de fevereiro de 2014 09h20min09s UTC-3, lennart escreveu: > > Hey Reginaldo, > > select a field you want to chart from the sidebar, hit the little cog > and press "Generate chart". Note that this is of course only possible > for numeric values. > > You can also get the same data via the REST APIs, that is true. :) > > Let me know if you need help with anything. > > Thanks, > Lennart > > On Wed, Feb 19, 2014 at 9:01 PM, Reginaldo Russinholi > > > wrote: > > Hi, > > > > I'd like to generate a chart using a custom field value, that is inside > the > > messages sent to Graylog2, but using Graylog Web Interface I see no way > to > > do this. > > > > Is there a way to do this? Is it possible retrieve the custom field > value > > searching 'fieldhistogram' by using the Graylog2 REST API? > > > > Regards, > > > > Reginaldo Russinholi > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "graylog2" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to graylog2+u...@googlegroups.com . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [graylog2] How to use custom field value in field chart
Hi Lennart, I found the problem, and it was not on chart generation but in the application that was sending messages to Graylog. There was a bug that sent all messages in a time interval with the same 'timestamp' value. So all messages sent in a minute, for example, was grouped in one single X-axis in the chart and making me believe that there was a problem with chart generation. Sorry for taking your time with this issue and thanks for your help. Reginaldo Russinholi Em quinta-feira, 20 de fevereiro de 2014 11h04min50s UTC-3, Reginaldo Russinholi escreveu: > > > Hi Lennart, > > I did that, but the generate chart only display the mean, count, total, > maximum or minimum value of my custom field inside the defined interval > (minute, hour and etc.) and I wanted to generate a chart with the absolute > field value. Of course, if I have 1 message per minute the mean will be > equal to the absolute value, but what if I have a lot of messages during a > minute and I want a chart to see how the values are varying from a message > to other. > > About the REST API, I tried to use > "/search/universal/relative/fieldhistogram" but I couldn't find a way to > get the value of my custom field, it seems that this method only accept the > values 'mean','count', 'total' in the field parameter. Is there any > documentation about that I can learn from? > > Thanks, > > Reginaldo Russinholi > > Em quinta-feira, 20 de fevereiro de 2014 09h20min09s UTC-3, lennart > escreveu: >> >> Hey Reginaldo, >> >> select a field you want to chart from the sidebar, hit the little cog >> and press "Generate chart". Note that this is of course only possible >> for numeric values. >> >> You can also get the same data via the REST APIs, that is true. :) >> >> Let me know if you need help with anything. >> >> Thanks, >> Lennart >> >> On Wed, Feb 19, 2014 at 9:01 PM, Reginaldo Russinholi >> wrote: >> > Hi, >> > >> > I'd like to generate a chart using a custom field value, that is inside >> the >> > messages sent to Graylog2, but using Graylog Web Interface I see no way >> to >> > do this. >> > >> > Is there a way to do this? Is it possible retrieve the custom field >> value >> > searching 'fieldhistogram' by using the Graylog2 REST API? >> > >> > Regards, >> > >> > Reginaldo Russinholi >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to graylog2+u...@googlegroups.com. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[graylog2] Re: [ANNOUNCE] Graylog2 v0.20.0 has been released
Good job ! Thanks a lot. -- Sébastien www.graylog2.fr On Wednesday, February 19, 2014 7:24:30 PM UTC+1, lennart wrote: > > Hey everybody, > > we are so happy to announce that we just released Graylog2 v0.20.0 > after almost a full year of work. > > You can find the release announcement page here: > http://graylog2.org/wow/such/0.20.0 > > With this as a foundation we'll be releasing regular updates with new > features based on the many requests we already got by you. > > Thank you very much for helping us so much in the last months. You are > awesome! > > Have a great day, > Lennart (in behalf of the whole TORCH team) > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[graylog2] Re: plan to use elasticsearch's routing feature in next graylog2 release ??
Hi,
I tried to resume the modifications in github .
https://github.com/cactus44/graylog2-server/commit/feef28aa3314488958a52de84c964c4564971bce
thx
Regards
Le mercredi 19 février 2014 11:52:05 UTC+1, Kay Röpke a écrit :
>
> Hi!
>
> That's a really good idea :)
> Can you add an issue to github, if there isn't one yet?
> This change seems relatively easy to add, so I think there's a good chance
> we can add it to the next version.
>
> Thanks,
> Kay
>
> On Tuesday, February 18, 2014 12:08:37 PM UTC+1, cactus wrote:
>>
>> Hello Lennart,
>>
>>
>>
>> Can i suggest you to use the elasticsearch’s routing feature in next
>> release graylog2 release . I have got a huge volume of logs (about 2To of
>> data in the ES cluster) with a lot of streams in graylog2, and using
>> the stream_id as routing key in ES , seems to be, from my point of view ,
>> a good idea to improve the response time when doing search requests
>> inside a stream. I have successfully made minor modifications in the
>> source code of GL2 (0.12 but should also work in 0.20) and were able to use
>> the stream_id as routing_key . I also change a little bit the ES
>> mapping.
>>
>> {
>>
>> "_default_": { "_routing": {
>>
>> "required": true,
>>
>> "path": "streams"
>>
>> }
>>
>> }
>>
>> }
>>
>> It’s working !! Documents are indexed in ES, the routing_key is equal
>> to the value of the stream_id .The last thing to do , is to do some
>> modifications in the graylog2-webinterface in order to use the routing_key
>> when doing searches. In fact; the webinterface should be able get the
>> automatically the object-id of stream you are sitting and passes it as
>> argument in the search request. Done the modification , in the
>> webinterface in ruby , it’s seam to work. I’ve not found for moment
>> what exactly to modify , for the webinterface in play
>>
>>
>>
>>
>> Thx ,Regards
>>
>> Guillaume
>>
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[graylog2] Re: graylog2 radio not working
I have found the problem, it was my in my server configuration. Input --> create a new "Graylog2 Radio Input" *Node(s) to spawn input on:* Select the graylog server (not the radio server) *ZooKeeper address*: IP from the zookeeper of the radio server Thanks Em terça-feira, 18 de fevereiro de 2014 12h03min49s UTC-3, André Coelho escreveu: > > I need some help setting up radios servers. > > I have followed the tutorial on: > http://support.torch.sh/help/kb/graylog2-server/using-graylog2-radio-v020x > > My Graylog2 server sees the new radio server, but does not receive any > message from it > > Graylog is using global inputs that are supposed to be working on the > radios, so I have not configured any specific radio input. > > I have sent a test message from the radio server to the port 10516 (this > port is configured on the graylogserver and shows up on the radio server): > > *echo "Hello, I was received via a radio node. :)" | nc -w 1 -u 127.0.0.1 > 10516* > > > This message appears in the graylog2-radio.log. But graylogserver does not > get it. > > > These are my configuration files: > > *kafka server.properties* > > broker.id=0 > port=*9092*(this number was other) > num.network.threads=2 > > num.io.threads=2 > socket.send.buffer.bytes=1048576 > socket.receive.buffer.bytes=1048576 > socket.request.max.bytes=104857600 > log.dirs=/tmp/kafka-logs > num.partitions=2 > log.flush.interval.messages=1 > log.flush.interval.ms=1000 > log.retention.hours=168 > log.segment.bytes=536870912 > log.cleanup.interval.mins=1 > zookeeper.connect=localhost:2181 > zookeeper.connection.timeout.ms=100 > > > *kafka zookeeper.properties* > > dataDir=/tmp/zookeeper > clientPort=2181 > maxClientCnxns=0 > > > */etc/graylog2-radio.conf* > > node_id_file = /etc/graylog2-radio-node-id > graylog2_server_uri = http://mygraylog2server:12900/ > rest_listen_uri = http://127.0.0.1:12950/ > kafka_brokers = localhost:*9092* > kafka_producer_type = async > kafka_batch_size = 200 > kafka_batch_max_wait_ms = 250 > kafka_required_acks = 0 > processbuffer_processors = 5 > processor_wait_strategy = blocking > ring_size = 1024 > > > Any help is appreciated. > > Thanks > > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[graylog2] What am I missing here? ERROR: No ElasticSearch master was found.
Just recently updating graylog and ES and I can't seem to get it to work.
It looks like ES fires up ok:
[root@graylog2-server]# curl -XGET
'http://10.80.1.3:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
However when I try start graylog I get the following error;
2014-02-20 04:41:36,048 ERROR: org.graylog2.Main -
ERROR: No ElasticSearch master was found.
I'm not using unicast and I've got the correct name set in my graylog2.conf
and elasticsearch.yml. (I've attached them too)
Does anyone have any ideas?
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
# If you are running more than one instances of graylog2-server you have to
select one of these
# instances as master. The master will perform some periodical tasks that
non-masters won't perform.
is_master = true
# The auto-generated node ID will be stored in this file and read after
restarts. It is a good idea
# to use an absolute file path here if you are starting graylog2-server from
init scripts or similar.
node_id_file = /etc/graylog2-server-node-id
# You MUST set a secret to secure/pepper the stored user passwords here. Use at
least 64 characters.
# Generate one by using for example: pwgen -s 96
password_secret = 123123123123123123123123123123123
# the default root user is named 'admin'
# root_username = admin
# You MUST specify a hash password for the root user (which you only need to
initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If
you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 =
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951
# Set plugin directory here (relative or absolute)
plugin_dir = plugin
# REST API listen URI. Must be reachable by other graylog2-server nodes if you
run a cluster.
rest_listen_uri = http://127.0.0.1:12900/
# REST API transport address. Defaults to first non-loopback IPv4 system
address and port 12900.
# This will be promoted in the cluster discovery APIs and other nodes may try
to connect on this
# address. (see rest_listen_uri)
#rest_transport_uri = http://192.168.1.1:12900/
# Enable CORS headers for REST api. This is necessary for JS-clients accessing
the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources
from the server.
# This is disabled by default. Uncomment the next line to enable it.
#rest_enable_cors = true
# Enable GZIP support for REST api. This compresses API responses and therefore
helps to reduce
# overall round trip times. This is disabled by default. Uncomment the next
line to enable it.
#rest_enable_gzip = true
# Embedded elasticsearch configuration file
# pay attention to the working directory of the server, maybe use an absolute
path here
#elasticsearch_config_file = /etc/graylog2-elasticsearch.yml
elasticsearch_max_docs_per_index = 2000
# How many indices do you want to keep?
# elasticsearch_max_number_of_indices*elasticsearch_max_docs_per_index=total
number of messages in your setup
elasticsearch_max_number_of_indices = 20
# Decide what happens with the oldest indices when the maximum number of
indices is reached.
# The following strategies are availble:
# - delete # Deletes the index completely (Default)
# - close # Closes the index and hides it from the system. Can be re-opened
later.
retention_strategy = delete
# How many ElasticSearch shards and replicas should be used per index? Note
that this only applies to newly created indices.
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog2
# Do you want to allow searches with leading wildcards? This can be extremely
resource hungry and should only
# be enabled with care. See also:
http://support.torch.sh/help/kb/graylog2-web-interface/the-search-bar-explained
allow_leading_wildcard_searches = false
# settings to be passed to elasticsearch's client (overriding those in the
provided elasticsearch_config_file)
# all these
# this must be the same as for your elasticsearch cluster
#elasticsearch_cluster_name = graylog2
# you could also leave this out, but makes it easier to identify the graylog2
client instance
#elasticsearch_node_name = graylog2-server
Re: [graylog2] What am I missing here? ERROR: No ElasticSearch master was found.
Hey Tom,
it looks like you commented out both "elasticsearch_config_file" and
all "elasticsearch_*" config variables. This leads to your
graylog2-server running with default configuration parameters that
will most probably not match the elasticsearch.yml of your
ElasticSearch nodes.
I recommend you fill out the elasticsearch_* config variables in your
graylog2.conf and try it again. The "elasticsearch_config_file" method
should be skipped for beginners.
Generally please follow:
*
http://support.torch.sh/help/kb/graylog2-server/installing-graylog2-server-v0200-on-nix-systems
*
http://support.torch.sh/help/kb/graylog2-server/configuring-and-tuning-elasticsearch-for-graylog2-v0200
Hope that helps!
Thanks,
Lennart
On Thu, Feb 20, 2014 at 6:46 PM, Tom Taylor wrote:
> Just recently updating graylog and ES and I can't seem to get it to work.
>
> It looks like ES fires up ok:
>
> [root@graylog2-server]# curl -XGET
> 'http://10.80.1.3:9200/_cluster/health?pretty=true'
>
> {
>
> "cluster_name" : "graylog2",
>
> "status" : "green",
>
> "timed_out" : false,
>
> "number_of_nodes" : 1,
>
> "number_of_data_nodes" : 1,
>
> "active_primary_shards" : 0,
>
> "active_shards" : 0,
>
> "relocating_shards" : 0,
>
> "initializing_shards" : 0,
>
> "unassigned_shards" : 0
>
> }
>
>
> However when I try start graylog I get the following error;
>
> 2014-02-20 04:41:36,048 ERROR: org.graylog2.Main -
>
>
>
> ERROR: No ElasticSearch master was found.
>
>
> I'm not using unicast and I've got the correct name set in my graylog2.conf
> and elasticsearch.yml. (I've attached them too)
>
>
> Does anyone have any ideas?
>
> --
> You received this message because you are subscribed to the Google Groups
> "graylog2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [graylog2] Is elasticsearch-1.0.0 not supported?
No, we do not support ElasticSearch 1.0.0 yet because it came out when we were in the RC phase of our own 0.20.0 series already. We do of course push that with priority now. Thanks, Lennart On Thu, Feb 20, 2014 at 10:07 PM, Jon Dison wrote: > I updated my elasticsearch from 90.11 to 1.0.0 and now graylog2-server > refuses to start. I rolled back the installation to 90.11 and it now starts > again with the same elasticsearch.yml file in both cases. > I verified that elasticsearch starts and listens on 9200 and 9300 in both > cases. I have attached the graylog-server startup debug along with the > elasticsearch.yml > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [graylog2] Is elasticsearch-1.0.0 not supported?
Thanks... So is that something we'd be looking for in like a 0.20.1 version or not until 0.30.0 ? Curious what changed between elasticsearch 0.90 and 1.00 that doesn't allow graylog2 to work any longer. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
