Make mdc properties top-level attributes in decanter elasticsearch appender

[dschulten]

Decanter produces mdc attributes as a nested structure. The behaviour in the
official  Elasticsearch ecs json format
and with the 
logstash-logback-encoder
   is different: the
mdc attributes become top-level fields.

I have found the behaviour of decanter only on the
https://github.com/vy/log4j2-logstash-layout.

Is it possible to pull up the mdc fields so that they become top-level
attributes?

A workaround would of course be to put logstash in front of Elasticsearch,
but I would rather avoid that.

Cheers,
Dietrich





--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

Limiting the decanter message size for elasticsearch

[dschulten]

Hi,

the other we ran into a size problem with Decanter. We received the
following from ES:

 

Now I wonder if it is possible to limit the message size somehow. I am
unsure how it happened, apparently there was a set of log events which was
so large as the message claims (475 MB).

In org.apache.karaf.decanter.collector.elasticsearch.cfg one can define a
from and max value to "square" the query. Could that be used to prevent this
situation?

* is the *query *property necessary so that *from *and *max * can have any
effect?
* how do I define a query?
* what is a valid value for *from*? A timestamp and if so, in which format?
* can *max *be used to limit the number of log events which will be sent in
one go?

Cheers,
Dietrich




--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

Decanter Elasticsearch ReST appender network traffic

[dschulten]

Hi,

I have combined 

decanter-collector-log
decanter-appender-elasticsearch-rest

and decanter sends the log events to my elasticsearch nicely. However, I am
a bit concerned about network traffic and blocking rest requests. I am not
entirely sure, but the code just seems to make blocking http requests to ES.

The ES client benchmark
https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client
shows that up to 30.000 ingestions/s could be doable under optimal
circumstances, that seems not all that much.

How does the ES Rest Appender send the log events? What if ES ingestion
slows down?
Does the Rest Appender do anything to cope with heavy load on the network or
reduce the call frequency?
Are there any advisable strategies which have been proven to work?

Best
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html

Re: Karaf community Meetup?

[dschulten]

Hi,

I was able to add myself to the Doodle - does that mean I can attend just
like that? Or should I register somewhere?

Best,
Dietrich



--
Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html