Hi, That’s a good question ;)
Currently, Decanter Elasticsearch appender sends each event followed by a refresh. It directly uses Elasticsearch REST native client via performRequest. A possible improvement is to use bulk but it means that we might have some delay between when the event actually occurs and when it’s stored in Elasticsearch. Using bulk would dramatically decrease network usage. I was thinking to introduce a property on the appender to allow user to have the choice between bulk or atomic Elasticsearch update. Regards JB > Le 18 févr. 2020 à 08:01, dschulten <dietrich.schul...@escalon.de> a écrit : > > Hi, > > I have combined > > decanter-collector-log > decanter-appender-elasticsearch-rest > > and decanter sends the log events to my elasticsearch nicely. However, I am > a bit concerned about network traffic and blocking rest requests. I am not > entirely sure, but the code just seems to make blocking http requests to ES. > > The ES client benchmark > https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client > shows that up to 30.000 ingestions/s could be doable under optimal > circumstances, that seems not all that much. > > How does the ES Rest Appender send the log events? What if ES ingestion > slows down? > Does the Rest Appender do anything to cope with heavy load on the network or > reduce the call frequency? > Are there any advisable strategies which have been proven to work? > > Best > Dietrich > > > > -- > Sent from: http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html
