Yes but I was thinking the case where we are sending a lot of request without pause during slow down ingestion, may be it can have an impact on the ES side to deal with the ingestion. If we add a temporization, it can help ES server to deal more easily with the slow down and backup to normal state.
Sometimes it could end with a OOM because the ES slowdown never stop and EventAdmin become full. regards, François fpa...@apache.org Le 18/02/2020 à 20:05, Jean-Baptiste Onofré a écrit : > I think slow down is not a big deal as eventadmin dispatcher act as a > buffer (with timeout). > > So the current state is already good. The only drawback is about the > number of append / refresh performed on ES. That?s why bulk could be > interesting. > > Regards > JB > > Le mar. 18 f?vr. 2020 ? 15:31, Francois Papon > <francois.pa...@openobject.fr> a ?crit : > > Hi, > > There is some improvement on the ES7 client so may be we could update the > ES appender. > > Another solution is that we could detect slow down ingestion and add a > temporization but I also like the bulk usage :) > > regards, > > Fran?ois > fpa...@apache.org <mailto:fpa...@apache.org> > > Le 18/02/2020 ? 09:04, Jean-Baptiste Onofre a ?crit : > > Hi, > > That?s a good question ;) > > Currently, Decanter Elasticsearch appender sends each event followed > by a refresh. > > It directly uses Elasticsearch REST native client via performRequest. > > A possible improvement is to use bulk but it means that we might have > some delay between when the event actually occurs and when it?s stored in > Elasticsearch. Using bulk would dramatically decrease network usage. > I was thinking to introduce a property on the appender to allow user > to have the choice between bulk or atomic Elasticsearch update. > > Regards > JB > > Le 18 f?vr. 2020 ? 08:01, dschulten > <dietrich.schul...@escalon.de> <mailto:dietrich.schul...@escalon.de> a ?crit : > > Hi, > > I have combined > > decanter-collector-log > decanter-appender-elasticsearch-rest > > and decanter sends the log events to my elasticsearch nicely. > However, I am > a bit concerned about network traffic and blocking rest requests. > I am not > entirely sure, but the code just seems to make blocking http > requests to ES. > > The ES client benchmark > > https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client > <https://www.elastic.co/de/blog/benchmarking-rest-client-transport-client> > shows that up to 30.000 ingestions/s could be doable under optimal > circumstances, that seems not all that much. > > How does the ES Rest Appender send the log events? What if ES > ingestion > slows down? > Does the Rest Appender do anything to cope with heavy load on the > network or > reduce the call frequency? > Are there any advisable strategies which have been proven to work? > > Best > Dietrich > > > > -- > Sent from: > http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html > <http://karaf.922171.n3.nabble.com/Karaf-User-f930749.html> >
