I was going to ask about utilizing HTTP/2 and its multiplexing capabilities as a potential way of helping. I don't know if we're doing this or not. Vert.X 3.3 and above support HTTP/2.
Another option, aside from the Kafka approach, if we decide to look in that direction, might be to leverage the vertx event bus for these backend communications. On 2024/08/29 14:07:33 Francisco Javier Tirado Sarti wrote: > Hi Enrique, > Thanks for the quick response. > Ill open an issue to modify the current http connector and add an option to > group http events. There are a couple of challenges, but they are mostly > implementation details. > > On Thu, Aug 29, 2024 at 3:29 PM Enrique Gonzalez Martinez < > [email protected]> wrote: > > > Hi Francisco, > > Composing a large message should be ok. There is no need for awareness > > as long as you create the collection in a different way. I mean > > If you are using one collector can create this composed message and > > the end can unroll the message. Users should be aware which one is > > using. and default should be the current system. > > > > So far so good for me. It should not have a large impact. As far as I can > > see. > > > > 1. composed message > > 2. modify the collector > > 3. endpoint being able to detect the composed message and route to the > > right internal services. > > > > There is a concern about the throughput of the data index though. As > > probably the problem is more related to the database cycle of > > updating. This part should be investigated further in the future in > > another thread. > > If you have something to share about those tests we can discuss it in > > another ML thread. > > > > Cheers :) > > > > El jue, 29 ago 2024 a las 11:53, Francisco Javier Tirado Sarti > > (<[email protected]>) escribió: > > > > > > Hi all, > > > > > > This rationale applies in a deployment scenario where we have > > > *runtimes *running > > > in a pod and *data index* is running in another pod and quarkus http > > > connector (based on Smallrye) is used for sending events from *runtimes > > *into > > > *data index* > > > > > > In that deployment scenarios, after process instance events were split > > with > > > https://github.com/apache/incubator-kie-issues/issues/249, an http > > request > > > to start a process over runtimes pod is followed by a variable number of > > > http request to data index pod in order to send these multiple events > > (one > > > per event). > > > > > > That affects system overall throughput, because *runtimes *eventually > > start > > > queuing http requests since* data index* cannot handle them at the same > > > speed. > > > > > > Apart from the obvious solution, to use a different communication > > mechanism > > > to send events (kafka or the so called single write, which basically > > > consist on runtimes instance updating data index tables within the same > > db > > > transaction, which obviously implies that both runtimes and dataindex use > > > the same db), I feel we should try to improve the HTTP communication > > > scenario by grouping http request. > > > > > > Since EventPublisher already has the events of a UnitOfWork already > > grouped > > > into a collection, but the Smallrye interface accepts one message per > > call; > > > in order to group them, I foresee these two option: > > > 1) Consolidate EventPublisher event collection into one single smallrye > > > message > > > 2) Change the http quarkus connector to group the messages that are being > > > received one by one using a time heuristic approach (as old legacy JMS > > used > > > to do) > > > > > > First approach is optimal from an application point of view but implies > > > changing current consumers to be aware of the fact that a message can > > > contain an array of DataEvent. > > > Second approach is transparent to existing consumers (they still receive > > > one message per event), but implies some delay before the http request is > > > performed by the emitter. > > > > > > Wdyt? > > > > > > > > -- > > Saludos, Enrique González Martínez :) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
