Hi Enrique, Thanks for the quick response. Ill open an issue to modify the current http connector and add an option to group http events. There are a couple of challenges, but they are mostly implementation details.
On Thu, Aug 29, 2024 at 3:29 PM Enrique Gonzalez Martinez < [email protected]> wrote: > Hi Francisco, > Composing a large message should be ok. There is no need for awareness > as long as you create the collection in a different way. I mean > If you are using one collector can create this composed message and > the end can unroll the message. Users should be aware which one is > using. and default should be the current system. > > So far so good for me. It should not have a large impact. As far as I can > see. > > 1. composed message > 2. modify the collector > 3. endpoint being able to detect the composed message and route to the > right internal services. > > There is a concern about the throughput of the data index though. As > probably the problem is more related to the database cycle of > updating. This part should be investigated further in the future in > another thread. > If you have something to share about those tests we can discuss it in > another ML thread. > > Cheers :) > > El jue, 29 ago 2024 a las 11:53, Francisco Javier Tirado Sarti > (<[email protected]>) escribió: > > > > Hi all, > > > > This rationale applies in a deployment scenario where we have > > *runtimes *running > > in a pod and *data index* is running in another pod and quarkus http > > connector (based on Smallrye) is used for sending events from *runtimes > *into > > *data index* > > > > In that deployment scenarios, after process instance events were split > with > > https://github.com/apache/incubator-kie-issues/issues/249, an http > request > > to start a process over runtimes pod is followed by a variable number of > > http request to data index pod in order to send these multiple events > (one > > per event). > > > > That affects system overall throughput, because *runtimes *eventually > start > > queuing http requests since* data index* cannot handle them at the same > > speed. > > > > Apart from the obvious solution, to use a different communication > mechanism > > to send events (kafka or the so called single write, which basically > > consist on runtimes instance updating data index tables within the same > db > > transaction, which obviously implies that both runtimes and dataindex use > > the same db), I feel we should try to improve the HTTP communication > > scenario by grouping http request. > > > > Since EventPublisher already has the events of a UnitOfWork already > grouped > > into a collection, but the Smallrye interface accepts one message per > call; > > in order to group them, I foresee these two option: > > 1) Consolidate EventPublisher event collection into one single smallrye > > message > > 2) Change the http quarkus connector to group the messages that are being > > received one by one using a time heuristic approach (as old legacy JMS > used > > to do) > > > > First approach is optimal from an application point of view but implies > > changing current consumers to be aware of the fact that a message can > > contain an array of DataEvent. > > Second approach is transparent to existing consumers (they still receive > > one message per event), but implies some delay before the http request is > > performed by the emitter. > > > > Wdyt? > > > > -- > Saludos, Enrique González Martínez :) > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
