Hi Francisco, Composing a large message should be ok. There is no need for awareness as long as you create the collection in a different way. I mean If you are using one collector can create this composed message and the end can unroll the message. Users should be aware which one is using. and default should be the current system.
So far so good for me. It should not have a large impact. As far as I can see. 1. composed message 2. modify the collector 3. endpoint being able to detect the composed message and route to the right internal services. There is a concern about the throughput of the data index though. As probably the problem is more related to the database cycle of updating. This part should be investigated further in the future in another thread. If you have something to share about those tests we can discuss it in another ML thread. Cheers :) El jue, 29 ago 2024 a las 11:53, Francisco Javier Tirado Sarti (<[email protected]>) escribió: > > Hi all, > > This rationale applies in a deployment scenario where we have > *runtimes *running > in a pod and *data index* is running in another pod and quarkus http > connector (based on Smallrye) is used for sending events from *runtimes *into > *data index* > > In that deployment scenarios, after process instance events were split with > https://github.com/apache/incubator-kie-issues/issues/249, an http request > to start a process over runtimes pod is followed by a variable number of > http request to data index pod in order to send these multiple events (one > per event). > > That affects system overall throughput, because *runtimes *eventually start > queuing http requests since* data index* cannot handle them at the same > speed. > > Apart from the obvious solution, to use a different communication mechanism > to send events (kafka or the so called single write, which basically > consist on runtimes instance updating data index tables within the same db > transaction, which obviously implies that both runtimes and dataindex use > the same db), I feel we should try to improve the HTTP communication > scenario by grouping http request. > > Since EventPublisher already has the events of a UnitOfWork already grouped > into a collection, but the Smallrye interface accepts one message per call; > in order to group them, I foresee these two option: > 1) Consolidate EventPublisher event collection into one single smallrye > message > 2) Change the http quarkus connector to group the messages that are being > received one by one using a time heuristic approach (as old legacy JMS used > to do) > > First approach is optimal from an application point of view but implies > changing current consumers to be aware of the fact that a message can > contain an array of DataEvent. > Second approach is transparent to existing consumers (they still receive > one message per event), but implies some delay before the http request is > performed by the emitter. > > Wdyt? -- Saludos, Enrique González Martínez :) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
