Hi Tiago and Francisco,
Thanks a lot for your input - this is really helpful!
@Tiago, I ran the test again with your latest fix included, and the results are
much better now:
+ No more exceptions, no more deadlocks
+ All requests were successful
+ Performance increased (*)
- I see this WARN message [1] for each created process
If it helps (perhaps for further improvements), here are some numbers from
Hibernate regarding inserts/updates (for 1002 created process instances):
4 insert into definitions_addons
1 insert into definitions
24 insert into definitions_nodes_metadata
12 insert into definitions_nodes
10020 insert into nodes
4008 insert into processes_addons
1002 insert into processes
168336 update nodes set ...
23046 update processes set ...
@Francisco - I will look out for your PR.
(*) My tests are far from being a full load test, it really just scratches the
surface to see how things behave with a bit of concurrency. If anyone has
pointers on existing load tests, benchmarks, etc, please let me know.
Thanks!
Martin
[1]
https://github.com/apache/incubator-kie-kogito-runtimes/blob/main/addons/common/persistence/jdbc/src/main/java/org/kie/kogito/persistence/jdbc/JDBCProcessInstances.java#L65
________________________________________
From: Francisco Javier Tirado Sarti <[email protected]>
Sent: Tuesday, November 21, 2023 12:06 PM
To: [email protected]
Cc: [email protected]
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
persistence addon
Hi Martin,
I have a task to review performance of
ProcessInstanceNodeDataEventMerger
My idea is to reduce the number of delete inserts when processing events
and try to do it incremental.
That should improve performance.
PS:
I was planning to send an e-mail tomorrow announcing that in case you were
already working on a fix for that. I assume you are not and I would be
sending a PR soon.
On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler <[email protected]>
wrote:
> I looked into the new examples using data-index persistence addon - Neus'
> PR#1813 [1] for serverless and Pere's branch [2] for workflow (great job
> both!) - and they work without issues using single requests. However, under
> some load (I used 'ab' for testing with a light concurrency of 10 parallel
> requests) I ran into the following problems:
>
> (1) Large number of insert/delete calls (eg. for tables such as nodes,
> definitions, etc)
>
> (2) Hibernate OptimisticLockExceptions / StaleStateExceptions
>
> (3) DB deadlocks
>
> (4) Error responses, slow response times
>
> The reason I am reaching out with this topic here is to find out if we are
> aware of this issue, and if someone is already looking into or being
> assigned to it?
>
> Thanks,
> Martin
>
> [1] https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> [2]
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
