Kevan Jahanshahi created UNOMI-753: -------------------------------------- Summary: Avoid loading items in RAM when possible, use updateByScript or scroll queries Key: UNOMI-753 URL: https://issues.apache.org/jira/browse/UNOMI-753 Project: Apache Unomi Issue Type: Improvement Reporter: Kevan Jahanshahi
There is place in Unomi code base where we load all profiles in RAM, for example: [https://github.com/apache/unomi/blob/master/extensions/lists-extension/services/src/main/java/org/apache/unomi/services/UserListServiceImpl.java#L88-L97] {code:java} List<Profile> profiles = persistenceService.query(query, null, Profile.class);{code} This can lead to OOM issue depending on the number of profiles in ES. But it can also lead to inconsistencies due to the usage of bulkProcessor update: {code:java} persistenceService.update(p, Profile.class, "systemProperties", profileSystemProperties);{code} the bulk processor is retaining the actions and will push them asynchronously. If a profile is modified in the meantime the bulkprocessor is retaining a profile to be updated, then when the bulk is performed it will overrides profile data, and this can lead to inconsistencies. 2 risks identified here, with this kind of pattern: * OOM: in case a lot of profiles in the user list * inconsistency due to asynchronous bulkprocessor overwritting the profiles systemProperties Others places may be impacted, and should be fixed in current ticket. To identify other places: - check usages of *persistenceService.update()* One usage have been updated already: Take a look at *MergeProfilesOnPropertyAction.reassignPersistedBrowsingDatasAsync* -- This message was sent by Atlassian Jira (v8.20.10#820010)