[
https://issues.apache.org/jira/browse/JAMES-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713154#comment-17713154
]
Benoit Tellier commented on JAMES-3777:
---------------------------------------
Regarding `enriching the event model` we in my opinion...
-> Do not need to change command. This guaranties no changes in upper level
APIs (JMAP) which is a nice roperty to get... (Avoids breaking changes).
-> Do not need to capture the full complexity of "list updates". We likely can
delivery very nice results by handling "prepending rules" and "post-pending
rules" which likely would capture the behavior of implemented UIs, and default
to "reset rules" when no smarter incremental representation can be found.
This saves the effort of capturing complicated changes within the aggregate.
We can advise our users against updates other than those done incrementally.
Finally if I am asked "how to capture the full complexity of list updates
incrementally" I would likely resort to "capturing incremental updates of
element presents (as a set)" and record elements order on each event (JSON list
of the ids). Still a quadratic complexity but it should nonetheless
significantly reduce event size, and complement nicely the above
prepend-postpend proposal. I am not planning to contribute this, just
documenting the possibility to do it.
> Event sourcing - O[n²] storage for filters
> ------------------------------------------
>
> Key: JAMES-3777
> URL: https://issues.apache.org/jira/browse/JAMES-3777
> Project: James Server
> Issue Type: Improvement
> Affects Versions: 3.7.0
> Reporter: Benoit Tellier
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Symptoms
> ```
> Largest Partitions:
> [FilteringRule/x...@linagora.com] 44952069 (45.0 MB)
> ```
> Every time this guy sends an email we load 45 MB of JSON, which can yield
> big performance impact.
> h2. What?
> We implemented event sourcing with reset. Given rule A, B if we want to
> persist rule C then we store a "reset to A, B, C" event.
> So, if we want to store N filter, the resulting structure with have a size
> depending of O[n²] which proves to be barely sustainable.
> h2. How to fix
> Coming back to O[n] likely would help.
> Implement filter addition / removal both at the storage and JMAP layer
> h2. Alternatives
> h3. The read projection
> Currently we are loading the full history, building the aggregate each time
> we process emails, and performing SERIAL lightweight transactions. Which is
> very common. And impactfull.
> It would be possible to introduce read projection, maintained by a
> subscriber to the event source, that would allow efficiently reading current
> filters for a given user.
> This mean the history would be loaded only upon writes, which are rare.
> Impact: yet another table. Also the solution is local to this usage and does
> not help other event sourcing usages.
> h3. Event sourcing snapshots
> Augment James event sourcing implementation with a Snapshot mechanism.
> Upon reading history, we would start reading available snapshots, then read
> the history from that snapshot.
> Event store would be responsible of taking snapshots. Even a one change out
> of 10 would do the job here.
> This implies being able to serialize state. This implies an additional table
> for storing event sourcing snapshots.
> My take on it: going `O[n2` -> `O[n]` will likely be a good enough mitigation
> that we don't need to grow the complexity of the event sourcing code.
> On the other hand, this ewould harden event sourcing code and likely lift
> most of the limitation for adoptions on the mailboxes write path (to enforce
> mailbox name unicity constraint).
> Note that both solutions are not exclusive.
> h3. The dirty fix
> For filters the history prior reset event can be dropped, this can be used to
> solve the immediate problem, even if it is not very clean.
> h1. Proposal
> - Implement a read projection
> - Implement addition / removal patches to filtering event sourcing aggregate
> - Don't implement event sourcing snapshots now
> And also... Remove the obligation to configure JMAP filtering mailet inside
> JMAP servers: after all this extension is not standard...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
