2018-07-25 16:14 GMT+02:00 Guillaume Delhumeau < guillaume.delhum...@xwiki.com>:
> After more thinking and discussions with Thomas, it appears solution B > (creating a table with Hibernate) is the only valuable option. These > preferences are not real data that would need to be stored on the user > page. They are "temporary" configurations for each user. > > In addition, it would permit to optimize the HQL query by linking elements > of the 2 tables instead of introducing 700 "OR location = xxx". This > optimization is actually not a nice-to-have thing, but a real requirement > for scalability. > As a drawback, it means users' configurations would not be exportable in a XAR like the rest of the content... > 2018-07-20 21:05 GMT+03:00 Guillaume Delhumeau < > guillaume.delhum...@xwiki.com>: > >> XWIKI-15455 <http:///browse/XWIKI-15455> (that I have just created) >> invalidates both Solution A and C since it requires to store a Date along >> with each location, so we cannot store several locations in the same >> Notification Filter Preferences object. >> >> 2018-07-19 11:03 GMT+02:00 Guillaume Delhumeau < >> guillaume.delhum...@xwiki.com>: >> >>> FYI, I'll continue with the solution A, which is very close to solution >>> C, and I'll try to use the less memory possible. >>> >>> 2018-07-18 11:27 GMT+02:00 Guillaume Delhumeau < >>> guillaume.delhum...@xwiki.com>: >>> >>>> Corresponding JIRA issue: https://jira.xwiki.org/browse/XWIKI-15445 >>>> >>>> 2018-07-18 11:07 GMT+02:00 Guillaume Delhumeau < >>>> guillaume.delhum...@xwiki.com>: >>>> >>>>> Hi. >>>>> >>>>> [TL;DR] >>>>> >>>>> This thread is about the way we store notification filter preferences >>>>> for each user. The constraint is there can be a lot of them (700 is a >>>>> number a user has recently reported). So how should we store them? >>>>> >>>>> [Full text] >>>>> >>>>> = Definition = >>>>> >>>>> So what is a filter preference? It's a generic object that can store >>>>> many elements, such as a page locations, application names, event types, >>>>> etc... They describe a configuration about a given filter for a given >>>>> user. >>>>> For example, a filter preference can say "for the ScopeNotificationFilter >>>>> and the user A, include the location Main.WebHome" as it could be "for the >>>>> UserNotificationFilter and the user A, exclude the user SPAM". It's >>>>> generic. >>>>> >>>>> The main usage is for page locations (ScopeNotificationFilter). By >>>>> default, we have the "autowatch" mode enabled. It means every time a user >>>>> modifies a page, a filter preference for this page and this user is >>>>> created. So if a user modifies 700 pages, he gets 700 filter preferences. >>>>> >>>>> = How are they stored = >>>>> >>>>> Currently, we have a simple implementation. There is a generic XClass >>>>> called "XWiki.Notifications.Code.NotificationFilterPreferenceClass". >>>>> For each preference, we add an XObject on the user page. It's that simple. >>>>> But it also means that if a users have 700 filter preferences, she also >>>>> gets 700 XObjects on her page, and 700 revisions of that page. Which is a >>>>> pain: it takes a lot of place in the document's cache, and it's heavy to >>>>> load (lot of SQL queries needed). So we have a big problem here. >>>>> >>>>> = Possible solutions = >>>>> >>>>> == A: Minimize the number of xobjects needed for >>>>> ScopeNotificationFilter == >>>>> >>>>> Currently, one location is represented by 1 filter preference. But >>>>> most filter preferences are very similar. They almost all say "for the >>>>> ScopeNotificationFilter, for all event types, for all applications, the >>>>> filter preference is enabled". The only different part is the actual >>>>> location. But the "location" field is itself a LIST stored with the >>>>> "relational storage" option. So we can take advantage of it and store >>>>> similar preferences into 1 single object. >>>>> >>>>> 1 object with 700 locations instead of 700 objects with 1 location. >>>>> >>>>> However, it's a bit harder than this. Event if the >>>>> NotificationFilterPreferences is generic and can contains many locations, >>>>> the ScopeNotificationFilter expect it to concern only one location (and >>>>> then it perform complex operations to sort the filters preferences >>>>> according to a hierarchy). The UI in the user profile makes the same >>>>> assumption so it does not handle multiple locations in the same >>>>> preferences >>>>> object. Refactoring this is not simple and cannot be done for 10.6. >>>>> >>>>> === Variation 1: store only 1 xobject, but make the API return 700 >>>>> preferences objects anyway === >>>>> >>>>> This is the variation I am prototyping. Actually it's ok if the >>>>> filters and the UI expect only 1 location into the preferences object. All >>>>> we have to do is to "smash" the xobject into many >>>>> NotificationFilterPreferences objects that we need internally. It would >>>>> simply be the responsibility of the Store to detect similarities and to >>>>> save the minimal amount of XObjects to store a bunch of preferences. >>>>> >>>>> But it means being very smart when loading, creating, updating and >>>>> deleting a preference. Not having one xobject per filter preference >>>>> introduces complexity, and complexity can lead to bugs. Again, according >>>>> to >>>>> the time frame, it's hard to implement. >>>>> >>>>> === Variation 2: use custom mapping === >>>>> >>>>> Probably the easiest solution that would help making less SQL queries. >>>>> The idea is to have a SQL table for notification filter preferences and >>>>> bind the XObjects to that table. It would still use a lot of place in the >>>>> document's cache but be more efficient on the database level. >>>>> >>>>> === Other Problem 1: it still creates page revisions === >>>>> >>>>> As long as we store the filter preferences with xobjects, we create >>>>> page revisions. We can get rid of those by using some internal API to not >>>>> create a revision when we save an xobject but I wonder if it's what users >>>>> want. If a user tries to rollback some changes and don't see all filter >>>>> preferences it concerns, I think it's not very transparent. >>>>> >>>>> === Other Problem 2: Document's cache === >>>>> >>>>> Sometime we load the a user document to get the avatar of the user, >>>>> her name, etc... So we load user documents very frequently, even if the >>>>> user is not connected! Having 700 filters in the document and cache them >>>>> with the document even if we don't need them is a big waste of memory. >>>>> >>>>> == B: Implement a completely new store with Hibernate == >>>>> >>>>> A bit like having a custom mapping. We could create a SQL table and >>>>> implement an API to handle it. Then, no xobjects would be involved. >>>>> >>>>> Some drawbacks: >>>>> * we need to write a custom cache as well. >>>>> * the user cannot modify her preferences using the wiki principles >>>>> (xobjects all the way). >>>>> >>>>> == C: Refactor the UI and the ScopeNotificationFilter so they do not >>>>> assume 1 filter preference = 1 location == >>>>> >>>>> This option is still possible. Probably the best because creating 1 >>>>> filter preferences object per location is an obvious waste of memory. A >>>>> refactoring of the UI is needed anyway, because we currently have no way >>>>> to >>>>> remove a bunch of filter preferences easily (users have to delete the 700 >>>>> filters preferences manually) so we can kill 2 birds with the same stone. >>>>> >>>>> But again, it requires some work. >>>>> >>>>> = Conclusion = >>>>> >>>>> That's it. All possible solutions require development effort that is >>>>> hardly possible to make before 10.6 (and even 10.7, considering I would >>>>> probably be the one implementing it and I'm not fulltime on the subject >>>>> and >>>>> I have holidays soon). >>>>> >>>>> Writing this email helped me to see the problem with perspective. I >>>>> think solution C may be the best. But any opinion is good to hear (except >>>>> if you propose something even more complex than I do :p). >>>>> >>>>> Thanks, >>>>> >>>>> Guillaume >>>>> >>>>> >>>> >>>> >>>> -- >>>> Guillaume Delhumeau (guillaume.delhum...@xwiki.com) >>>> Research & Development Engineer at XWiki SAS >>>> Committer on the XWiki.org project >>>> >>> >>> >>> >>> -- >>> Guillaume Delhumeau (guillaume.delhum...@xwiki.com) >>> Research & Development Engineer at XWiki SAS >>> Committer on the XWiki.org project >>> >> >> >> >> -- >> Guillaume Delhumeau (guillaume.delhum...@xwiki.com) >> Research & Development Engineer at XWiki SAS >> Committer on the XWiki.org project >> > > > > -- > Guillaume Delhumeau (guillaume.delhum...@xwiki.com) > Research & Development Engineer at XWiki SAS > Committer on the XWiki.org project > -- Guillaume Delhumeau (guillaume.delhum...@xwiki.com) Research & Development Engineer at XWiki SAS Committer on the XWiki.org project
