| jcrespo added a comment. |
This is my view of the issue, based on the comments above.
Short term:
- If the number of watchlist items for the users is less than N (N to be determined), join Watchlist -> recentchanges, then "suffer" a small in-memory sort (I belive this is the current situation)
- Else, if the number of recentchanges is less than M, join recentchanges -> Watchlist (we may need to force the JOIN direction)
- Else, scan recentchanges in inverse timestamp order, paged in small batches of size S, and join in the direction small recentchanges batch -> watchlist. It will not be more efficient, but it will get results faster if they are very common (and queries will not timeout).
Long term, I do not see elastic helping (it could help with advanced search, like tags/multiple filtering options, but I do not see it for live querying per user- the point is to avoid "prerendering" of watchlist and always get live results). However, I proposed in the past have a separate subsystem to track global changes and usage (for thing like wikidata and commons), separate from the main metadata databases. Polluting 85%-95% of recentchanges is a bit too much. By making it shared between projects but a different "master", maybe it could be more efficient than now, were probably lots of large recentchanges are being appended to every single project. Querying 2 services could be faster and definitely would improve the amount of writes per master, too. If that is a large scope goal, we could create a "wikibaserecentchanges" for wikibase clients, and separate the entries for each source. It will not be pretty, but at least it would allow people with a large watchlist to make rcs work disabling the "show wikidata changes".
Cc: Ladsgroup, Lsanabria, Josve05a, Bawolff, greg, Dominicbm, Vriullop, Jmabel, Fae, IKhitron, Johan, Herzi.Pinki, jmatazzoni, Trizek-WMF, Mattflaschen-WMF, KTC, Framawiki, zhuyifei1999, Marostegui, aaron, Andrei_Stroe, Turbojet, Rsocol, Strainu, Jwh, Pyb, Darwinius, Arbnos, Jdforrester-WMF, TheDJ, gerritbot, VladXe, Kf8, Liuxinyu970226, Jay8g, TerraCodes, Iniquity, jcrespo, Reedy, Catrope, Vicpeters, Demidenko, MaxBioHazard, Aklapper, GoranSMilovanovic, Abiyoyo, QZanden, Vali.matei, Jack_who_built_the_house, Poyekhali, Volker_E, Izno, Wong128hk, Luke081515, Wikidata-bugs, Base, aude, GWicke, El_Grafo, Gryllida, putnik, Steinsplitter, Mbch331, Krenair
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
