| Addshore lowered the priority of this task from "High" to "Normal". Addshore added a comment. |
Setting this to normal now as the on server caching has reduced the traffic to the individual server, although it is still higher than other servers, although only 2-3x greater rather than 4+x greater.
The on server cache should provide steadier rates of access and spikes in mw requests should not result in spike in these requests to memcached.
We could increase the local cache from 10 seconds to 5 seconds and should see a decrease in cache traffic again.
We could also think about sharding the data in some way.
We could
- Have n keys containing the same data, and randomly choose one to load the data from when needed. This would share the load across the cache cluster more evenly.
- Shard based on data type. The lookups don't know what data type they want to load when requesting the data, so they would just have to load a shard, see if the property lies within the shard before trying the next one. This again would reduce the traffic on a single server but would result in more requests overall to get the same data.
- Shard based on property ID. Simply speaking we could shove the first 1-500 properties in 1 key, the next 500-1000 properties in another key. This would distribute the data among multiple cache servers reducing traffic. Also it would allow us to know which shard we want to look at instead of having to guess. Instead of hard coding this / using the numeric representations of property IDs we could also shard based on some hash function on the serialized property ID.
- have a key per property, I don't think this is a great idea, this would result it a much greater number of hits, single requests might need info on any number of properties. Currently the code doesn;t have a way of knowing how many properties we are going to request before we start requesting them. Of course we could re work this..
TASK DETAIL
EMAIL PREFERENCES
To: Addshore
Cc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, CucyNoiD, Nandana, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg
Cc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, CucyNoiD, Nandana, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
