Re: [Wikitech-l] Using wiki pages as databases

2013-02-22 Thread Johnuniq
On Feb 20, 2013 at 3:54 pm, Tim Starling wrote:
 The idea of storing a database in a large string literal could
 be made to be fairly efficient and user-friendly if a helper
 module was written to do parsing and a binary search.

I have implemented the above suggestion with some promising results.
Packing a large table in a string and unpacking it on demand appears
to work well, and the data is accessed as if it were stored in a
standard table. Using the table from Wiktionary Module:Languages
mentioned earlier in this thread, testing shows that accessing the
packed data is 20 times faster. Info is at

http://test2.wikipedia.org/wiki/User_talk:Johnuniq#Big_tables

Johnuniq

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Using wiki pages as databases

2013-02-19 Thread Johnuniq
On Feb 19, 2013 at 9:11 PM, MZMcBride wrote:
 https://en.wikipedia.org/wiki/Module:Convertdata

I'm guilty of that, and what's been worrying me is that there are
hundreds more units to add. Some guidance on using Lua as a database
would be very desirable.

Quick tests suggest that if {{convert}} is used 100 times on a page
(where that template invokes Module:Convert, which requires
Module:Convertdata), then Convertdata is loaded 100 times. I've
wondered if there might be a pragma in a module like that to set read
only (at least a promise of read only, even if it were not enforced),
then more aggressively cache the bytecode so it is loaded once only
per page render, or even once only until the cache memory is flushed.

Or, if performance due to such module abuse is a problem, the data
could be split into, say, ten modules, and the code accessing the data
could work out which of the smaller data modules needed to be
required. I'm not going to worry about that until I have to, but some
guidance would be good.

I just had a quick look at one test page which invokes the module 66
times, and the NewPP limit report in the html source says Lua time
usage: 0.324s (5 ms/invoke).
http://en.wikipedia.org/wiki/Template:Convert/testcases/bytype/time

Johnuniq

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l