Hi,
On 3/28/23 14:07, Roy Smith wrote:
Hmmm. What I'm doing requires Page.expand_text(), which looks like it
does a Page.get() followed by a Site.expand_text(), and it's the later
which actually takes most of the time. That becomes an
action=expandtemplates API call
<https://www.mediawiki.org/w/api.php?action=help&modules=expandtemplates>, which I don't see any way to batch.
Naively you can batch action=expandtemplates by putting unique markers
in between each fragment of wikitext if you know the template will emit
balanced wikitext (this part is key).
For example:
{{foo1}}
--mycoolbot123--
{{foo2}}
--mycoolbot123--
{{foo3}}
Then split the response on your marker and sync up with your list of
pages :)
I suppose it's also worth asking what you're using expand_text() for in
the first place, to see if there's a better way to do whatever it is you
want to :)
For real concurrency I'll refer you to the blog post I wrote last year
about this:
<https://blog.legoktm.com/2022/01/21/building-fast-wikipedia-bots-in-rust.html>.
Yes it's about Rust, but the same concepts can easily be replicated in
Python.
-- Kunal / Legoktm
_______________________________________________
pywikibot mailing list -- pywikibot@lists.wikimedia.org
Public archives at
https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/FFCASNJI6YIR75KYT4TLZIURKAAX3PHX/
To unsubscribe send an email to pywikibot-le...@lists.wikimedia.org
