[pywikibot] How to make mwparserfromhell recognize __TOC__

If I do: wikicode = mwparserfromhell.parse("This is\n__TOC__\nan article.") I get back a single Text node. For what I'm doing, it would be really convenient to have the __TOC__ be parsed as it's own node, i.e. instead of a single Text node with 'This is\n__TOC__\nan article.', I'd like to

[pywikibot] Re: Path between a category and an article

I once went down a similar rathole. I was trying to find all article which were (recursively) in Category:Music. That's when I discovered just what a mess the wiki category graph is. At first glance, it looks like a tree, but it's not. It's not even a DAG. It has cycles! And not just "it

[pywikibot] Re: Argument to IsRedirectPageError?

tation is wrong. I will > try to find out what was really meant; maybe the compat release can give a > hint. > > Thanks for investigating. > > xqt > > > > Am 31.03.2024 um 17:34 schrieb Roy Smith : > > > > The docs for BasePage.get() say: > > >

[pywikibot] Argument to IsRedirectPageError?

The docs for BasePage.get() say: pywikibot.exceptions.IsRedirectPageError – The page is a redirect. The argument of the exception is the title of the page it redirects to. but that doesn't seem to be what's actually happening: > from pywikibot import Site, Page > from pywikibot.exceptions

[pywikibot] Getting Node location in mwparserfromhell?

If I've got a Node out of Wikicode.filter(), is there a way to map that back to a location in the original parsed wikitext? Ideally, character offsets for the start and end of the text that makes up the Node. ___ pywikibot mailing list --

[pywikibot] Variable not allowed in type expression?

I'm trying to single-step through pywikibot using vscode. I've got: certifi==2024.2.2 charset-normalizer==3.3.2 idna==3.6 iniconfig==2.0.0 mwparserfromhell==0.6.6 packaging==23.2 pluggy==1.4.0 pytest==8.0.2 pytest-mock==3.12.0 pywikibot==8.6.0 requests==2.31.0 urllib3==2.2.1 vscode: Version:

[pywikibot] Testing against a docker-based wiki?

I periodically struggle with finding a good way to write unit tests for a pywikibot app. I've come to the conclusion that using unittest.mock to patch calls into pywikibot is just not the way to go. It's certainly hermetic, but the overhead of building tests is so high, it doesn't seem worth

[pywikibot] Re: Problems submitting a patch

the years, but somehow managed to never use git until fairly recently and I'm still getting up to speed on ithe fancier stuff. Thanks for the help. > On Jun 26, 2023, at 1:44 PM, YiFei Zhu wrote: > > On Mon, Jun 26, 2023 at 8:51 AM Roy Smith <mailto:r...@panix.co

[pywikibot] Re: Testing against live wikis

> On Jun 26, 2023, at 4:17 PM, Merlijn van Deen (valhallasw) > wrote: > > , there is a massive value in tests that test the integration with Wikimedia > wikis: it's great if mocked API calls work, but if it breaks against the real > API, Pywikibot is not doing its job for the user.

[pywikibot] Problems submitting a patch

I'm trying to follow the directions at https://www.mediawiki.org/wiki/Gerrit/Tutorial#How_to_submit_a_patch I made a branch (roysmith-T340447) and committed my patch there. Then I did "git pull --rebase origin master", but

[pywikibot] Testing against live wikis

I noticed that category_tests.py was failing for me. The breakage was due to this edit on enwiki , and fixed in ce301a2

[pywikibot] Re: the test environment?

> On May 15, 2023, at 3:11 PM, YiFei Zhu wrote: > > having tests depend on user-supplied user-config.py is just > ... broken. I hear you! So, still trying to find my way around the test suite, I see appveyor.yml. I don't know anything about Appveyor, but it looks like this is building a

[pywikibot] the test environment?

The past few days, I've worked on T336630 and T336624, both of which describe failures I was seeing which other people weren't. In both cases, it turned out to be caused by something in my user-config.py. Many of these tests also rely on external resources (i.e. live testing against testwiki

[pywikibot] test_follow_redirects fails

I've checked out /pywikibot/core.git. When I run the tests, I get a number of failures: FAILED tests/http_tests.py::TestHttpStatus::test_follow_redirects - AssertionError: '//en.wikipedia.org/wiki/Main_Page' not found in 'https://en.wikipedia.org/wiki/Main%20Page' FAILED

[pywikibot] Re: Place for Wikipedia in the framework

I'm relatively new to pywikibot, so I'm not really familiar with the culture of the project, i.e. which new features are accepted and which are rejected. But, yes, I did comment on the phab ticket that I thought this didn't belong as a part of the Page class. Let me give some of my more

[pywikibot] Re: Parallel gets?

one is the one that counts. > On Mar 28, 2023, at 10:57 PM, John wrote: > > If you are looking to check for image/file usage it’s better to query the api > for just image used images instead of trying to parse wiki text > > On Tue, Mar 28, 2023 at 10:29 PM Roy Smith <

[pywikibot] Re: Parallel gets?

On Mar 28, 2023, at 9:09 PM, Kunal Mehta wrote: > I suppose it's also worth asking what you're using expand_text() for in the > first place, to see if there's a better way to do whatever it is you want to > :) That's a fair question. What I'm doing is looking at DYK nominations to evaluate

[pywikibot] Re: Parallel gets?

Without the screenshot this time > On Mar 28, 2023, at 2:01 PM, Roy Smith wrote: > > Hmmm. What I'm doing requires Page.expand_text(), which looks like it does a > Page.get() followed by a Site.expand_text(), and it's the later which > actually takes most of the time

[pywikibot] Re: Parallel gets?

Mostly for the sake of the archives, I ended up with this , which is pretty much straight out of the example in the python library manual: > def _is_approved(nom): > return

[pywikibot] Re: Parallel gets?

gt; > Best > xqt > >> Am 27.03.2023 um 22:06 schrieb John : >> >>  >> I’ve not checked in the v3+ version but there used to be a preload page >> generator that batch retrieved stuff from the API. Then pass the preloaded >> page object on to the

[pywikibot] Parallel gets?

I need to issue a bunch of Page.get() requests in parallel. My understanding is that pywikibot uses the requests library which is incompatible with async_io, so that's out. So what do people use? Threading ? Or, I see there's an async_io

[pywikibot] How to get enclosing section with mwparserfromhell?

To my surprise, wikicode.get_parent() does not get you the section a node is part of: > import mwparserfromhell as mwp > > text = """==foo== > {{Template:Foo}} > """ > wikicode = mwp.parse(text) > print(wikicode.get_tree()) > > print('++') > > node = wikicode.nodes[-2] >

[pywikibot] Re: How to avoid leaving extra blank lines when removing templates?

tected the > template you want to delete, set a flag, and delete following text nodes, > until the next template node is detected. > > On Tue, 21 Mar 2023, 18:28 Roy Smith, <mailto:r...@panix.com>> wrote: > I've got some code which is essentially: >

[pywikibot] How to avoid leaving extra blank lines when removing templates?

I've got some code which is essentially: > wikicode = mwp.parse(self.page.get()) > for node in wikicode.filter_templates(recursive=False, matches=title): > wikicode.remove(node) > self.page.text = str(wikicode) > self.page.save() which works, but it

[pywikibot] Failing unit tests?

I'm gearing up to do some work (hopefully dive into fixing https://phabricator.wikimedia.org/T326650). I've gotten as far as closing the repo and running the existing unit tests. I get 4 failures: FAILED tests/make_dist_tests.py::TestMakeDist::test_handle_args - AssertionError:

[pywikibot] Re: Multiple users in user-config.py?

Hmmm. I was reading through that code earlier today, and I assumed I was just in the wrong place. I got as far as: >exec(compile(f.read(), _filename, 'exec'), _exec_globals) and paused. I had been wondering about how user-config.py was getting imported, since that's not a valid module

[pywikibot] Re: Multiple users in user-config.py?

: Bináris > Gesendet: 24.02.2023 04:27 > An: Pywikibot discussion list > Betreff: [pywikibot] Re: Multiple users in user-config.py? > > > > Roy Smith mailto:r...@panix.com>> ezt írta (időpont: 2023. > febr. 24., P, 0:42): > My bot needs to be abl

[pywikibot] Re: Multiple users in user-config.py?

__init__ > raise NoUsernameError( > pywikibot.exceptions.NoUsernameError: ERROR: username for wikipedia:test is > undefined. > If you have a username for that site, please add a line to user config file > (user_config.py) as follows: > usernames['wikipedia']['test'] = 'myUser

[pywikibot] Multiple users in user-config.py?

My bot needs to be able to run as either of two different accounts depending on the task. DYKToolsBot for most things, DYKToolsAdminBot for specific tasks that need admin rights. How do I do this? I'm not using the pwb.py driver layer; I've got my own top-level driver code and just import

[pywikibot] Re: Checkuser

It's based on mwclient instead of pywikibot, but I do have some code that uses the checkuserlog API. https://github.com/roysmith/spi-tools/blob/2080f436d6aeb7d49346241c3dfa356d479b2d19/wiki_interface/wiki.py#L389

[pywikibot] Re: Unable to commit

. > On Feb 20, 2023, at 12:21 PM, Roy Smith wrote: > > I've used a few VCS's in my time. Unless I've forgotten something, RCS, CVS, > SVN, ClearCase, Perforce, hg, and most recently git. I can't argue that git > is complicated. Probably more complicated than most people need.

[pywikibot] Re: Unable to commit

I've used a few VCS's in my time. Unless I've forgotten something, RCS, CVS, SVN, ClearCase, Perforce, hg, and most recently git. I can't argue that git is complicated. Probably more complicated than most people need. > On Feb 20, 2023, at 12:11 PM, i...@gno.de wrote: > > Hi Binaris, > > I

[pywikibot] Re: Alternative tags for italic and bold with mwparserfromhell?

Cool, thanks. > On Feb 6, 2023, at 8:05 PM, Kunal Mehta wrote: > > Hi, > > On 2/6/23 18:59, Roy Smith wrote: >> That only works on the content of an entire page, doesn't it? I've just got >> a short snippet of wikitext that I've already parsed out of a page.

[pywikibot] Re: Alternative tags for italic and bold with mwparserfromhell?

That only works on the content of an entire page, doesn't it? I've just got a short snippet of wikitext that I've already parsed out of a page. Specifically, a proposed DYK hook, such as: > that '''[[Edward A. Hanna]]''' wanted to eliminate the [[New York State > Assembly]] while he served

[pywikibot] Alternative tags for italic and bold with mwparserfromhell?

I want to turn wikitext into HTML for display on a web front-end I'm building. For what I'm doing, all I need is a few constructs like wiklinks, bold, and italic, which I'm able to do with a smallish amount of mwparserfromhell code. The one annoyance I've got now is I'm using bootstrap

[pywikibot] How to ignore all the non-visible stuff?

[[:en:Template:Did you know/Queue/NextPrep]] contains: > 4 > {{documentation|content=This number indicates the next DYK prep set to move > into the queue.}} > What I want to get is just the "4". Is Page.extract() what I'm looking for? Experimentally, it does what I want, but it's not clear

[pywikibot] Re: Text between two comments?

start and end represent comment objects you found from > wikicode.filter_comments() > start_index = wikicode.index(start) > end_index = wikicode.index(end) > inside = wikicode.nodes[start_index:end_index] > > > On Thu, Feb 2, 2023 at 3:39 PM Roy Smith <mailto:r...@panix.com>> wrote

[pywikibot] Text between two comments?

I'm trying to parse DYK prep area templates, for example Template:Did you know/Preparation area 3 . Unfortunately, these are more like flat text files than any kind of nicely structured data. The stuff of interest is

[pywikibot] Pruning category search?

Is there a way to recurse through a category, but excluding specific sub-cats? For example, I want to find all the templates in [[Category:People and person infobox templates]], except that I don't want to recurse into [[Category:Styles infobox templates]].

[pywikibot] Re: Viztracer

I see my original message was held up in moderation due to the large attachment. I've uploaded the image to File:Screenshot of viztracer output.png <https://commons.wikimedia.org/wiki/File:Screenshot_of_viztracer_output.png> > On Jan 13, 2023, at 6:38 PM, Roy Smith wrote: >

[pywikibot] Re: Simplifying logging?

I've opened https://phabricator.wikimedia.org/T326650 <https://phabricator.wikimedia.org/T326650> on this. > On Jan 4, 2023, at 7:25 PM, Roy Smith wrote: > > PS, quoting from > https://docs.python.org/3.9/howto/logging.html#logging-advanced-tutorial > <https://d

[pywikibot] Re: Simplifying logging?

you might well interfere with their ability to carry out > unit tests and deliver logs which suit their requirements. > On Jan 4, 2023, at 7:14 PM, Roy Smith wrote: > > Pywikibot, even if you're just using it as a library, configures its own > complicated logging structure: >

[pywikibot] Simplifying logging?

Pywikibot, even if you're just using it as a library, configures its own complicated logging structure: o "pywiki" | Level Level 11 | Propagate OFF | Handler (INFO)> | Level INFO | Filter | Formatter fmt='%(message)s%(newline)s' datefmt=None |

[pywikibot] OAuth errors?

I found these in my dykbot-cron.err file. What causes this? Is it something to worry about? WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: ERROR: Retrying failed OAuth authentication for wikipedia:en: The

[pywikibot] Next node?

Let's say I've got the following wikitext: Some stuff More stuff {{MyTemplate}} Blah Blah I want to find {{MyTemplate}} but only if it's immediately followed by , and insert something after the comment. Is there anything that does that kind of pattern matching? I can use mwparserfromhell to

[pywikibot] Re: How to install in toolforge?

handy. > On Dec 29, 2022, at 3:43 AM, Kunal Mehta wrote: > > Hi, > > On 12/28/22 10:31, Roy Smith wrote: >> How do people install their bots into toolforge? In the past, I've just >> done a "git pull" to the bastion host and run with the code directl

[pywikibot] How to install in toolforge?

How do people install their bots into toolforge? In the past, I've just done a "git pull" to the bastion host and run with the code directly out of the source tree, but I'm not sure that's a good plan. My current project is a more complicated, with a bot that I'm going to run on krb as a

[pywikibot] Re: What IDE for Mac desktop?

Well, I've been using vs code for a month now, and I gotta say (much to my surprise) I'm really happy with it. I keep learning new tricks. TIL that you can configure the file explorer to ignore things like build temp directories, egg-info files, etc. But even cooler than that, there's a

[pywikibot] How to edit template parameters?

I'm trying to implement the idea that was suggested in this edit. I see how I can find the existing {{DYK-Tools-Bot was here}}, with Page.templatesWithParams(). But once I've

[pywikibot] Re: What's with the APISite.encodings() magic?

only e.g. site.encodings() > calls site.family.encodings(site.code). > > Best > xqt > >> Am 12.12.2022 um 02:50 schrieb Roy Smith : >> >> I've been beating my head against the wall trying to figure out how to >> build a Mock of APISite. If I do the obvious: &g

[pywikibot] What's with the APISite.encodings() magic?

I've been beating my head against the wall trying to figure out how to build a Mock of APISite. If I do the obvious: site = mocker.MagicMock(spec=APISite) I end up with: > AttributeError: Mock object has no attribute 'encodings' Yet, if I hand-build a Site object, sure enough it does:

[pywikibot] Re: pywikibot is making a mockery of my tests

I'll just chalk that up to "Every time I finally think I understand how unittest.mock works, I discover something new". > On Nov 29, 2022, at 10:49 AM, Roy Smith wrote: > > That's a good observation, thanks, but fixing that doesn't change anything. > This: > > def t

[pywikibot] Re: pywikibot is making a mockery of my tests

be changed. Either change ` assert > site.xxyyzz ` to ` assert mock_Site.xxyyzz ` or ` assert > mock_IPv4Address.xxyyzz ` to ` assert ip.xxyyzz `. > > On Tue, Nov 29, 2022 at 5:19 AM Roy Smith <mailto:r...@panix.com>> wrote: > I've got: > > platform darwin -- Pyt

[pywikibot] pywikibot is making a mockery of my tests

I've got: platform darwin -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0 rootdir: /Users/roy/dev/dyk-tools, configfile: src/pages/pytest.ini plugins: mock-3.10.0, socket-0.5.1 When I run the following: def test_ip(mocker): mock_IPv4Address = mocker.patch('ipaddress.IPv4Address',

[pywikibot] Re: What IDE for Mac desktop?

Es because, despite once > being a bit of an open source zealot, I've been thoroughly snared by MS. > > On Sat, 26 Nov 2022 at 20:47, Roy Smith <mailto:r...@panix.com>> wrote: > I never thought I'd ever write this, but after close to 40 years of using > emacs for everythi

[pywikibot] What IDE for Mac desktop?

I never thought I'd ever write this, but after close to 40 years of using emacs for everything, I'm thinking of switching to a real IDE for python development. My latest evolution is emacs with elpy, which is pretty powerful as these things go, but I seem to spend more time configuring emacs

[pywikibot] Application testing strategies?

I'm diving into my first serious PWB application. I'm using it as a library to replace mwclient. I'm not sure how to approach testing. I'd really like something which is isolated from making actual API calls (i.e. hermetic unit tests). Does there exist a mechanism to do that? What I've

[pywikibot] Re: How to analyze block logs?

There's a bunch of block reason templates such as {{blocked proxy}}, {{webhostblock}}, etc. They don't always get used, but they're good things to look for before you fall back to just comment_text like '%proxy%' I don't think you need to check for ipb_expiry != 'infinity'. People who know

[pywikibot] Re: How to analyze block logs?

You may be able to do this with a SQL query against the database. https://quarry.wmcloud.org/query/68543 I think this does what you want, with the proviso that once a block expires, it falls out of the ipblocks table, so you'll be able to see blocks that

[pywikibot] Re: authenticate as bot and sysop

I'm a total newbie at pywikibot, but let me address the general issue of putting credentials in config files. Be very wary of how you store credentials. You should make sure they're in a file which is read-protected, and not in source control. Otherwise the entire world will have access to

[pywikibot] Does total always mean the same thing?

I'm just starting to use pywikibot. I see that many of the methods of site have an optional "total" argument. In some places, it's documented as: > if not None, limit the generator to yielding this many items in total In other methods, it just says: > number of pages to return In some

[pywikibot] pywikibot vs mwclient?

I've been using mwclient for eons. I'm wondering if it's worth my switching to pywikibot. What's different/better about pywikibot that would justify the switch? Is pywikibot a fork of mwclient or a clean start? I found some comparisons on