Bug#877103: ITP: python-rebulk -- Define simple search patterns in bulk to perform advanced matching on any string

[Etienne Millon]

Package: wnpp
Severity: wishlist
Owner: Etienne Millon 

* Package name: python-rebulk
  Version : 0.9.0
  Upstream Author : Rémi Alvergnat (https://github.com/Toilal)
* URL : https://github.com/Toilal/rebulk/
* License : MIT
  Programming Lang: Python
  Description : Define simple search patterns in bulk to perform advanced 
matching on any string

ReBulk is a python library that performs advanced searches in strings
that would be hard to implement using re module or String methods
only.

It includes some features like Patterns, Match, Rule that allows
developers to build a custom and complex string matcher using a
readable and extendable API.

This library is a dependency for new versions of python-guessit,
already in Debian and maintained by myself in the DPMT.

I intend to maintain it there as well, and comaintainers are more than
welcome.

Thanks

-- 
Etienne Millon

Re: providing sphinx3-* binaries

[Antoine Beaupré]

On 2017-09-28 19:59:12, Dmitry Shachnev wrote:
> On Thu, Sep 28, 2017 at 08:27:20AM -0400, Antoine Beaupré wrote:
>> > And moving Python 3 packages to /usr/bin/sphinx3-build or something like
>> > that will mean diverging from upstream (see below).
>>
>> Note that we already diverge from upstream in numerous places in Python,
>> first and foremost in the naming of the Python binary itself.
>
> How do we diverge there? In my opinion we follow PEP 394 quite closely:
> https://www.python.org/dev/peps/pep-0394/

Ah. Oops. :) I guess I was thinking of virtualenvs...

>> > Good suggestion, I have submitted a pull request upstream:
>> > https://github.com/sphinx-doc/sphinx/pull/4092
>>
>> Excellent - I meant to do that but ran out of time writing the email in
>> the first place. :)
>>
>> I see, however, it's not getting too much traction.. But they have a
>> fair point - I didn't realize there was a difference between variables
>> from the environment and from the commandline in Make. Maybe it's good
>> enough as it is then.
>
> It is not ‘they’, it is our Simon :) He also replied on this mailing list,
> just did not CC you.

Oops again. :) I replied there as well.

[...]

>> This is why I mention pybuild: maybe *that* is where docs should be
>> built? I am not sure. In any case, I feel there's a shim missing here,
>> and there's a good occasion to leverage the automation we have to
>> generate proper build deps and build-time commands.
>
> Maybe pybuild can do this, but it then needs to make some assumptions
> about the doc source path, build path, wished formats, etc. For finding source
> path it can probably use something like this:
>
> https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/setup_command.py#L111
>
> But it is up to pybuild maintainer (Piotr) to decide whether he wants this
> feature or not.

Well, it would sure be nice to have some place for this, of course.

[...]

>> > Also there is no such thing as sphinx3-build upstream (unlike other 
>> > packages
>> > that follow this convention). So all upstream projects will try to use
>> > sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
>> > override this command anyway, you can use two existing ways, I don’t see a
>> > need for the third one.
>>
>> The reasoning here is that is is more discoverable, namely through
>> commandline completion, and is consistent with the /usr/bin/python*
>> binary naming conventions.
>
> I have just figured out that there *is* bash completion for python3 -m
> syntax.

What i meant is that "sphinx" doesn't show there is a python2 vs
python3 version.

[...]

> Now it is explicit:
> https://anonscm.debian.org/cgit/python-modules/packages/sphinx.git/commit/?id=5b2efffcaae8c915

Excellent, thanks again.

A.

-- 
Pour marcher au pas d'une musique militaire, il n'y a pas besoin de
cerveau, une moelle épinière suffit.
- Albert Einstein

Re: providing sphinx3-* binaries

[Dmitry Shachnev]

On Thu, Sep 28, 2017 at 08:27:20AM -0400, Antoine Beaupré wrote:
> > And moving Python 3 packages to /usr/bin/sphinx3-build or something like
> > that will mean diverging from upstream (see below).
>
> Note that we already diverge from upstream in numerous places in Python,
> first and foremost in the naming of the Python binary itself.

How do we diverge there? In my opinion we follow PEP 394 quite closely:
https://www.python.org/dev/peps/pep-0394/

> > Good suggestion, I have submitted a pull request upstream:
> > https://github.com/sphinx-doc/sphinx/pull/4092
>
> Excellent - I meant to do that but ran out of time writing the email in
> the first place. :)
>
> I see, however, it's not getting too much traction.. But they have a
> fair point - I didn't realize there was a difference between variables
> from the environment and from the commandline in Make. Maybe it's good
> enough as it is then.

It is not ‘they’, it is our Simon :) He also replied on this mailing list,
just did not CC you.

> > And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
> > but it operates on already built and installed documentation, so it cannot
> > help with building.
>
> I guess this is what I mean should be automated better. Every time I
> used --with=sphinxdoc, I expected it to build the docs and then
> remembered, when the produced package lacked any docs whatsoever, that I
> needed to build things myself.
>
> A simple --with=sphinxdoc should be sufficient to build sphinx docs. We
> shouldn't expect upstream to do that step in setup.py, as it is rarely
> hooked into the main build. Yes, there's the `build_sphinx` command, but
> it's not hooked into `setup.py build` except in some rare project.

Sphinxdoc is a helper to run after install, not a build system. So it is
not a place for such code.

> This is why I mention pybuild: maybe *that* is where docs should be
> built? I am not sure. In any case, I feel there's a shim missing here,
> and there's a good occasion to leverage the automation we have to
> generate proper build deps and build-time commands.

Maybe pybuild can do this, but it then needs to make some assumptions
about the doc source path, build path, wished formats, etc. For finding source
path it can probably use something like this:

https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/setup_command.py#L111

But it is up to pybuild maintainer (Piotr) to decide whether he wants this
feature or not.

> > What is the problem with using 
> > /usr/share/sphinx/scripts/python3/sphinx-build
> > explicitly if you want to force Python 3?
>
> It's hard to find. It's unusal for a user-facing command to be absent
> from the path yet to expect users to find it, especially when one
> version *is* available in the path.

As I said ealier, for most cases just using sphinx-build which is on PATH
should be fine. In advanced cases, it should not be hard to run ‘dpkg -L
python3-sphinx’ to figure out where its scripts are, or use the -m syntax.

> Sure. I just found out about this while writing the email, after almost
> a decade of maintaining python packages. That's a clear symptom of bad
> discovery, although the upstream Makefile changes may help with that
> eventually.

Actually quite many packages tend to provide the -m syntax and even prefer
it to scripts in /usr/bin/. Examples from packages I maintain: -m markdown,
-m keyring, -m nose.

> > Also there is no such thing as sphinx3-build upstream (unlike other packages
> > that follow this convention). So all upstream projects will try to use
> > sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
> > override this command anyway, you can use two existing ways, I don’t see a
> > need for the third one.
>
> The reasoning here is that is is more discoverable, namely through
> commandline completion, and is consistent with the /usr/bin/python*
> binary naming conventions.

I have just figured out that there *is* bash completion for python3 -m
syntax.

It is not difficult for me to provide the sphinx3-* binaries, but that might
create false assumptions for developers that it is available on other systems
too. It is better to use the -m syntax because it is more universal and works
on any OS.

> > I won’t say that this convention is usual. There are many packages named
> > python-* or python3-* which provide executable scripts.
>
> I would argue that is a strange practice: I tend to ignore packages that
> start with python* unless I am programming something. As a user, they
> are basically useless because I am supposed to code something to use
> those libraries. Now, granted, sphinx is a little special because it
> *is* a programmer-oriented library, but keep in mind it can be used for
> non-programs as well (see the Debian policy for example).
>
> > Let me summarize what you, in my opinion, should do as a maintainer of
> > packages using Sphinx:
> >
> > 1) If your project is not using autodoc: just build-depend on python3-sphinx
> > 

Re: providing sphinx3-* binaries

[Antoine Beaupré]

On 2017-09-28 01:03:27, Dmitry Shachnev wrote:
> Hi Antoine, and thanks for the detailed mail!
>
> On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
>> We just had a short conversation on the #debian-devel IRC channel
>> regarding the upcoming Python 2 EOL, in the context of the Sphinx
>> packages.
>>
>> [...]
>>
>> So that's the first problem I have. The other problem we identified is
>> that the /usr/bin/sphinx-build symlink is not owned by any package.
>>
>> $ LANG=C dpkg -S /usr/bin/sphinx-build
>> dpkg-query: no path found matching pattern /usr/bin/sphinx-build
>>
>> That's a rather bad practice... We shouldn't install those symlinks in
>> postinst. They should be alternatives or conflicts or *something*. In
>> 2013, the sphinx maintainer [explained][1] the following:
>>
>> > True for docutils; however, sphinx doesn't use alternatives, and it
>> > doesn't do so for good reasons.
>> >
>> > The alternatives mechanisms is only suitable if both commands are
>> > compatible, i.e. their behavior doesn't vary with Python version. This
>> > is the case for rst2html and friends, but no so much for
>> > sphinx-build. The problem is that sphinx-build can import 3rd-party
>> > Python code, which is not necessarily compatible with both Python 2
>> > and 3.
>>
>> I understand that reasoning, but I don't think I agree with it. Or at
>> least, I don't think that, in the long term, it should be something that
>> blocks us.
>
> I think the reason why we should not use alternatives still applies. The
> current approach is conservative and it guarantees that when a package has
> python-sphinx in build-dependencies, /usr/bin/sphinx-build will always use
> Python 2. With alternatives there is no such guarantee.

Understood, fair point.

> And moving Python 3 packages to /usr/bin/sphinx3-build or something like
> that will mean diverging from upstream (see below).

Note that we already diverge from upstream in numerous places in Python,
first and foremost in the naming of the Python binary itself.

>> I am not sure what the solution is, but I would offer the following:
>>
>>  1. there should be a more easy way to override SPHINXBUILD in
>> quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
>> instead of SPHINXBUILD=
>
> Good suggestion, I have submitted a pull request upstream:
> https://github.com/sphinx-doc/sphinx/pull/4092

Excellent - I meant to do that but ran out of time writing the email in
the first place. :)

I see, however, it's not getting too much traction.. But they have a
fair point - I didn't realize there was a difference between variables
from the environment and from the commandline in Make. Maybe it's good
enough as it is then.

>>  2. similarly, --with=sphinx should do the right thing depending on the
>> detected pybuild Python version, that is, call
>> /usr/share/sphinx/scripts/python{,3}/sphinx-build directly depending
>> on which version we are building for.
>
> I do not understand this. You mentioned pybuild, so you are talking about
> setup.py based build systems, not Makefile based ones, right?

True. Well, technically, I refer to dh_sphinxdoc, but from what I could
tell, that doesn't seem to actually *build* the documentation, but just
clean it up, something I didn't expect.

> If setup.py calls build_sphinx automatically, then there is nothing else
> to do. If it does not, then you should call ‘setup.py build_sphinx’ with
> explicit interpreter, i.e. python or python3.
>
> The only thing where pybuild may help is when you want to run Sphinx for
> all supported Python versions, then you can use its COMMAND syntax.
>
> And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
> but it operates on already built and installed documentation, so it cannot
> help with building.

I guess this is what I mean should be automated better. Every time I
used --with=sphinxdoc, I expected it to build the docs and then
remembered, when the produced package lacked any docs whatsoever, that I
needed to build things myself.

A simple --with=sphinxdoc should be sufficient to build sphinx docs. We
shouldn't expect upstream to do that step in setup.py, as it is rarely
hooked into the main build. Yes, there's the `build_sphinx` command, but
it's not hooked into `setup.py build` except in some rare project.

This is why I mention pybuild: maybe *that* is where docs should be
built? I am not sure. In any case, I feel there's a shim missing here,
and there's a good occasion to leverage the automation we have to
generate proper build deps and build-time commands.

>>  3. sphinx packages should follow the "python vs python3"
>> convention. right now, when you run the "python" binary, you get
>> Python 2. if you want Python 3, you run "python3". I don't know if
>> there are plans to change this, but that's irrelevant at this stage:
>> sphinx should follow convention and call python2 sphinx when you run
>> "sphinx-build" and python3 sphinx when 

Re: providing sphinx3-* binaries

[Simon McVittie]

On Thu, 28 Sep 2017 at 01:03:27 +0300, Dmitry Shachnev wrote:
> On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
> >  1. there should be a more easy way to override SPHINXBUILD in
> > quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
> > instead of SPHINXBUILD=
> 
> Good suggestion, I have submitted a pull request upstream:
> https://github.com/sphinx-doc/sphinx/pull/4092

I'm not sure this is necessary. If a Makefile has

SPHINXBUILD = sphinx-build
# or := or ::=

some-target:
$(SPHINXBUILD) some-arguments

then you can override it on the command-line:

make SPHINXBUILD="python3 -m sphinx"

without needing the ?= syntax. The ?= syntax is only necessary if you want
to pick up a value from the environment:

export SPHINXBUILD="python3 -m sphinx"; make

or if your Makefile has complicated logic that may or may not have set it
already:

ifeq(...)
SPHINXBUILD = $(PYTHON) -m sphinx
endif

SPHINXBUILD ?= sphinx-build

The Makefiles generated by GNU Automake, which are very verbose but
generally also very good about doing the right thing in the face of
strange make(1) quirks, use plain "=".

(I've said the same on the upstream PR.)

Regards,
smcv