Re: packaging problems with python3.5 and python 3.6

[Steve Langasek]

On Wed, Aug 02, 2017 at 12:10:05PM -0400, ba...@debian.org wrote:
> On Jul 30, 2017, at 02:41, Steve Langasek  wrote:

> >> https://wiki.debian.org/Python/LibraryStyleGuide where it states:
> >>You'll want to have at least the following build dependencies:
> >>...
> >>python3-all
> >> but the
> >>python[,3]-all-dev
> >> are never mentioned. So sorry, I didn't know about them and searched
> >> for the required header files, and added the respective packages.

> > Ok, that was the first hit I got but I discounted it because it spoke of
> > packaging "libraries", not modules or extensions.

> “Libraries” is typically the term we’ve used to describe things you import 
> from other things.  In Python, everything you import is a module, and 
> packages are just special kinds of modules, whereas extensions are module 
> implemented in a compiled language instead of native Python.  It may not be 
> the best nomenclature, but it’s what we’ve been using for a long time.

> > Perhaps someone should make this wiki page a redirect to the python policy
> > manual?  https://www.debian.org/doc/packaging-manuals/python-policy/

> No, because LibraryStyleGuide *isn’t* policy, it’s an opinionated
> collection of best practices, and it is better suited for a wiki because
> those best practices do change over time.

> Policy is what you MUST do, and the barrier for changing policy is — and
> needs to be — higher.

Except the python packaging policy already mentions build dependencies:

"The python3-all-dev should be used when building extensions for any or all
Python 3 versions.  The python-all-dev should be used when building
extensions for any or all Python 2 versions."

  
https://www.debian.org/doc/packaging-manuals/python-policy/ap-build_dependencies.html

But this is then shadowed by the less precise/complete recommendation in the
wiki.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: packaging problems with python3.5 and python 3.6

[barry]

On Jul 30, 2017, at 02:41, Steve Langasek  wrote:

>> https://wiki.debian.org/Python/LibraryStyleGuide where it states:
>>  You'll want to have at least the following build dependencies:
>>  ...
>>  python3-all
>> but the
>>  python[,3]-all-dev
>> are never mentioned. So sorry, I didn't know about them and searched
>> for the required header files, and added the respective packages.

> Ok, that was the first hit I got but I discounted it because it spoke of
> packaging "libraries", not modules or extensions.

“Libraries” is typically the term we’ve used to describe things you import from 
other things.  In Python, everything you import is a module, and packages are 
just special kinds of modules, whereas extensions are module implemented in a 
compiled language instead of native Python.  It may not be the best 
nomenclature, but it’s what we’ve been using for a long time.

> Perhaps someone should make this wiki page a redirect to the python policy
> manual?  https://www.debian.org/doc/packaging-manuals/python-policy/

No, because LibraryStyleGuide *isn’t* policy, it’s an opinionated collection of 
best practices, and it is better suited for a wiki because those best practices 
do change over time.

Policy is what you MUST do, and the barrier for changing policy is — and needs 
to be — higher.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> The snippet you quoted is not specific to extension modules but to the
> use of the autodoc feature, which requires the modules to be in the
> PYTHONPATH. The `sys.path.insert` hack is just here so that you don't
> have to specify PYTHONPATH yourself when running the upstream Makefile.

I just says that this snipset is in the conf.py and I yes we need this for 
autodoc.

> I don't understand how setup.py / build options are affecting the Sphinx
> documentation. You are supposed to either call the generator manually
> via sphinx-build (which the style guide recommends), or using the
> upstream Makefile (which upstream often does).

autodoc introspect the module so you need a working modules from python, whcih 
is avhieved via the right PYTHONPATH.
So, during the build phase you can gives  options on the python setup.py 
command line.
This has an effect on the location where the module  and the extensions are 
built.

build seems to be the default location BUT when we use pybuild it is under 
.pybuild

So the default PYTHONPATH does not work.

there is no way for sphinx to know where the module and its extensions are 
excepty by providing the right PYTHONPATH.

Except if sphinx would provide a build_sphinx command whcih take care of this 
problem and take into account the parameters of the setup.py.

> As far as reliability is concerned, my reference here is upstream. If
> upstream can produce the docs, then I should be able to do it too. And
> they don't have pybuild for that.

Yes but sometime there si some hack done by the upstream in conf.py or setup.py
in order to look for the module at the standards path (under build/) which 
doesn not work with pybuild.

> Why would you build the docs for each supported Python version,
> considering you'll end up shipping only one instance of the generated
> HTML? I am probably missing something here.

I use this as a sort of unit test. And it is interesting because I catche a 
bunch of python2/python3 issues like this.
Then I can find issues between sphinx running on python2 or python3.

This is at least a good way to check thaht all the modules can be laoded.

autodoc is interesting for this.

> You might as well generate the docs just once with the default Python 2
> or Python 3 interpreter, the same way you would do without extension
> modules, no?

> Pardon my insistence, but I really fail to understand what issues you
> are referring to.

incompatibilities vetween python2 and python 3 code.
peoples trie to  write code python2 and python3 compatible but this is not 
thaht easy...

> More than I expected, indeed.

> And amongst these 643 packages, how many have such large build times
> that the overhead of an additional inplace call would be considered
> prohibitive, I wonder?

I do not know, I just think that we should be able to automatically generate 
packages for python modules from pip in order to
reduce the effort of pacakging, So there should be only one way to build the 
documentation , manpages.
Escpecillay wheren there is extensions.


> Besides, the docs will typically be processed by arch-all builders
> (provided you use -indep targets), so arch-any builds won't even feel
> that overhead, right?

Yes except that this is ot because --inplace work that the result of python 
setup.py build are identical.

it is possible to  forget a module referenced by another module in the setup.py.
It workd with --inplace but it does not work for real after build ans 
installation.

Cheers

Fred


I could be wrong though.

Re: building manpages via setup.py

[Ghislain Vaillant]

On 02/08/17 10:45, PICCA Frederic-Emmanuel wrote:

First, that's very speculative. Second, that's upstream's problem.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))

You need to customize the sys.path in order to find the extensions.


The snippet you quoted is not specific to extension modules but to the 
use of the autodoc feature, which requires the modules to be in the 
PYTHONPATH. The `sys.path.insert` hack is just here so that you don't 
have to specify PYTHONPATH yourself when running the upstream Makefile.



This path can change depending on the setup.py build options so this is not 
reliable.


I don't understand how setup.py / build options are affecting the Sphinx 
documentation. You are supposed to either call the generator manually 
via sphinx-build (which the style guide recommends), or using the 
upstream Makefile (which upstream often does).


As far as reliability is concerned, my reference here is upstream. If 
upstream can produce the docs, then I should be able to do it too. And 
they don't have pybuild for that.



Why? All you need is *one* occurrence of the extension modules somewhere
in the PYTHONPATH in order to generate the docs. Chances are that's how
upstream generates them.


Because by experience I find issue in the build system and the python code when 
building the doc for multiple
interpreter (python2/python3 differences...)


Why would you build the docs for each supported Python version, 
considering you'll end up shipping only one instance of the generated 
HTML? I am probably missing something here.


You might as well generate the docs just once with the default Python 2 
or Python 3 interpreter, the same way you would do without extension 
modules, no?


Pardon my insistence, but I really fail to understand what issues you 
are referring to.



Found a total of 643 reverse build-depend(s) for python-all-dev.

not that small



How is the ratio over all the Python packages? I suspect very small.


Found a total of 1968 reverse build-depend(s) for python-all.

not that small 32 % ;)


More than I expected, indeed.

And amongst these 643 packages, how many have such large build times 
that the overhead of an additional inplace call would be considered 
prohibitive, I wonder?


Besides, the docs will typically be processed by arch-all builders 
(provided you use -indep targets), so arch-any builds won't even feel 
that overhead, right?


I could be wrong though.

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> First, that's very speculative. Second, that's upstream's problem.

> The upstream Makefile and conf.py are likely generated by Sphinx itself
> via sphinx-quickstart. Did your upstream tinker with them that much that
> they cannot be trusted?

No this is just that does not fit well with extension.

> Some upstream do use a build_sphinx command, but it is far from common
> and it does not solve the extension module problem.

Yes this is what they are doing. extension are part of the python ec-system but 
it seems thaht this is not well suported out of the box by sphinx.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))

You need to customize the sys.path in order to find the extensions.

This path can change depending on the setup.py build options so this is not 
reliable.

> Why? All you need is *one* occurrence of the extension modules somewhere
> in the PYTHONPATH in order to generate the docs. Chances are that's how
> upstream generates them.

Because by experience I find issue in the build system and the python code when 
building the doc for multiple
interpreter (python2/python3 differences...)

> > Found a total of 643 reverse build-depend(s) for python-all-dev.
> >
> > not that small

> How is the ratio over all the Python packages? I suspect very small.

Found a total of 1968 reverse build-depend(s) for python-all.

not that small 32 % ;)


Cheers

Fred

Re: building manpages via setup.py

[Ghislain Vaillant]

On 02/08/17 09:55, PICCA Frederic-Emmanuel wrote:

PYTHONPATH=. sphinx-build -N -b html  



One can also use the sphinx-generated Makefile if available:



PYTHONPATH=$(CURDIR) $(MAKE) -C  html



Both are simple one-liners and do not rely on pybuild.


Yes it works but this is fragile since the organisation of the module can 
change in the sources.


First, that's very speculative. Second, that's upstream's problem.

I have not seen many upstream projects playing with the layout of their 
modules from one version to the next. If that's the case, then there are 
worst things to be worried about (API breakage for instance) than the docs.



at least the .pybuild directory is under the responsability of pybuild and we 
should use pybuild instead of relying
on the maintainer snipset. (typo error, change during the time.)


The upstream Makefile and conf.py are likely generated by Sphinx itself 
via sphinx-quickstart. Did your upstream tinker with them that much that 
they cannot be trusted?



It would be nice if the doc generation in python shold be standardize.


Some upstream do use a build_sphinx command, but it is far from common 
and it does not solve the extension module problem.



If it does not cost much to build the extension inplace, then the
simplest option is to prefix one of these calls above with:



python3 setup.py build_ext --inplace


when you have multiple verison of the interpreter you prefer to avoid --inplace.


Why? All you need is *one* occurrence of the extension modules somewhere 
in the PYTHONPATH in order to generate the docs. Chances are that's how 
upstream generates them.


I fail to picture how this is an issue in practice considering your 
build override will run the isolated pybuilds first and the sphinx call 
last.



If the cost is prohibitive, which arguably applies to a very limited set
of packages (yours included) then you would use pybuild for that, as
Piotr kindly suggested.


Yes it depends olsa of the arch. Some are really slow


That's unfortunate, indeed.


Considering the rarity of this use case though, I wonder whether it is
worth adding a separate section to the style guide.


Found a total of 643 reverse build-depend(s) for python-all-dev.

not that small


How is the ratio over all the Python packages? I suspect very small.

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> PYTHONPATH=. sphinx-build -N -b html  

> One can also use the sphinx-generated Makefile if available:

> PYTHONPATH=$(CURDIR) $(MAKE) -C  html

> Both are simple one-liners and do not rely on pybuild.

Yes it works but this is fragile since the organisation of the module can 
change in the sources.

at least the .pybuild directory is under the responsability of pybuild and we 
should use pybuild instead of relying
on the maintainer snipset. (typo error, change during the time.)

It would be nice if the doc generation in python shold be standardize.

> If it does not cost much to build the extension inplace, then the
> simplest option is to prefix one of these calls above with:

> python3 setup.py build_ext --inplace

when you have multiple verison of the interpreter you prefer to avoid --inplace.

> If the cost is prohibitive, which arguably applies to a very limited set
> of packages (yours included) then you would use pybuild for that, as
> Piotr kindly suggested.

Yes it depends olsa of the arch. Some are really slow

> Considering the rarity of this use case though, I wonder whether it is
> worth adding a separate section to the style guide.

Found a total of 643 reverse build-depend(s) for python-all-dev.

not that small

cheers

Fred

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> At the end of the day, it is just a matter of providing an appropriate
> PYTHONPATH, regardless of whether pybuild is used or not.

Yes but to avoid the multiplications of way to provide this PYTHONPATH.

Is it possible to have the recommended way which works for modules and 
extensions.

once agreed, we should put this in the wiki

Cheers

Fred

Re: building manpages via setup.py

[Ghislain Vaillant]

On 02/08/17 09:19, PICCA Frederic-Emmanuel wrote:

At the end of the day, it is just a matter of providing an appropriate
PYTHONPATH, regardless of whether pybuild is used or not.


Yes but to avoid the multiplications of way to provide this PYTHONPATH.


For the vast majority of packages, the current method listed in 
LibraryStyleGuide applies, i.e.:


PYTHONPATH=. sphinx-build -N -b html  

One can also use the sphinx-generated Makefile if available:

PYTHONPATH=$(CURDIR) $(MAKE) -C  html

Both are simple one-liners and do not rely on pybuild.


Is it possible to have the recommended way which works for modules and 
extensions.


If it does not cost much to build the extension inplace, then the 
simplest option is to prefix one of these calls above with:


python3 setup.py build_ext --inplace

If the cost is prohibitive, which arguably applies to a very limited set 
of packages (yours included) then you would use pybuild for that, as 
Piotr kindly suggested.



once agreed, we should put this in the wiki


Considering the rarity of this use case though, I wonder whether it is 
worth adding a separate section to the style guide.


Ghis

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> Perhaps the LibraryStyleGuide should be updated to reflect on this
> change? I believe we are still advising explicit http_proxy /
> https_proxy exports prior to running sphinx-build.

And running sphinx-build does not work expecially if there is extensions in the 
documentation.
sphinx-build should be run via pybuild in order to know about the build_dir.

right ?

Fred

Re: building manpages via setup.py

[Ghislain Vaillant]

On 02/08/17 09:03, PICCA Frederic-Emmanuel wrote:

Perhaps the LibraryStyleGuide should be updated to reflect on this
change? I believe we are still advising explicit http_proxy /
https_proxy exports prior to running sphinx-build.


And running sphinx-build does not work expecially if there is extensions in the 
documentation.
sphinx-build should be run via pybuild in order to know about the build_dir.

right ?


At the end of the day, it is just a matter of providing an appropriate 
PYTHONPATH, regardless of whether pybuild is used or not.


The default is $(CURDIR) for the vast majority of packages. But, it may 
also be something else if extension packages are involved, or if the 
package directory is under a different folder than the root directory, 
such as src/.


Ghis

Re: building manpages via setup.py

[Ghislain Vaillant]

On 02/08/17 08:44, Piotr Ożarowski wrote:

[PICCA Frederic-Emmanuel, 2017-08-02]

you can drop it, PYTHONPATH and http_proxy should be set by pybuild


Is it true  for jessie

I need to support jessie and stretch

And even debian7...


I didn't test it even for unstable, but IIRC pybuild exports those in
all steps since a long time ago. You'll know after first builds...


Perhaps the LibraryStyleGuide should be updated to reflect on this 
change? I believe we are still advising explicit http_proxy / 
https_proxy exports prior to running sphinx-build.


Ghis

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

> export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man

Thanks, The only drawback I see with this solution is that I want to run

dh_auto_build 2 times,
 - 1 for the arch part (pyhon modules, extensions and manpages)
 - 2 for the indep part (doc build)

nevertheless thans a lot it is flexxible enought for my case.

> you can drop it, PYTHONPATH and http_proxy should be set by pybuild

Is it true  for jessie

I need to support jessie and stretch

And even debian7...


Fred

Re: building manpages via setup.py

[Piotr Ożarowski]

[PICCA Frederic-Emmanuel, 2017-08-02]
> > you can drop it, PYTHONPATH and http_proxy should be set by pybuild
> 
> Is it true  for jessie
> 
> I need to support jessie and stretch
> 
> And even debian7...

I didn't test it even for unstable, but IIRC pybuild exports those in
all steps since a long time ago. You'll know after first builds...


PS please don't CC me, I'm subscribing the list
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: updating packages

[Michael Fladischer]

On 2017-08-01 11:58, Christopher Hoskin wrote:
> What's the plan for moving them to unstable? Are they still using git
> pq rather than git-dpm?

No plan yet for an upload to unstable. amqp and kombu have quite a few
reverse dependencies and I did no have the time yet to try a rebuild on
them.

All update packages have been converted to gbp pq.

Cheers,
-- 
Michael Fladischer
Fladi.at



signature.asc
Description: OpenPGP digital signature

Re: building manpages via setup.py

[Piotr Ożarowski]

[PICCA Frederic-Emmanuel, 2017-08-02]
> > if you want to test all requested Python interpreters:
> 
> I prefer this solution in order to check that the built extensions are 
> working for all the python interpreters.
> The doc use autodoc so it is nice to have this functionnality
> 
> | override_dh_auto_build:
> |   dh_auto_build -- --after-build '{interpreter} setup.py build_man'
> 
> 
> It is possible to replace this by 
> 
> export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'

export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man

(i.e. without quotes if you want to use Make's export)

> > there's no need to prepend --after-build argument with
> > "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir 
> > automatically
> 
> Do you mean that I should remove the PYTHONPATH in this ?

you can drop it, PYTHONPATH and http_proxy should be set by pybuild

> override_dh_sphinxdoc
> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>   PYBUILD_SYSTEM=custom \
>   PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} 
> http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" 
> dh_auto_build  # HTML generator
>   dh_installdocs "doc/build/html" -p python-pyfai-doc
>   dh_sphinxdoc -O--buildsystem=pybuild
> endif
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

Hello piotr

> if you want to test all requested Python interpreters:

I prefer this solution in order to check that the built extensions are working 
for all the python interpreters.
The doc use autodoc so it is nice to have this functionnality

| override_dh_auto_build:
|   dh_auto_build -- --after-build '{interpreter} setup.py build_man'


It is possible to replace this by 

export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'

> there's no need to prepend --after-build argument with
> "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically

Do you mean that I should remove the PYTHONPATH in this ?

override_dh_sphinxdoc
ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
PYBUILD_SYSTEM=custom \
PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} 
http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" 
dh_auto_build  # HTML generator
dh_installdocs "doc/build/html" -p python-pyfai-doc
dh_sphinxdoc -O--buildsystem=pybuild
endif



--
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645