from:"Tomas Orsava"

Disk space minimization of Python

2020-10-14 Thread Tomas Orsava


Hi,
I have opened an upstream discussion about disk space minimization of 
Python, take a look:


https://discuss.python.org/t/disk-space-minimization-for-python-distributors/5447

All the best,
Tomas
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

New pipenv 2020.6.2 ready for testing in a copr repo

2020-06-29 Thread Tomas Orsava


Hi,
yet newer upstream version of pipenv, 2020.6.2, was released and is 
ready for you in copr:


https://copr.fedorainfracloud.org/coprs/g/python/pipenv/

All the best,
Tomas

On 5/28/20 5:11 PM, Tomas Orsava wrote:

Hi,
upstream released a brand new version of `pipenv` this morning, and 
we've packaged it for you to try out in copr:


    https://copr.fedorainfracloud.org/coprs/g/python/pipenv/

Please report any issues in Fedora's Bugzilla and in the text please 
indicate that it's regarding this new version.


All the best,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to 
python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

2020-06-11 Thread Tomas Orsava

On 6/9/20 12:15 PM, Petr Viktorin wrote:

On 2020-06-08 12:15, Tomas Orsava wrote:

On 6/8/20 11:58 AM, Petr Viktorin wrote:

On 2020-06-05 16:12, Tomas Orsava wrote:

On 6/5/20 2:26 PM, Petr Viktorin wrote:

On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in
the document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:

### Dist-info metadata

Each Python package **MUST** include *Package Distribution
Metadata* conforming to [PyPA
specifications](https://packaging.python.org/specifications/)
(specifically, [Recording installed
distributons](https://packaging.python.org/specifications/recording-installed-packages/)).

This applies to libraries (e.g. `python-requests`) as well as
tools (e.g. `ansible`).

> XXX what with splitting into subpackages? 1) dist-info
always installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry,
fedpkg)

>
> e.g.
> When software is split into several subpackages, it is OK to
only ship metadata in one built RPM.

dist-info usually corresponds to an importable module, so let's
say that it SHOULD be in the same subpackage as the importable
module?

But if you split that module, which submodule does the dist-info
go to?

I'd leave it to the packager; all these cases are different.

That's why I suggested the SHOULD, because there will be
exceptions. But I think these guidelines might be read by people
who will not be actively aware of the relationship between
dist-info and a Python importable module, so I would add guidance
that these should be together if possible.

Please suggest the wording you'd like to see.
The one you have has the problem that there can be more importable
modules in one project. It doesn't give any guidance for what to
do if you split the project.

Of course, if you put importable module(s) in one subpackage and
documentation in another, the dist-info should be with the module(s).

I'm trying to read this guide through the eyes of someone starting
out with Fedora and/or Python, so I'd rather not assume people know
these details.

How about:

When software is split into several subpackages, it is OK to only
ship metadata in one built RPM. If the project contains an
importable module(s), the metadata SHOULD be included in the same
subpackage as the (main) importable module.

(First sentence already was in the text, I included it for context.)

I added it. I put the **SHOULD** sentence near the top of the
section to align to the organization of the document: rules first,
explanations/examples under them.

Looks good, thank you.

All in all, really nice document. Let me know if you could use more
help with it!

Tomas

Well, any of the XXX could use help :)
Except maybe the links; those can be dded when we convert from Markdown.

Is there some way to do reviews? If someone just replaces XXX with text,
it's hard to notice.

Tomas

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Monthly highlights from the Red Hat's Python-maint team, May 2020

2020-06-11 Thread Tomas Orsava


Hi,
we're the Python-maint team from Red Hat, and we'd like to experiment 
with sharing with you what we are working on from month to month. There 
are some things we should not disclose (such as embargoed CVEs), but the 
majority of what we do is out in public. Please let us know if you find 
this useful and whether you'd like us to continue.


So here are the highlights of what we've been working on this May, 2020:

*Fedora:*

 * Python updated to Python 3.9 in rawhide (Fedora 33)

 * ~3000 source packages rebuilt

 * ~100 reamain problematic (fail to build)

 * python3X packages renamed to python3.X in rawhide

 * Python RPM depndency generators:

 * Backported proposed upstream changes, added new test suite to Fedora CI

 * Working on handling of proposed Python extras subpackages

 * pipenv continually being rebased to new versions, currently 2020.5.28

 * Test in copr:https://copr.fedorainfracloud.org/coprs/g/python/pipenv/

 * Provided feedback and fixes upstream

 * pytest updated to 5 on rawhide (compat package with pytest 4 introduced)

 * pip updated to version 20.1, changes in Python RPM macros were
   needed to workaround problems with new PEP 610

 * https://www.python.org/dev/peps/pep-0610/

 * New RPM macros for easier packaging: %py_provides, %pytest and
   %python3_shebang_fix

 * 
%py_provides:https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/52

 * 
%python3_shebang_fixhttps://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/58

 * %pytesthttps://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/55

 * Improved %pypi_source macro

 * Removes tildes (`~`) from %version so that it works with
   alpha/beta/dev releases out of the box

 * https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/57

 * Python 3.8.3 updated across Fedora releases

 * pypy and pypy 3 updated to versions 7.3.1

 * provided feedback upstream for new Cython alpha release


*CPython:*

 * CPython 3.9b1 was released:
   seehttps://docs.python.org/dev/whatsnew/3.9.html

 * Fixing a CPython type reference count issue found by PySide

 * Python 3.9 is now FIPS compatible upstream.

 * Proof-of-concept of subinterpreters running in parellel with one GIL
   per interpreter: "Subinterpreters 4x faster than sequential
   execution or threads on CPU-bound
   
workload"https://mail.python.org/archives/list/python-...@python.org/thread/S5GZZCEREZLA2PEMTVFBCDM52H4JSENR/

 * Progress on subinterpreter isolation work: most free lists are now
   per-interpreter.https://bugs.python.org/issue40521

 * Progress on hiding implementation details from the C API: Py_TYPE(),
   Py_REFCNT() and Py_SIZE() can no longer be used as l-value. Use
   Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE()
   
instead.https://mail.python.org/archives/list/capi-...@python.org/thread/HYFQHLGNCRFBOMOOV2WUT46CQPXV5VQ7/

 * New --hardlink-dupes option in compileall: "If two .pyc files with
   different optimization level have the same content, use hard links
   to consolidate duplicate
   
files."https://docs.python.org/dev/library/compileall.html#cmdoption-compileall-hardlink-dupes

 * Helped to get features merged before the 3.9 feature freeze:
   removeprefix/removesuffix (PEP 616), new zoneinfo module (PEP 615).
   Fixed buildbot failures and a few reference leaks.

 * Python Steering Council Community Address with our team member
   Victor Stinner:https://www.youtube.com/watch?v=xX8fGuh4T_o

*
*
*Red Hat Enterprise Linux:*

 * Overhauling the FIPS patch for python3 in RHEL 7

 * Created an Insights rule candidate that verifies Python alternatives

 * Backporting fix for Python's CVE-2019-16056 to older RHEL 7 versions

 * For RHEL9 preparations -- see the Fedora section


*Red Hat Software Collections:*

 * Python 3.8 from RHSCL 3.5 added for s2i-python-container upstream


*Misc:*

 * Releasing a GitHub action to run Tox on
   Fedora:https://github.com/marketplace/actions/python-tox-on-fedora


All the best,
Python-maint

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Draft of New Python Packaging Guidelines

2020-06-08 Thread Tomas Orsava

On 6/8/20 11:58 AM, Petr Viktorin wrote:

On 2020-06-05 16:12, Tomas Orsava wrote:

On 6/5/20 2:26 PM, Petr Viktorin wrote:

On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the
document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:

### Dist-info metadata

This applies to libraries (e.g. `python-requests`) as well as
tools (e.g. `ansible`).

> XXX what with splitting into subpackages? 1) dist-info always
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry,
fedpkg)

>
> e.g.
> When software is split into several subpackages, it is OK to
only ship metadata in one built RPM.

dist-info usually corresponds to an importable module, so let's
say that it SHOULD be in the same subpackage as the importable
module?

But if you split that module, which submodule does the dist-info
go to?

I'd leave it to the packager; all these cases are different.

Of course, if you put importable module(s) in one subpackage and
documentation in another, the dist-info should be with the module(s).

I'm trying to read this guide through the eyes of someone starting
out with Fedora and/or Python, so I'd rather not assume people know
these details.

How about:

(First sentence already was in the text, I included it for context.)

I added it. I put the **SHOULD** sentence near the top of the section
to align to the organization of the document: rules first,
explanations/examples under them.

Looks good, thank you.

All in all, really nice document. Let me know if you could use more help
with it!

Tomas

## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check`
section and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the
result to solve a build failure.

As an exception, you **MAY** disable tests with an appropriate
`%if` conditional (e.g. bcond) when
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).

I would like to see either that the bcond SHOULD be named `tests`
(or possibly `check`), or if that's too strong, at least
recommend these two as common bcond names.

I agree, but it should be a Fedora-wide guideline.

True. Does anyone know if this is already in progress somewhere? I
remember it being talked about. Otherwise I guess the best way will
be for me to open a new thread about this.

Please do.

Here we go:
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/443LZIX4XLZL6NMF3M7HAGHKPNA4TRYN/
>>
In the meantime, I would at least list these as common bcond names,
as Fedora-wide guidelines might take a while.

The Python guidelines will also take a while, and I'd like to avoid
giving different guidelines than what we come up with for Fedora.

Makes sense. I've added an "XXX" note to the text so we don't forget
to address this later.

Thanks!

Re: Draft of New Python Packaging Guidelines

2020-06-05 Thread Tomas Orsava

On 6/5/20 2:26 PM, Petr Viktorin wrote:

On 2020-06-05 11:51, Tomas Orsava wrote:

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the
document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:

### Dist-info metadata

This applies to libraries (e.g. `python-requests`) as well as
tools (e.g. `ansible`).

> XXX what with splitting into subpackages? 1) dist-info always
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry,
fedpkg)

>
> e.g.
> When software is split into several subpackages, it is OK to
only ship metadata in one built RPM.

dist-info usually corresponds to an importable module, so let's say
that it SHOULD be in the same subpackage as the importable module?

But if you split that module, which submodule does the dist-info go to?
I'd leave it to the packager; all these cases are different.

That's why I suggested the SHOULD, because there will be exceptions.
But I think these guidelines might be read by people who will not be
actively aware of the relationship between dist-info and a Python
importable module, so I would add guidance that these should be
together if possible.

Of course, if you put importable module(s) in one subpackage and
documentation in another, the dist-info should be with the module(s).

I'm trying to read this guide through the eyes of someone starting out
with Fedora and/or Python, so I'd rather not assume people know these
details.

How about:

(First sentence already was in the text, I included it for context.)

If this is not the case, the packager **MUST** contact upstream
about this. The goal is to get the project name registered or
blocked on PyPI, or to otherwise ensure the rule is followed.

> XXX Note that project names that were in Fedora but not on PyPI
when these guidelines were proposed are *blocked* as we discuss
how they should be handled.
> This prevents potential trolls, but also legitimate owners, from
taking them.

This is necessary to protect users, avoid confusion, and enable
automation as Fedora and upstream ecosystems grow more integrated.

As always, specific exceptions can be granted by FPC (XXX link
exceptions rules).

> XXX Write an automated check for this.

## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result
to solve a build failure.

As an exception, you **MAY** disable tests with an appropriate
`%if` conditional (e.g. bcond) when
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).

I would like to see either that the bcond SHOULD be named `tests`
(or possibly `check`), or if that's too strong, at least recommend
these two as common bcond names.

I agree, but it should be a Fedora-wide guideline.

True. Does anyone know if this is already in progress somewhere? I
remember it being talked about. Otherwise I guess the best way will
be for me to open a new thread about this.

Please do.

Here we go:
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/443LZIX4XLZL6NMF3M7HAGHKPNA4TRYN/

In the meantime, I would at least list these as common bcond names,
as Fedora-wide guidelines might take a while.

The Python guidelines will also take a while, and I'd like to avoid
giving different guidelines than what we come up with for Fedora.

Makes sense. I've added an "XXX" note to the text so we don't forget to
address this later.

Re: Draft of New Python Packaging Guidelines

2020-06-05 Thread Tomas Orsava

On 6/5/20 2:22 PM, Petr Viktorin wrote:

On 2020-06-05 13:58, Tomas Orsava wrote:

On 6/5/20 1:43 PM, Miro Hrončok wrote:

On 05. 06. 20 11:51, Tomas Orsava wrote:

[...]

I see what you mean.
On the other hand, that's a pretty horrible error message format
(posting in it's entirety for others to consider).

Is there no better way to achieve this? For a few packages it's ok,
but I would be weary of introducing this to too many packages.

In the proposal there's talk of blocking the name on PyPI. Is this
the way the blocking will be achieved?

You can talk to PyPI admins to block packages.

But all in all, I think what "fedora", "ldap" or "microsoft are doing
is the best option right now.
I started some discussion upstream, if you want to read it:
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a-k-a-pypi-namesquatting-e-g-for-fedora-packages/4045/2

(But let's discuss Fedora issues here first.)

Ah, that's sad. Thanks for raising the topic upstream, hopefully it'll
improve down the road.

As fer Fedora: Do I understand it correctly that:
- the names of Python packages in Fedora have been now blocked on PyPI
using the admin intervention, and
- we want to advise people to get those names on PyPI and either publish
the projects there or do this fedora/ldap-sort of namesquatting?

I agree it's the least-worst option now. However, it will need a
detailed instructions.

Re: Draft of New Python Packaging Guidelines

2020-06-05 Thread Tomas Orsava


On 6/5/20 1:43 PM, Miro Hrončok wrote:

On 05. 06. 20 11:51, Tomas Orsava wrote:

## PyPI parity

Every Python package in Fedora **SHOULD** also be available on 
[the Python Package Index](https://pypi.org) (PyPI).


The command `pip install PROJECTNAME` **MUST** install the same 
package (possibly in a different version), install nothing, or 
fail with a reasonable error message.



What should I imagine under "reasonable error message"?


One that explains the situation. Not a segfault.
I think it's fine to leave this to the packager.



Maybe I'm missing some context, but I'm lost. Are we proposing to add 
new functionality to pip that somehow handles this? Or how could the 
packager influence pip's error message?


Try:

    $ pip install fedora



|$ pip install fedora||
||Defaulting to user installation because normal site-packages is not 
writeable||

||Collecting fedora||
||  Downloading fedora-0.tar.gz (2.0 kB)||
||Building wheels for collected packages: fedora||
||  Building wheel for fedora (setup.py) ... error||
||  ERROR: Command errored out with exit status 1:||
||   command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; 
sys.argv[0] = '"'"'/tmp/pip-install-d5gw3ura/fedora/setup.py'"'"'; 
__file__='"'"'/tmp/pip-install-d5gw3ura/fedora/setup.py'"'"';f=getattr(tokenize, 
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', 
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' 
bdist_wheel -d /tmp/pip-wheel-baiub5ca||

||   cwd: /tmp/pip-install-d5gw3ura/fedora/||
||  Complete output (72 lines):||
||  running bdist_wheel||
||  running build||
||  running build_py||
||||
||  # `fedora` on PyPI||
||
||  This dummy project is not installable.||
||  You probably want `python-fedora` instead.||
||
||
||  ## pyproject.toml tool.fedora namespace||
||
||  The [PEP 518](https://www.python.org/dev/peps/pep-0518/#tool-table) 
declares||
||  that a project can use the subtable `tool.$NAME` if `pyproject.toml` 
if,||

||  and only if, they own the entry for `$NAME` in the Cheeseshop/PyPI.||
||
||  That's what this project is for.||
||  We own the entry for `fedora`, so we could use `tool.fedora` in .||
||
||  ## python-fedora||
||
||  The Fedora Infra's 
[python-fedora](https://github.com/fedora-infra/python-fedora)||

||  project provides an importable module named `fedora`.||
||
||  This goes against the convention that PyPI distribution names should||
||  match the module names.||
||  But, python-fedora pre-dates wide use of that convention, and the 
issue||

||  is hard to fix now.||
||
||  Please install `python-fedora` to get the Fedora Infra's package.||
||
||  ### Please: Don't install packages blindly||
||
||  When you see the exception:||
||
||  ```||
||  ModuleNotFoundError: No module named 'foo'||
||  ```||
||
||  … please research the actual requirements instead of going directly 
for||

||  `pip install foo`.||
||  The project (distribution) name may differ from the module(s) it||
||  provides.||
||
||||
||  Traceback (most recent call last):||
||    File "", line 1, in ||
||    File "/tmp/pip-install-d5gw3ura/fedora/setup.py", line 38, in 
||

||  zip_safe=False,||
||    File 
"/home/torsava/.local/lib/python3.7/site-packages/setuptools/__init__.py", 
line 144, in setup||

||  return distutils.core.setup(**attrs)||
||    File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup||
||  dist.run_commands()||
||    File "/usr/lib64/python3.7/distutils/dist.py", line 966, in 
run_commands||

||  self.run_command(cmd)||
||    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in 
run_command||

||  cmd_obj.run()||
||    File 
"/home/torsava/.local/lib/python3.7/site-packages/wheel/bdist_wheel.py", 
line 223, in run||

||  self.run_command('build')||
||    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in 
run_command||

||  self.distribution.run_command(command)||
||    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in 
run_command||

||  cmd_obj.run()||
||    File "/usr/lib64/python3.7/distutils/command/build.py", line 135, 
in run||

||  self.run_command(cmd_name)||
||    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in 
run_command||

||  self.distribution.run_command(command)||
||    File "/usr/lib64/python3.7/distutils/dist.py"

Re: Draft of New Python Packaging Guidelines

2020-06-05 Thread Tomas Orsava

On 6/5/20 11:26 AM, Petr Viktorin wrote:

On 2020-06-03 21:49, Tomas Orsava wrote:

Hi,
I have left a few notes about the text itself as comments in the 
document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:

### Dist-info metadata

Each Python package **MUST** include *Package Distribution Metadata* 
conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)). 

This applies to libraries (e.g. `python-requests`) as well as tools 
(e.g. `ansible`).

> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, fedpkg)
>
> e.g.
> When software is split into several subpackages, it is OK to only 
ship metadata in one built RPM.

dist-info usually corresponds to an importable module, so let's say 
that it SHOULD be in the same subpackage as the importable module?

But if you split that module, which submodule does the dist-info go to?
I'd leave it to the packager; all these cases are different.

That's why I suggested the SHOULD, because there will be exceptions. But 
I think these guidelines might be read by people who will not be 
actively aware of the relationship between dist-info and a Python 
importable module, so I would add guidance that these should be together 
if possible.

The metadata takes the form of a `dist-info` directory installed in 
`%{python3_sitelib}` or `%{python3_sitearch}`, and contains 
information that tools like 
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) 
use to introspect installed libraries.

> XXX example %files with manual dist-info entry

Note that some older tools instead put metadata in an `egg-info` 
directory, or even a single file.

This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates an `egg-info` 
directory or file, please contact Python SIG.

> XXX We need a better solution before we go out of beta.

As an exception, the Python standard library **MAY** ship without 
this metadata.

## PyPI parity

Every Python package in Fedora **SHOULD** also be available on [the 
Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same 
package (possibly in a different version), install nothing, or fail 
with a reasonable error message.

What should I imagine under "reasonable error message"?

One that explains the situation. Not a segfault.
I think it's fine to leave this to the packager.

Maybe I'm missing some context, but I'm lost. Are we proposing to add 
new functionality to pip that somehow handles this? Or how could the 
packager influence pip's error message?

If this is not the case, the packager **MUST** contact upstream 
about this. The goal is to get the project name registered or 
blocked on PyPI, or to otherwise ensure the rule is followed.

> XXX Note that project names that were in Fedora but not on PyPI 
when these guidelines were proposed are *blocked* as we discuss how 
they should be handled.
> This prevents potential trolls, but also legitimate owners, from 
taking them.

This is necessary to protect users, avoid confusion, and enable 
automation as Fedora and upstream ecosystems grow more integrated.

As always, specific exceptions can be granted by FPC (XXX link 
exceptions rules).

> XXX Write an automated check for this.

## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result 
to solve a build failure.

As an exception, you **MAY** disable tests with an appropriate `%if` 
conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping). 

I would like to see either that the bcond SHOULD be named `tests` (or 
possibly `check`), or if that's too strong, at least recommend these 
two as common bcond names.

I agree, but it should be a Fedora-wide guideline.

True. Does anyone know if this is already in progress somewhere? I 
remember it being talked about. Otherwise I guess the best way will be 
for me to open a new thread about this.

In the meantime, I would at least list these as common bcond names, as 
Fedora-wide guidelines might take a while.

A popular testing tool, and one which is well integrated in Fedora, 
is `tox`. Upstream, it is commonly used to test against multiple 
Python versions. In a Fedora package, BuildRequire test dependencies 
(see *Test dependencies* below) an

Re: Draft of New Python Packaging Guidelines

2020-06-03 Thread Tomas Orsava

Hi,
I have left a few notes about the text itself as comments in the document.

Comments about the subject matter are inlined below:

On 4/30/20 3:41 PM, Petr Viktorin wrote:

### Dist-info metadata

Each Python package **MUST** include *Package Distribution Metadata* 
conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)).

This applies to libraries (e.g. `python-requests`) as well as tools 
(e.g. `ansible`).

> XXX what with splitting into subpackages? 1) dist-info always 
installed, 2) dist-info installed trough a metapackage?

> * Ideally, do the split upstream
> * Consider package split between library & tool (see poetry, fedpkg)
>
> e.g.
> When software is split into several subpackages, it is OK to only 
ship metadata in one built RPM.

dist-info usually corresponds to an importable module, so let's say that 
it SHOULD be in the same subpackage as the importable module?

The metadata takes the form of a `dist-info` directory installed in 
`%{python3_sitelib}` or `%{python3_sitearch}`, and contains 
information that tools like 
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) 
use to introspect installed libraries.

> XXX example %files with manual dist-info entry

Note that some older tools instead put metadata in an `egg-info` 
directory, or even a single file.

This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates an `egg-info` 
directory or file, please contact Python SIG.

> XXX We need a better solution before we go out of beta.

As an exception, the Python standard library **MAY** ship without this 
metadata.

## PyPI parity

Every Python package in Fedora **SHOULD** also be available on [the 
Python Package Index](https://pypi.org) (PyPI).

The command `pip install PROJECTNAME` **MUST** install the same 
package (possibly in a different version), install nothing, or fail 
with a reasonable error message.

What should I imagine under "reasonable error message"?

If this is not the case, the packager **MUST** contact upstream about 
this. The goal is to get the project name registered or blocked on 
PyPI, or to otherwise ensure the rule is followed.

> XXX Note that project names that were in Fedora but not on PyPI when 
these guidelines were proposed are *blocked* as we discuss how they 
should be handled.
> This prevents potential trolls, but also legitimate owners, from 
taking them.

This is necessary to protect users, avoid confusion, and enable 
automation as Fedora and upstream ecosystems grow more integrated.

As always, specific exceptions can be granted by FPC (XXX link 
exceptions rules).

> XXX Write an automated check for this.

## Tests

### Running tests

If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.

You **MAY** exclude specific failing tests.

You **MUST NOT** disable the entire testsuite or ignore the result to 
solve a build failure.

As an exception, you **MAY** disable tests with an appropriate `%if` 
conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).

I would like to see either that the bcond SHOULD be named `tests` (or 
possibly `check`), or if that's too strong, at least recommend these two 
as common bcond names.

A popular testing tool, and one which is well integrated in Fedora, is 
`tox`. Upstream, it is commonly used to test against multiple Python 
versions. In a Fedora package, BuildRequire test dependencies (see 
*Test dependencies* below) and run `tox` with:

```
%tox
```

### Test dependencies

One part of the Python packaging ecosystem that is still not 
standardized is specifying test dependencies (and development 
dependencies in general).

The best practice to specify tests is using an extra (XXX link to 
section above, which should be fleshed out) like `[test]` or `[dev]`. 
In this case, upstream's instructions to install test dependencies 
might look like `pip install -e.[test]`.

Projects using `tox` usually specify test dependencies in a 
`tox`-specific format: a 
[requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) 
key in the configuration.

Both forms are handled by the `%pyproject_buildrequires` macro, see 
below.

If upstream does not use either form, list test dependencies as manual 
*BuildRequires* in the `spec` file.

Should these be manually listed as Fedora built RPM names 
(python3-testdep) or using the dist tag `python3dist(testdep)`?

Overall, a great step forward from the old guidelines!
Tomas
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs

Brand new pipenv 2020.5.28 ready for testing in a copr repo

2020-05-28 Thread Tomas Orsava


Hi,
upstream released a brand new version of `pipenv` this morning, and 
we've packaged it for you to try out in copr:


    https://copr.fedorainfracloud.org/coprs/g/python/pipenv/

Please report any issues in Fedora's Bugzilla and in the text please 
indicate that it's regarding this new version.


All the best,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Redesigning the %python_provide macro from scratch

2020-05-26 Thread Tomas Orsava


On 5/26/20 11:52 AM, Miro Hrončok wrote:

On 26. 05. 20 11:29, Tomas Orsava wrote:

On 5/25/20 7:42 PM, Miro Hrončok wrote:

On 25. 05. 20 18:33, Tomas Orsava wrote:

On 4/19/20 4:55 PM, Miro Hrončok wrote:
4) Make it so that for given arguments, the macro will only expand 
to something once per build. Hence when you use it with package 
name, the automatic provides won't re-add the same provide again. 
This also means you cannot have 2 different (sub)packages provide 
the same name-version-release, but that shall be very very very 
uncommon need and can always be workarounded somehow if needed.



I thought multiple identical provides were just unified and didn't 
pose a problem. So what is the reason to focus on this 
once-per-build expansion?


Not if one if manual and one from a generator. See:

https://github.com/rpm-software-management/rpm/issues/1166



Interesting. One thing not mentioned is, does this cause any issues 
besides the visual issue of the provide being listed twice?


Technical issues? No. Createrepo merges those, so in the repo, no 
difference.


But rpmlint is very loud about those. It tells you: This provide you 
put in is not needed (which is great except it only applies to rawhide 
and we don't want packagers start removing it and merging the removal 
to older branches).


Side note: Once Fedora 32 goes EOL, we can change the behavior, so 
rpmlint starts bugging packagers about this again.



Ah, makes sense.

Overall, good work!

Tomas




7) Support arbitrary names. Only provide the given name and 
nothing else if not "recognized".



Given the limitations, I agree with this choice.


What limitations?



I was referring to this bit from the other thread:

 > we cannot error out with arbitrary package names, so we currently 
need to pre-filter the arguments before using them


Did I misunderstand it?


Not really. I did. Nothing to discuss here, these aren't the droids 
you're looking for... move along.



___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Redesigning the %python_provide macro from scratch

2020-05-26 Thread Tomas Orsava


On 5/25/20 7:42 PM, Miro Hrončok wrote:

On 25. 05. 20 18:33, Tomas Orsava wrote:

On 4/19/20 4:55 PM, Miro Hrončok wrote:
4) Make it so that for given arguments, the macro will only expand 
to something once per build. Hence when you use it with package 
name, the automatic provides won't re-add the same provide again. 
This also means you cannot have 2 different (sub)packages provide 
the same name-version-release, but that shall be very very very 
uncommon need and can always be workarounded somehow if needed.



I thought multiple identical provides were just unified and didn't 
pose a problem. So what is the reason to focus on this once-per-build 
expansion?


Not if one if manual and one from a generator. See:

https://github.com/rpm-software-management/rpm/issues/1166



Interesting. One thing not mentioned is, does this cause any issues 
besides the visual issue of the provide being listed twice?





7) Support arbitrary names. Only provide the given name and nothing 
else if not "recognized".



Given the limitations, I agree with this choice.


What limitations?



I was referring to this bit from the other thread:

> we cannot error out with arbitrary package names, so we currently 
need to pre-filter the arguments before using them


Did I misunderstand it?





Usage example:

  %package -n python3-setuptools
  %python_provides python3-pkg_resources

Resulting provides:

  python3-setuptools = 46.6.6-6.fc33
  python-setuptools = 46.6.6-6.fc33
  python39-pkg_resources = 46.6.6-6.fc33
  python3-pkg_resources = 46.6.6-6.fc33
  python-pkg_resources = 46.6.6-6.fc33
  python39-pkg_resources = 46.6.6-6.fc33



What provides the names `python-setuptools`


The generator.



Oh right, I thought you were only talking about the provides from the 
macro. Makes sense.





 and `python39-pkg_resources`?

%py_provides python3-pkg_resources

(Note that it is python3.9-pkg_resources now.)

From the code [0] it looks like the current rawhide implementation of 
%py_provides doesn't.
On the other hand, it might be a good idea to also provide the names 
for the given %subpackage name (with an option to disable this). In 
that case, the macros should also without any parameters at all.


I don't get this part, sorry.



Irrelevant due to my misunderstanding above.

Tomas




[0] 
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.python-srpm#_162 





___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: Redesigning the %python_provide macro from scratch

2020-05-25 Thread Tomas Orsava


On 4/19/20 4:55 PM, Miro Hrončok wrote:

Hello Python packagers.

After touching the %python_provide topic in:

https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/SSJLPWSGFGPYRSHXQZDR7JNQXSDGGX3Z/ 



I have realized several things I don't like about %python_provide:

1) It must be used conditionally (it is not defined in 
python-srpm-macros).
That means you always wrap it in %{?python_provide:...} in order to 
have a "valid" specfile even when the macro is not yet defined (e.g. 
during SRPM creation in Koji or on a packager's machine without 
python-rpm-macros installed).


2) You cannot use it with arbitrary versions. Suppose you package 
python3-foo 1.0 but you want to provide python3-bar 2.0 for some 
reason -- this is not very common, but it happens. %python_provide 
only takes the "name" as an argument, always using %{?epoch} 
%{version} and %{release}.


3) You need to both add a virtual provide + use the macro. Suppose you 
want to provide python3-pkg_resources from python3-setuptools. 
Currently, you need to add:


  Provides: python3-pkg_resources = %{version}-%{release}
  %{?python_provide:%python_provide python3-pkg_resources}

4) When used with (sub)package name, it generates a duplicate 
dependency on Fedora 33+ (and an rpmlint error).


5) It produces Obsoletes, but that might no longer be necessary nor 
desired.


6) Broken expectations about %_isa. It used to add %_isa provides 
based on wrong data, it no longer does that (except on old releases 
and EPELs), can be used manually with name%{?_isa}, but not on the old 
releases.


7) Undocuemnted error handling (e.g. the macro expands to nothing when 
used with pypy-foo, but errors when used with foo).



Hence, I was thinking (for the sake of backwards compatibility) to 
provide a new mechanism to do this and preserve the old macro as is, 
deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 
9 goes EOL (or never, which is basically the same from today's 
perspective).


The new macro should solve the problems from above, my current (quick, 
untested) ideas are:



1) Define the macro in python-srpm-macros. No need to use it 
conditionally. We can backport it to EPEL 8 and define a "stub" macro 
in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 
goes EOL, we can make the macro behave consistently across all Fedora 
versions.)


2) Accept version identifier as an optional argument, use %{?epoch} 
%{version} and %{release} as default. (See for example %{pypi_source} 
on how this can be done.)


3) Make the macro also produce the provide for the given name and 
document that. E.g. when you call it with python3-pkg_resources, it 
also provides python3-pkg_resources (not only python-pkg_resources etc.).


4) Make it so that for given arguments, the macro will only expand to 
something once per build. Hence when you use it with package name, the 
automatic provides won't re-add the same provide again. This also 
means you cannot have 2 different (sub)packages provide the same 
name-version-release, but that shall be very very very uncommon need 
and can always be workarounded somehow if needed.



I thought multiple identical provides were just unified and didn't pose 
a problem. So what is the reason to focus on this once-per-build expansion?





5) No obsoletes with the new macro. Packagers use manual obsoletes 
when desired.


6) Document clearly that there is no %{?_isa} (and there is no 
"backwards compatibility" load to carry). When absolutely desired, 
packagers can call the macro with %{name}%{?_isa}.


7) Support arbitrary names. Only provide the given name and nothing 
else if not "recognized".



Given the limitations, I agree with this choice.





As a bonus, I think the current if-elif-elif-elif-elif code can be 
replaced with lua patterns (imagine regex).



As always, this leaves us with the name problem, but I'd very much 
like to use %python_provides (note the s). The only problem I see is 
that it is likely to be mistaken for the old one, but IMHO it 
shouldn't really hurt that much.


Usage example:

  %package -n python3-setuptools
  %python_provides python3-pkg_resources

Resulting provides:

  python3-setuptools = 46.6.6-6.fc33
  python-setuptools = 46.6.6-6.fc33
  python39-pkg_resources = 46.6.6-6.fc33
  python3-pkg_resources = 46.6.6-6.fc33
  python-pkg_resources = 46.6.6-6.fc33
  python39-pkg_resources = 46.6.6-6.fc33



What provides the names `python-setuptools` and 
`python39-pkg_resources`? From the code [0] it looks like the current 
rawhide implementation of %py_provides doesn't.
On the other hand, it might be a good idea to also provide the names for 
the given %subpackage name (with an option to disable this). In that 
case, the macros should also without any parameters at all.


[0] 
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.python-srpm#_162


Thanks for a nifty new macro!
Tomas





Another example:

Monthly highlights from the Red Hat's Python-maint team, April 2020

2020-05-04 Thread Tomas Orsava


Hi,
we're the Python-maint team from Red Hat, and we'd like to experiment 
with sharing with you what we are working on from month to month. There 
are some things we should not disclose (such as embargoed CVEs), but the 
majority of what we do is out in public. Please let us know if you find 
this useful and whether you'd like us to continue.


So here are the highlights of what we've been working on this April, 2020.


*Red Hat Enterprise Linux:*

 * Updating python38-pip to support manylinux2014 in future RHEL8

 * Fixing CVE-2020-10108 and CVE-2020-10109 in python-twisted-web on
   RHEL 6 and 7

 * Fixing CVE-2019-20477 and CVE-2019-20477 in PyYAML on RHEL 8 and
   RHSCL 3.5

 * Fixing CVE-2020-8492 in python3 on RHEL 7 and 8

 * For RHEL9 preparations—see the Fedora section


*Red Hat Software Collections:*

 * upcoming rh-python38 collection

 * Implementing FIPS support

 * Fixing CVE-2020-8492

 * Updating python-pip to 19.3.1 in order to support manylinux2014


*Fedora:*

 * Updated Python 2 to the ultimate version (2.7.18)

 * Python RPM generators were reworked and submitted to rpm upstream

 * https://github.com/rpm-software-management/rpm/pull/1195

 * The *last* Python 3.8 FTBFS Fedora 32 package (upm) fixed

 * Phoronix reported Python is a lot faster in Fedora 32 (most likely
   due to --no-semantic-interposition):

 * 
https://www.phoronix.com/scan.php?page=article&item=fedora-32-benchmarks&num=5

 * Automation for the %files section in Python packages
   (pyproject-rpm-macros) now exists

 * https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/44

 * Several small imporvement ideas to make Python packaging easier were
   shared

 * %py_provide

 * 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/6DGWPIRP7AYBZP5XEB67YP263P6Q6WTB/

 * %pytest

 * 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/XLPDSH362PJKMJCAYOXNJNV53Y66EF6B/

 * Proposal to rename pythonXY to pythonX.Y was shared

 * 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/VIUS7WMQMDX6H2WEIH7TVTMBB6SUHY7E/

 * The rebuilds of the newly-released Python 3.9.0a6 have started in
   Python 3.9 copr

 * https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

 * The beta version of pip 20.1 was tested in Fedora and preparations
   have started for shipping the final version

 * https://src.fedoraproject.org/rpms/python-pip/pull-request/62

 * Update to Sphinx 3 is ongoing and testing has started in copr, there
   are still packages that block this effort

 * Tracking bugzilla:https://bugzilla.redhat.com/show_bug.cgi?id=1783776

 * Testing of Cython 3.0 alpha has started in copr

 * Proposed a draft of new Python Packaging Guidelines for Fedora

 * 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/ZCNUQBJLDUJUJXK2EOPP2MWL6FJKLBPS/


*CPython:*

 * CVE-2020-8492 fixed, denial of service in urllib HTTP basic
   authentication:

 * https://bugs.python.org/issue39503

 * test_gdb now automatically skips tests when it detects that Python
   binary has been optimized:

 * https://bugs.python.org/issue40019

 * "PEP: Modify the C API to hide implementation details" proposed to
   python-dev:

 * 
https://mail.python.org/archives/list/python-...@python.org/thread/HKM774XKU7DPJNLUTYHUB5U6VR6EQMJF/#HBOKY4WKIV7Z4Y5RSYISEU7D5ABI2B2I

 * Victor Stinner's mentee Dong-hee Na is promoted to a core developer!

 * antigravity module adjusted for FIPS mode:

 * https://github.com/python/cpython/pull/19520

 * https://xkcd.com/353/

 * https://xkcd.com/426/

 * IDLE has a high resolution icon now, to be included in Fedora

 * Progress on isolating subinterpreters: the signal handler has been
   fixed on Windows.

 * Added a RHEL8 x86_64 FIPS buildbot to the upstream CI fleet

 * Some aarch64 upstream builders provided by python-main are now
   officially on the stable list, meaning we can say now that upstream
   officially supports aarch64  on Fedora, RHEL7 and RHEL8

 * https://github.com/python/buildmaster-config/pull/186


*Misc:*

 * pyp2rpm development/maintenance has been successfully transferred to
   the community

 * Delivered an open-source talk at Red Hat Czech Open House

All the best,
Python-maint

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

Re: DNF: "There are following alternatives to this package"

2018-09-13 Thread Tomas Orsava


On 09/13/2018 06:43 PM, Mathieu Bridon wrote:

On Thu, 2018-09-13 at 18:17 +0200, Tomas Orsava wrote:

We'd like to propose a new functionality for dnf: When a user tries
to install a package XYZ and dnf doesn't find it, dnf would recommend
them alternative packages. These offered packages would advertise
that they are an alternative for XYZ using a specially formatted
Provides tag.

For example, packages `python2-requests` and `python3-requests`
would both have the following tag:

  Provides: alternative-for(python-requests) = %{version}-
%{release}

(Possibly via the already existing and widespread %python_provide
macro in the Python case.)

And when the user would try to install `python-requests`, dnf would
look for packages with the relevant Provides tag and display them:

  # dnf install python-requests
* There are following alternatives to this package:
python2-requests python3-requests
  No match for argument: python-requests
  Error: Unable to find a match

This would be very similar to an already existing functionality that
searches for lowercase package names:

  # dnf install python3-REQUESTS
* Maybe you meant: python3-requests
  No match for argument: python3-REQUESTS
  Error: Unable to find a match

(That functionality is broken in some versions—RHBZ#1628514—so you
might not see this result at the moment.)

What are your thoughts?

It's neat, but it doesn't help catch typos, which seems like the most
probably case where I'd want such a feature.

How about instead of a scheme based on provides, just quickly check if
a package has a "similar" name?

Basically extend the existing check for lowercase you mention.


Catching typos would be useful, and I'd certainly support that addition, 
but this doesn't really scratch the itch I'm trying to solve.
We want to tell the user "these are exactly the alternatives available 
to you". Not guess, not rely on some typo resolution. We know what the 
alternatives are, here they are.


I've talked with people working on dnf and this looks like an easy 
mechanism to add, and it will be more reliable for the use cases it covers.


Tomas



   $ git stats
   git: 'stats' is not a git command. See 'git --help'.
   
   The most similar command is

   status



___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

DNF: "There are following alternatives to this package"

2018-09-13 Thread Tomas Orsava


Hi!
We'd like to propose a new functionality for dnf: When a user tries to 
install a package XYZ and dnf doesn't find it, dnf would recommend them 
alternative packages. These offered packages would advertise that they 
are an alternative for XYZ using a specially formatted Provides tag.


For example, packages `python2-requests` and `python3-requests` would 
both have the following tag:


    Provides: alternative-for(python-requests) = %{version}-%{release}

(Possibly via the already existing and widespread %python_provide macro 
in the Python case.)


And when the user would try to install `python-requests`, dnf would look 
for packages with the relevant Provides tag and display them:


    # dnf install python-requests
  * There are following alternatives to this package: 
python2-requests python3-requests

    No match for argument: python-requests
    Error: Unable to find a match

This would be very similar to an already existing functionality that 
searches for lowercase package names:


    # dnf install python3-REQUESTS
  * Maybe you meant: python3-requests
    No match for argument: python3-REQUESTS
    Error: Unable to find a match

(That functionality is broken in some versions—RHBZ#1628514—so you might 
not see this result at the moment.)


What are your thoughts?

One possible addition would be to include a parameter for weight in the 
Provides tag, so that python3-requests could be listed as the preferred 
option before python2-requests.


All the best,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org

What happened to Platform-Python?

2017-11-22 Thread Tomas Orsava


Hey everybody!
Some of you may have noticed RPMs called `platform-python-foo` on your 
Fedora 27, and even others may have noticed that these have recently 
been removed from rawhide. This is expected, read on if you'd like to 
know more.



Fedora 27

The `platform-python-*` RPMs are part of a F27 [Fedora Change] called 
the Platform Python Stack. This change was aimed primarily to support 
the F27 Modular Server release, specifically to create a completely 
separate Python stack that would be parallelly installable with Python 
from the `python3` module. This Platform-Python stack was needed to run 
DNF that lived in the Platform module, so that Fedora would continue 
working even when no modules were installed. The change was also 
deployed to mainline Fedora so that DNF would not behave differently on 
the two platforms.
In the end, however, the change was not successful. To make the 
Platform-Python stack completely separate, its files had to be forced to 
non-standard locations which, among other things, created enormous 
problems when trying to build C extensions for it. On top of this, 
Platform turned out to house many more Python tools and building all the 
Python dependencies for them was just too costly in this manner. F27 
Modular Server therefore ended up shipping the normal Python 3 stack in 
the Platform module, and `platform-python-*` packages were made obsolete 
in rawhide. Though they are not actually used for anything, the packages 
shall remain in F27 as it would be risky (and probably impossible) to 
remove them from a live release.


[Fedora Change] https://fedoraproject.org/wiki/Changes/Platform_Python_Stack


Future

The need for a Python stack in the Platform is still there, however, and 
thus we'll need to revise Platform-Python for F28. Currently we're 
trying to make the Platform-Python and the modular Python stacks share 
one internal implementation behind the scenes, eliminating the need for 
a costly new independent stack that has to be maintained separately and 
in non-standard paths. We'd like to post our current plan to 
python-devel ML soon.



All the best,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Warning before sudo pip'ing?

2017-02-22 Thread Tomas Orsava


Hi!
The warning when sudo pip installing is deployed in rawhide!

Package version: python-pip-9.0.1-6.fc26.

Cheers,
Tomas


On 02/10/2017 06:35 PM, Tomas Orsava wrote:

Hi!
On the last FESCo meeting while discussing the sudo pip Fedora 
[Change], maxamillion proposed that it might be useful to issue a 
warning when a user tries to run pip with root privileges--as in most 
cases it's not what they should be doing (`pip install --user` is 
usually more appropriate).


What do you think?

[Change] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Regards,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to 
python-devel-le...@lists.fedoraproject.org

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Mass-rebuild hiccup

2017-02-15 Thread Tomas Orsava


Hi!
It appears that there's been a bug in rpm during the time of Fedora 26 mass
rebuild. The [bug] resulted in some packages failing to be rebuilt and some
being built without .pyc files inside __pycache__ directories.

[bug]https://bugzilla.redhat.com/show_bug.cgi?id=1411588

Together with ignatenkobrain and sgallagh who alerted us to the issue, we've
managed to narrow down affected packages to Python 3–only packages that 
don't

drag Python 2 into the buildroot and don't use standard Python build/install
macros.

In the end the following 22 packages were rebuilt:

antimony (antimony-0.9.3-0.3.20161128git41a770.fc26.src.rpm)
btest (btest-0.54-7.fc26.src.rpm)
coccinelle (coccinelle-1.0.6-6.fc26.src.rpm)
devscripts (devscripts-2.17.1-3.fc26.src.rpm)
dnf-plugin-spacewalk (dnf-plugin-spacewalk-2.6.1-3.fc26.src.rpm)
dnfdragora (dnfdragora-0.0.0-0.111.git20170207.783aede.fc26.src.rpm)
gausssum (gausssum-3.0.1.1-4.fc26.src.rpm)
liborcus-python3 (liborcus-0.12.1-3.fc26.src.rpm)
loook (loook-0.8.1-7.fc26.src.rpm)
mock (mock-1.3.3-2.fc26.src.rpm)
mock-lvm (mock-1.3.3-2.fc26.src.rpm)
mock-scm (mock-1.3.3-2.fc26.src.rpm)
policycoreutils-devel (policycoreutils-2.5-21.fc26.src.rpm)
python3-fuckit (python-fuckit-4.8.0-8.fc26.src.rpm)
python3-louis (liblouis-2.6.2-8.fc26.src.rpm)
python3-lxc (lxc-2.0.7-1.fc26.1.src.rpm)
python3-meh-gui (python-meh-0.44-4.fc26.src.rpm)
python3-pillow-qt (python-pillow-4.0.0-2.fc26.src.rpm)
python3-pylint-gui (pylint-1.6.4-3.fc26.src.rpm)
python3-qt5-devel (python-qt5-5.7.1-4.fc26.src.rpm)
python3-uwsgidecorators (uwsgi-2.0.14-8.fc26.src.rpm)
speedtest-cli (speedtest-cli-0.3.2-7.fc26.src.rpm)

Regards,
Tomas Orsava

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Warning before sudo pip'ing?

2017-02-10 Thread Tomas Orsava

I should add that thanks to the aforementioned Fedora [Change] using 
`sudo pip` should not be dangerous as it could have been before, so the 
warning would be just for good measure.


[Change] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Tomas


On 02/10/2017 06:35 PM, Tomas Orsava wrote:

Hi!
On the last FESCo meeting while discussing the sudo pip Fedora 
[Change], maxamillion proposed that it might be useful to issue a 
warning when a user tries to run pip with root privileges--as in most 
cases it's not what they should be doing (`pip install --user` is 
usually more appropriate).


What do you think?

[Change] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Regards,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to 
python-devel-le...@lists.fedoraproject.org

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Warning before sudo pip'ing?

2017-02-10 Thread Tomas Orsava


Hi!
On the last FESCo meeting while discussing the sudo pip Fedora [Change], 
maxamillion proposed that it might be useful to issue a warning when a 
user tries to run pip with root privileges--as in most cases it's not 
what they should be doing (`pip install --user` is usually more 
appropriate).


What do you think?

[Change] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Regards,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

portingdb.xyz is back!

2016-12-27 Thread Tomas Orsava

We're pleased to announce that the Python 3 PortingDB is back at it's usual 
location:

http://portingdb.xyz/

Happy New Year!
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

2016-12-12 Thread Tomas Orsava


On 12/12/2016 05:39 AM, Nick Coghlan wrote:

On 11 December 2016 at 01:33, Donald Stufft  wrote:

On Dec 10, 2016, at 8:10 AM, Nick Coghlan  wrote:


P.S. For folks wondering what the problem with "--user" is on
Debian/Ubuntu, as far as I know it's mainly the fact that
"~/.local/bin" isn't on PATH by default, so scripts installed via
"--user" aren't automatically available.

FWIW, I think Nathaniel? got the uh, Bash? Maintainers on Debian to add it
to the default skel for users.

Ah, nice - that's very good to hear, as it removes one of the barriers
to making "--user" the default behaviour upstream in pip.

However, it also reminded me of our past discussions about defaulting
to --user in pip, and re-reading the relevant issue brought me to:
https://github.com/pypa/pip/issues/1668#issuecomment-107016411

The specific proposal there was to:

1. Add the `--global` flag to pip upstream, so folks can always force
environment global installs by doing:

 pip3 install --global 

2. Change the default install behavior in the Fedora system pip
package to include "--user" (and require an explicit "--global" to
override that)

The burden would then be on Fedora as the distro to ensure that "pip3
install " still offered a good user experience,
such as making sure that the per-user bin directory was on PATH by
default.

Actually doing that would mean taking Donald's first pass at
implementing the `--global` switch, bringing it up to merge-ready
standards, and then incorporating the pip release containing the
feature into Fedora: https://github.com/pypa/pip/pull/2418

The benefit of that approach is that it would not only solve the
conflict between dnf and pip at the Fedora level, but also move the
packaging user experience forward for the Python ecosystem as a whole
(by making the `--global` switch available in anticipation of a future
upstream switch to `--user` as the default).


It would be a partial solution I think. People who first try sudo-less 
`pip3 install` would be covered, but there are great many people that 
already go with `sudo pip3 install` right away, as well as many install 
guides on the Internet that advise to do so. In that case pip would 
probably install it under /root, which wouldn't be accessible from 
user-run python.


Tomas
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

2016-12-09 Thread Tomas Orsava


On 12/07/2016 06:21 PM, Stephen John Smoogen wrote:

On 7 December 2016 at 10:27, Tomas Orsava  wrote:

On 12/07/2016 01:56 PM, Neal Gompa wrote:

On Wed, Dec 7, 2016 at 7:53 AM, Michal Cyprian 
wrote:

- system-python (i.e. what all programs installed via DNF will use) is
limited to site-packages under /usr, so DNF-installed software is unaffected
by anything installed with pip

system-python is not intended for this use-case. It was designed to
support DNF itself and other related system software. It's also
intentionally set up to not have everything that regular python3 has.
I would really hesitate to allow this to be used for more than that,
because then we're back to square one, again...


I think there's been a slight miscommunication: system-python will be used
instead of python3 only when building/installing packages inside spec files.
The built packages themselves depend on will be run by python3.

There has been some sort of miscommunication because from my layman
take on things, what Neal was what system-python was only meant to be
used for .. for exactly the reason he said.


System-python is a rather complex and evolving topic, so let my try to 
explain it a bit deeper (as I understand it):


Up to this proposal, the system-python binary 
(/usr/libexec/system-python) has been simply a direct copy of the 
python3 executable (/usr/bin/python3{,.5}) [0]. The usefulness of 
system-python thus does not come from the executable you use, but from 
the `system-python` package that contains it. The difference between the 
`python3` and `system-python` packages is that the former requires the 
full Python 3 standard library (package `python3-libs`), whereas the 
`system-python` package requires only a smaller subset thereof (package 
`system-python-libs`). Thus saving space when Python is only needed for 
system tools.


This proposal constitutes a second evolutionary step of system-python: 
We now want to slightly modify how the two executables behave. Python 3 
will continue as it always has—you will be able to import modules 
whether you installed them through dnf or pip/setuptools. Contrary to 
that, system-python will become more limited: It will only recognize 
Python modules installed through dnf, and as a result, your system tolls 
(dnf et al.) will become impervious to problems stemming from 
non-functional/problematic modules installed using sudo pip, which up to 
now could simply overwrite crucial modules installed by dnf.


To implement this modification, the locations of dnf-installed and sudo 
pip–installed modules need to be separated. The default location where 
Python decides to install new modules is contained within the 
executable, and thus Python 3's install location will be the one where 
sudo pip–installed modules belong, while system-python will install 
modules to a path where dnf-installed modules ought to go. Therefore if 
you run sudo pip, Python 3 executable will be invoked and pip will 
install modules to the appropriate place. And in spec files of Python 
RPM packages, the %{__python3} macro will be modified so that 
system-python is invoked instead, and the modules will be installed to 
the appropriate location for dnf-installed modules.


A good question to ask at this point is: Is it ok to use system-python 
in spec files like this when it requires only a limited subset of the 
Python 3 standard library? The confusing part is that while 
system-python requires only a limited stdlib, it will use the full 
standard library if it is installed. And when it comes to Python RPM 
packages, it will always be installed because: 1. At build time all 
Python 3 packages have to use `BuildRequires: python3-devel` which 
brings in a full stdlib complement, and 2. at run time the resulting RPM 
packages will depend on the `python(abi) = 3.X`, which is only satisfied 
by the `python3` package and it's full stdlib.


I hope this may shine a bit of light on the issue.

[0] 
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?id=ddb16c68d95603cc11233c520544adf92c3741fe#n1061

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

2016-12-07 Thread Tomas Orsava


On 12/07/2016 02:47 PM, Konstantin Zemlyak wrote:

Michal Cyprian wrote:

there is a long-standing problem that `sudo pip install` cannot be 
safely used in Fedora. Many users don't know about this and break 
python packages on theirs systems. Packages installed using this 
command can conflict and overwrite Python rpm packages.
This is a major problem and we have seen several systems broken by 
people using "sudo pip". Unfortunately, telling people to not use it 
will not work: "sudo pip" appears in documentation of too many 
projects online.


The default install location of pip/distutils-installed packages 
depends on the value of the

  sys.prefix variable [4].


But this default can be trivially changed by distro.

There were several discussions on Bugzilla [1] and the pypa-dev 
mailing list [2].

Interesting solutions were conceived at the CPython level:
 - addition of sys.local_prefix [2]
 - simplification of CPython startup sequence [4]
Unfortunately none of them were realized and both solutions require 
many changes of Python and Python Standard Library.


But you don't need to do either.

One can probably either modify python package itself or create another 
one with two files with following contents:


/usr/lib64/pythonX.Y/distutils/distutils.cfg:

[install]
install-lib = /usr/local/lib/pythonX.Y

/usr/lib64/pythonX.Y/sitecustomize.py:

import site
site.addsitedir('/usr/local/lib/pythonX.Y')

Or whatever custom-packages directory you want to redirect it to.

That's it. First file will change default location of installed 
packages. Second one will add it to site dirs to support imports from 
those local packages. Maybe sitecustomize.py will need few tweaks to 
sitedirs order to make it work better.


This is documented in "legacy" section of python docs on distutils:

https://docs.python.org/3.5/install/#custom-installation



I'm not sure this would produce the desired effect, that is:
- system-python: installs to and imports only from /usr/lib*/pythonX.Y
- python3: installs to /usr/local/lib*/pythonX.Y but loads from both 
/usr/... and /usr/local/...

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: Making sudo pip Safe

2016-12-07 Thread Tomas Orsava


On 12/07/2016 01:56 PM, Neal Gompa wrote:

On Wed, Dec 7, 2016 at 7:53 AM, Michal Cyprian  wrote:

- system-python (i.e. what all programs installed via DNF will use) is limited 
to site-packages under /usr, so DNF-installed software is unaffected by 
anything installed with pip

system-python is not intended for this use-case. It was designed to
support DNF itself and other related system software. It's also
intentionally set up to not have everything that regular python3 has.
I would really hesitate to allow this to be used for more than that,
because then we're back to square one, again...


I think there's been a slight miscommunication: system-python will be 
used instead of python3 only when building/installing packages inside 
spec files. The built packages themselves depend on will be run by python3.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

PortingDB is temporarily moved to a new address

2016-12-02 Thread Tomas Orsava


Hi!
PortingDB—a tool providing an overview of the Python 3 porting progress 
for Fedora packages—is temporarily unavailable through it's regular 
address [0] due to some DNS issues.


You can access PortingDB here in the meantime: 
http://portingdb-encukou.rhcloud.com/


We apologize for the inconvenience.
Tomas Orsava

[0] http://portingdb.xyz/
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

2016-12-01 Thread Tomas Orsava


On 12/01/2016 02:36 PM, Igor Gnatenko wrote:

We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.


Yeah, absolutely. When it's battle-tested and solid, it will make 
packaging that much easier.



Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.


Oh that's clever, I was wondering how things like that are done!


I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...


AFAIK the best way to get the guidelines updated is to create an 
accompanying Fedora Change. [0]
I sadly don't have free cycles to take that on, as I'm currently 
involved in 2 upcoming Fedora Changes, nevertheless I will gladly 
provide any help and guidance you might need!


[0] https://fedoraproject.org/wiki/Changes/Policy
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

2016-11-30 Thread Tomas Orsava


On 11/30/2016 02:44 PM, Neal Gompa wrote:

On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:

I don't think the depgen should be enabled by default, at least not in the
foreseeable future. IIRC it's not that well implemented—e.g. I believe it
doesn't read requirements.txt for example (but I might be wrong).
There will be a lot of cases where the generated requirements are
incomplete, or contain unnecessary entries, etc. I think it should remain an
opt-in.


According to various Python people, we're not actually supposed to
read requirements.txt. That file is explicitly designed for people to
individualized deployments. The proper place to get this information
is from the egg-info/dist-info data, which is what we read. The fact
that some people abuse requirements.txt and have it read in by their
setup.py is beside the point. Whatever the setup.py (thus
pip/easy_install/etc.) says it needs, the generator will dutifully
report.


The fact remains in too many cases it will need to be manually adjusted, 
it won't be foolproof.
Therefore I argue it would be better for it to be an opt-in so that new 
packagers don't immediately have to jump in into debugging a depgen they 
have no clue how really works.

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: [RFC] RPM's Python dependency generator

2016-11-30 Thread Tomas Orsava


On 11/30/2016 08:04 AM, Igor Gnatenko wrote:

Hi,

in short, it reads egg metadata and can generate Provides (which we
already do now), Requires (which I want to talk about) and Recommends
(which I don't care atm).

Let's take simple package -- aiohttp.
https://bugzilla.redhat.com/show_bug.cgi?id=1381750

As you can see, since some version multidict requirement was bumped to

= 2.0 and async_timeout requirement was added. Currently we specify

all requirements during initial package and usually without versions
which is breaking after some time.

So, let's try it (I will skip unrelated parts).
Before:
python(abi) = 3.5
python3-chardet
python3-multidict
After:
python(abi) = 3.5
python3.5dist(async-timeout)
python3.5dist(chardet)
python3.5dist(multidict) >= 2.0

Without more complicated packages (MNE, nipy, nilearn, moss) it's
getting much more harder since I have there 10+ deps.

We can't really track all changes in upstream code, so if we will
enable dependency generator, it will do this work for us. Note that we
can't just enable it in RPM, because it will break a lot of packages
due to:
* missing requires in egg metadata
* extra requires in egg metadata (e.g. windows-modules)

I would propose plan:
1. Create macro which will enable/disable depgen (e.g %python_enable_depgen)


That would be cool!


2. Start enabling depgen and porting things (somehow reuse
portingdb.xyz probably?) and submitting patches upstream


While portingdb is open source and you can definitely set up a new 
instance, I think that would be an overkill. Whether a package supports 
Python 3 or not is highly relevant to other packages, since they depend 
on one another.
However, whether a package manually lists its Requires or uses a script 
to do that is not relevant to other packages, therefore I don't see the 
benefit of a portingdb portal here.



3. In 1-2 releases I hope we can use it for major amount of packages
4. Enable depgen by default in RPM, add disabling depgen for remaining packages


I don't think the depgen should be enabled by default, at least not in 
the foreseeable future. IIRC it's not that well implemented—e.g. I 
believe it doesn't read requirements.txt for example (but I might be wrong).
There will be a lot of cases where the generated requirements are 
incomplete, or contain unnecessary entries, etc. I think it should 
remain an opt-in.


Regards,
Tomas
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

PEP: Distributing a Subset of the Standard Library

2016-10-21 Thread Tomas Orsava


Hi!

We have prepared a PEP that aims to standardize and improve what Fedora 
and other distributions have been doing for a while, that is splitting 
parts of Python's standard library to separate optional packages. 
Feedback is welcome:


https://fedora-python.github.io/pep-drafts/pep-A.html


 Abstract

Python is sometimes being distributed without its full standard library. 
However, there is as of yet no standardized way of dealing with 
importing a missing standard library module. This PEP proposes a 
mechanism for identifying which standard library modules are missing and 
puts forth a method of how attempts to import a missing standard library 
module should be handled.


I'm leaving for PTO lasting till November 17th, after which we want to 
send this to the PEP editors upstream.


Yours aye,
Tomas Orsava

___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: python34 for epel6 ?

2016-10-17 Thread Tomas Orsava


On 10/17/2016 08:52 AM, Nick Coghlan wrote:

On 15 October 2016 at 02:00, Tomas Orsava  wrote:

Hi!

I'm not able to find it myself, and nobody chimed in so far. Have you had
any luck?

OK, I filed https://bugzilla.redhat.com/show_bug.cgi?id=1385471

There's a mini-rant in the "Additional Info" section regarding RPATH,
as not setting that appropriately on the SCL Python runtime binaries
is causing the bulk of these dynamic linking problems, so if it was
set properly, the current issues would almost all just go away and you
wouldn't have to find and patch all the Python projects that assume
'sys.executable' can be used the same way it can upstream and then
hope that nobody does "pip install --upgrade" on a patched downstream
component :)
Thank you, I wasn't sure I would be able to formulate it as clearly as 
you could!


Yours aye,
Tomas
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: python34 for epel6 ?

2016-10-14 Thread Tomas Orsava


Hi!

I'm not able to find it myself, and nobody chimed in so far. Have you 
had any luck?


Yours aye,
Tomas


On 10/05/2016 06:37 PM, Nick Coghlan wrote:

On 5 October 2016 at 21:45, Tomas Orsava  wrote:

On 10/04/2016 03:10 PM, Nick Coghlan wrote:

I bring that up, as one of the other challenges with the way SCLs
currently work is that the upstream convention of copying
"sys.executable" into the shebang to generate a script that runs in
the current Python doesn't work as expected.

That would be cool, but it's not planned as of yet. Maybe we should create
an RFE for it?

I think there may be one already, as I seem to recall suggesting
checking whether or not 'pipsi' works as a possible test case (pipsi =
"pip script installer", which creates a virtualenv for the installed
package and its dependencies, then ensures the virtualenv is activated
when any installed scripts are run.

However, I can't find it in a quick look at Bugzilla, so it may have
just been an email thread rather than a Bugzilla RFE.

Cheers,
Nick.


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Updating Python in Fedora 24

2016-10-11 Thread Tomas Orsava


Hi!
I'm planning to update Python in Fedora 24 from version 3.5.1 to 3.5.2 [0].

It transpired that Python 3.5 made a rather unfortunate change of making 
the `os.urandom()` function blocking by default, which is causing 
problems during the boot process. [1] This was thankfully reverted in 
Python 3.5.2.


After a discussion on #fedora-python, the consensus is to update Python 
in F24 rather than backport the specific patches. As a bonus, Python 
3.5.2 brings with it a lot of unrelated bugfixes as well [0].


If you have a serious reason against updating Python in F24, speak now 
or forever hold your peace.


[0] https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-2
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1383060

Yours aye,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: python34 for epel6 ?

2016-10-05 Thread Tomas Orsava


On 10/04/2016 03:10 PM, Nick Coghlan wrote:

On 3 October 2016 at 23:36, Tomas Orsava  wrote:

On 09/27/2016 08:43 AM, Nick Coghlan wrote:

P.P.S. I realize rh-python34 is available on RHSCL, but it didn't seem to
support "python3" in scripts...

Script shebang lines can be supported via SCLs, but they need to run a
wrapper script that implicitly enables the SCL, rather than just being
a symlink to the SCL's Python executable. I don't recall the exact
incantation myself, but hopefully someone else will be able to chime
in with that.

Right now you need to create and use a wrapper script that contains:

 #!/bin/bash
 /usr/bin/scl enable rh-python34 -- python3 "$@"

However, when this BZ [0] gets into RHEL 7.4 (there's also talk of 7.3
Z-stream), you'll be able to enable the collection right in the shebang like
so:

 #!/usr/bin/scl enable rh-python34 -- python3

Nice. Would that come with a corresponding update to the pip and
setuptools versions in RHSCL such that their wrapper script generators
did the right thing?

I bring that up, as one of the other challenges with the way SCLs
currently work is that the upstream convention of copying
"sys.executable" into the shebang to generate a script that runs in
the current Python doesn't work as expected.
That would be cool, but it's not planned as of yet. Maybe we should 
create an RFE for it?


Tomas



Cheers,
Nick.


___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: python34 for epel6 ?

2016-10-03 Thread Tomas Orsava


On 09/27/2016 08:43 AM, Nick Coghlan wrote:

P.P.S. I realize rh-python34 is available on RHSCL, but it didn't seem to
support "python3" in scripts...

Script shebang lines can be supported via SCLs, but they need to run a
wrapper script that implicitly enables the SCL, rather than just being
a symlink to the SCL's Python executable. I don't recall the exact
incantation myself, but hopefully someone else will be able to chime
in with that.

Right now you need to create and use a wrapper script that contains:

#!/bin/bash
/usr/bin/scl enable rh-python34 -- python3 "$@"

However, when this BZ [0] gets into RHEL 7.4 (there's also talk of 7.3 
Z-stream), you'll be able to enable the collection right in the shebang 
like so:


#!/usr/bin/scl enable rh-python34 -- python3

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1369788

Regards,
Tomas Orsava
___
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org

Re: PEP: Distributing a Subset of the Standard Library

2016-09-08 Thread Tomas Orsava


On 09/07/2016 06:13 PM, Nick Coghlan wrote:

On 7 September 2016 at 19:30, Tomas Orsava  wrote:

On 09/06/2016 08:25 PM, Nick Coghlan wrote:

Very interesting, although I see a pragmatic problem with trying to
check for explicitly missing packages only after checking for the
standard library ones: the default import system doesn't make a clear
distinction as to which sys.path entries refer to the standard library
and which refer to other directories (like site-packages), so you
can't readily intercept processing after the standard library is
checked but before the rest of sys.path is processed.

I have created a proposal for the Reference Implementation which you can
find here:
https://github.com/torsava/cpython/pull/1

It works by checking for the .missing.py files in every directory on the
sys.path. Therefore the packager can put it into any directory they use for
stdlib, and it will work (that directory of course already is on the
sys.path). And because it checks for the .missing.py file only after it
tried every other possible file extension in each directory, it won't shadow
actually installed stdlib modules.

Oh, nice, that's very cool, and way simpler than anything I was thinking of :)

Cheers,
Nick.


Credit goes to Petr Viktorin, the author of the idea!

Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: PEP: Distributing a Subset of the Standard Library

2016-09-07 Thread Tomas Orsava


Hi!

On 09/06/2016 08:25 PM, Nick Coghlan wrote:

On 7 September 2016 at 02:41, Tomas Orsava  wrote:

Hi!

I'm currently writing a PEP titled "Distributing a Subset of the Standard
Library" to standardize and hopefully improve the behavior of Python without
the its full standard library. This is relevant to Fedora, as we exclude
several standard library modules into separate optional packages (e.g.
python3-tkinter).

I have a draft of the first two sections: Motivation and Specification.
https://fedora-python.github.io/pep-drafts/pep-A.html

Very interesting, although I see a pragmatic problem with trying to
check for explicitly missing packages only after checking for the
standard library ones: the default import system doesn't make a clear
distinction as to which sys.path entries refer to the standard library
and which refer to other directories (like site-packages), so you
can't readily intercept processing after the standard library is
checked but before the rest of sys.path is processed.
I have created a proposal for the Reference Implementation which you can 
find here:

https://github.com/torsava/cpython/pull/1

It works by checking for the .missing.py files in every directory on the 
sys.path. Therefore the packager can put it into any directory they use 
for stdlib, and it will work (that directory of course already is on the 
sys.path). And because it checks for the .missing.py file only after it 
tried every other possible file extension in each directory, it won't 
shadow actually installed stdlib modules.


Tomas Orsava

However, sys.meta_path *does* let you explicitly block imports before
the default machinery is tried by raising ImportError from find_spec:
https://docs.python.org/3/reference/import.html#the-meta-path

Now, I'm making the assumption that what we need is a model whereby
the base install includes files that tells Python "these stdlib pieces
might be missing", and then the other packages can install files that
mean those "these pieces are missing" markers don't get processed.

One possible way to do that as a pre-import check injected into the
start of sys.meta_path would be to maintain a set of static
"module_name.optional" files in the standard library directory that
included:

- a relative file path to stat to indicate that the optional module is installed
- an import error message to raise if its not found

If the new hook either didn't see an optional module entry, or if it
checked and the file path was present, it would allow the import to
continue as normal. However, if it found the optional declaration file
and the file path missing, it would raise ImportError with the given
message.

CPython by default would provide optional declarations for the modules
that are optional in the upstream build process, then downstreams like
Fedora may add more for things like tkinter, idlelib, and the test
suite.

The downside of this approach is that it does mean the initial import
of optional modules would be a touch slower (since the file path to
stat and the error message would need to be read from file), whereas
the version in the draft PEP has the virtue of having no impact on
import time for modules that are available, even when starting from a
completely cold sys.modules cache.

So if we did want to enable the draft proposal in the PEP, we'd need
to look at proposing a special directory to hold the "missing" markers
that was always placed just after the standard library directory on
sys.path, and then defining a custom path_hook to process it:
https://docs.python.org/3/reference/import.html#path-entry-finders

Cheers,
Nick.


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: PEP: Distributing a Subset of the Standard Library

2016-09-06 Thread Tomas Orsava

I forgot to include a link to a previous discussion of this topic on the 
python-dev upstream mailing list:


https://mail.python.org/pipermail/python-dev/2016-July/145534.html

Tomas


On 09/06/2016 06:41 PM, Tomas Orsava wrote:

Hi!

I'm currently writing a PEP titled "Distributing a Subset of the 
Standard Library" to standardize and hopefully improve the behavior of 
Python without the its full standard library. This is relevant to 
Fedora, as we exclude several standard library modules into separate 
optional packages (e.g. python3-tkinter).


I have a draft of the first two sections: Motivation and Specification.
https://fedora-python.github.io/pep-drafts/pep-A.html

The source can be found here: https://github.com/fedora-python/pep-drafts

All the best,
Tomas Orsava


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org 


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

PEP: Distributing a Subset of the Standard Library

2016-09-06 Thread Tomas Orsava


Hi!

I'm currently writing a PEP titled "Distributing a Subset of the 
Standard Library" to standardize and hopefully improve the behavior of 
Python without the its full standard library. This is relevant to 
Fedora, as we exclude several standard library modules into separate 
optional packages (e.g. python3-tkinter).


I have a draft of the first two sections: Motivation and Specification.
https://fedora-python.github.io/pep-drafts/pep-A.html

The source can be found here: https://github.com/fedora-python/pep-drafts

All the best,
Tomas Orsava


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: [EPEL-devel] python34 for EPEL6

2016-08-25 Thread Tomas Orsava



On 08/24/2016 11:39 PM, Neal Gompa wrote:

On Wed, Aug 24, 2016 at 5:38 PM, Orion Poplawski  wrote:

I have no idea if there is any interest in this or not.  I managed to get the
EPEL7 python34 package to build on EL6 with a few modifications.  Is there any
interest in supporting this?


I think the Koji people would be interested in this, as it would help
them in moving Koji to Python 3. They have a requirement for Koji to
be able to run on EL6, so this could help unblock moving to Python 3.
I might be wrong, but I believe I have heard at Flock that Koji is 
actually trying to maintain support of EL5 still!

___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Update python-django EPEL7

2016-08-16 Thread Tomas Orsava

As far as I understand, someone could package it as a new package, e.g. 
python-django18.



On 08/16/2016 07:11 PM, Igor Gnatenko wrote:

It's against policies. Currently python-django has version 1.6. And
1.8 is major release.

On Tue, Aug 16, 2016 at 6:17 PM, Germano Massullo
 wrote:

Hello, I would like to ask if it is possible to update python-django
for EPEL7 to, for example, 1.8 version.
I am actually going to fill a Review Request for
python-django-netjsongraph[1], and among its requirements, there is
python-django-rest-framework that cannot actually be packaged for
EPEL7 due python-django actual version

Thank you

[1]: https://github.com/interop-dev/django-netjsongraph
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org




___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Python 3.5.2

2016-08-16 Thread Tomas Orsava


Hi!
I'm happy to inform you that I've rebased Python 3 in Fedora rawhide to 
a new version 3.5.2 up from 3.5.1.
The official build has just finished [0], and should show up in rawhide 
repos in the upcoming days.


I invite everyone to try it out, if all goes well I'll push the update 
to Fedora 25 as well.


[0] http://koji.fedoraproject.org/koji/taskinfo?taskID=15274894

Good day to all,
Tomas Orsava
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Rebasing Python, deleting patches

2016-08-15 Thread Tomas Orsava


Hi,
I'm currently rebasing Python to version 3.5.2 for Fedora 25.

As many of the existing patches are no longer necessary I needed to 
delete or disable them. I looked through the git history of python and 
python3 packages, and there isn't a clear consensus on which method is 
preferred. While python3 package so far only has patches deleted, python 
package had many patches that were just disabled/commented out, but not 
deleted.


I decided deleting no longer useful patches is the better option, not 
only because it was a more prevalent method in the past, but also 
because it keeps the spec file clean for the years to come. No data is 
actually lost as it remains in git history.


Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

2016-08-15 Thread Tomas Orsava


On 08/12/2016 05:40 PM, Petr Viktorin wrote:
The idea with pyp2rpm is to work with the Python Packaging Authority 
so that the upstream metadata *can* be converted automatically. 
Ideally Fedora packagers would fix packaging problems upstream, rather 
than maintaining spec files.

Ideas for a tool for *updating* spec files are also floating around.

There's already rebase-helper (it's an interactive tool as well as an 
automatic one) that specializes in updating spec files.

___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Automatic Provides: Discussion summary and plan

2016-08-11 Thread Tomas Orsava


Hi!
No, this automatic Provides generator does not create "python(name)". 
Only "pythonX.Ydist()" and "pythonXdist()".



On 08/10/2016 07:09 PM, Avram Lubkin wrote:
Is an unversioned python(name) also being created automagically? I 
manually create these for the command line tools which support 
multiple Python versions, where the python2-name and python3-name 
package both provide python(name) and then the name package requires 
python(name) with a recommends for one or the other.



On Wed, Aug 10, 2016 at 12:38 PM, Petr Viktorin > wrote:



To help automatic building RPMs from native Python packages, we
need an
automatic way to record the software's upstream name (dist name, name
on PyPI) in Fedora packages.

For this, we are using RPM's automatic dependency generator written by
Per Øyvind Karlsen and Neal Gompa, which automatically scans
".dist-info" and similar files and generates virtual Provides,
currently in the form "pythonX.Ydist(name)".
(The generator was originally developed for Mandriva and Mageia;
finally
it's coming to Fedora as well!)
The Fedora Change page for this feature is here:

https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages



Unfortunately, while implementing this, we failed to understand
how the
virtual provides work with source RPMs. This leads to the situation
that, as currently planned, the automatic provides are only useful in
"Requires" lines, not in "BuildRequires".

The problem is that if a requirement like "python3.5dist(name)" is
present in a SRPM, the SRPM can't be rebuilt on systems with other
Python versions, which would provide, say, "python3.6dist(name)".
For these systems, the SRPM would need to be rebuilt, which not all
tools do.

Igor Gnatenko (ignatenkobrain) explained this on IRC on
#fedora-python,
after which we had a long discussion about the problem and possible
solutions. We're sorry for not getting this earlier -- despite several
people mentioning it (including Neal who wrote the generator)!


The fix is to enable "--majorver-provides" in the dependency
generator,
so that each package will provide two forms of the virtual provide:
python3.5dist(name)
python3dist(name)
The full version should then be used for runtime Requires, while the
one with major version only should be used for BuildRequires.

We'll wrap this up in easy-to use (and hopefully future-proof) macros:

* One macro for getting the canonical (normalized) dist-name:
%{python_dist_name NAME}

* Four macros for adding Requires and BuildRequires lines (which
use the
  python_dist_name macro above):

  %{requires_python3_dist NAME} => Requires: python3.Ydist(name)
  %{buildrequires_python3_dist NAME} => BuildRequires:
python3dist(name)
  and similarly for Python 2.

An alternative would be macros that don't include the keyword
"Requires:" or
"BuildRequires:". This would result in specfiles with lines like:
BuildRequires: %{python3_dist_br name}
Requires: %{python3_dist name}
This would be susceptible to people copying the Requires line, adding
"Build" to the front, and forgetting to change the macro as well,
leading to a hard-to-catch failure (working binary RPMs but SRPMs that
fail to rebuild on other distro versions).
Macros that include the keyword are more robust to copy-pasting.

Since the %buildrequires_python3_dist and %python_dist_name macros
will
be used in the SRPM, they'll need to go in macros.python-srpm to be
present in the default buildroot.
The %requires_python3_dist macro (and %pythonX_version) can live in
macros.pythonX.


Following is the new road plan:


# Plan for Fedora 25:
* The 5 macros will be created and deployed.
* The --majorver-provides swich for the Provides generator in
Fedora RPM will
  be enabled
* Fedora Packaging Guidelines for Python will be amended:
* The addition of the pythonX.Ydist(..) tags will be explained.
* The %{python_dist_name} macro will be advertised.
* The %{requires_pythonX_dist} macros will be advertised for
use in
  generating Requires tags.
* It will be explained that, at this time, it is impossible to
  generate BuildRequires without the providing package being
  rebuilt, so the %{buildrequires_pythonX_dist} macros will be
  discouraged for now.
* See if we can get in another targeted mass rebuild. If yes, continue
  with the f26 plan early.


# Plan for Fedora 26:
* All Provides will be regenerated in the regular Fedora 26 mass
rebuild
* Change Python guidelines so the %{buildrequires_pythonX_dist} macros
  are now e

Re: Automatic Provides: Discussion summary and plan

2016-08-11 Thread Tomas Orsava




Yeah, I really wish I had actually pushed through the macro work I had
done last year.  You can see that at https://pagure.io/python-macros

A spec would look like this:
https://pagure.io/python-macros/blob/master/f/test1.spec

And most of that is actually implemented.  Note the almost complete
absence of version-specific bits.  That package builds for an arbitrary
number of system python versions automatically.

That looks incredible! Why didn't it see the light of day? Time 
constraints or some technical issues?


Maybe it could be revived?

Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Problems with scripts in a common spec file

2016-05-30 Thread Tomas Orsava


Hi!


On 05/27/2016 07:44 PM, John Dennis wrote:

On 05/27/2016 10:10 AM, Tomas Orsava wrote:

I think the python2-XXX package in the examples is missing something
like this:

Requires: %{_bindir}/sample-exec

Make sense?


I believe there is a misunderstanding. In your first message you said
"But the guidelines require the py3 version of the script in the py2
package." That is incorrect.

The guidelines only require you to package the Python 3 version of the
script. They are not very clear on where you should put it, but the only
logical place is of course the Python 3 subpackage (or a new subpackage
with only the executable script that relies on the Python 3 subpackage).

You should never put the Python 3 executable in the Python 2 subpackage.


That's fine. But unless one of two things are done the guidelines as 
they currently stand will leave you with a broken python2-XXX package. 
You either have to copy the Py2 version of the script to 
%{_bindir}/sample-exec-2.7 after the %py2_install runs instead of 
deleting the scripts as is currently recommended. -OR- you need to add 
a Requires on the %{_bindir}/sample-exec to the python2-XXX package as 
I suggested above.


If one of these 2 things are not done then someone installing just the 
python2-XXX version of the package (what I expect might be a common 
user action) will not have a script that can execute.
That is as it should be. The application should be provided in only one 
subpackage, not both, unless it behaves differently on Python 2 and 
Python 3 (example would be e.g. pip that does behave differently). If 
the user wants the application, they will have to install the Python 3 
subpackage.


Also neither set of guidelines include examples of setting up links if 
you elect to use versioned scripts (e.g. %{_bindir}/sample-exec-2.7, 
%{_bindir}/sample-exec-3.5). The guidelines seem to suggest it's best 
to avoid versioned scripts which I would agree with, if that is the 
case then of the two options discussed above only the Requires fix 
will work.
The guidelines are indeed flawed. But the Python RPM Packaging Guide 
(not part of the guidelines) does include examples of this [0]. However, 
it's only explained in the 4th chapter—chapter for packages where the 
executable behaves differently depending on the Python version (2 or 3). 
In all other cases you should not use versioned executables as the 
executable shall be included in only one version.


[0] http://python-rpm-porting.readthedocs.io/en/latest/tools.html
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Reg- binclock porting from python 2 to python 3

2016-05-30 Thread Tomas Orsava


Hi Rajesh!

the `binclock` project is a bit problematic. As you can see in the note 
on the left side on the PortingDB, the upstream for this project is 
abandoned, emails bounce. If you go look into the Bugzilla link on the 
same page, you'll also find that someone has made a Python 3 patch, but 
the maintainer is not responding.


The best thing you could do is to fork the upstream (i.e. create a new 
repo on GH/BitBucket/GitLab and put the current source files there), and 
then use the patch on BZ to port it to Python 3. Test if it works, and 
then post to the Bugzilla that the software has a new upstream it should 
switch to.


Alternatively, you can try your luck on another package.
Are you interested more in porting Python code or Fedora spec files for 
Python projects?


Thank you for your initiative!
Tomas


[0] http://fedora.portingdb.xyz/pkg/binclock/


On 05/29/2016 09:24 AM, Rajeshkumarpothiappan wrote:

Hi Fedora Team,

  I would like to contribute in porting binclock from python 2 to 
python 3.

I found http://fedora.portingdb.xyz/pkg/binclock/
https://admin.fedoraproject.org/pkgdb/package/rpms/binclock/

Please guide me where to start with or whom to contact and how to .

by
Rajeshkumar P



___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org


___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Problems with scripts in a common spec file

2016-05-27 Thread Tomas Orsava


Hi!

On 05/26/2016 07:38 PM, John Dennis wrote:

On 05/26/2016 08:24 AM, Tomas Orsava wrote:

Hi,

those are very good questions to which you should be able to find
answers on the Python RPM Porting Guide [0]. You are right that this
should be better covered in the packaging guidelines, sadly the process
of changing them is rather problematic so no one yet had the time to
update them.

[0] http://python-rpm-porting.readthedocs.io/en/latest/


Thanks Tomas, I was aware of this document as well. However I believe 
both documents contain the same mistake.


Here is the problem:

Only the python3-XXX package installs the (Py3) script. Unless the 
python2-XXX package requires the script neither the script nor the Py3 
modules/packages required by the (Py3) script will be installed when 
you install either the python-XXX or python2-XXX package (assuming the 
virtual provides of python-XXX points to python2-XXX as is currently 
the case).


I think the python2-XXX package in the examples is missing something 
like this:


Requires: %{_bindir}/sample-exec

Make sense?

I believe there is a misunderstanding. In your first message you said 
"But the guidelines require the py3 version of the script in the py2 
package." That is incorrect.


The guidelines only require you to package the Python 3 version of the 
script. They are not very clear on where you should put it, but the only 
logical place is of course the Python 3 subpackage (or a new subpackage 
with only the executable script that relies on the Python 3 subpackage).


You should never put the Python 3 executable in the Python 2 subpackage.

Kind regards,
Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

New PyPI URL format for tarballs/sources

2016-05-26 Thread Tomas Orsava


Hi!
The Python Package Index (PyPI) has decided to change the format of the 
URLs to download tarballs/sources. The new format is not predictable, 
because part of the URL is a hash of the contents of the file. [0]


This means that for the vast majority of Python packages (those using 
PyPI to download sources) the Source0 link needs to be updated.


The maintainer can either go to PyPI with each version update and copy 
the URL containing the hash, or use the new URL redirector provided by 
PyPI with a predictable format (like the previous one was):


https://files.pythonhosted.org/packages/source/p/positional/positional-1.1.0.tar.gz 



The URL redirector will have longterm support [1], so I believe it's the 
better choice.



Maybe it would be beneficial to also work the change into the RPM 
rebase-helper [2] which does automatic scratch builds when a new version 
of software is detected upstream? Does anyone have experience with this 
project?



[0] 
https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package 

[1] 
https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package#comment-27734791 


[2] https://github.com/phracek/rebase-helper

Great day to all,
Tomas Orsava
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Problems with scripts in a common spec file

2016-05-26 Thread Tomas Orsava


Hi,

those are very good questions to which you should be able to find 
answers on the Python RPM Porting Guide [0]. You are right that this 
should be better covered in the packaging guidelines, sadly the process 
of changing them is rather problematic so no one yet had the time to 
update them.


[0] http://python-rpm-porting.readthedocs.io/en/latest/

Kind regards,
Tomas



On 05/26/2016 12:18 AM, John Dennis wrote:

With reference to the guidelines for Python packaging found here:

https://fedoraproject.org/wiki/Packaging:Python

Specifically the material concerning executables in /usr/bin.

In the "Example common spec file" section is this comment.

# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.

But this does not work if you only install the python2-XXX package 
instead of both the python2-XXX and python3-XXX packages. Here is why.


distutils will adjust the shbang interpreter line in installed scripts 
to be interpreter specific, e.g. you get one of:


#!/usr/bin/python2

or

#!/usr/bin/python3

But the guidelines require the py3 version of the script in the py2 
package. Thus if you install just the python2-XXX package you'll end 
up with a script that runs the /usr/bin/python3 interpreter whose 
sys.path only includes library locations for py3! Thus the script 
cannot execute because it's missing its libraries (because the 
libraries were installed in the py2 locations).


How is this supposed to be addressed?

One possible workaround is to add a requirement on the py3 package in 
the py2 package. But that is silly because it completely negates 
everything in the py2 package (everything will run as if it were py3 
after installing the py2 package and the py2 files will never be 
referenced).


Perhaps the issue is better rephrased as "How does one package Python 
when there are no version specific differences/requirements?" If this 
is covered in the guidelines it's not obvious and needs clarification. 
I assume this means using the generic %{_python} macros instead of 
dual use of the version specific macros so that everything corresponds 
to the system default version of Python. Correct?





___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Problems with script installation in RPM's

2016-05-12 Thread Tomas Orsava


On 05/12/2016 05:16 PM, Miro Hrončok wrote:

On 12.5.2016 15:24, John Dennis wrote:

On 05/11/2016 11:54 AM, Petr Viktorin wrote:

On 05/11/2016 05:16 PM, John Dennis wrote:
The workaround I came up with is to delay the execution of 
%py3_build by

at least 1 second by inserting a sleep in-between the %py2_build and
%py3_build macros in the spec file, like this:

%py2_build
sleep 1
%py3_build

Actually I think sleeping 2 seconds might be safer given the 1 second
resolution on the timestamps.


A better workaround would be 'rm /usr/bin/xyz' instead of the sleep.


I understand where you're coming from Petr but let me offer a reason for
why a short sleep might be preferable.

In the interest of robustness it's best to avoid requiring someone to
enumerate exceptions. File lists change as do other aspects of the
build. It's very easy to miss enumerating every necessary exception on
every update to the spec file. If a simple short sleep is sufficient to
avoid manual enumeration then the minor inefficiency of the sleep seems
a price well paid.



Well, on the other hand, if setuptools/distutils decides that it will 
NEVER override a file, sleep stops working, rm works.


With rm, what's happening is crystal clear. With sleep, not so much.

rm is more explicit, explicit is better than implicit.

That seems like the best argument and solution, I'll amend the Python 
RPM Porting Guide [PyRPG] accordingly.


[PyRPG] http://python-rpm-porting.readthedocs.io/
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Inconsistencies in the Fedora Packaging Guidelines for Python

2016-03-22 Thread Tomas Orsava

> On Mon, Mar 21, 2016 at 05:35:03PM -0400, Neal Gompa wrote:
> Ah, OK. It didn't at some point and I didn't check.
> It seems that the Guidelines:Python page could still use some
> editing. I think most of the info is there, but it's not very clear.
> 
> In particular, the multiple-executables case is again very prominent
> (as this thread shows), and it's really applicable to a miniscule
> percentage of packages (literally: sphinx, pytest, nosetest, a bunch
> of python development and installation tools. There's a spattering
> of other random packages, which might be mistakes. E.g. python-nibabel
> also provides versioned executables, but that I don't think there's a
> good reason for that). The way that Guidelines are written only serves
> to confuse packagers.

So, I think we're in agreement that the example is not ideal and can be very 
confusing. As Robert pointed out, there's already a ticket for the change at: 
https://fedorahosted.org/fpc/ticket/558, however it appears to have been 
inactive for months. Would you know what can be done to get the example fixed?

Tomas
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Re: Inconsistencies in the Fedora Packaging Guidelines for Python

2016-03-21 Thread Tomas Orsava

> But this section explicitly only covers the rare case where a py2 and
> py3 version of something provide _different_ functionality.

> The example spec is for the normal case. That's why it's up above the
> "Avoiding collisions between the python 2 and python 3 stacks" section.
> That section covers a specific case which applies to only a few packages.

Well, here's what the guidelines say about the normal case:
"If the executables provide the same functionality independent of whether they 
are run on top of Python 2 or Python 3, then only the Python 3 version of the 
executable should be packaged."

Since the spec file does package both p2 and p3 versions of the executable, I 
infer that we are indeed dealing with the case where it is beneficial to 
package both p2 and p3 version of the executable, in which case my first post 
applies and the example is wrong.

So in summary, either the example is wrong by packaging both p2 and p3 versions 
of the executable, or it is wrong by packaging the p3 version of the executable 
as the default one.
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

Inconsistencies in the Fedora Packaging Guidelines for Python

2016-03-21 Thread Tomas Orsava

Hi,
according to the "Fedora Packaging Guidelines for Python" [0] the "Example 
common spec file" [1] contained in those very same guidelines is wrong.

Specifically, the section "Naming" [2] in the guidelines says the following:
(1) "Both python 2 and python 3 variants must provide symlinks with a '-X' and 
'-X.Y' suffix (python runtime major version, or python runtime major.minor 
version), unless upstream already provides appropriately versioned executables 
without the dash."
(2) "The unversioned executable must be the python2 version."
(3) "For example, the python3 version of "coverage" must ship executables 
/usr/bin/coverage-3 and /usr/bin/coverage-3.4 (assuming python3 is currently 
version 3.4), while the python2 version must provide /usr/bin/coverage, 
/usr/bin/coverage-2 and /usr/bin/coverage-2.7 (assuming python2 version 2.7)."

So given that, the "Example common spec file" [1] has the following problems:
* The unversioned executable ('%{_bindir}/sample-exec') is provided by the 
python3- subpackage instead of the python2- subpackage.
* In the %install section, the order of execution of the %py2_install and 
%py3_install macros is wrong, since it leads to python3- subpackage providing 
the unversioned executable. The comment in the section should be modified 
accordingly as well.
* Neither python2- nor python3- subpackages provide symlinks to the '-X' 
suffix, i.e. 'sample-exec-2' and 'sample-exec-3'

[0] https://fedoraproject.org/wiki/Packagingython
[1] https://fedoraproject.org/wiki/Packagingython#Example_common_spec_file
[2] https://fedoraproject.org/wiki/Packagingython#Naming

Thank you for your time,
Tomas Orsava
___
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org

57 matches

Mail list logo