Re: MBF for deprecating Python2 usage

[Scott Kitterman]

Dropped d-devel.

On August 3, 2017 9:08:10 PM EDT, ba...@debian.org wrote:
>On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
>> 
>> While at DebCamp, Stefano Rivera and I sat down to analyze what needs
>to be done
>> to deprecate Python2 usage within the distribution.  It might not be
>possible to
>> drop Python2 for the next release, but there are still too many
>issues with
>> packages.  For now we identified some categories which need fixing.
>These are
>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal,
>resulting in
>> more than 3000 bug reports.  That's a high number, on the other hand
>we won't
>> make much progress if the issues are not named.  My intent is to
>bring that up
>> in the Python BoF next week at DebConf and then filing these bug
>reports with
>> the user tags mentiond on the wiki page.
>
>Great to hear that you guys talked about it.
>
>Just a quick note that PEP 394 discussions have revived, lead by the
>Fedora folks.  Please do check out the new thread, especially if you
>have opinions about what /usr/bin/python should do once Python 2.7 is
>EOL.
>
>https://mail.python.org/pipermail/linux-sig/2017-August/thread.html
>
>Cheers,
>-Barry

Read it.  I remain completely convinced that /usr/bin/python pointing at a 
python3 version is utterly wrong and a disservice to our users.  Even after we 
remove python2.7, people will be locally compiling it and using it for a decade.

Such a change would be actively user hostile.

When python2.7 goes away, /usr/bin/python should go too.

Before python2.7 goes away, any python3 scripts/apps need to work with 
/usr/bin/python3.  Making them work with /usr/bin/python too doesn't actually 
help anything.  All it does is look pretty for us old timers.

I've said before and I've yet to see anything to cause me to have any doubt on 
the matter, that as long as I am one of the python-defaults maintainers, 
/usr/bin/python will not point at a python3 version.

Scott K

Re: MBF for deprecating Python2 usage

[barry]

On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
> 
> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be 
> done
> to deprecate Python2 usage within the distribution.  It might not be possible 
> to
> drop Python2 for the next release, but there are still too many issues with
> packages.  For now we identified some categories which need fixing. These are
> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting 
> in
> more than 3000 bug reports.  That's a high number, on the other hand we won't
> make much progress if the issues are not named.  My intent is to bring that up
> in the Python BoF next week at DebConf and then filing these bug reports with
> the user tags mentiond on the wiki page.

Great to hear that you guys talked about it.

Just a quick note that PEP 394 discussions have revived, lead by the Fedora 
folks.  Please do check out the new thread, especially if you have opinions 
about what /usr/bin/python should do once Python 2.7 is EOL.

https://mail.python.org/pipermail/linux-sig/2017-August/thread.html

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP

MBF for deprecating Python2 usage

[Matthias Klose]

While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be done
to deprecate Python2 usage within the distribution.  It might not be possible to
drop Python2 for the next release, but there are still too many issues with
packages.  For now we identified some categories which need fixing. These are
documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting in
more than 3000 bug reports.  That's a high number, on the other hand we won't
make much progress if the issues are not named.  My intent is to bring that up
in the Python BoF next week at DebConf and then filing these bug reports with
the user tags mentiond on the wiki page.

Matthias

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

Ok, If I replace 

'{interpreter} setup.py build_man'

 by

'env; setup.py build_man'

I get this

HOME=/home/picca
PYTHONPATH=/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build

but If I read the code of 

def create_pydistutils_cfg(func):
"""distutils doesn't have sane command-line API - this decorator creates
.pydistutils.cfg file to workaround it

hint: if you think this is plain stupid, please don't read
distutils/setuptools/distribute sources
"""

def wrapped_func(self, context, args, *oargs, **kwargs):
fpath = join(args['home_dir'], '.pydistutils.cfg')
if not exists(fpath):
with open(fpath, 'w', encoding='utf-8') as fp:
lines = ['[clean]\n',
 'all=1\n',
 '[build]\n',
 'build-lib={}\n'.format(args['build_dir']),
 '[install]\n',
 'force=1\n',
 'install-layout=deb\n',
 'install-scripts=/usr/bin\n',
 'install-lib={}\n'.format(args['install_dir']),
 '[easy_install]\n',
 'allow_hosts=None\n']
log.debug('pydistutils config file:\n%s', ''.join(lines))
fp.writelines(lines)
context['ENV']['HOME'] = args['home_dir']
return func(self, context, args, *oargs, **kwargs)

we can see that the HOME should be home_dir

HOME should be /home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7

I think that there is a problem here.

right ?

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

So I am sltill investigating

with some debug output, I can see this the first build part gives

dh_auto_build -- --after-build '{interpreter} setup.py build_man'
pybuild --build -i python{version} -p 2.7 --after-build "{interpreter} 
setup.py build_man"
D: pybuild pybuild:474: version: 2.20170125
D: pybuild pybuild:475: ['/usr/bin/pybuild', '--build', '-i', 
'python{version}', '-p', '2.7', '--after-build', '{interpreter} setup.py 
build_man']
D: pybuild pybuild:36: cfg: Namespace(after_build='{interpreter} setup.py 
build_man', after_clean=None, after_configure=None, after_install=None, 
after_test=None, before_build=None, before_clean=None, before_configure=None, 
before_install=None, before_test=None, build_args=None, build_only=True, 
clean_args=None, clean_only=False, configure_args=None, configure_only=False, 
custom_tests=False, destdir='debian/tmp', detect_only=False, 
dir='/home/picca/Debian/silx/silx', disable=None, ext_destdir=None, 
ext_pattern='\\.so(\\.[^/]*)?$', install_args=None, install_dir=None, 
install_only=False, interpreter=['python{version}'], list_systems=False, 
name='silx', quiet=False, really_quiet=False, system=None, test_args=None, 
test_nose=False, test_only=False, test_pytest=False, test_tox=False, 
verbose=True, versions=['2.7'])
D: pybuild pybuild:103: detected build system: distutils (certainty: 61%)
I: pybuild base:184: /usr/bin/python setup.py build 
D: pybuild tools:217: invoking: /usr/bin/python setup.py build 
INFO:silx.setup:Use setuptools
running build
running build_py
creating /home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build/silx
<- right path

the after part execute the build_man part BUT

I: pybuild pybuild:242: python2.7 setup.py build_man
D: pybuild tools:217: invoking: python2.7 setup.py build_man
INFO:silx.setup:Use setuptools
running build_man
running build_py
creating build/lib.linux-i386-2.7
creating build/lib.linux-i386-2.7/silx <- ko


So the after part does not seems to take into account the .pydistutils.cfg file 
???

Is it normal ?

RE:building manpages via setup.py

[PICCA Frederic-Emmanuel]

Hello Piotr,

I am struggling, with the build system.

I will speak about this solution

> | override_dh_auto_build:
> |   dh_auto_build -- --after-build '{interpreter} setup.py build_man'

the code instrumented of the BuildMan is this one

class BuildMan(Command):
"""Command to build man pages"""
user_options = []

def initialize_options(self):
pass

def finalize_options(self):
pass

def run(self):
build = self.get_finalized_command('build')
path = sys.path
path.insert(0, os.path.abspath(build.build_lib))

print("PYTHONPATH", os.environ.get("PYTHONPATH", None))
print("sys.path", sys.path, os.path.exists(build.build_lib))
env = dict((str(k), str(v)) for k, v in os.environ.items())
env["PYTHONPATH"] = os.pathsep.join(path)
print("env[PYTHONPATH]", env["PYTHONPATH"])

import subprocess

status = subprocess.call(["mkdir", "-p", "build/man"])
if status != 0:
raise RuntimeError("Fail to create build/man directory")

try:
import tempfile
import stat
script_name = None

# help2man expect a single executable file to extract the help
# we create it, execute it, and delete it at the end

# create a launcher using the right python interpreter
script_fid, script_name = tempfile.mkstemp(prefix="%s_" % PROJECT, 
text=True)
script = os.fdopen(script_fid, 'wt')
script.write("#!%s\n" % sys.executable)
script.write("import runpy\n")
script.write("runpy.run_module('%s', run_name='__main__')\n" % 
PROJECT)
script.close()

# make it executable
mode = os.stat(script_name).st_mode
os.chmod(script_name, mode + stat.S_IEXEC)

# execute help2man
p = subprocess.Popen(["help2man", script_name, "-o", 
"build/man/silx.1"], env=env)
status = p.wait()
if status != 0:
raise RuntimeError("Fail to generate man documentation")
finally:
# clean up the script
if script_name is not None:
os.remove(script_name)


When I run the build it generate a pydistconfig file wit this content

D: pybuild plugin_distutils:55: pydistutils config file:
[clean]
all=1
[build]
build-lib=/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build
[install]
force=1
install-layout=deb
install-scripts=/usr/bin
install-lib=/usr/lib/python2.7/dist-packages
[easy_install]
allow_hosts=None


But when it comes to the --after step, I get this output

('PYTHONPATH', '/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build')
('sys.path', ['/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7', 
'/home/picca/Debian/silx/silx', 
'/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build', 
'/usr/lib/python2.7', '/usr/lib/python2.7/plat-i386-linux-gnu', 
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', 
'/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', 
'/usr/lib/python2.7/dist-packages', 
'/usr/lib/python2.7/dist-packages/PILcompat', 
'/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7'], 
False)
('env[PYTHONPATH]', 
'/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7:/home/picca/Debian/silx/silx:/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build:/usr/lib/python2.7:/usr/lib/python2.7/plat-i386-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PILcompat:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/lib/pymodules/python2.7')

So as you can see the path returned by build.build_lib
seems to be

/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7'

and not

/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build (provided via pydist)

So my question is what is wrong.

self.get_finalized_command('build')

doesn not take into account the pydistconfig file ?


Thanks for your help


Frederic