Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

On 2022-02-01 18:37:02 -0500, Tom Lane wrote:
> So I'm going to go ahead with patching; but does anyone want
> to change their vote?  (We can always "git revert".)

+1 for going ahead

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Noah Misch  writes:
> On Mon, Jan 31, 2022 at 05:18:47PM -0500, Tom Lane wrote:
>> If nobody else has weighed in by tomorrow, I'll backpatch to v10.

> Works for me.  I agree wanting Python 3.12 w/ PG10.latest is far more likely
> than wanting Python 2.6 or 3.1.  If someone lodges a non-academic complaint,
> we could have back branches fallback to the old way if they detect a Python
> version needing the old way.  I doubt anyone will complain.

I started to do that, but paused when the patch failed on v12, which
I soon realized is because our minimum requirement before v13 was
Python 2.4 not 2.6.  That means we're moving the goalposts a bit
further in the old branches than this discussion was presuming.

I don't think this changes the conclusion any: there's still little
chance that anyone wants to build PG against such old Python versions
in 2022.  So I'm going to go ahead with patching; but does anyone want
to change their vote?  (We can always "git revert".)

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Noah Misch]

On Mon, Jan 31, 2022 at 05:18:47PM -0500, Tom Lane wrote:
> [ I was hoping for more opinions, but I guess nobody cares but us ]
> 
> Andres Freund  writes:
> > On 2022-01-27 17:53:02 -0500, Tom Lane wrote:
> >> So now we need to discuss whether we want to back-patch this.
> >> Pros: avoid configure warning now (not worth much); avoid outright
> >> build failure on Python 3.12+ in future.
> >> Cons: breaks compatibility with Python 2.6 and 3.1.

> If nobody else has weighed in by tomorrow, I'll backpatch to v10.

Works for me.  I agree wanting Python 3.12 w/ PG10.latest is far more likely
than wanting Python 2.6 or 3.1.  If someone lodges a non-academic complaint,
we could have back branches fallback to the old way if they detect a Python
version needing the old way.  I doubt anyone will complain.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

[ I was hoping for more opinions, but I guess nobody cares but us ]

Andres Freund  writes:
> On 2022-01-27 17:53:02 -0500, Tom Lane wrote:
>> So now we need to discuss whether we want to back-patch this.
>> Pros: avoid configure warning now (not worth much); avoid outright
>> build failure on Python 3.12+ in future.
>> Cons: breaks compatibility with Python 2.6 and 3.1.

> How about adding a note about the change to this set of minor releases, and
> backpatch in the next set?

Meh.  Nobody looks at minor release notes to find out what will happen
in some other minor release.  Moreover, the sort of people who might
be adversely affected are probably not absorbing every minor release
right away, so they'd very likely not see the advance warning anyway.

> I don't see much point in worrying somebody still building plpython with 2.6,
> given its age. I feel a tad more compassion with a future self that wants to
> build a by-then EOL version of postgres, and plpython fails to build. We
> didn't commit to keeping plpython building, but it's in my default build
> script, so ...

Hmm, well, we're certainly not making this change in pre-v10 releases,
so I'm not sure that changing v10 will make things much easier for your
future self.  But it's unusual for us to make back-patching decisions
on the sort of basis I proposed here, so I'm okay with just going back
to v10 instead.

> I vote for backpatching all the way either now, or after the next set of minor
> releases is tagged.

If nobody else has weighed in by tomorrow, I'll backpatch to v10.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-27 17:53:02 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > Thanks! Looks pretty good so far. Including on machines that were broken in
> > take 1...
>
> Just about all of the buildfarm has reported in now, and it's all good.
> So now we need to discuss whether we want to back-patch this.
>
> Pros: avoid configure warning now (not worth much); avoid outright
> build failure on Python 3.12+ in future.
>
> Cons: breaks compatibility with Python 2.6 and 3.1.

How about adding a note about the change to this set of minor releases, and
backpatch in the next set?

2.6 has been out of support since October 29, 2013. 2.7 was released
2010-07-03 and has been EOL 2020-01-01.


> There are probably not many people using current Postgres builds
> with 2.6 or 3.1, but we can't rule out that there are some; and
> moving the compatibility goalposts in minor releases is generally
> not nice.  On the other hand, it's very foreseeable that somebody
> will want to build our back branches against 3.12 once it's out.

I don't see much point in worrying somebody still building plpython with 2.6,
given its age. I feel a tad more compassion with a future self that wants to
build a by-then EOL version of postgres, and plpython fails to build. We
didn't commit to keeping plpython building, but it's in my default build
script, so ...


> We could also wait until closer to 2023 before doing anything,
> but I fear we'd forget until complaints start to show up.
> I'd rather get this done while it's front-of-mind.

I vote for backpatching all the way either now, or after the next set of minor
releases is tagged.

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> Thanks! Looks pretty good so far. Including on machines that were broken in
> take 1...

Just about all of the buildfarm has reported in now, and it's all good.
So now we need to discuss whether we want to back-patch this.

Pros: avoid configure warning now (not worth much); avoid outright
build failure on Python 3.12+ in future.

Cons: breaks compatibility with Python 2.6 and 3.1.

There are probably not many people using current Postgres builds
with 2.6 or 3.1, but we can't rule out that there are some; and
moving the compatibility goalposts in minor releases is generally
not nice.  On the other hand, it's very foreseeable that somebody
will want to build our back branches against 3.12 once it's out.

3.12 is scheduled to start beta in roughly May of 2023 (assuming
they hold to their annual release cadence, which seems like a
good bet).

The compromise I propose is to back-patch into branches that
will still be in-support at that point, which are v11 and up.
v10 will be dead, and it's perhaps a shade more likely than the
later branches to be getting used with hoary Python versions,
so I think the odds favor not changing it.

We could also wait until closer to 2023 before doing anything,
but I fear we'd forget until complaints start to show up.
I'd rather get this done while it's front-of-mind.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Julien Rouhaud]

Hi,

Sorry I somehow missed that email.

On Mon, Jan 24, 2022 at 7:53 AM Tom Lane  wrote:
>
> Now, most of those are BSD machines --- but lapwing isn't.
> It says
>
> checking for python... (cached) /usr/bin/python
> configure: using python 3.6.9 (default, Jan 14 2022, 06:45:55)
> checking for Python distutils module... yes
> checking Python configuration directory... 
> /usr/local/lib/python3.6/config-3.6m-i386-linux-gnu
> checking Python include directories... -I/usr/local/include/python3.6m
> checking how to link an embedded Python application... -L/usr/local/lib 
> -lpython3.6m -lpthread -ldl  -lutil -lrt -lm
>
> Not sure what to make of that --- maybe that's a handmade build?

Yes it is.  Lapwing is a vanilla debian 7, and the backports only
offers python 3.2.  So in anticipiation to the switch to meson I
compiled the latest python 3.6, as it's supposed to be the oldest
supported version.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

On 2022-01-25 18:33:55 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > On 2022-01-25 12:45:15 -0500, Tom Lane wrote:
> >> I guess next steps are to revert f032f63e7 and then retry e0e567a10
> >> with that change.  Who's going to do the honors?
> 
> > I assume Peter is done working for the day. I'm stuck in meetings and stuff
> > for another 2-3 hours.  I can give it a go after that, unless you do so
> > before.
> 
> OK, will do.

Thanks! Looks pretty good so far. Including on machines that were broken in
take 1...

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> On 2022-01-25 12:45:15 -0500, Tom Lane wrote:
>> I guess next steps are to revert f032f63e7 and then retry e0e567a10
>> with that change.  Who's going to do the honors?

> I assume Peter is done working for the day. I'm stuck in meetings and stuff
> for another 2-3 hours.  I can give it a go after that, unless you do so
> before.

OK, will do.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-25 12:45:15 -0500, Tom Lane wrote:
> As of now, 92 buildfarm animals have reported results from f032f63e7.
> Every single one of them reports that all the different methods you
> tested give the same answer.  So it looks to me like we should just
> go with get_config_var('INCLUDEPY') and be happy.
> 
> I guess next steps are to revert f032f63e7 and then retry e0e567a10
> with that change.  Who's going to do the honors?

I assume Peter is done working for the day. I'm stuck in meetings and stuff
for another 2-3 hours.  I can give it a go after that, unless you do so
before.

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> It's a little bit too soon to decide that INCLUDEPY is reliably equal
> to that, but if it still looks that way tomorrow, I'll be satisfied.

As of now, 92 buildfarm animals have reported results from f032f63e7.
Every single one of them reports that all the different methods you
tested give the same answer.  So it looks to me like we should just
go with get_config_var('INCLUDEPY') and be happy.

I guess next steps are to revert f032f63e7 and then retry e0e567a10
with that change.  Who's going to do the honors?

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Yeah, I was just noticing that.  It looks like the whole business
> with checking both get_python_inc(False) and get_python_inc(True)
> has been useless from the start: none of the buildfarm animals report
> more than one -I switch in "checking Python include directories".

Also, that appears to be true even in the oldest runs that vendikar
still has data for, back in 2015.  So it's not something that they
cleaned up recently.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> ... But none of the systems report a get_python_inc(False) differing from
> get_python_inc(True), or from the value of INCLUDEPY. So I don't see a reason
> for why it'd not?

Yeah, I was just noticing that.  It looks like the whole business
with checking both get_python_inc(False) and get_python_inc(True)
has been useless from the start: none of the buildfarm animals report
more than one -I switch in "checking Python include directories".

It's a little bit too soon to decide that INCLUDEPY is reliably equal
to that, but if it still looks that way tomorrow, I'll be satisfied.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 19:49:57 -0800, Andres Freund wrote:
> > To avoid too noisy breakages, we could have python.m4 emit INCLUDEPY and 
> > then
> > search the bf logs in a day or three?
> 
> Maybe something like the attached? Not particularly nice, but should give us
> most of the relevant information?

FWIW, so far all 73 animals that reported on HEAD that they ran with the
change and that currently detect as with_python='yes', find Python.h via
INCLUDEPY the same as via get_python_inc(). This includes systems like gadwall
where sysconfig.get_path('include') returned the erroneous
/usr/local/include/python2.7.

Of course that doesn't guarantee in itself that Python.h is usable that
way. But none of the systems report a get_python_inc(False) differing from
get_python_inc(True), or from the value of INCLUDEPY. So I don't see a reason
for why it'd not?

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> Also note that python-config is itself a Python script that uses 
> sysconfig and includes code like this:

>  elif opt in ('--includes', '--cflags'):
>  flags = ['-I' + sysconfig.get_path('include'),
>   '-I' + sysconfig.get_path('platinclude')]

> So this would just do the same thing we are already doing anyway.

It used to look like that, but at least in my 3.6.8 installation
on RHEL8, it's been rewritten to be a shell script that doesn't
depend on sysconfig at all.

The result is sufficiently bletcherous that you'd have thought
they'd reconsider getting rid of sysconfig :-(.  Also, it
definitely won't work on Windows.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-24 21:48:21 +0100, Peter Eisentraut wrote:
> Also note that python-config is itself a Python script that uses sysconfig
> and includes code like this:

Huh. It's a shell script on my debian system. Looks like the python source
tree has both. Not sure what / who decides which is used.

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 24.01.22 03:53, Andres Freund wrote:

On 2022-01-23 21:31:52 -0500, Tom Lane wrote:

Andres Freund  writes:

No, not really. There generally seems to be very little documentation about
what one is supposed to use when embedding python (rather than building a
python module). The only thing I really see is:
https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems
which says to use python-config.

Yeah :-(.  I don't really want to go there, because it will break
existing setups.

Yea, it seems to introduce a whole set of new complexities (finding python
from python-config, mismatching python-config and explicitly specified python,
...). And it doesn't exist on windows either :(.


Also note that python-config is itself a Python script that uses 
sysconfig and includes code like this:


elif opt in ('--includes', '--cflags'):
flags = ['-I' + sysconfig.get_path('include'),
 '-I' + sysconfig.get_path('platinclude')]

So this would just do the same thing we are already doing anyway.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 18:53:01 -0800, Andres Freund wrote:
> I don't really understand what the various "platform" variables / paths are
> supposed to do.

The code says:


> def get_python_inc(plat_specific=0, prefix=None):
> """Return the directory containing installed Python header files.
> 
> If 'plat_specific' is false (the default), this is the path to the
> non-platform-specific header files, i.e. Python.h and so on;
> otherwise, this is the path to platform-specific header files
> (namely pyconfig.h).
> ...

Looking at the code for get_python_inc() in 2.7, it seems that plat_specific
toggles between CONFINCLUDEPY and INCLUDEPY. Except on windows, where it uses
get_path('include') for both. sysconfig.py sets INCLUDEPY to
get_path('include') on windows, so we'd be good with INCLUDEPY there.



> To avoid too noisy breakages, we could have python.m4 emit INCLUDEPY and then
> search the bf logs in a day or three?

Maybe something like the attached? Not particularly nice, but should give us
most of the relevant information?

Greetings,

Andres Freund
diff --git i/config/python.m4 w/config/python.m4
index f51d23c3d43..46829b5b117 100644
--- i/config/python.m4
+++ w/config/python.m4
@@ -68,6 +68,27 @@ if test "$PORTNAME" = win32 ; then
 fi
 AC_MSG_RESULT([$python_includespec])
 
+python_ways=`${PYTHON} -c "
+import distutils.sysconfig as ds
+import os
+import sysconfig as s
+print('ds.get_python_inc(False): %s' % ds.get_python_inc(False))
+print('s path include: %s' % s.get_path('include'))
+print('ds.get_python_inc(True): %s' % ds.get_python_inc(True))
+print('s path platinclude: %s' % s.get_path('platinclude'))
+print('ds var INCLUDEPY: %s' % ds.get_config_var('INCLUDEPY'))
+print('s var INCLUDEPY: %s' % s.get_config_var('INCLUDEPY'))
+print('ds var CONFINCLUDEPY: %s' % ds.get_config_var('CONFINCLUDEPY'))
+print('s var CONFINCLUDEPY: %s' % s.get_config_var('CONFINCLUDEPY'))
+print('')
+print('ds get_python_inc(False)/Python.h exists: %s' % os.path.exists(os.path.join(ds.get_python_inc(False), 'Python.h')))
+print('ds get_python_inc(True)/Python.h exists: %s' % os.path.exists(os.path.join(ds.get_python_inc(True), 'Python.h')))
+print('s var INCLUDEPY/Python.h exists: %s' % os.path.exists(os.path.join(ds.get_config_var('INCLUDEPY'), 'Python.h')))
+print('s var CONFINCLUDEPY/Python.h exists: %s' % os.path.exists(os.path.join(ds.get_config_var('CONFINCLUDEPY'), 'Python.h')))
+"` 2>/dev/null
+AC_MSG_NOTICE([python include paths, different approaches:
+$python_ways])
+
 AC_SUBST(python_majorversion)[]dnl
 AC_SUBST(python_version)[]dnl
 AC_SUBST(python_includespec)[]dnl

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> To avoid too noisy breakages, we could have python.m4 emit INCLUDEPY and then
> search the bf logs in a day or three?

+1, it'd give us some info without breaking the farm.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 21:31:52 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > No, not really. There generally seems to be very little documentation about
> > what one is supposed to use when embedding python (rather than building a
> > python module). The only thing I really see is:
>
> > https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems
>
> > which says to use python-config.
>
> Yeah :-(.  I don't really want to go there, because it will break
> existing setups.

Yea, it seems to introduce a whole set of new complexities (finding python
from python-config, mismatching python-config and explicitly specified python,
...). And it doesn't exist on windows either :(.


> Getting back to the INCLUDEPY solution: I see nothing equivalent
> to that for the "platform-dependent include directory".  But maybe
> we don't really need that?  Not clear.

I don't really understand what the various "platform" variables / paths are
supposed to do. I think it might only differ when compiling stuff as part of
(or against) the python source tree.

To avoid too noisy breakages, we could have python.m4 emit INCLUDEPY and then
search the bf logs in a day or three?

Not that that's any guarantee, but it's maybe worth a bit that INCLUDEPY is
one of the few wars that sysconfig.py explicitly computes for windows builds 
[2].

Greetings,

Andres Freund

[2] https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L487

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> No, not really. There generally seems to be very little documentation about
> what one is supposed to use when embedding python (rather than building a
> python module). The only thing I really see is:

> https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems

> which says to use python-config.

Yeah :-(.  I don't really want to go there, because it will break
existing setups.  An example is that on a few machines I have
pointed the build to non-default Python installations by doing
things like
ln -s /path/to/desired/python ~/bin/python3
So there's no matching python-config in my PATH at all.
Yeah, I can change that, but that would be a dealbreaker for
back-patching this, I think.

Getting back to the INCLUDEPY solution: I see nothing equivalent
to that for the "platform-dependent include directory".  But maybe
we don't really need that?  Not clear.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 20:50:23 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > I think this might be problem on our own end, actually. The 
> > distutils.sysconfig
> > code did
> > a = '-I' + distutils.sysconfig.get_python_inc(False)
> > b = '-I' + distutils.sysconfig.get_python_inc(True)
> > which the patch upthread  changed to
> > +a = '-I' + sysconfig.get_path('include')
> > +b = '-I' + sysconfig.get_path('platinclude')
> > but I think that's possibly not quite the right translation?
>
> I don't buy it.  The sysconfig documentation says pretty clearly
> that get_path('include') and get_path('platinclude') are supposed
> to return the directories we want, and there's nothing there
> suggesting that we ought to magically know to look in a
> non-default scheme.

I'm not really convinced. Note that the whole thing is prefixed with

  Every new component that is installed using distutils or a Distutils-based
  system will follow the same scheme to copy its file in the right places.

and then

  Each scheme is itself composed of a series of paths and each path has a
  unique identifier. Python currently uses eight paths:

and that get_path()'s documentation says:

  If scheme is provided, it must be a value from the list returned by
  get_scheme_names(). Otherwise, the default scheme for the current platform is
  used.

(with some 2.7 vs 3.x differences)

The list of schemas explicitly includes stuff like posix_home, posix_user,
nt_user, which all won't contain python.h in 'include'.  I don't see anything
implying scheme on some platform isn't *_user or such.


> > But even so, it seems using sysconfig.get_config_vars('INCLUDEPY') or such
> > seems like it might be a better translation than the above
> > sysconfig.get_path() stuff?
>
> Can you find ANY documentation suggesting that INCLUDEPY is
> meant as a stable API for outside code to use?  That seems
> far more fragile than anything else we've discussed, even
> if it happens to work today.

No, not really. There generally seems to be very little documentation about
what one is supposed to use when embedding python (rather than building a
python module). The only thing I really see is:

https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems

which says to use python-config.

Note that we are already using get_config_vars() for LIBDIR, LDLIBRARY,
LDVERSION, VERSION, LIBS, ... which all seem equally undocumented.


> I remain of the persuasion that these Debian packages are
> broken.  The fact that they've not perpetuated the scheme
> into their python3 packages shows that they came to the
> same conclusion.

Yea, I don't like it at all.


Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> I think this might be problem on our own end, actually. The 
> distutils.sysconfig
> code did
> a = '-I' + distutils.sysconfig.get_python_inc(False)
> b = '-I' + distutils.sysconfig.get_python_inc(True)
> which the patch upthread  changed to
> +a = '-I' + sysconfig.get_path('include')
> +b = '-I' + sysconfig.get_path('platinclude')
> but I think that's possibly not quite the right translation?

I don't buy it.  The sysconfig documentation says pretty clearly
that get_path('include') and get_path('platinclude') are supposed
to return the directories we want, and there's nothing there
suggesting that we ought to magically know to look in a
non-default scheme.

(I do note that the documentation says there's no direct
equivalent to what get_python_inc does, which is scary.)

> But even so, it seems using sysconfig.get_config_vars('INCLUDEPY') or such
> seems like it might be a better translation than the above
> sysconfig.get_path() stuff?

Can you find ANY documentation suggesting that INCLUDEPY is
meant as a stable API for outside code to use?  That seems
far more fragile than anything else we've discussed, even
if it happens to work today.

I remain of the persuasion that these Debian packages are
broken.  The fact that they've not perpetuated the scheme
into their python3 packages shows that they came to the
same conclusion.  We should not be inventing usage patterns
based on a belief that it's supposed to work like this,
because what we'll mainly get out of that is failures on
other platforms.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 19:00:41 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > On 2022-01-23 18:31:44 -0500, Tom Lane wrote:
> >> Yeah, but we don't know whether there are any versions of the Debian
> >> packaging in which they fixed the file layout, so that 'posix_local'
> >> actually does describe the layout.
> 
> > I think posix_local try to achieve something different than what you assume 
> > it
> > does. It's intended to return the location to which "locally" intalled 
> > python
> > extension install their files (including headers) - after having the problem
> > that such local python package installations overwrite (and thus broke) 
> > files
> > installed via the system mechanism.
> 
> Okay, but surely they'd have thought of packages that just want to find
> out where the system Python headers are?

I think this might be problem on our own end, actually. The distutils.sysconfig
code did
a = '-I' + distutils.sysconfig.get_python_inc(False)
b = '-I' + distutils.sysconfig.get_python_inc(True)

which the patch upthread  changed to

+a = '-I' + sysconfig.get_path('include')
+b = '-I' + sysconfig.get_path('platinclude')

but I think that's possibly not quite the right translation?

The recommended way to find flags to compile against python appears to be the
python$version-config binary. To which we might not want to switch.

But even so, it seems using sysconfig.get_config_vars('INCLUDEPY') or such
seems like it might be a better translation than the above
sysconfig.get_path() stuff?

For me that returns more sensible paths.

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> On 2022-01-23 18:31:44 -0500, Tom Lane wrote:
>> Yeah, but we don't know whether there are any versions of the Debian
>> packaging in which they fixed the file layout, so that 'posix_local'
>> actually does describe the layout.

> I think posix_local try to achieve something different than what you assume it
> does. It's intended to return the location to which "locally" intalled python
> extension install their files (including headers) - after having the problem
> that such local python package installations overwrite (and thus broke) files
> installed via the system mechanism.

Okay, but surely they'd have thought of packages that just want to find
out where the system Python headers are?  Having this be the default
behavior seems like it breaks as much as it fixes.  (Of course, maybe
that's why they gave up on it.)

Anyway, I don't mind trying your second suggestion

scheme = sysconfig._get_default_scheme()
# Work around Debian / Ubuntu returning paths not useful for finding python 
headers
if scheme == 'posix_local':
scheme = 'posix_prefix'
sysconfig.get_path('include', scheme = scheme)

If it doesn't work everywhere, we can adjust it later.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Yeah, but we don't know whether there are any versions of the Debian
> packaging in which they fixed the file layout, so that 'posix_local'
> actually does describe the layout.

Actually ... scraping the buildfarm to see what we're currently
finding shows that the following machines are reporting that
/usr/local/include/pythonN.N really is the include directory:

conchuela
curculio
florican
gombessa
jabiru
lapwing
loach
longfin
marabou
morepork
peripatus
plover

Now, most of those are BSD machines --- but lapwing isn't.
It says

checking for python... (cached) /usr/bin/python
configure: using python 3.6.9 (default, Jan 14 2022, 06:45:55) 
checking for Python distutils module... yes
checking Python configuration directory... 
/usr/local/lib/python3.6/config-3.6m-i386-linux-gnu
checking Python include directories... -I/usr/local/include/python3.6m
checking how to link an embedded Python application... -L/usr/local/lib 
-lpython3.6m -lpthread -ldl  -lutil -lrt -lm

Not sure what to make of that --- maybe that's a handmade build?

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 18:31:44 -0500, Tom Lane wrote:
> Andres Freund  writes:
> > On 2022-01-23 18:11:41 -0500, Tom Lane wrote:
> >> Anyway, trying to figure out whether we're on a Debian package with this
> >> mistake doesn't seem any cleaner than what I proposed.  (In particular,
> >> blindly changing to a different scheme without a check to see what's really
> >> in the filesystem seems doomed to failure.)
> 
> > If we make it depend on _get_default_scheme() == 'posix_local' that 
> > shouldn't
> > be a risk, because that's the debian addition...
> 
> Yeah, but we don't know whether there are any versions of the Debian
> packaging in which they fixed the file layout, so that 'posix_local'
> actually does describe the layout.

I think posix_local try to achieve something different than what you assume it
does. It's intended to return the location to which "locally" intalled python
extension install their files (including headers) - after having the problem
that such local python package installations overwrite (and thus broke) files
installed via the system mechanism.

So posix_local works "by design" if it returns paths in /usr/local that do not
contain a python installation. If it did, it wouldn't achieve the goal.

It's definitely crappily documented. And probably not a great approach as a
whole. But...

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> On 2022-01-23 18:11:41 -0500, Tom Lane wrote:
>> Anyway, trying to figure out whether we're on a Debian package with this
>> mistake doesn't seem any cleaner than what I proposed.  (In particular,
>> blindly changing to a different scheme without a check to see what's really
>> in the filesystem seems doomed to failure.)

> If we make it depend on _get_default_scheme() == 'posix_local' that shouldn't
> be a risk, because that's the debian addition...

Yeah, but we don't know whether there are any versions of the Debian
packaging in which they fixed the file layout, so that 'posix_local'
actually does describe the layout.  I do not think that we are wise
to suppose we know which scheme to use without a check on what's
actually there.

I could go for "if we don't see Python.h where it's claimed to be,
try again with scheme = posix_prefix".  But I'm still not convinced
that that's noticeably cleaner than the hack I suggested.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> The relevant part of distutils-install-layout.diff explaining this is:

> +(0)
> +   Starting with Python-2.6 Debian/Ubuntu uses for the Python which comes 
> within
> +   the Linux distribution a non-default name for the installation directory. 
> This
> +   is to avoid overwriting of the python modules which come with the 
> distribution,
> +   which unfortunately is the upstream behaviour of the installation tools.

Yeah, I figured that the explanation was something like that.  Too bad
they didn't get it right.

I stopped to wonder if maybe the problem is that sysconfig.py is from the
"different distribution" that they're worried about here, but it doesn't
look like it:

tgl@rpi3:~$ dpkg -S /usr/lib/python2.7/sysconfig.py
libpython2.7-minimal:armhf: /usr/lib/python2.7/sysconfig.py
tgl@rpi3:~$ dpkg -S /usr/include/python2.7/Python.h 
libpython2.7-dev:armhf: /usr/include/python2.7/Python.h

Oh well.  For a moment there I thought maybe this was a "missing
dev package" kind of problem, but it's hard to come to any other
conclusion than "packager screwed up".

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 18:11:41 -0500, Tom Lane wrote:
> It may be that one of the other "scheme" values accurately describes
> Debian's actual layout of this package.  I didn't check, because the
> scheme is defined to be platform-specific.

posix_prefix does, as far as I can see.


> Specifying a particular value for it would therefore break other platforms.

Hence the suggestion to only force posix_prefix when posix_local (the debian
invention) otherwise would get used...


> Anyway, trying to figure out whether we're on a Debian package with this
> mistake doesn't seem any cleaner than what I proposed.  (In particular,
> blindly changing to a different scheme without a check to see what's really
> in the filesystem seems doomed to failure.)

If we make it depend on _get_default_scheme() == 'posix_local' that shouldn't
be a risk, because that's the debian addition...

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Andres Freund  writes:
> On 2022-01-23 16:06:21 -0500, Tom Lane wrote:
>> +'posix_local': {
>> +'stdlib': '{base}/lib/python{py_version_short}',
>> +'platstdlib': '{platbase}/lib/python{py_version_short}',
>> +'purelib': 
>> '{base}/local/lib/python{py_version_short}/dist-packages',
>> +'platlib': 
>> '{platbase}/local/lib/python{py_version_short}/dist-packages',
>> +'include': '{base}/local/include/python{py_version_short}',
>> +'platinclude': '{platbase}/local/include/python{py_version_short}',
>> +'scripts': '{base}/local/bin',
>> +'data': '{base}/local',
>> +},
>> +'deb_system': {
>> +'stdlib': '{base}/lib/python{py_version_short}',
>> +'platstdlib': '{platbase}/lib/python{py_version_short}',
>> +'purelib': '{base}/lib/python{py_version_short}/dist-packages',
>> +'platlib': '{platbase}/lib/python{py_version_short}/dist-packages',
>> +'include': '{base}/include/python{py_version_short}',
>> +'platinclude': '{platbase}/include/python{py_version_short}',
>> +'scripts': '{base}/bin',
>> +'data': '{base}',
>> +},
>> 'posix_home': {

> Hm. It seems the intent of the different paths you show is that we can specify
> which type of path we want. The one to locally installed extensions, or the
> distribution ones. So we'd have to specify the scheme to get the other include
> path?

It may be that one of the other "scheme" values accurately describes
Debian's actual layout of this package.  I didn't check, because the
scheme is defined to be platform-specific.  Specifying a particular
value for it would therefore break other platforms.  Anyway, trying
to figure out whether we're on a Debian package with this mistake
doesn't seem any cleaner than what I proposed.  (In particular,
blindly changing to a different scheme without a check to see
what's really in the filesystem seems doomed to failure.)

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 16:06:21 -0500, Tom Lane wrote:
> (I don't know
> enough about Debian packaging to find the smoking gun though;
> what apt-get claims is the source package contains no trace of
> this diff.)  There's no sign of comparable changes in
> /usr/lib/python3.5/sysconfig.py on the same machine, either.

FWIW, here's the steps to find it (on a debian 9 instance):

dpkg -S /usr/lib/python2.7/sysconfig.py
libpython2.7-minimal:amd64: /usr/lib/python2.7/sysconfig.py
mkdir /tmp/aptsrc
cd /tmp/aptsrc
apt-get source libpython2.7-minimal
root@283a48b8d701:/tmp/aptsrc# grep -lR sysconfig.py python2.7*/debian/
python2.7-2.7.13/debian/changelog
python2.7-2.7.13/debian/patches/distutils-install-layout.diff
python2.7-2.7.13/debian/patches/mangle-fstack-protector.diff
python2.7-2.7.13/debian/patches/ext-no-libpython-link.diff
python2.7-2.7.13/debian/patches/multiarch.diff
python2.7-2.7.13/debian/patches/issue9189.diff
python2.7-2.7.13/debian/patches/debug-build.diff
python2.7-2.7.13/debian/patches/distutils-sysconfig.diff
python2.7-2.7.13/debian/patches/disable-some-tests.patch

The relevant part of distutils-install-layout.diff explaining this is:

+(0)
+   Starting with Python-2.6 Debian/Ubuntu uses for the Python which comes 
within
+   the Linux distribution a non-default name for the installation directory. 
This
+   is to avoid overwriting of the python modules which come with the 
distribution,
+   which unfortunately is the upstream behaviour of the installation tools. The
+   non-default name in :file:`/usr/local` is used not to overwrite a local 
python
+   installation (defaulting to :file:`/usr/local`).

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andres Freund]

Hi,

On 2022-01-23 16:06:21 -0500, Tom Lane wrote:
> I wrote:
> > Based on the buildfarm results so far, the problem can be described
> > as "some installations say /usr/local when they should have said /usr".
> > I experimented with the attached delta patch and it fixes the problem
> > on my Debian 9 image.  (I don't know Python, so there may be a better
> > way to do this.)  We'd have to also bump the minimum 3.x version to
> > 3.2, but that seems very unlikely to bother anyone.
> 
> I did a little more digging into this.  The python2 package on
> my Deb9 (actually Raspbian) system says it is 2.7.13, but
> /usr/lib/python2.7/sysconfig.py is different from what I find in
> a virgin Python 2.7.13 tarball, as per attached diff.  I conclude
> that somebody at Debian decided that Python should live under
> /usr/local, and changed sysconfig.py to match, but then failed
> to adjust the actual install scripts to agree, because there is
> certainly nothing installed under /usr/local.  (I don't know
> enough about Debian packaging to find the smoking gun though;
> what apt-get claims is the source package contains no trace of
> this diff.)  There's no sign of comparable changes in
> /usr/lib/python3.5/sysconfig.py on the same machine, either.

> +'posix_local': {
> +'stdlib': '{base}/lib/python{py_version_short}',
> +'platstdlib': '{platbase}/lib/python{py_version_short}',
> +'purelib': '{base}/local/lib/python{py_version_short}/dist-packages',
> +'platlib': 
> '{platbase}/local/lib/python{py_version_short}/dist-packages',
> +'include': '{base}/local/include/python{py_version_short}',
> +'platinclude': '{platbase}/local/include/python{py_version_short}',
> +'scripts': '{base}/local/bin',
> +'data': '{base}/local',
> +},
> +'deb_system': {
> +'stdlib': '{base}/lib/python{py_version_short}',
> +'platstdlib': '{platbase}/lib/python{py_version_short}',
> +'purelib': '{base}/lib/python{py_version_short}/dist-packages',
> +'platlib': '{platbase}/lib/python{py_version_short}/dist-packages',
> +'include': '{base}/include/python{py_version_short}',
> +'platinclude': '{platbase}/include/python{py_version_short}',
> +'scripts': '{base}/bin',
> +'data': '{base}',
> +},
>  'posix_home': {


Hm. It seems the intent of the different paths you show is that we can specify
which type of path we want. The one to locally installed extensions, or the
distribution ones. So we'd have to specify the scheme to get the other include
path?

andres@awork3:~$ python2
Python 2.7.18 (default, Sep 24 2021, 09:39:51)
[GCC 10.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/local/include/python2.7'
>>> sysconfig.get_path('include', 'posix_prefix')
'/usr/include/python2.7'
>>> sysconfig.get_path('include', 'posix_local')
'/usr/local/include/python2.7'

So it seems we could do something like

sysconfig.get_path('include', 'posix_prefix' if os.name == 'posix' else os.name)

or

scheme = sysconfig._get_default_scheme()
# Work around Debian / Ubuntu returning paths not useful for finding python 
headers
if scheme == 'posix_local':
scheme = 'posix_prefix'
sysconfig.get_path('include', scheme = scheme)

Greetings,

Andres Freund

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Based on the buildfarm results so far, the problem can be described
> as "some installations say /usr/local when they should have said /usr".
> I experimented with the attached delta patch and it fixes the problem
> on my Debian 9 image.  (I don't know Python, so there may be a better
> way to do this.)  We'd have to also bump the minimum 3.x version to
> 3.2, but that seems very unlikely to bother anyone.

I did a little more digging into this.  The python2 package on
my Deb9 (actually Raspbian) system says it is 2.7.13, but
/usr/lib/python2.7/sysconfig.py is different from what I find in
a virgin Python 2.7.13 tarball, as per attached diff.  I conclude
that somebody at Debian decided that Python should live under
/usr/local, and changed sysconfig.py to match, but then failed
to adjust the actual install scripts to agree, because there is
certainly nothing installed under /usr/local.  (I don't know
enough about Debian packaging to find the smoking gun though;
what apt-get claims is the source package contains no trace of
this diff.)  There's no sign of comparable changes in
/usr/lib/python3.5/sysconfig.py on the same machine, either.

So I think this can fairly be characterized as brain-dead packaging
error, and we should just hack around it as per my previous patch.

In other news, I switched prairiedog and gaur to python 3.2.

regards, tom lane

--- Python-2.7.13/Lib/sysconfig.py	2016-12-17 15:05:06.0 -0500
+++ /usr/lib/python2.7/sysconfig.py	2018-09-26 14:42:22.0 -0400
@@ -16,6 +16,26 @@ _INSTALL_SCHEMES = {
 'scripts': '{base}/bin',
 'data': '{base}',
 },
+'posix_local': {
+'stdlib': '{base}/lib/python{py_version_short}',
+'platstdlib': '{platbase}/lib/python{py_version_short}',
+'purelib': '{base}/local/lib/python{py_version_short}/dist-packages',
+'platlib': '{platbase}/local/lib/python{py_version_short}/dist-packages',
+'include': '{base}/local/include/python{py_version_short}',
+'platinclude': '{platbase}/local/include/python{py_version_short}',
+'scripts': '{base}/local/bin',
+'data': '{base}/local',
+},
+'deb_system': {
+'stdlib': '{base}/lib/python{py_version_short}',
+'platstdlib': '{platbase}/lib/python{py_version_short}',
+'purelib': '{base}/lib/python{py_version_short}/dist-packages',
+'platlib': '{platbase}/lib/python{py_version_short}/dist-packages',
+'include': '{base}/include/python{py_version_short}',
+'platinclude': '{platbase}/include/python{py_version_short}',
+'scripts': '{base}/bin',
+'data': '{base}',
+},
 'posix_home': {
 'stdlib': '{base}/lib/python',
 'platstdlib': '{base}/lib/python',
@@ -129,7 +149,7 @@ def is_python_build():
 _PYTHON_BUILD = is_python_build()
 
 if _PYTHON_BUILD:
-for scheme in ('posix_prefix', 'posix_home'):
+for scheme in ('posix_prefix', 'posix_local', 'deb_system', 'posix_home'):
 _INSTALL_SCHEMES[scheme]['include'] = '{projectbase}/Include'
 _INSTALL_SCHEMES[scheme]['platinclude'] = '{srcdir}'
 
@@ -163,8 +183,11 @@ def _expand_vars(scheme, vars):
 
 def _get_default_scheme():
 if os.name == 'posix':
-# the default scheme for posix is posix_prefix
-return 'posix_prefix'
+# the default scheme for posix on Debian/Ubuntu is posix_local
+# FIXME: return dist-packages/posix_prefix only for
+#   is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__
+# is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
+return 'posix_local'
 return os.name
 
 def _getuserbase():
@@ -231,11 +254,19 @@ def _parse_makefile(filename, vars=None)
 done[n] = v
 
 # do variable interpolation here
-while notdone:
-for name in notdone.keys():
+variables = list(notdone.keys())
+
+# Variables with a 'PY_' prefix in the makefile. These need to
+# be made available without that prefix through sysconfig.
+# Special care is needed to ensure that variable expansion works, even
+# if the expansion uses the name without a prefix.
+renamed_variables = ('CFLAGS', 'LDFLAGS', 'CPPFLAGS')
+
+while len(variables) > 0:
+for name in tuple(variables):
 value = notdone[name]
 m = _findvar1_rx.search(value) or _findvar2_rx.search(value)
-if m:
+if m is not None:
 n = m.group(1)
 found = True
 if n in done:
@@ -246,23 +277,48 @@ def _parse_makefile(filename, vars=None)
 elif n in os.environ:
 # do it like make: fall back to environment
 item = os.environ[n]
+
+elif n in renamed_variables:
+if name.startswith('PY_') and name[3:] in 

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> The early returns are not great: we have about half a dozen machines
> so far that are finding python3, and reporting sane-looking Python
> include paths, but not finding Python.h.  They're all Linux-oid
> machines, so I suppose what is going on is that they have the base
> python3 package installed but not python3-dev or local equivalent.

> I want to leave that patch in place long enough so we can get a
> fairly full survey of which machines are OK and which are not,
> but I suppose I'll have to revert it tomorrow or so.  We did
> promise the owners a month to adjust their configurations.

I have now reverted that patch, but I think this was a highly
worthwhile bit of reconnaissance.  It identified 18 animals
that had incomplete python3 installations (versus only 13
that definitely or possibly lack python3 altogether).  Their
owners most likely thought they were already good to go for the
changeover, so without this experiment we'd have had a whole lot
of buildfarm red when the real change is made.

I've notified the owners of these results.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Andrew Dunstan]

On 1/18/22 19:18, Tom Lane wrote:
> I wrote:
>> Are we sure it's an issue within Python, rather than something we
>> could dodge by invoking sysconfig differently?  It's hard to believe
>> that sysconfig could be totally unfit for the purpose of finding out
>> the include path and would remain so for multiple years.
> I dug up a Debian 9 image and found that I could reproduce the problem
> against its python2 (2.7.13) installation, but not its python3 (3.5.3):
>
> $ python2 -m sysconfig | grep include
> include = "/usr/local/include/python2.7"
> platinclude = "/usr/local/include/python2.7"
> ...
> $ python3 -m sysconfig | grep include
> include = "/usr/include/python3.5m"
> platinclude = "/usr/include/python3.5m"
> ...
>
> Looking at the buildfarm animals that failed this way, 10 out of 11
> are using python 2.x.  The lone exception is Andrew's prion.  I wonder
> if there is something unusual about its python3 installation.



It's an Amazon Linux instance, and using their packages, which seem a
bit odd (there's nothing in /usr/local/include). Maybe we should be
looking at INCLUEPY?


[ec2-user@ip-172-31-22-42 bf]$ python3 -m sysconfig | grep include
    include = "/usr/local/include/python3.6m"
    platinclude = "/usr/local/include/python3.6m"
    CONFIG_ARGS = "'--build=x86_64-redhat-linux-gnu'
'--host=x86_64-redhat-linux-gnu' '--target=x86_64-amazon-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/var/lib' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--enable-ipv6' '--enable-shared'
'--with-computed-gotos=yes' '--with-dbmliborder=gdbm:ndbm:bdb'
'--with-system-expat' '--with-system-ffi'
'--enable-loadable-sqlite-extensions' '--with-dtrace' '--with-valgrind'
'--without-ensurepip' '--enable-optimizations'
'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu'
'target_alias=x86_64-amazon-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC
-fwrapv  ' 'LDFLAGS= -g  ' 'CPPFLAGS= '
'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'"
    CONFINCLUDEDIR = "/usr/include"
    CONFINCLUDEPY = "/usr/include/python3.6m"
    INCLDIRSTOMAKE = "/usr/include  /usr/include/python3.6m"
    INCLUDEDIR = "/usr/include"
    INCLUDEPY = "/usr/include/python3.6m"


I have upgraded it to python 3.8, but got similar results.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=  
writes:
> On Wed, Jan 19, 2022 at 9:40 PM Tom Lane  wrote:
>> Done.  (I couldn't find any equivalent logic in the MSVC build scripts
>> though; is there something I missed?)

> MSVC will use the path configured in src\tools\msvc\config.pl 
> $config->{"python"},
> there is no ambiguity.

Ah, right.  We have only three active Windows animals that are building
with python, and all three are using 3.something, so that side of the
house seems to be ready to go.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Peter Eisentraut  writes:
>> On 19.01.22 01:18, Tom Lane wrote:
>>> ... I'm tempted to resurrect the
>>> idea of changing configure's probe order to "python3 python python2"
>>> in the meantime, just so we can see how much of the buildfarm is ready
>>> for that.

>> This seems sensible in any case, given that we have quasi-committed to 
>> enforcing Python 3 soon.

> Done.

The early returns are not great: we have about half a dozen machines
so far that are finding python3, and reporting sane-looking Python
include paths, but not finding Python.h.  They're all Linux-oid
machines, so I suppose what is going on is that they have the base
python3 package installed but not python3-dev or local equivalent.

I want to leave that patch in place long enough so we can get a
fairly full survey of which machines are OK and which are not,
but I suppose I'll have to revert it tomorrow or so.  We did
promise the owners a month to adjust their configurations.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Juan José Santamaría Flecha]

On Wed, Jan 19, 2022 at 9:40 PM Tom Lane  wrote:

>
> Done.  (I couldn't find any equivalent logic in the MSVC build scripts
> though; is there something I missed?)
>
> MSVC will use the path configured in src\tools\msvc\config.pl 
> $config->{"python"},
there is no ambiguity.

Regards,

Juan José Santamaría Flecha

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> On 19.01.22 01:18, Tom Lane wrote:
>> ... I'm tempted to resurrect the
>> idea of changing configure's probe order to "python3 python python2"
>> in the meantime, just so we can see how much of the buildfarm is ready
>> for that.

> This seems sensible in any case, given that we have quasi-committed to 
> enforcing Python 3 soon.

Done.  (I couldn't find any equivalent logic in the MSVC build scripts
though; is there something I missed?)

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 19.01.22 01:18, Tom Lane wrote:

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3.  I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.


This seems sensible in any case, given that we have quasi-committed to 
enforcing Python 3 soon.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Anyway, based on these results, we might have better luck switching to
> sysconfig after we start forcing python3.

On the other hand, that answer is not back-patchable, and we surely
need a back-patchable fix, because people will try to build the
back branches against newer pythons.

Based on the buildfarm results so far, the problem can be described
as "some installations say /usr/local when they should have said /usr".
I experimented with the attached delta patch and it fixes the problem
on my Debian 9 image.  (I don't know Python, so there may be a better
way to do this.)  We'd have to also bump the minimum 3.x version to
3.2, but that seems very unlikely to bother anyone.

regards, tom lane

diff --git a/config/python.m4 b/config/python.m4
index 8ca1eaa64b..c65356c6ac 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -56,13 +56,20 @@ AC_MSG_RESULT([$python_configdir])
 
 AC_MSG_CHECKING([Python include directories])
 python_includespec=`${PYTHON} -c "
-import sysconfig
-a = '-I' + sysconfig.get_path('include')
-b = '-I' + sysconfig.get_path('platinclude')
+import sysconfig, os.path
+a = sysconfig.get_path('include')
+b = sysconfig.get_path('platinclude')
+# Some versions of sysconfig report '/usr/local/include'
+# when they should have said '/usr/include'
+if not os.path.exists(a + '/Python.h'):
+aalt = a.replace('/usr/local/', '/usr/', 1)
+if os.path.exists(aalt + '/Python.h'):
+a = aalt
+b = b.replace('/usr/local/', '/usr/', 1)
 if a == b:
-print(a)
+print('-I' + a)
 else:
-print(a + ' ' + b)"`
+print('-I' + a + ' -I' + b)"`
 if test "$PORTNAME" = win32 ; then
 python_includespec=`echo $python_includespec | sed 's,[[\]],/,g'`
 fi
diff --git a/configure b/configure
index 9c856cb1d5..f88db9467d 100755
--- a/configure
+++ b/configure
@@ -10370,13 +10370,20 @@ $as_echo "$python_configdir" >&6; }
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking Python include directories" >&5
 $as_echo_n "checking Python include directories... " >&6; }
 python_includespec=`${PYTHON} -c "
-import sysconfig
-a = '-I' + sysconfig.get_path('include')
-b = '-I' + sysconfig.get_path('platinclude')
+import sysconfig, os.path
+a = sysconfig.get_path('include')
+b = sysconfig.get_path('platinclude')
+# Some versions of sysconfig report '/usr/local/include'
+# when they should have said '/usr/include'
+if not os.path.exists(a + '/Python.h'):
+aalt = a.replace('/usr/local/', '/usr/', 1)
+if os.path.exists(aalt + '/Python.h'):
+a = aalt
+b = b.replace('/usr/local/', '/usr/', 1)
 if a == b:
-print(a)
+print('-I' + a)
 else:
-print(a + ' ' + b)"`
+print('-I' + a + ' -I' + b)"`
 if test "$PORTNAME" = win32 ; then
 python_includespec=`echo $python_includespec | sed 's,[\],/,g'`
 fi

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

I wrote:
> Are we sure it's an issue within Python, rather than something we
> could dodge by invoking sysconfig differently?  It's hard to believe
> that sysconfig could be totally unfit for the purpose of finding out
> the include path and would remain so for multiple years.

I dug up a Debian 9 image and found that I could reproduce the problem
against its python2 (2.7.13) installation, but not its python3 (3.5.3):

$ python2 -m sysconfig | grep include
include = "/usr/local/include/python2.7"
platinclude = "/usr/local/include/python2.7"
...
$ python3 -m sysconfig | grep include
include = "/usr/include/python3.5m"
platinclude = "/usr/include/python3.5m"
...

Looking at the buildfarm animals that failed this way, 10 out of 11
are using python 2.x.  The lone exception is Andrew's prion.  I wonder
if there is something unusual about its python3 installation.

Anyway, based on these results, we might have better luck switching to
sysconfig after we start forcing python3.  I'm tempted to resurrect the
idea of changing configure's probe order to "python3 python python2"
in the meantime, just so we can see how much of the buildfarm is ready
for that.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> I don't have a clear idea how to fix this in the long run.  We would 
> perhaps need to determine at which points the various platforms had 
> fixed this issue in their Python installations and select between the 
> old and the new approach based on that.  Seems messy.

Are we sure it's an issue within Python, rather than something we
could dodge by invoking sysconfig differently?  It's hard to believe
that sysconfig could be totally unfit for the purpose of finding out
the include path and would remain so for multiple years.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 18.01.22 16:24, Tom Lane wrote:

Peter Eisentraut  writes:

Also, considering the failure on prairiedog, I do see now on
 that the sysconfig
module is "New in version 3.2".  I had interpreted the fact that it
exists in version 2.7 that that includes all higher versions, but
obviously there were multiple branches involved, so that was a mistaken
assumption.


Hm.  I installed 3.1 because we claim support for that.  I don't mind
updating to 3.2 (as long as we adjust the docs to match), but it seems
kinda moot unless you figure out a solution for the include-path
issue.  I see that platforms as recent as Debian 10 are failing,
so I don't think we can dismiss that as not needing fixing.


I have reverted this for now.

I don't have a clear idea how to fix this in the long run.  We would 
perhaps need to determine at which points the various platforms had 
fixed this issue in their Python installations and select between the 
old and the new approach based on that.  Seems messy.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> Also, considering the failure on prairiedog, I do see now on 
>  that the sysconfig 
> module is "New in version 3.2".  I had interpreted the fact that it 
> exists in version 2.7 that that includes all higher versions, but 
> obviously there were multiple branches involved, so that was a mistaken 
> assumption.

Hm.  I installed 3.1 because we claim support for that.  I don't mind
updating to 3.2 (as long as we adjust the docs to match), but it seems
kinda moot unless you figure out a solution for the include-path
issue.  I see that platforms as recent as Debian 10 are failing,
so I don't think we can dismiss that as not needing fixing.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 09.12.21 14:31, Tom Lane wrote:

Peter Eisentraut  writes:

On 02.12.21 08:20, Peter Eisentraut wrote:

Buildfarm impact:
gaur and prariedog use Python 2.6 and would need to be upgraded.



Tom, are you planning to update the Python version on these build farm
members?  I realize these are very slow machines and this might take
some time; I'm just wondering if this had registered.


I can do that when it becomes necessary.  I've got one eye on the meson
conversion discussion, which will kill those two animals altogether;
so it seems possible that updating their Pythons now would just be
wasted effort depending on what lands first.


I saw that the Python installations on gaur and prairiedog had been 
updated, so I committed this patch.  As the buildfarm shows, various 
platforms have problems with this, in particular because they point to 
the wrong place for the include directory.  AFAICT, in most cases this 
appears to have been fixed in more recent editions of those platforms 
(e.g., Debian unstable members pass but older releases don't), so at 
least the approach was apparently not wrong in principle.  But 
obviously, this leaves us in a mess.  I will revert this patch in a bit, 
after gathering a few more hours of data.


Also, considering the failure on prairiedog, I do see now on 
 that the sysconfig 
module is "New in version 3.2".  I had interpreted the fact that it 
exists in version 2.7 that that includes all higher versions, but 
obviously there were multiple branches involved, so that was a mistaken 
assumption.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> On 02.12.21 08:20, Peter Eisentraut wrote:
>> Buildfarm impact:
>> gaur and prariedog use Python 2.6 and would need to be upgraded.

> Tom, are you planning to update the Python version on these build farm 
> members?  I realize these are very slow machines and this might take 
> some time; I'm just wondering if this had registered.

I can do that when it becomes necessary.  I've got one eye on the meson
conversion discussion, which will kill those two animals altogether;
so it seems possible that updating their Pythons now would just be
wasted effort depending on what lands first.

regards, tom lane

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 02.12.21 08:20, Peter Eisentraut wrote:

Buildfarm impact:

gaur and prariedog use Python 2.6 and would need to be upgraded.


Tom, are you planning to update the Python version on these build farm 
members?  I realize these are very slow machines and this might take 
some time; I'm just wondering if this had registered.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Peter Eisentraut]

On 02.12.21 19:22, Tom Lane wrote:

That's surely no problem in HEAD, but as you say, it is an issue for
the older branches.  How difficult would it be to teach configure to
try both ways, or adapt based on its python version check?


I think it wouldn't be unreasonable to do that.  I'll look into it.

Re: Replace uses of deprecated Python module distutils.sysconfig

[Tom Lane]

Peter Eisentraut  writes:
> With Python 3.10, configure spits out warnings about the module 
> distutils.sysconfig being deprecated and scheduled for removal in Python 
> 3.12:

Bleah.

> This patch changes the uses in configure to use the module sysconfig 
> instead.  The logic stays the same.  (It's basically the same module but 
> as its own top-level module.)
> Note that sysconfig exists since Python 2.7, so this moves the minimum 
> required version up from Python 2.6.

That's surely no problem in HEAD, but as you say, it is an issue for
the older branches.  How difficult would it be to teach configure to
try both ways, or adapt based on its python version check?

> I suggest leaving the backbranches alone for now.  At the moment, we 
> don't even know whether additional changes will be required for 3.12 
> (and 3.11) support, so the overall impact isn't known yet.  In a few 
> months, we will probably know more about this.

Agreed, this is a moving target so we shouldn't be too concerned
about it yet.

regards, tom lane