[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[STINNER Victor]

STINNER Victor  added the comment:

Thank you Michael Felt for your big "Implement find_library() support in 
ctypes/util for AIX" contribution, commit 
c5ae169e1b73315672770517bf51cf8464286c76!

I close the issue.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt  added the comment:

imho - this should have status "closed"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Ned Deily]

Ned Deily  added the comment:

It looks like c5ae169e1b73315672770517bf51cf8464286c76 broke find_library on 
macOS; see Issue33281.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt  added the comment:

will open a new PR for ./Doc/library/ctypes.rst

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:


New changeset c0919c27c6442aa147ae559a936b1b8deb4ee783 by Mariatta in branch 
'master':
bpo-26439: Convert %s in Lib/ctypes/_aix.py to f-strings. (GH-4986)
https://github.com/python/cpython/commit/c0919c27c6442aa147ae559a936b1b8deb4ee783


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Mariatta Wijaya]

Change by Mariatta Wijaya :


--
pull_requests: +4872

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Haubenwallner]

Michael Haubenwallner  added the 
comment:

Within this context, the "svr4" label originates in the "-bsvr4" AIX linker 
flag, and actually is another (yet fully documented by the ld(1) man page) 
method for creating shared libraries on AIX to support filename based shared 
library versioning, which is known as the DT_SONAME tag with the ELF binary 
format.

For details please refer to the GCC install doc:
https://gcc.gnu.org/install/configure.html#WithAixSoname

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt  added the comment:

SVR4 - stands for AT System V Release 4 - the beginning (as I understand it) 
of shared libraries as (indivudual) .so files in UNIX. SVR3 (and earlier) used 
.a files (aka archives with members inside).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[STINNER Victor]

STINNER Victor  added the comment:

> Although I'm unable to double check for the moment, feels like the "SVR4" 
> support still is incomplete: (...)

Sorry, I don't know what is SVR4. Is it a version of AIX? If it's rarely used, 
maybe a different issue should be opened to ask for an enhancement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Haubenwallner]

Michael Haubenwallner  added the 
comment:

Although I'm unable to double check for the moment, feels like the "SVR4" 
support still is incomplete: Remember that even the libNAME.so file may be an 
archive, or a symlink to an archive, with the real Shared Object as member 
having the F_LOADONLY flag set and usually named shr.o or shr_64.o, besides an 
Import File shr.imp or shr_64.imp, which actually is used at linktime and 
referring to the real Shared Object: This is necessary to emulate the 
"DT_SONAME" feature seen with ELF shared libraries.

For reference, please have a look at the shared libraries created by recent 
libtool when --with-aix-soname=svr4 configure option is set.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[STINNER Victor]

STINNER Victor  added the comment:

Michael Felt: I merged your PR ✨ ! Thank you very much for your patience and 
perseverance. 80 comments on this bug, 117 comments and 19 commits in the PR... 
wow! AIX is not dead 

I leave the bug open since Martin, Mariatta and me were interested to make 
further minor coding style changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c5ae169e1b73315672770517bf51cf8464286c76 by Victor Stinner 
(Michael Felt) in branch 'master':
bpo-26439 Fix ctypes.util.find_library failure on AIX (#4507)
https://github.com/python/cpython/commit/c5ae169e1b73315672770517bf51cf8464286c76


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter  added the comment:

Michael Felt: if you still want the code compatible with Python 2 and 3 (and 
others are happy with that), I suggest documenting that in a code comment.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Haubenwallner]

Changes by Michael Haubenwallner :


--
nosy: +haubi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

OOPS: I have a ... above, meant to be a link to a message.

I also needed to 'patch' util.py - with 32-bit build with something like this:
--- src/python-2.7.13/Lib/ctypes/util.py2016-12-17 20:05:05 +
+++ python-2.7.13.0/Lib/ctypes/util.py  2017-01-13 13:29:12 +
@@ -299,6 +299,10 @@
 print cdll.LoadLibrary("libcrypto.dylib")
 print cdll.LoadLibrary("libSystem.dylib")
 print cdll.LoadLibrary("System.framework/System")
+elif sys.platform[:3] == "aix":
+from ctypes import CDLL
+RTLD_MEMBER =  0x0004
+print CDLL("libc.a(shr.o)", RTLD_MEMBER)
 else:
 print cdll.LoadLibrary("libm.so")
 print cdll.LoadLibrary("libcrypt.so")


And 64-bit with:

1: elif sys.platform[:3] == "aix":
1: from ctypes import CDLL
1: RTLD_MEMBER =  0x0004
1: print CDLL("libc_64.a(shr.o)", RTLD_MEMBER)

The respective output is:

root@x064:[/data/prj/python/python-2.7.13.0]./python Lib/ctypes/util.py
None
None
None


root@x064:[/data/prj/python/python-2.7.12.0]./python Lib/ctypes/util.py
None
None
None


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

I would like to say:
a) I am happy this is being considered for Python3-3.7

b) I still believe it is a "bug" plain and simple - it (find_library()) cannot 
work in "normal" situations. Why "IBM" or others never addressed this is beyond 
me - but - imho - that is a poor argument for not fixing it.

FYI: As I have documented in ... this does not mean that ctypes.CDLL is 
incapable of loading a library. If you know the 'magic', aka what is not being 
provided by this "implementation dependent" module - it is possible to load 
either a .so file (actually a file by any name as long as it is a shared 
archive) - or an AIX-style .a archive with shared libraries stored internally 
as "archive members" - again, any member name is acceptable.

The bug: The "default code" behind an "else:" block depends on the presence of 
the program "/sbin/ldconfig" to create a list of shared libraries. Normally, 
this program is not installed on an AIX system (certainly not production - 
maybe a system with gcc environment and using the gcc ld rather than the AIX ld 
(the latter being normal).

Using python -m trace -C /tmp --count Lib/src/util.py The following (key) 
counts are reported.

This is all that is being called - so when /sbin/ldonfig is not installed 
(which is usual) - so this code can only return 'None'.

Environment: AIX 5.3, AIX 6.1, AIX 7.1 - all identical

root@x064:[/]ls -l /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

michael@x071:[/home/michael]ls /sbin/ld*
ls: cannot access '/sbin/ld*': A file or directory in the path name does not 
exist.
(GNU coreutils ls here)

root@x062:[/]ls /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

So, obviously the sys.popen() call below will always 'fail'.


The basic trace from util.py:
1: if os.name == "posix":
   # find and load_version
1: print find_library("m")
1: print find_library("c")
1: print find_library("bz2")


And how this is called:

   else:

1: def _findSoname_ldconfig(name):
3: import struct
3: if struct.calcsize('l') == 4:
   machine = os.uname()[4] + '-32'
   else:
3: machine = os.uname()[4] + '-64'
3: mach_map = {
3: 'x86_64-64': 'libc6,x86-64',
3: 'ppc64-64': 'libc6,64bit',
3: 'sparc64-64': 'libc6,64bit',
3: 's390x-64': 'libc6,64bit',
3: 'ia64-64': 'libc6,IA-64',
   }
3: abi_type = mach_map.get(machine, 'libc6')

   # XXX assuming GLIBC's ldconfig (with option -p)
3: expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), 
abi_type)
3: f = os.popen('LC_ALL=C LANG=C /sbin/ldconfig -p 2>/dev/null')
3: try:
3: data = f.read()
   finally:
3: f.close()
3: res = re.search(expr, data)
3: if not res:
3: return None
   return res.group(1)

1: def find_library(name):
3: return _findSoname_ldconfig(name) or 
_get_soname(_findLib_gcc(name))


And with python2-2.7.13 - the counts start the same - but the result is the 
same - by definition 'None' because subprocess.Popen() also
does not have any output to search...

1: if os.name == "posix":
   # find and load_version
1: print find_library("m")
1: print find_library("c")
1: print find_library("bz2")

   else:

1: def _findSoname_ldconfig(name):
3: import struct
3: if struct.calcsize('l') == 4:
3: machine = os.uname()[4] + '-32'
   else:
   machine = os.uname()[4] + '-64'
3: mach_map = {
3: 'x86_64-64': 'libc6,x86-64',
3: 'ppc64-64': 'libc6,64bit',
3: 'sparc64-64': 'libc6,64bit',
3: 's390x-64': 'libc6,64bit',
3: 'ia64-64': 'libc6,IA-64',
   }
3: abi_type = mach_map.get(machine, 'libc6')

   # XXX assuming GLIBC's ldconfig (with option -p)
3: expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), 
abi_type)

3: env = dict(os.environ)
3: env['LC_ALL'] = 'C'
3: env['LANG'] = 'C'
3: null = open(os.devnull, 'wb')
3: try:
3: with null:
3: p = subprocess.Popen(['/sbin/ldconfig', '-p'],
3:   stderr=null,
3:   stdout=subprocess.PIPE,
3:   env=env)
3:

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Just some minor comments on aix-library.161004.patch:

Instead of _util.py, I wonder if the new file should have a different name, 
like _util_common.py, to avoid being too similar to util.py.

+def get_shared(input):
+"""Internal support function: examine the get_dumpH() output and
+return a list of all shareable objects indicated in the output the
+character "[" is used to strip off the path information.

Needs a newline or new sentance to separate “output” and “the character”.

+def get_legacy(members):
+[. . .]
+# shr.o is the preffered name so we look for shr.o first

Spelling: preferred [single F, double R]

+def get_version(name, members):
+"""[. . .]
+Before the GNU convention became the standard scheme regardless of
+binary size AIX packagers used GNU convention "as-is" for 32-bit
+archive members but used an "distinguishing" name for 64-bit members.

Should be: a "distinguishing" [not “an”]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Besides correcting a small error, also my attempt to follow the
guidelines to not import *, but to actually specify all bits
that are to be imported.

This is a patch compered to Python-3.6b1

--
hgrepos: +359
Added file: http://bugs.python.org/file44963/aix-library.161004.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


Added file: http://bugs.python.org/file44962/aix-modules.161004.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Have a way to have both 64-bit and 32-bit modes working.

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python 
`pwd`/util.py
m   :: None
c   :: libc.a(shr_64.o)
bz2 :: libbz2.a(libbz2.so.1)
crypt   :: libcrypt.a(shr_64.o)
crypto  :: 
c   :: 

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python 
`pwd`/util.py
m   :: None
c   :: libc.a(shr.o)
bz2 :: libbz2.a(libbz2.so)
crypt   :: libcrypt.a(shr.o)
crypto  :: 
c   :: 

Will post patch asap. Going to shorten some lines going well over 76-char 
lime-limit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Curious.

When in 32-bit mode changing line 15 of _aix.py to

   +14  import re, os, sys
   +15  # from . import util

The Lib/ctypes/util.py works.

In 64-bit mode it does not:
instead:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
m   :: None
c   :: libc.a(shr_64.o)
Traceback (most recent call last):
  File "util.py", line 355, in 
test()
  File "util.py", line 330, in test
print("bz2\t:: %s" % find_library("bz2"))
  File "util.py", line 104, in find_library
return aix.find_library(name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 255, in 
find_library
(base, member) = find_shared(libpaths, name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 247, in 
find_shared
member = get_member(re.escape(name), members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 189, in 
get_member
member = get_version(name, members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 170, in 
get_version
return util._last_version(versions, '.')
NameError: name 'util' is not defined

+
When the comment is removed, i.e.
from . import util

both 32 and 64-bit report:
root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in 
import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in 

from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in 

import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

This last condition also occurs in Python2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

I have spent the last two hours trying to run the test - however, it fails with:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in 
import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in 

from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in 

import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

I have noticed several issues with the file that used to be named just
./build/_sysconfigdata.py

but is now: ./build/lib.aix-5.3-3.6/_sysconfigdata_m_aix5_.py

I am guessing something is wrong there - I am going to try copying only _aix.py 
to the Python2 branch and see if it works there -- and also dig deeper into 
what is going wrong with Python3.6*

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

On 01-Oct-16 08:44, Martin Panter wrote:
> Martin Panter added the comment:
>
> Hi Michael, I have done some cleanup and modifications to your patch. The 
> result is in aix-library.161001.patch, which has all the changes, i.e. it is 
> not based on another patch.
Thanks.
>   More significant changes I made:
>
> * Change getExecLibPath_aix() and find_parts() to return a list object, 
> rather than building a colon-separated string only to be pulled apart again
> * Escape dots in get_legacy() regular expressions, so that they no longer 
> match [shr_64xo], [shrxo], etc.
> * Make get_dumpH() return the a list of (object, objectinfo) tuples, where 
> objectinfo is a list of lines; avoids building multiline strings and then 
> splitting them apart again
> * Rewrite get_exactMatch() and get_version() without nested inline “for” 
> loops; use RE capture group
> * Reuse util._last_version() instead of copying the _num_version() function
> * Use lower case B for liB in get_member(). This means e.g. libcrypto.so is 
> now preferred over libcrypto.so.1.0.0.
That was a typo - to be sure I was still finding the versioned ones (the 
previous ones had had a bug that they no longer found the "standard" 
one. I forgot to remove (rather save file before the diff command) - you 
see everything!
>
> I did test it a bit on Linux with faked dump -H output, but I may have made 
> mistakes that I did not pick up.
Will apply the patch, build in 32 and 64 bit modes, and respond.
>
> Also, this still needs documentation, and I think some more tests for the 
> test suite exercising various aspects of find_library() would be nice if 
> possible.
Working on that - have been posting some questions on python-list as I 
want to build an interface with an AIX performance library (libperfstat).
Without this library one must run a command and then do some string 
manipulation, and sometimes call a second command once that has been 
found (e.g., uuid calls to get a MAC address, cloud-init to get 
boottime) - things that - with a library do not need a subprocess at all.

But I shall also write up the AIX dlopen() process to explain how both 
.so and .a(member.so) works (as the argument to CDLL).
>
> Another thing: in the last few patches, you dropped the definition of 
> RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a 
> temporary thing?
Temporary thing - as I keep hoping for inclusion in Python2 and I recall 
you not wanting to add that into _ctypes on Python2.
>
> --
> versions:  -Python 3.6
> Added file: http://bugs.python.org/file44902/aix-library.161001.patch
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Hi Michael, I have done some cleanup and modifications to your patch. The 
result is in aix-library.161001.patch, which has all the changes, i.e. it is 
not based on another patch. More significant changes I made:

* Change getExecLibPath_aix() and find_parts() to return a list object, rather 
than building a colon-separated string only to be pulled apart again
* Escape dots in get_legacy() regular expressions, so that they no longer match 
[shr_64xo], [shrxo], etc.
* Make get_dumpH() return the a list of (object, objectinfo) tuples, where 
objectinfo is a list of lines; avoids building multiline strings and then 
splitting them apart again
* Rewrite get_exactMatch() and get_version() without nested inline “for” loops; 
use RE capture group
* Reuse util._last_version() instead of copying the _num_version() function
* Use lower case B for liB in get_member(). This means e.g. libcrypto.so is now 
preferred over libcrypto.so.1.0.0.

I did test it a bit on Linux with faked dump -H output, but I may have made 
mistakes that I did not pick up.

Also, this still needs documentation, and I think some more tests for the test 
suite exercising various aspects of find_library() would be nice if possible.

Another thing: in the last few patches, you dropped the definition of 
RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a 
temporary thing?

--
versions:  -Python 3.6
Added file: http://bugs.python.org/file44902/aix-library.161001.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Roundup Robot]

Roundup Robot added the comment:

New changeset 01885f78b299 by Martin Panter in branch '2.7':
Issue #26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/01885f78b299

New changeset 0db4403e62c4 by Martin Panter in branch '3.5':
Issue #26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/0db4403e62c4

New changeset 4b7e51998a90 by Martin Panter in branch '3.6':
Issue #26439: Merge ctypes doc from 3.5 into 3.6
https://hg.python.org/cpython/rev/4b7e51998a90

New changeset f496fb6bf4a0 by Martin Panter in branch 'default':
Issue #26439: Merge ctypes doc from 3.6
https://hg.python.org/cpython/rev/f496fb6bf4a0

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Sigh - missed the feature cutoff that would have made this easier to get into 
python...

Anyway - have learned a few new things about python def: syntax and removed 
some bits that I thought were suitable for variable "initialization" - but tend 
to be static (not what I was thinking).

Also, "new and improved" comments about what the code is doing.

Thank you all for your patience and feedback - especially Martin!

--
versions: +Python 3.7
Added file: http://bugs.python.org/file44831/Python3.6.Lib.ctypes.160926.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Not always as elegant as I would wish (do not like the idea of "while 1:" and 
later a break...

But, all in all, much improved by redoing the processing of the subprocess 
output and getting rid of more noise.

Hope this meets your approval!

p.s. This is a patch compared to previous patch. If a diff compared to 
Python-3.6.0a4 I can remake the patch.

--
Added file: http://bugs.python.org/file44366/Python3.6.Lib.ctypes.160904.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

On 04/09/2016 06:11, Martin Panter wrote:
> I do not know whether to fix the annotation (has 64 preceded by any number of 
> underscores), or whether to fix the regular expression (_?64).

The later - _?64. Working on this today. Thank you for the correction.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Regarding the regular expressions, I (or someone else unfamiliar with AIX) may 
be able to adjust them if you can explain what you are trying to achieve. Take 
the first one I commented on 

 for example. You added a comment:

# '\[%s_*64\.so\]' % name, -> has either _64 or 64 added to name

As written, this will match many strings including

[___64.so]

However the annotation leads me to belive you want it to match two cases only:

[64.so]
[_64.so]

I do not know whether to fix the annotation (has 64 preceded by any number of 
underscores), or whether to fix the regular expression (_?64).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

On 27-Aug-16 09:11, Martin Panter wrote:
> Martin Panter added the comment:
>
> The documentation is in RST format in the Doc/ directory. For basic stuff, 
> you can just copy the syntax from existing text, or see e.g. 
>  
> for more hints. For this change, I think we need to mention in 
> Doc/library/ctypes.rst:
>
> * the special AIX changes for find_library(); include “.. versionchanged::” 
> (or maybe versionadded? I’m not sure) notice
> * the change for the CDLL constructor, also with versionchanged
>
> Perhaps also add an entry to Doc/whatsnew/3.6.rst.
>
> Patch 160823 does not addressed many of my previous comments. Please have a 
> closer look. I can make simple changes to simplify the code myself, but I 
> don’t really know what to do about the questionable regular expressions, for 
> instance.
As far as regular expressions go, that will always be difficult for me 
aka questionable for you. Every language has it's nuances and I never 
seem to get them right.

>
> Also, see . What was wrong with 
> the cases supported by your original patches?
As you clearly pointed out, there were no prior cases showing their 
need. I was taking an unbiased approach in that I knew no previous code 
well. I was just trying to make it work and was trying to think of cases 
were it could go wrong. In short, I was trying to solve issues that do 
not exist. People accept whatever libFOO.so points to as a symbolic link 
- and expect libSOO.so to be the name of the shared library that 
dlopen() is going to open somewhere.

Once I understood the boundaries of find_library("foo") I limited myself 
to only resolve in the way the compile-time resolution is done. As such, 
find_library("c") is the equivalent of -lc and needs to resolve to 
libc.a(shr.o) - period (32-bit) or libc.a(shr_64.o) (64-bit).

The linker is not looking for a particular member name: it only uses 
-lFOO to find libFOO.a, and it it exists it looks for a symbol. The 
member name that contains the symbol is what it stores in the executable 
(displayed via dump -H). So, getting back to this patch and packaging 
conventions.

Basically, libtool has standardised how members are named, i.e., 
versioning. In most cases with OSS packages the only member-name that 
gets stored is the same name that libFOO.so would be a symbolic link to. 
Again, the compiler-linker only needs the name of the archive. The 
member name within the archive is irrelevant - from a compiler/linker 
perspective.

My goal for find_library() is to find the most likely name based on some 
legacy "rules" from the way IBM packaged libraries starting over 20 
years ago (aka AIX 4 standards) that are alive today to support binary 
compatibility - as much as possible. Mainly, that is relevant for 
libraries/archives such as libc.a. For new libraries, especially OSS 
packages built using the GNU autotools (and finishing with libtool) 
there are other conventions.

Finally, we had different goals - my focus was on writing something that 
matches other platforms python2 behavior, not on writing a new syntax 
specific for Python3.6 and AIX. If I were to do be writing a new syntax 
I would prefer that it also work for other platforms. Something 
different for only one platform feels wrong - imho.

In closing:
a) regular expressions and me is always a headache for someone. please 
accept my apologies if I have given you a headache.
b) I had assumed abilities for find_library() (from studying the output 
of ldconfig and trying to follow the regular expressions in the current 
code) that are not used. To assume makes an ass of u and me - especially me.
c) I also apologize for not meeting your expectations. I cut code, 
rather than defend it, as you were correct that is was not based on 
meeting the needs of general aka current practice.

I am not trying to redesign find_library(). My hope is that most python 
code would work asis - if they follow conventions aka general practice 
(not equivalent to best practice as best depends (in part) on your goal).
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

The documentation is in RST format in the Doc/ directory. For basic stuff, you 
can just copy the syntax from existing text, or see e.g. 
 for 
more hints. For this change, I think we need to mention in 
Doc/library/ctypes.rst:

* the special AIX changes for find_library(); include “.. versionchanged::” (or 
maybe versionadded? I’m not sure) notice
* the change for the CDLL constructor, also with versionchanged

Perhaps also add an entry to Doc/whatsnew/3.6.rst.

Patch 160823 does not addressed many of my previous comments. Please have a 
closer look. I can make simple changes to simplify the code myself, but I don’t 
really know what to do about the questionable regular expressions, for instance.

Also, see . What was wrong with 
the cases supported by your original patches?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Changes by Martin Panter :


Added file: https://bugs.python.org/file44237/Python3.6.ctypes.160823.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

patch to include RTLD_MEMBER in Modules/_ctypes and also "document", or not 
document, RTLD_NOW

--
Added file: 
http://bugs.python.org/file44206/Python3.6.Modules._ctypes.160823.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

New patch (diff) - that I hope address the comments made in the previous 
submission

--
Added file: http://bugs.python.org/file44205/Python3.6.Lib.ctypes.160823.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

For the record, I presume you are referring to Issue 27435.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Thanks again for your help.

Note: I started a issue# is because too much of what I was submitting here was 
"extension" and not "correction".

Hopefully, the new issue# will be cleaner.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

I left a few more comments, but I see that many of my earlier comments (both in 
Rietveld and main bug thread) still apply.

I’m still struggling to understand all the special cases for the find_library() 
argument. You seem to have added even more recently (although I think you 
removed some as well). Can you point to any documentation, command lines, etc 
demonstrating why Python should support:

find_library("libNAME")
find_library("search:paths/NAME")
find_library("NAME.so*")
find_library("/NAME")  # Not sure if this case is even relevant

Perhaps it would help if you explained what a typical AIX compiler or linker 
command line looks like. Or if you updated the documentation 
 to 
explain the cases for AIX.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Uploading a combined patch that should work with Rietveld, and I will leave 
some comments.

I don’t think there is much that I am comfortable changing in Python 2.7. In 
general it is a bad idea to add new features that are only going to be 
available in e.g. 2.7.13+ and won’t work in older versions of 2.7. One rule of 
thumb is if a change would require additional documentation, then it is 
probably a new feature. In this case, find_library() is returning something 
that is not a plain filename or path name, so the documentation also needs 
changing.

If we added a version of find_library() that returned only file or path names 
(and not archive members!), I might consider that acceptable for 2.7, because 
it is filling in missing AIX functionality, and sticks to the current 
documentation. But it sounds like most libraries are archive members, so I can 
understand if that sort of change is not worth the effort.

--
Added file: http://bugs.python.org/file43479/Python2.Lib.ctypes.160611.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

updated patch for Python 2.7 (shall adopt and submit new patch for Python3 
based on this - next week)

used gnu diff to generate diff output.

Additional tests in util.py are for demo only, would expect them to be removed, 
however, the additional test in test/test_loading.py could remain.

--
Added file: http://bugs.python.org/file43426/Python2.Lib.ctypes.160611.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

The patch for Python2 - however, only now see the change in selections for 
version. Will need to redo/test Lib/ctypes/*.py in newer version.

Note also, the additional tests in util.py are for my testing - I do not expect 
them to stay, but I do want you aware of how I am testing - see also change in 
Lib/ctypes/test/test_loading.py

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

_aixutil.py to be paired with Python2.Lib.ctypes.16.06.11.patch

--
Added file: http://bugs.python.org/file43425/_aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

On 6/12/2016 11:41 AM, Martin Panter wrote:
> .
> Michael, how are you supposed to apply your latest patch? I have Gnu Patch 
> 2.7.5, but even in the best case it doesn’t find the files to patch:
>
> $ patch -p1 -n < python.Lib.ctypes.160608.patch
> can't find file to patch at input line 2
> The text leading up to this was:
> --
> |diff -r Python-2.7.11/Lib/ctypes/__init__.py 
> python-2.7.11.5/Lib/ctypes/__init__.py
> --
I must have forgotten the u (-r rather than -ru) (read mistyped, as I 
"always" use the -u option)
> Perhaps can you make a “unified” diff (-u option) like you did in your 
> previous patches? Whatever you did with Python3.issue26439.160511.patch 
> seemed to work best.
Will redo all of them in about 24 hours.
> Also, since your patches add new functionality (the automatic archive member 
> detection), they will have to be for Python 3, not 2.

Well, let me know what needs to be deleted - I see the member detection 
as the equivalent of reading the output of "ldconfig -p" to search for 
where an archive might be, or compareable with the gcc command (forgot 
the options) to say where it is looking.

Not going into Python2 misses the whole point from my perspective.

Unfortunate (for my perspective) is that noone else is saying anything 
one way or the other.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

.
Michael, how are you supposed to apply your latest patch? I have Gnu Patch 
2.7.5, but even in the best case it doesn’t find the files to patch:

$ patch -p1 -n < python.Lib.ctypes.160608.patch
can't find file to patch at input line 2
The text leading up to this was:
--
|diff -r Python-2.7.11/Lib/ctypes/__init__.py 
python-2.7.11.5/Lib/ctypes/__init__.py
--

Perhaps can you make a “unified” diff (-u option) like you did in your previous 
patches? Whatever you did with Python3.issue26439.160511.patch seemed to work 
best.

Also, since your patches add new functionality (the automatic archive member 
detection), they will have to be for Python 3, not 2.

===

Tuple vs "archive(member)" string being an API change:
> name = "libcrypto.a(libcrypto.so.1.0.0)"
> name = ("libcrypto.a", "libcrypto.so.1.0.0")
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)

IMO all three options are API changes. Even though the first one is still a 
string, we are changing the meaning of the string. It no longer always means a 
plain file or library name, and you lose the ability to indicate a plain 
library file when the special format is used (even though that ability is 
unlikely to be used much).

===

Yes, please remove the internal variables from the signatures. Local variables 
always override the global scope anyway, so it makes no difference. And if you 
use the wrong kind of object (mutable object) as a default value in the 
signature, it can actually lead to subtle bugs.

I am happy to try modifying your code (once we get the patch format etc figured 
out). That is probably the quickest way to fix up some of the problems that are 
obvious to me, e.g. variables in signatures, subprocess driving, etc.

===

With my understanding, find_library("FOO") should not recognize 
/usr/lib/FOO.so, it should only look for /usr/lib/libFOO.so, etc. So it should 
return None. Does linking with -lFOO recognize /usr/lib/FOO.so for you?

Similarly, find_library("../some/where/Foo.so") shouldn’t work. You wouldn’t 
pass that with -l to the linker, I don’t think.

--
versions: +Python 3.6 -Python 2.7, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

On 6/10/2016 6:33 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Will try to change the existing code from os.popen to subprocess (Issue 
> 26439) to set a better example for new code like this
>
> --
> dependencies: +avoid using a shell in ctypes.util: replace os.popen with 
> subprocess
>
> ___
> Python tracker 
> 
> ___

I have added comments to _aixutil.py - but while doing so I came up with 
some thoughts re "special cases", where

a) the member is not in a archive (i.e., looking for a file)

b) the filename does not begin with "lib"

c) CDLL.LoadLibrary(name) succeeds

So, given that /usr/lib is default search path, and file FOO.so exists 
as /usr/lib/FOO.so - should find_library(name) return name, or None.

d) name = ../some/where/Foo.so; CDLL.LoadLibrary("../some/where/Foo.so") 
works - should find_library(name) return name

e) assume find_library("../some/where/Foo") returns 
../some/where/libFoo.so, or None - again, given that 
../some/where/libFoo.a does not exist

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

* You may also just modify my "attempt" in _aixutil.py

I recall you said something about the p.wait() being a potential to hang, but I 
also want to be sure the subprocess has exited properly - as it is not 
something to be running in parallel - and be sure that the "lines" I return are 
complete.

And, I see I have more underscores to remove :)

* I shall work on the comments you requested asap.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Will try to change the existing code from os.popen to subprocess (Issue 26439) 
to set a better example for new code like this

--
dependencies: +avoid using a shell in ctypes.util: replace os.popen with 
subprocess

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

aixutil.py renamed as _aixutil.py and other changes in response to Martin's 
comments of 4 June

delta of changes to __init__.py, util.py, and test/test_loading.py

--
Added file: http://bugs.python.org/file43312/python.Lib.ctypes.160608.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

aixutil.py renamed as _aixutil.py and other changes in response to Martin's 
comments of 4 June

--
Added file: http://bugs.python.org/file43311/_aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

Answering this again - now that the new patch is ready.

On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked 
> for how others handle this, and I found Libtool’s LTDL library, and Apache 
> Portable Runtime (APR). Both have a similar (but stricter) automatic addition 
> based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
>
> So I guess I can accept this way, if you think it is better than my other 
> ideas:
a) I think it is more in line with what I perceive as normal practice
  libFOO = CDLL(find_library("FOO"))
As CDLL() has always had a 'simple' string as input and not a "tuple".
I have also added some lines to test/test_loading.py to test direct 
calling of CDLL() with fixed strings and a test of os.maxsize
and in util.py - but using e.g., CDLL(find_library("c") as behavior is 
dependent on 32 or 64-bit mode

depending on mode - different output:
  note: find_library("libssl64") is expected to return None - as it 
would be "abnormal" to have an archive libssl64.a or a file libssl64.so

cd Lib/ctypes
../../python util.py
# 32-bit mode:
None
libc.a(shr.o)
libbz2.a(libbz2.so)
find_library("c")returns: libc.a(shr.o)
find_library("libc") returns: libc.a(shr.o)
find_library("libssl")   returns: libssl.a(libssl.so)
find_library("libssl64") returns: None
find_library("ssl")  returns: libssl.a(libssl.so)
find_library("libiconv") returns: libiconv.a(libiconv.so.2)
find_library("intl") returns: libintl.a(libintl.so.8)
libcrypt.a(shr.o)



# 64-bit mode:
None
libc.a(shr_64.o)
libbz2.a(libbz2.so.1)
find_library("c")returns: libc.a(shr_64.o)
find_library("libc") returns: libc.a(shr_64.o)
find_library("libssl")   returns: libssl.a(libssl64.so.1.0.0)
find_library("libssl64") returns: None
find_library("ssl")  returns: libssl.a(libssl64.so.1.0.0)
find_library("libiconv") returns: libiconv.a(shr4_64.o)
find_library("intl") returns: libintl.a(libintl.so.1)
libcrypt.a(shr_64.o)



>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not 
> conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)
>
> libcrypto = CDLL(name)
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm 
> that each behaviour is intended:
I was being "Q" here, not changing the aixutils.py (now _aixutils.py). 
My intent is to be comparable with other "posix" behaviors.
So, if you believe I am still not compatible with their behavior, 
forgive me - but also shoot me!
>
> A) If I have a file called "crypto" in the current directory, 
> find_library("crypto") returns "crypto". This does not seem right. On Linux, 
> “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or 
> a dot (.), so find_library("glib-2.0") is almost equivalent to 
> find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file 
> exists. Just like in A), this does not seem right to me.
All should be seen as bugs, and I hope I coded it correctly to not do 
this anymore.
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be 
> confused with a public API. If you follow my earlier suggestion of renaming 
> the new file to _aixutil.py (so it is obvious it is not a public module), 
> then you can freely write “import re, os, sys”, etc.
I had missed, certainly not understood the context, before. aixutil.py 
is now _aixutil.py. Originally I had done this to make the diff in 
util.py much simplier, but also because I incorrectly thought CDLL() was 
frequently called with "foo" or "libfoo". In short, trying to prevent a 
non-existent problem.
__init__.py delta is also much much simpler to grasp.
> * No need to add the internal variable names to the function signatures. Just 
> write find_library(name), and if you need to initialize a variable, do that 
> in the body.
Oops - not removed those yet. That was done to be sure there was no 
global scope interference. If you feel it is vital they be removed - 
will be done.
>
> * I suggest to go over all the regular expressions, and either change them to 
> plain string searching, or make sure special characters and external 
> variables are escaped as necessary. A comment explaining what the RE is 
> trying to do might help too.
Ugh. I actually 

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked 
> for how others handle this, and I found Libtool’s LTDL library, and Apache 
> Portable Runtime (APR). Both have a similar (but stricter) automatic addition 
> based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
The "stricter" from libtool is an excellent idea - I should done that as 
well, but I really hate string manipulation :)

I hope that the intent at least more clear, and should anyone ever 
complain about it not opening a file named: libFoo.a(libFOO.so) as a 
filename - by design, not accepted as a filename.
> So I guess I can accept this way, if you think it is better than my other 
> ideas:
>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not 
> conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)
>
> libcrypto = CDLL(name)
Isn't this more of an API change - name is no longer just a string?
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm 
> that each behaviour is intended:
I have to catch a plane - will get back on these. Short - if I have a 
potential bug, then needs to be improved. More later.
>
> A) If I have a file called "crypto" in the current directory, 
> find_library("crypto") returns "crypto". This does not seem right. On Linux, 
> “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or 
> a dot (.), so find_library("glib-2.0") is almost equivalent to 
> find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file 
> exists. Just like in A), this does not seem right to me.
>
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be 
> confused with a public API. If you follow my earlier suggestion of renaming 
> the new file to _aixutil.py (so it is obvious it is not a public module), 
> then you can freely write “import re, os, sys”, etc.
>
> * No need to add the internal variable names to the function signatures. Just 
> write find_library(name), and if you need to initialize a variable, do that 
> in the body.
>
> * I suggest to go over all the regular expressions, and either change them to 
> plain string searching, or make sure special characters and external 
> variables are escaped as necessary. A comment explaining what the RE is 
> trying to do might help too.
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Okay here are some more thoughts about your latest patch:

## Automatic RTLD_MEMBER ##

I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked for 
how others handle this, and I found Libtool’s LTDL library, and Apache Portable 
Runtime (APR). Both have a similar (but stricter) automatic addition based on 
detecting the archive(member) notation:

http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
https://github.com/apache/apr/commit/c27f8d2

So I guess I can accept this way, if you think it is better than my other ideas:

# Always uses RTLD_MEMBER, never loads a plain file outside an archive
name = "libcrypto.a(libcrypto.so.1.0.0)"

# Other options, that could be returned by find_library() and would not 
conflict with a plain file name
name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)

libcrypto = CDLL(name)

## find_library() modes ##

In your find_library() function, you still have three parts. Can you confirm 
that each behaviour is intended:

A) If I have a file called "crypto" in the current directory, 
find_library("crypto") returns "crypto". This does not seem right. On Linux, 
“gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.

B) You are still stripping bits off the library name if it contains “lib” or a 
dot (.), so find_library("glib-2.0") is almost equivalent to 
find_library("b-2"). Isn’t this a bug?

C) find_library("crypto") will return "/usr/lib/crypto" if such a file exists. 
Just like in A), this does not seem right to me.

## Other things ##

* You don’t need to prefix most names with underscores, unless they could be 
confused with a public API. If you follow my earlier suggestion of renaming the 
new file to _aixutil.py (so it is obvious it is not a public module), then you 
can freely write “import re, os, sys”, etc.

* No need to add the internal variable names to the function signatures. Just 
write find_library(name), and if you need to initialize a variable, do that in 
the body.

* I suggest to go over all the regular expressions, and either change them to 
plain string searching, or make sure special characters and external variables 
are escaped as necessary. A comment explaining what the RE is trying to do 
might help too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

without aixutil.py - hopefully better for Mercurial import

--
Added file: http://bugs.python.org/file43074/Python2.Lib.ctypes.160531.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

As a separate file in the hope it re-fits into Mercurial better

--
Added file: http://bugs.python.org/file43073/aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

On 28-May-16 04:04, Martin Panter wrote:
> Martin Panter added the comment:
>
> Strictly speaking, Python 2.7 never had special support for RTLD_MEMBER or 
> find_library(). That is why I am unsure about many of these changes being 
> done in 2.7. They seem more like new features than bug fixes. Especially once 
> you start talking about a smart CDLL() that tests for a file and then falls 
> back to RTLD_MEMBER.
Well, actually, within AIX I was intending to "just use" RTLD_MEMBER 
when "the string" ended in a ")" as that is the correct additional mode 
bit for an established interface dlopen(). I perhaps do not understand 
(certainly did not) the goal of CDLL(), better Lib/ctypes

As far as Python-2.7 is concerned - I would like to see it "in" because 
there is a lot of Python-2(.7) code out there that will not work without 
manageling AIX at a low level (i.e., current practice is to extract .so 
members from archives so dlopen() will be able to open them.
>
> Whether you represent an archive member as an (archive, member) tuple, or as 
> an "archive(member)" string, or something else, it is still an extension of 
> the API of CDLL(), and arguably also of find_library().
I would want to disagree. A string is a string, regardless of the 
characters in it. And, I have verified - if a file 
/usr/lib/libFOO.a(libFoo.so.X) exists dlopen() will open it when 
RTLD_MEMBER is not ORed into the mode.
> I think I have already tried to explain that on Linux, 
> find_library("libm.so") is far from normal usage (although may be technically 
> legal if there was a library called liblibm.so.so).
find_library("libm.so") is far from normal usage: again, my 
misunderstanding - I was reading as much as possible from various 
sources - and my head was spinning.
In util.py there was no "libFOO.so", only "FOO" and "libFOO" - so my 
find_library() tests are not correct. Agreed.

>
> Also, on Linux I wouldn’t recommend CDLL("libcrypt.so"), which relies on a 
> file only intended for build time. On Debian, the glibc library 
>  has separate packages for 
> shared libraries (libc6) and development libraries (libc6-dev). It should be 
> possible to run programs with libc6 installed, but without libc6-dev. Looking 
> at the file lists for “i386” architecture, libc6-dev includes:
>
> /usr/lib/i386-linux-gnu/libcrypt.a
> /usr/lib/i386-linux-gnu/libcrypt.so
My expectation (misconception perhaps) is that Linux uses a .a archive 
for static linking, and .so shared-library for dynamic-loading. The AIX 
loader/linker looks first in .a archives for shared libraries (when 
static is not specified) and if no member is found (member name is not 
relevant for the search, only the symbol and the object size (32 or 64 
bit)). If not found in libFOO.a then a search for a specific file is 
done libFOO.so.

An assumption of mine is: if I specified (not usual on AIX, based on my 
experience) -lc.so.6 AIX would look for an archive named libc.so.6.a 
first. If found, would look for a member in that archive. If not found 
as a .a archive and/or not found as a file at all then it will look for 
a file libFOO.so.6

>
> It symbolically links to /lib/i386-linux-gnu/libcrypt.so.1, which is provided 
> by libc6:
>
> /lib/i386-linux-gnu/libcrypt-2.19.so
> /lib/i386-linux-gnu/libcrypt.so.1
>
> So on Debian (and with Linux in general), CDLL("libcrypt.so") may work in 
> some cases, but you should normally use CDLL("libcrypt.so.1") instead.
Agreed - CDLL("libFOO.so") is not preferred. But, should it be refused? 
Or, if it is accepted on Linux - and works when the (symbolic) file 
exists - must it be refused on AIX? If refusing it on AIX is what it 
takes to make the patch acceptable - then code it to refuse it is.
>
> Usually people just upload a new version of the original patch (original 
> changes, plus new changes, in one file). I think it would be too confusing 
> looking at differences between two patches.
So, just provide the new delta between "2.7.11(.0) and what I have, not 
the changes between patches. Coming up tomorrow.

In short, my head was spinning and I thought I saw problems that are not 
meant to exist, in particular find_library("libFOO.so") - that should be 
coded to reject I guess. I would still like to let something like 
"libc.so" be creative, as libc.so will never exist, by default, on AIX. 
My guess is that is a legacy thing from AIX4. (and I think the default 
name is shr.o when using the ld flag -G to create a shared object aka 
shared library)
>
> --
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing 

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Strictly speaking, Python 2.7 never had special support for RTLD_MEMBER or 
find_library(). That is why I am unsure about many of these changes being done 
in 2.7. They seem more like new features than bug fixes. Especially once you 
start talking about a smart CDLL() that tests for a file and then falls back to 
RTLD_MEMBER.

Whether you represent an archive member as an (archive, member) tuple, or as an 
"archive(member)" string, or something else, it is still an extension of the 
API of CDLL(), and arguably also of find_library().

I think I have already tried to explain that on Linux, find_library("libm.so") 
is far from normal usage (although may be technically legal if there was a 
library called liblibm.so.so).

Also, on Linux I wouldn’t recommend CDLL("libcrypt.so"), which relies on a file 
only intended for build time. On Debian, the glibc library 
 has separate packages for 
shared libraries (libc6) and development libraries (libc6-dev). It should be 
possible to run programs with libc6 installed, but without libc6-dev. Looking 
at the file lists for “i386” architecture, libc6-dev includes:

/usr/lib/i386-linux-gnu/libcrypt.a
/usr/lib/i386-linux-gnu/libcrypt.so

It symbolically links to /lib/i386-linux-gnu/libcrypt.so.1, which is provided 
by libc6:

/lib/i386-linux-gnu/libcrypt-2.19.so
/lib/i386-linux-gnu/libcrypt.so.1

So on Debian (and with Linux in general), CDLL("libcrypt.so") may work in some 
cases, but you should normally use CDLL("libcrypt.so.1") instead.

Usually people just upload a new version of the original patch (original 
changes, plus new changes, in one file). I think it would be too confusing 
looking at differences between two patches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

A bug was found, and has been corrected - but I am unclear on how to best 
submit the differences.

Editing the current "review" code is not a good idea, as there are many 
differences.

Please recommend correct course of action.

p.s. I will have also removed the imports for _ctypes and the use of LIBPATH

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

See below between points.

On 12-May-16 17:45, Martin Panter wrote:
> Martin Panter added the comment:
>
> Thanks for trying to move the directory in the new patch, but it does not 
> seem to have been picked up by Rietveld. I have only seen patches generated 
> with Mercurial and Git work, but I don’t exactly know _why_ they work :)
>
> Here is a summary of what I would be comfortable adding to Python:
>
> 1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)
Understood. Should also add RTLD_NOW as it is referenced in comments, 
and used in the _global.c (imho).
> 2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases 
> like find_library("libintl.so") should fail. I think your earlier patches 
> were closer to this implementation than the later ones.
>
> I am a bit hesitant about the automatic behaviour of 
> CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be 
> better to let the caller specify RTLD_MEMBER explicitly. If a shared library 
> file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, 
> i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", 
> RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to 
> reduce the domain of a low-level API.
>
> I understand it would be good to have the return value of find_library() 
> consistent with the name accepted by CDLL(). Perhaps a new parameter format 
> would help, such as a tuple (archive, member).
I would expect a new tuple would be seen as new API (like 
_ctypes.RTLD_MEMBER), so it would never help in Python-2.7, which is 
still where I see most people asking for support with AIX.
My guidance was: find_library("m") is legal, as is 
find_library("libm.so"), as is CDLL("libcrypt.so").

These are all dependencies, at a low-level of how the system is 
configured, i.e., it is not uncommon for a GNU/Linux installation to 
have many symbolic links to resolve multiple names, and programmers use 
one, or more of those conventions. The find_library() and CDLL() calls 
work because of these conventions.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 19 02:53:48 2016 from xxx.yyy.zzz.net 

root@x066:~# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0


libcrypt.so.1

Looking at filesystem level:
root@x066:~# cd /lib64
root@x066:/lib64# ls -l libcrypt.*
lrwxrwxrwx 1 root root 16 Feb 22  2015 libcrypt.so.1 -> libcrypt-2.13.so 

root@x066:/lib64# cd /usr/lib64
root@x066:/usr/lib64# ls -l libcrypt.*
-rw-r--r-- 1 root root 62108 Feb 22  2015 libcrypt.a
lrwxrwxrwx 1 root root20 Feb 22  2015 libcrypt.so -> 
/lib64/libcrypt.so.1

So, CDLL.dlopen() finds /usr/lib64/libcrypt.so that is a symbolic link 
to /lib64/libcrypt.so.1 that is a symbolic link to 
/lib64/libcrypt-2.13.so . I see libcrypt.so as 
a "shorthand" for whatever is installed. As a porgrammer/user of 
CDLL.dlopen() I do not care what it finally resolves to.

Linux has it's conventions: one important one I observe when porting OSS 
packages is that "installation" and updates of libraries (or API's) 
provide "shortnames" in the form of libFOO.so  that are symbolic names 
to the final object. Unfortunately, there are exceptions - especially 
look at "libc.so"

root@x066:/usr/lib64# ls -l libc.so*
-rw-r--r-- 1 root root 243 Feb 22  2015 libc.so

root@x066:/usr/lib64# cd /lib64
root@x066:/lib64# ls -l libc.so*
lrwxrwxrwx 1 root root 12 Feb 22  2015 libc.so.6 -> libc-2.13.so
root@x066:/lib64# ls -l libc-2.13.so
-rwxr-xr-x 1 root root 1758016 Feb 22  2015 libc-2.13.so

Obviously, using shorthand for libc.so on Debian is not returning 
anything equivalent to libc.so.6 (look at the different sizes). So, a 
program that looks specifically for libc.so.6 will succeed on Debian, 
and fail on AIX (as it does not exist in that form) - and my expectation 
is that existing programs that code libc.so.6 will fail on AIX. However, 
I have extreme doubts that anyone actually codes the test - as (at least 
on AIX) libc is already loaded.

AIX:
root@x071:[/root]ldd /opt/bin/python
/opt/bin/python needs:
  /usr/lib/libc.a(shr.o)
  /usr/lib/libpthreads.a(shr_xpg5.o)
  /usr/lib/libpthreads.a(shr_comm.o)
  /usr/lib/libdl.a(shr.o)
  /unix
  /usr/lib/libcrypt.a(shr.o)

And, it seems, libc is also already loaded on Debian:
root@x066:/lib64# ldd /usr/bin/python
 linux-vdso32.so.1 =>  (0x0010)
 libpthread.so.0 => /lib/powerpc-linux-gnu/libpthread.so.0 
(0x0ffc5000)
 libdl.so.2 => /lib/powerpc-linux-gnu/libdl.so.2 (0x0ffa1000)
 libutil.so.1 => /lib/powerpc-linux-gnu/libutil.so.1 (0x0ff7e000)
 libz.so.1 => /lib/powerpc-linux-gnu/libz.so.1 (0x0ff47000)
 libm.so.6 => /lib/powerpc-linux-gnu/libm.so.6 (0x0fe79000)
 

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Thanks for trying to move the directory in the new patch, but it does not seem 
to have been picked up by Rietveld. I have only seen patches generated with 
Mercurial and Git work, but I don’t exactly know _why_ they work :)

Here is a summary of what I would be comfortable adding to Python:

1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)

2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases 
like find_library("libintl.so") should fail. I think your earlier patches were 
closer to this implementation than the later ones.

I am a bit hesitant about the automatic behaviour of 
CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be better 
to let the caller specify RTLD_MEMBER explicitly. If a shared library file 
literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, i.e. not 
inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW) 
succeed? I admit this is an unlikely scenario, but it seems bad to reduce the 
domain of a low-level API.

I understand it would be good to have the return value of find_library() 
consistent with the name accepted by CDLL(). Perhaps a new parameter format 
would help, such as a tuple (archive, member).

I am not comfortable with other aspects. I think I would have to see more 
discussion with other people to change my opinion:

1. CDLL("libintl.so") should not load “libintl.a(libintl.so.1)”. I understand 
you want this to help code written for Gnu or Linux automatically work on AIX, 
but it doesn’t feel correct and robust to me. Perhaps moving this sort of thing 
to a separate function or package would be better.

2. find_library("libintl.so") -> "libintl.a(libintl.so.1)". I would expect it 
to look for a shared library installed in something like 
"/usr/lib/liblibintl.so.a", unless I have misunderstood how compile-time 
linking (cc -llibintl.so) works.

3. find_library() should not set the LIBPATH environment variable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

used diff -rNu Python-2.7.11 Python-2.7.11.4 to generate

Note: content same as those from yesterday - except 
Lib/ctype/test/test_loading.py is no longer changed. No longer needed.

--
Added file: http://bugs.python.org/file42812/Python2.issue26439.160511.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

used diff -rNu Python-3.5.1 Python-3.5.1.1 to generate

--
Added file: http://bugs.python.org/file42811/Python3.issue26439.160511.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

re:

AIX: CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | RTLD_MEMBER)

Officially it would be
dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW | RTLD_MEMBER)

Further, that would only load the 32-bit version, as there is a legacy naming 
scheme for the 64-bit members (the same name may actually occur twice - once 
for each size)

Example: the ar command showing all members regardless of size:
root@x064:[/]ar -Xany tv /usr/lib/libcrypto.a
rwxr-xr-x 0/0 2967588 Jul 24 13:46 2015 libcrypto.so
rwxr-xr-x 0/0 2256131 Jul 24 13:43 2015 libcrypto.so.0.9.8
rwxr-xr-x 0/0 2967588 Jul 24 13:45 2015 libcrypto.so.1.0.0
rwxr-xr-x 0/0 3376521 Jul 24 13:46 2015 libcrypto64.so
rwxr-xr-x 0/0 2606185 Jul 24 13:44 2015 libcrypto64.so.0.9.8
rwxr-xr-x 0/0 3376521 Jul 24 13:45 2015 libcrypto64.so.1.0.0

Note: the default archives (ending in .so) are "same size, checksum, etc" as 
the .so.1.0.0 versioned members.

Also, this can be more complex - e.g., if LibreSSL is also included (I am 
re-working my packaging so that I will also have both 32 and 64-bit packaging, 
for now only 32-bit is available)

michael@x071:[/home/michael]ar -X32 tv /opt/lib/libcrypto.a
rwxr-xr-x 0/0 3060762 Jun 17 21:17 2015 libcrypto.so.33
rwxrwxr-x 0/0 2965597 Jul 25 16:57 2015 libcrypto.so
rwxrwxr-x 0/0 2253850 Jul 25 17:03 2015 libcrypto.so.0.9.8
rwxrwxr-x 0/0 2965597 Jul 25 17:03 2015 libcrypto.so.1.0.0

IMHO: this is a needless complication for a programmer using ctypes.cdll - but 
who am I?

>From the dlopen man page (aka InfoCenter)

Flags: (aka mode)
Specifies variations of the behavior of dlopen. Either RTLD_NOW or RTLD_LAZY 
must always be specified. Other flags may be OR'ed with RTLD_NOW or RTLD_LAZY.

RTLD_NOWLoad all dependents of the module being loaded and resolve all 
symbols.

RTLD_LAZY   Specifies the same behavior as RTLD_NOW. In a future release of 
the operating system, the behavior of the RTLD_LAZY may change so that loading 
of dependent modules is deferred of resolution of some symbols is deferred.

RTLD_MEMBER The dlopen subroutine can be used to load a module that is a 
member of an archive. The L_LOADMEMBER flag is used when the load subroutine is 
called. The module name FilePath names the archive and archive member according 
to the rules outlined in the load subroutine.

To be complete...

RTLD_GLOBAL Allows symbols in the module being loaded to be visible when 
resolving symbols used by other dlopen calls. These symbols will also be 
visible when the main application is opened with dlopen(NULL, mode).

RTLD_LOCAL  Prevent symbols in the module being loaded from being used when 
resolving symbols used by other dlopen calls. Symbols in the module being 
loaded can only be accessed by calling dlsym subroutine. 

If neither RTLD_GLOBAL nor RTLD_LOCAL is specified, the default is RTLD_LOCAL. 
If both flags are specified, RTLD_LOCAL is ignored.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

a) https://bugs.python.org/review/26439/#msg12, but getting HTML 500 error)

The call to find_library here is for "ease of use" with existing code who use, 
e.g., cdll("libcrypto.so"). This format fails unless someone has previously 
done, e.g.: (cd /usr/lib; ar -X32 x libcrypto.a; mkdir -p /usr/lib64; cd 
/usr/lib64; ar -X64 x ../lib/libcrypto.a)

What I do think would be a valid addition here is to look for a '/' in the name 
anywhere (i.e., path info in name, and skip aix.find_library(). IMHO - if a 
programmer is usiong a relative path - he/she should take full responsilibilty 
(and this is basically what find_library will return anyway)

However, If there is a syntax issue I can rework the aixutil.py so there are 
multiple entry points - e.g., find_library() finds the .a file, and 
find_member() finds an archive containing a member (and returns a suitable 
base(member) argument for dlopen() and/or add an (undocumented) argument to 
modify find_library() return value.

But I think that only creates complexity for the user/programmer - additional 
complexity is an additional for an error (of omission) to occur.

Here, as it is now, if find_library() was used to find the name, find_library() 
is not called again. Again, if there is a behavior you want to force - then it 
can be made stricter - but compatibility with existing programs (I have been 
looking at cloud-init and salt) would be weak/non-existent. And, if 
...startswith("aix") will be a very common addition to code.
Difficult to use is a reason programmers avoid a platform - and I would not 
like to see my favorite platform avoided because they had to add lots of 
"startswith("aix") - as one would always be forgotten, and then the attitude 
becomes - not going to bother with python on AIX. - my two bits.

p.s. Notice in the patches from yesterday - I have added the 'export' of 
RTLD_NOW and RTLD_MEMBER to Modules/_ctype/_ctype.c.
The "export" only occurs if the variable is defined, and the import only occurs 
in __init__.py behind 'startswith("aix")' blocks.

b) Yes, I can load and use GNU diff and resubmit (you can ignore them if you 
prefer - and I shall move the src/Python* to parallel with the build 
directories.

FYI: I tried to pip install Mercurial - but got an error message from an 
include file so the last bit did not compile. Problem for a later date - could 
be a user error on my part, or an AIX version dependency.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Patches: I’m not sure, but maybe it would help the Reitveld review system if 
the src/ directory in them was eliminated (e.g. src/Python-3.5.1 vs just 
Python-3.5.1). Also, if you are using Gnu diff, maybe you can try the -N 
(--new-file) option to include aixutil.py in the diff, and -x (--exclude) 
__pycache__ to avoid that directory.

Personally, I would find patches a lot easier to handle than a tar file, 
especially if I cannot figure out which bits of the tar file are original code 
and which are changed code.

It seems you want to CDLL() to accept names using the Linux/BSD libFOO.so.N 
convention, but load an AIX library with a related name. This sounds like a bad 
design to me, for multiple reasons. I would like to hear other people’s 
opinions before going in that direction.

This is my understanding of the CDLL() calls currently needed to load the Open 
SSL library on various platforms:

openssl 1.0.2.g-3 package on Arch Linux: CDLL("libcrypto.so.1.0.0")
Windows: CDLL("libeay32.dll")
OS X (Darwin): CDLL("libcrypto.1.0.0.dylib")
FreeBSD: CDLL("libcrypto.so.8")
AIX: CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | RTLD_MEMBER)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

On 5/8/2016 8:29 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Your new patch calls find_library() internally in CDLL(); why?
Because arguments that work for GNU (aka Linux, even though internally 
it is called "posix") will not work for AIX.
> My understanding is CDLL() is a fairly lightweight wrapper around the 
> dlopen() call.
For AIX, the dlopen() call is a front-end for the function 
initAndLoad(). I have not looked in depth at Module/_ctypes (only 
recently found it) and I do not know how much work is being done there 
for the different platforms.
> On Linux, you either pass a full library file name, or an SO-name.
Same for AIX: a full-path library name, relative-path library name, or a 
library name and the library search path (with optional additional 
directories provided via environment variables (LIBPATH or 
LD_LIBRARY_PATH - when it is a FILE, or an archive(member_name) (with 
archive part being full or relative, or stem) and RTLD_MEMBER ORed into 
the mode argument.
> Both these strings can be discovered for compiled objects using e.g.:
Likewise for AIX: - example: an executable - notice how many are named 
shr.o: how is cdll("shr.o") suppossed to know it is libc, or is it libcrypt?

root@x064:[/usr/lib]ldd /usr/sbin/sshd
/usr/sbin/sshd needs:
  /usr/lib/libc.a(shr.o)
  /usr/lib/libcrypto.a(libcrypto.so.1.0.0)
  /usr/lib/libz.a(libz.so.1)
  /usr/lib/libpam.a(shr.o)
  /usr/lib/libdl.a(shr.o)
  /unix
  /usr/lib/libcrypt.a(shr.o)
  /usr/lib/libpthreads.a(shr_xpg5.o)
  /usr/lib/libpthreads.a(shr_comm.o)

Or a file:
./perl-5.14.4/lib/5.14.4/aix-thread-multi-64int/auto/B/B.so needs:
  /usr/lib/libpthreads.a(shr_xpg5.o)
  /usr/lib/libc.a(shr.o)
  /unix
  /usr/lib/libpthreads.a(shr_comm.o)
  /usr/lib/libcrypt.a(shr.o)

Again, cdll("shr.o") is not going to find the correct archive.
> $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
>   linux-vdso.so.1 (0x7fff567fe000)
>   libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x7f598474c000)
>   libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x7f59842d4000)
>   . . .
>
> So in Python, the SO-name or full path can be used, but not the compile-time 
> name, unless you first pass it through find_library():

 CDLL("libcrypto.so.1.0.0")  # soname
> 
 CDLL("/usr/lib/libcrypto.so.1.0.0")  # Full path
> 
 CDLL("crypto")  # Compile-time name
> Traceback (most recent call last):
>File "", line 1, in 
>File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
>  self._handle = _dlopen(self._name, mode)
> OSError: crypto: cannot open shared object file: No such file or directory
 find_library("crypto")  # Some people pass the result of this to CDLL()
> 'libcrypto.so.1.0.0'
Thank you for the explanation of what should be used when. Maybe I 
worked too hard for the current result: (note - 64-bit this time! and 
new member names, even for libc.so)

michael@x071:[/usr/lib]python -m ctypes.util
None
libc.a(shr_64.o)
None
aix.find_library("libiconv.so") libiconv.a(shr4_64.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None



libcrypt.a(shr_64.o)

Back to Why though.

Ease of use, and portability with existing convention. My experience 
with just two python packages I wanted to port is that cdll("libFOO.so") 
is what is used, and works on GNU. I wanted that to work ASIS. I also 
noticed that sometimes code was explicit about the version (sometimes 
the latest was not right, and the code complained). So, since "short" 
names are preferred, and are portable - those are what programmers tend 
to use. Full-path-names imply an assumption.

dlopen() is not going to know that libc.so is actually libc.a(shr.o) - 
so I call find_library() for portability aka Ease of Use. While it may 
be slightly less performance I am hoping (assuming) cdll("foo") is not 
the most central part of any application - and the overhead is acceptable.

And thanks for the question. It is simple enough to remove now, should 
it be considered a sin.

> --
>
> ___
> Python tracker 
> 
> ___

--
nosy: +aixto...@gmail.com

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


Added file: 
http://bugs.python.org/file42805/Python3.Modules._ctypes.160510.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


Added file: http://bugs.python.org/file42804/Python3.Lib.ctypes.160510.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


Added file: 
http://bugs.python.org/file42803/Python2.Modules._ctypes.160510.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


Added file: http://bugs.python.org/file42802/Python2.Lib.ctypes.160510.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Spent more time today, testing - on 3.5.1, 3.4.4 and 2.7.11.

The patches would not apply on both Python2 and Python3, but the new file 
Lib/ctypes/aixutil.py is identical on all three versions.

History of patching (starting from version 3.5.1(.1) is my patched version (as 
are Python-3.4.4.1 and Python-2.7.11.4)

michael@x071:[/data/prj/aixtools/python/python-3.4.4.1]for i in 
../*patches/*patch^Jdo^Jcat $i | patch -p1 ^Jdone
patching file Lib/ctypes/__init__.py
Hunk #1 succeeded at 341 (offset 4 lines).
patching file Lib/ctypes/test/test_loading.py
patching file Lib/ctypes/util.py
Hunk #1 succeeded at 70 (offset -6 lines).
Hunk #2 succeeded at 256 (offset -6 lines).
Hunk #3 succeeded at 275 (offset -6 lines).
patching file Modules/_ctypes/_ctypes.c
Hunk #1 succeeded at 5508 (offset -1 lines).
michael@x071:[/data/prj/aixtools/python/python-3.4.4.1]

But, not with 2.7.11

michael@x071:[/data/prj/aixtools/python/python-2.7.11.4]for i in 
../*patches/*patch^Jdo^Jcat $i | patch -p1 --dry-run^Jdone
patching file Lib/ctypes/__init__.py
Hunk #1 succeeded at 355 (offset 18 lines).
patching file Lib/ctypes/test/test_loading.py
Hunk #1 succeeded at 38 (offset -5 lines).
patching file Lib/ctypes/util.py
Hunk #1 succeeded at 71 (offset -5 lines).
Hunk #2 FAILED at 262.
Hunk #3 FAILED at 274.
2 out of 3 hunks FAILED -- saving rejects to file Lib/ctypes/util.py.rej
patching file Modules/_ctypes/_ctypes.c
Hunk #1 FAILED at 5509.
1 out of 1 hunk FAILED -- saving rejects to file Modules/_ctypes/_ctypes.c.rej

So, I made new patches against 2.7.11 after patching manually.

After the builds:

==
aixtools.python:aixtools.python.man.en_US:2.7.11.4::I:T:N:man pages0::
aixtools.python:aixtools.python.rte:2.7.11.4::I:T:N:built 10-May-2016 1810 
UTC0::
==
root@x064:[/data/prj/aixtools/python/python-2.7.11.4]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-2.7.11.4/Lib/ctypes]../../python 
util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
('aix.find_library("libiconv.so")', 'libiconv.a(shr4.o)')
('aix.find_library("libintl.so")', 'libintl.a(libintl.so.1)')
('aix.find_library("libintl.so.1")', 'libintl.a(libintl.so.1)')
('aix.find_library("libintl.so.2")', None, ':: should be None!')
('aix.find_library("libintl.so.8")', None)



libcrypt.a(shr.o)
root@x064:[/data/prj/aixtools/python/python-2.7.11.4/Lib/ctypes]../../python 
test/test_loading.py
libc_name is libc.a(shr.o)
ss..sss
--
Ran 7 tests in 0.189s

OK (skipped=5)

==
aixtools.python:aixtools.python.man.en_US:3.4.4.1::I:T:N:man pages0::
aixtools.python:aixtools.python.rte:3.4.4.1::I:T:N:built 10-May-2016 1812 
UTC0::
==
root@x064:[/data/prj/aixtools/python/python-3.4.4.1]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-3.4.4.1/Lib/ctypes]../../python 
util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
aix.find_library("libiconv.so") libiconv.a(shr4.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None



libcrypt.a(shr.o)
root@x064:[/data/prj/aixtools/python/python-3.4.4.1/Lib/ctypes]../../python 
test/test_loading.py
libc_name is libc.a(shr.o)
ss..sss
--
Ran 7 tests in 0.323s

OK (skipped=5)

root@x064:[/data/prj/aixtools/python/python-3.5.1.1]installp -d . -L
aixtools.python:aixtools.python.man.en_US:3.5.1.0::I:T:N:man pages0::
aixtools.python:aixtools.python.man.en_US:3.5.1.1::I:T:N:man pages0::
aixtools.python:aixtools.python.rte:3.5.1.0::I:T:N:built 03-Mar-2016 0856 
UTC0::
aixtools.python:aixtools.python.rte:3.5.1.1::I:T:N:built 10-May-2016 1620 
UTC0::

And, with a 64-bit build

root@x064:[/data/prj/aixtools/python/python-3.5.1.1]installp -d . -L
aixtools.python:aixtools.python.man.en_US:3.5.1.0::I:T:N:man pages0::
aixtools.python:aixtools.python.man.en_US:3.5.1.1::I:T:N:man pages0::
aixtools.python:aixtools.python.rte:3.5.1.0::I:T:N:built 03-Mar-2016 0856 
UTC0::
aixtools.python:aixtools.python.rte:3.5.1.1::I:T:N:built 10-May-2016 1620 
UTC0::
root@x064:[/data/prj/aixtools/python/python-3.5.1.1]cd lib/ctypes
ksh: lib/ctypes:  not found.
root@x064:[/data/prj/aixtools/python/python-3.5.1.1]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-3.5.1.1/Lib/ctypes]../../python 
util.py
None
libc.a(shr_64.o)
None
aix.find_library("libiconv.so") libiconv.a(shr4_64.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None



libcrypt.a(shr_64.o)

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

New version of aixutil.py

++ Changes ++
* more comments
* re-worked the 'searches' for matches after adding changing the way
the output from _get_dumpH was read (now to an array using readlines).
Also, p.stdout.close() and p.wait() are done within the _get_dumpH routine.

Suggestions on how to have cleaner workings with the 'arrays of text' and 
searching through them is welcomed! It is passing the internal tests is all I 
will say for now.

--
versions: +Python 3.5
Added file: http://bugs.python.org/file42790/aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

New patch of the Lib/ctypes directory - BUT - this time as a delta based on 
3.5.0.

++ changes from last patch ++
* OSError gets raised (as expected) by test/test_loading.py
* test in util.py modified (libm is replaced by libc when "aix"), and
added an additional test for AIX (showing it finds the latest version
of libintl that is installed (frequently, there are two versions)
* moved RTLD_NOW and RTLD_MEMBER definitions to the "aix" blocks in __init__.py
* in test/test_loading.py added two more libraries to look for

--
Added file: http://bugs.python.org/file42789/python.Lib.ctypes.160509.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Your new patch calls find_library() internally in CDLL(); why? My understanding 
is CDLL() is a fairly lightweight wrapper around the dlopen() call. On Linux, 
you either pass a full library file name, or an SO-name. Both these strings can 
be discovered for compiled objects using e.g.:

$ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
linux-vdso.so.1 (0x7fff567fe000)
libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x7f598474c000)
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x7f59842d4000)
. . .

So in Python, the SO-name or full path can be used, but not the compile-time 
name, unless you first pass it through find_library():

>>> CDLL("libcrypto.so.1.0.0")  # soname

>>> CDLL("/usr/lib/libcrypto.so.1.0.0")  # Full path

>>> CDLL("crypto")  # Compile-time name
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: crypto: cannot open shared object file: No such file or directory
>>> find_library("crypto")  # Some people pass the result of this to CDLL()
'libcrypto.so.1.0.0'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

'''
call find_library("foo")
libc:  libc.a
libc.a:  libc.a
libc.so.6:  libc.a
libcrypto.so:  libcrypto.a
'''

The above don’t seem right to me, unless compiling with “cc -llibc.so.6” etc 
works on AIX.

'''
call cdll.LoadLibrary("foo")
m:  
libm.so:  
'''

These doesn’t look right. What happened to the library name?

With your new aixutil.py file, it might be good to give it an underscore (_) 
prefix, to indicate it is an internal module rather than part of the ctypes 
API. So your code would do

import ctypes._aixutil as aix

--
Added file: http://bugs.python.org/file42776/python.Lib.ctypes.160504.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

implements ctypes.aixutil.find_library()
In a separate file as both __init__.py as util.py needs it's logic.

--
Added file: http://bugs.python.org/file42713/aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

reworked patch. To assist port to Python3 that changes in __init__.py and 
util.py are minimal. There is a new file: aixutil.py

I have only tested on Python-2.7, so there may be issues for Python3. My goal 
is to have a single file for both versions.

The main change in util.py is lots of specific (behavior) tests that I have 
encountered (and initially failed) from projects using ctypes.cdll.

I feel confident that this will work "ASIS" with nearly all existing projects.

Note: normal return is not a full path name. A full path name is only returned 
when an archive + member could not be found, but a file with the name requested 
could be found. This "fullpath" is to be compatible with existing code 
'demanding' unpacked archives.

--
Added file: http://bugs.python.org/file42712/python.Lib.ctypes.160504.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Question - before I submit a patch.

A. Is there a PEP I should read re: ctypes/util and/or ctypes/cdll?

B. I show two different behaviors of responding - My question is, what does the 
community think should be the response? My preference is to bring the request 
back to it's stub - (strip lib from the beginning, and .so* or .a from the 
suffix, so that it is in it's -lFOO form - as if being offered to a linker 
(with -lFOO, we do not use -lFOO.so.6 (e.g., for libc.so.6).
I have seen a lot of variance in how cdll is used (in only two python based 
projects I am currently interested in porting). Calls are frequently made using 
some variation including .so somewhere in the name and/or code is basing 
decesions based on a the number (###) returned by find_library (e.g., 
libFOO.so.###, and ### MUST be one of a list - and is rejected if not a match 
(even if newer)). I would think if there is a version dependancy this should be 
a call to the library loaded rather than an external string.
In any case, - for one example - as libc.so.* will never exist, by default, on 
AIX, for ease of use I believe that translating libFOO.* to FOO and then doing 
the 'find' comes closest to what was intended (long long ago) - again, is there 
a PEP with guidance on this AND my preference, for AIX, is to look in .a 
archives first for a .so member, and then look for an AIX legacy member 
(shr.*.o) and then look for an external .so.* file.

With my current working version - this is the output of the test in util.py:

root@x064:[/data/prj/aixtools/python/python-2.7.11.3/Lib/ctypes]../../python 
./util.py
libm.a
libc.a
libbz2.a


libcrypt.a

Additional Tests for AIX
call find_library("foo")
c:  libc.a
c.a:  None
c.so:  None
libc:  libc.a
libc.a:  libc.a
libc.so.6:  libc.a
crypt:  libcrypt.a
crypto:  libcrypto.a
crypto.so:  None
libcrypto.so:  libcrypto.a

call cdll.LoadLibrary("foo")
m:  
libm.so:  
c:  
c.a:  
libc.a:  
libc.so.6:  
libc.so.9:  
bz2:  
libbz2:  
crypt:  
crypto:  
crypto.so:  
libcrypto.so:  
iconv:  
intl:  

The test looks like:

# test code

def test():
from ctypes import cdll
if os.name == "nt":
print cdll.msvcrt
print cdll.load("msvcrt")
print find_library("msvcrt")

if os.name == "posix":
# find and load_version
print find_library("m")
print find_library("c")
print find_library("bz2")

# getattr
##print cdll.m
##print cdll.bz2

# load
if sys.platform == "darwin":
print cdll.LoadLibrary("libm.dylib")
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
else:
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so")
print find_library("crypt")

if sys.platform.startswith("aix"):
print "\nAdditional Tests for AIX"
print "call find_library(\"foo\")"
print "c: ", find_library("c")
print "c.a: ", find_library("c.a")
print "c.so: ", find_library("c.so")
print "libc: ", find_library("libc")
print "libc.a: ", find_library("libc.a")
print "libc.so.6: ", find_library("libc.so.6")
print "crypt: ", find_library("crypt")
print "crypto: ", find_library("crypto")
print "crypto.so: ", find_library("crypto.so")
print "libcrypto.so: ", find_library("libcrypto.so")
###
print "\ncall cdll.LoadLibrary(\"foo\")"
print "m: ", cdll.LoadLibrary("m")
print "libm.so: ", cdll.LoadLibrary("libm.so")
print "c: ", cdll.LoadLibrary("c")
print "c.a: ", cdll.LoadLibrary("c.a")
print "libc.a: ", cdll.LoadLibrary("libc.a")
print "libc.so.6: ", cdll.LoadLibrary("libc.so.6")
print "libc.so.9: ", cdll.LoadLibrary("libc.so.9")
print "bz2: ", cdll.LoadLibrary("bz2")
print "libbz2: ", cdll.LoadLibrary("libbz2")
print "crypt: ", cdll.LoadLibrary("crypt")
print "crypto: ", cdll.LoadLibrary("crypto")
print "crypto.so: ", cdll.LoadLibrary("crypto.so")
print "libcrypto.so: ", cdll.LoadLibrary("libcrypto.so")
print "iconv: ", cdll.LoadLibrary("iconv")
print "intl: ", cdll.LoadLibrary("intl")

if __name__ == "__main__":
test()

Thank you for your considered comments. My goal is ease of use for (porting) 
existing projects to AIX, i.e., ideally without code change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

On 4/28/2016 11:56 PM, Michael Felt wrote:
> Michael Felt added the comment:
>
> I have not looked specifically, at least not that I remember, for differences 
> in util/ctypes in python2 and python3. Will do so tomorrow.
>
> I did just look briefly at the library, rather archive, built by default as 
> libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., 
> not using gcc) does not build a shared object, so cdll.LoadLibrary and/or 
> find_library will not find anything for python2.7.. Neither will m, or libm, 
> on a default AIX system (with no other gcc based packages installed - these 
> also install a gnu rte where the utilities and libs you mention might include.
>
> The few python packages I have found, packaged by others, tend to reload 
> everything yet again, not depending on anything that may already be there. 
> And to use shared libraries they are extracting the members from the .a 
> archives into two directories - when they support both 32 and 64-bit targets.
>
> My intent is to examine the program to discover where libraries should be and 
> find the member name that is most likely. Also, if LIBPATH is defined, those 
> directories are searched first for a match.
>
> In short, the key difference is to look at the program (probably python) for 
> the blibpath string in the application as well as python (from memory, sys.* 
> calls) to build a list of directories to search.
>
> findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o 
> members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..
>
> I need to check that findLibrary('foo.so') continues to work. At one time it 
> did, just have not looked at this for several weeks and I forget if it still 
> works. That is what I shall make sure stays in the "testing" part of the 
> patch.
>
> Michael
>
> --
>
> ___
> Python tracker 
> 
> ___

I am reworking the logic - as many use cdll.LoadLibrary without ever 
calling find_library, and then __init__.py breaks.

More asap.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

The obvious (but easy to fix) problem I forsee with Python 3 is the print() 
calls. If you use print("") and print(arg), that will work with both 2 and 3. 
There may be more complications with bytes vs text stdout if we change the 
os.popen() calls to use subprocess.Popen.

I didn’t mean to use libpython2.7 specifically. Substitute any shared library 
that is widely available across platforms; maybe “crypto” is a better example. 
CDLL(find_library("crypto")) loads "libcrypto.so.1.0.0" on my Linux computer. 
It looks like you got the equivalent working for AIX; I was just checking.

FWIW it looks like your parsing of sys.executable to find library search paths 
is similar to searching the runpath (or RPATH) on ELF files, as proposed in 
Issue 19317. And it seems AIX’s LIBPATH environment variable is similar to 
LD_LIBRARY_PATH on Linux, proposed to be searched in Issue 9998. Also, I 
understand the equivalent OS X environment variables 
DYLD_(FALLBACK)_LIBRARY_PATH are already used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

I have not looked specifically, at least not that I remember, for differences 
in util/ctypes in python2 and python3. Will do so tomorrow.

I did just look briefly at the library, rather archive, built by default as 
libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not 
using gcc) does not build a shared object, so cdll.LoadLibrary and/or 
find_library will not find anything for python2.7.. Neither will m, or libm, on 
a default AIX system (with no other gcc based packages installed - these also 
install a gnu rte where the utilities and libs you mention might include.

The few python packages I have found, packaged by others, tend to reload 
everything yet again, not depending on anything that may already be there. And 
to use shared libraries they are extracting the members from the .a archives 
into two directories - when they support both 32 and 64-bit targets.

My intent is to examine the program to discover where libraries should be and 
find the member name that is most likely. Also, if LIBPATH is defined, those 
directories are searched first for a match.

In short, the key difference is to look at the program (probably python) for 
the blibpath string in the application as well as python (from memory, sys.* 
calls) to build a list of directories to search.

findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o 
members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..

I need to check that findLibrary('foo.so') continues to work. At one time it 
did, just have not looked at this for several weeks and I forget if it still 
works. That is what I shall make sure stays in the "testing" part of the patch.

Michael

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

I don’t know anything about AIX specific stuff, but I left some general 
comments in the code review.

Is there any chance that AIX people would be relying on the current behaviour 
that I understand uses _findSoname_ldconfig() and _findLib_gcc()?

Is this new functionality covered by the test suite? E.g. in 
/Lib/ctypes/test/test_find.py, there are tests that call find_library() for GL, 
GLU, and gle. Are those libraries common on AIX?

As discussed in Issue 9998, it seems a lot of people use find_library() to help 
convert a build-time library name to a run-time shared library name that can be 
passed to CDLL() or LoadLibrary(). E.g. on Linux:

>>> find_library("python2.7")  # As used in cc . . . -lpython2.7
'libpython2.7.so.1.0'
>>> cdll.LoadLibrary("libpython2.7.so.1.0")


Does your patch support this kind of use case?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Changes by Martin Panter :


Added file: http://bugs.python.org/file42634/python.Lib.ctypes.160317.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Changes by Martin Panter :


Removed file: http://bugs.python.org/file42633/python.Lib.ctypes.160309.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

I notice the patch is against Python 2, and uses Python 2 specific syntax. I am 
unsure if this kind of change is acceptable for 2.7, or if it would have to 
only go into 3.6. Hopefully this version of the patch will notify the review 
system that it is against Python 2.

--
stage:  -> patch review
Added file: http://bugs.python.org/file42633/python.Lib.ctypes.160309.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[David Edelsohn]

David Edelsohn added the comment:

The most recent patch seems to follow AIX semantics correctly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

These are very different issues. However, this patch may resolve both!

ldconfig (-p if I recall) lists where (shared) libraries have been installed 
(imho, this is a GNU tool approach) - whereas AIX would use dump -H to find 
library paths embedded in a program and/or shared library.

Until this patch, to use shared libraries on AIX the members of an archive 
needed to be extracted from the .a archive, and for 64-bit members, a separate 
directory (e.g. /usr/lib64) is needed. With this patch find_library() (actually 
cdll.LoadLibrary() can load members from either both .so and .a libraries, as 
is normal for AIX.

So, in a way, this would also solve https://bugs.python.org/issue21826 as 
ldconfig is no longer needed (nor called) on AIX.

p.s. As it is well longer than a month - I would appreciate that someone 
actually look at the patch and tell me how it can be improved! :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Martin Panter]

Martin Panter added the comment:

Maybe Issue 21826 is relevant (slowdown on AIX caused by missing ldconfig)

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Ah, good news - the build is successful ONCE I open the file 
./Lib/ctypes/util.py and find the (hiding) tab characters and replace them with 
8 space characters. (And I had tried so hard to check for that in advance).

So, if you apply the patch - it may need some love.

I would be grateful for people (even just one!) looking at it.

Michael

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

fyi: just completed a test both as 32 and 64 bit build.

However, openssl-1.0.1.514 does not work in 64-bit mode (packaging error). 
Fortunately, openssl-1.0.1.515 (released 02-March-2016) fixes that.

In short, 64-bit build is dependent on openssl-1.0.1.515

Here is the corrected patch (tabs removed). name: python.Lib.ctypes.160317.patch

--
Added file: http://bugs.python.org/file42193/python.Lib.ctypes.160317.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

The patch works when installed on top of pre-compiled version (e.g., copy the 
two files to /opt/lib/python2.7/ctypes/

but there seems to be a nasty side-effect when trying to build.

make completes, but make install fails during a compile step.

Please assist with howto see how/why ctypes/util is impacting this part.

And - for my first adventure into python please provide the needed (expected) 
feedback.

Michael

make install DESTDIR=/var/aixtools/aixtools/python/2.7.11.2 > 
.buildaix/install.out

Listing /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/__init__.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/_exceptions.py 
...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/expatreader.py 
...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/handler.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/saxutils.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/xmlreader.py 
...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmllib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmlrpclib.py 
...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/zipfile.py 
...

So, zipfile.py seems to exit non-zero. :(

--
keywords: +patch
Added file: http://bugs.python.org/file42094/python.Lib.ctypes.160309.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

The following demonstrates the basics.

cdll.LoadLibrary will load a .so file if it can be located (you will have to 
believe me as I forgot to include the test in this last batch)

Further, when given the AIX format for loading a member of an archive
(e.g., libc.a(shr.o) or libcrypto.a(libcrypto.so), the dlload occurs as 
expected.

in ctypes/utils.py
This looks in the standard directories (as the example, but not complete, see 
below) to demonstrate how to respond.
Note: it is also looking for a .so file, and if it finds one, it returns the 
.so name, otherwise it looks for a .a file - and when it finds it, just appends 
the libNAME.so in the correct format.

What is not done, and probably different from ldconfig behavior!

1. The complete path is not returned (so cdll.LoadLibrary will still follow 
default search - which I hope includes, read uses, the default for the python 
executable

2. The member name is not verified for existence (e.g., libconv.a(libiconv.so) 
is not expected to find libiconv.a(libiconv.so.2)

3. Since member verification is not being performed, shr.o, shr4.o, shr*_64.o

And many many thanks for the strip -e/-E info. Hard to find!

root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/__init__.py 
/opt/lib/python2.7/ctypes/__init__.py
--- python-2.7.11/Lib/ctypes/__init__.py2015-12-05 19:46:56 +
+++ /opt/lib/python2.7/ctypes/__init__.py   2016-02-29 17:36:43 +
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -356,6 +357,14 @@
 if use_last_error:
 flags |= _FUNCFLAG_USE_LASTERROR
 
+if _sys.platform.startswith("aix"):
+RTLD_NOW= 0x0002
+RTLD_MEMBER = 0x0004
+mode |= RTLD_NOW
+if name is not None:
+if name[-1] == ')':
+   mode |= RTLD_MEMBER
+
 class _FuncPtr(_CFuncPtr):
 _flags_ = flags
 _restype_ = self._func_restype_


root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/util.py 
/opt/lib/python2.7/ctypes/util.py
--- python-2.7.11/Lib/ctypes/util.py2015-12-05 19:46:56 +
+++ /opt/lib/python2.7/ctypes/util.py   2016-02-29 17:42:41 +
@@ -84,6 +84,23 @@
 continue
 return None
 
+if os.name == "posix" and sys.platform.startswith("aix"):
+def _findLib_aix(name):
+paths='/usr/lib:/lib:/usr/lpp/xlC/lib'
+for dir in paths.split(":"):
+shlib = os.path.join(dir, "lib%s.so" % name)
+if os.path.exists(shlib):
+return shlib
+libfile = os.path.join(dir, "lib%s.a" % name)
+if os.path.exists(libfile):
+shlib = "lib%s.a(lib%s.so)" % (name, name)
+return shlib
+
+return None
+
+def find_library(name):
+return _findLib_aix(name)
+
 elif os.name == "posix":
 # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
 import re, tempfile, errno
@@ -267,7 +284,16 @@
 print cdll.LoadLibrary("libcrypto.dylib")
 print cdll.LoadLibrary("libSystem.dylib")
 print cdll.LoadLibrary("System.framework/System")
+elif sys.platform.startswith("aix"):
+print find_library("crypto")
+x = find_library("crypto")
+print cdll.LoadLibrary(x)
+print cdll.LoadLibrary("libcrypto.a(libcrypto.so)")
+print cdll.LoadLibrary("libc.a(shr.o)")
+print find_library("crypt")
+print cdll.LoadLibrary("libcrypt.a(shr.o)")
 else:
+print cdll.LoadLibrary("libc.a")
 print cdll.LoadLibrary("libm.so")
 print cdll.LoadLibrary("libcrypt.so")
 print find_library("crypt")


p.s. Anyone know what flag to set in vi so that tabs are never inserted and/or 
spaces converted to tabs. Python seems to hate tab indents.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

Oops: forgot the example output:

root@x064:[/data/prj/aixtools/src]python -m ctypes.util
libm.a(libm.so)
libc.a(libc.so)
libbz2.a(libbz2.so)
libcrypto.a(libcrypto.so)



libcrypt.a(libcrypt.so)


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[David Edelsohn]

David Edelsohn added the comment:

AIX traditionally used member names like shr.o or shr.o or 
shr.o insider the archive, with _64 designating a 64 bit object 
when there is a naming collision.

GNU libtool defaults to the SO name and version number insider the archive.

AIX objects (and shared objects) contain a bit in the header that specifies 32 
bit or 64 bit.  Both 32 bit and 64 bit objects are intended to be archived 
together.  The linker only processes objects of the correct mode.

AIX shared objects contain a bit that specifies if the object may be used at 
link-edit time or only should be used for loading.  This is controlled by the 
AIX strip -e/-E option (yes, I know, strange place to hide that option).

This combination of features allows all of the libraries to be placed in a 
single /usr/lib directory and all of the objects to be collected into a single 
archive, avoiding /usr/lib64 and explosion of shared objects and symbolic links 
clutter.  Various packages have created /usr/local/lib64 anyway using 
Linux/Solaris/SVR4-style naming.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

it was, partly, about the technical details - but I feel I have found the key 
bits - /full/path/libNAME.a(libNAME.so) + dlflags RTLD_NOW + RTLD_MEMBER for 
"native" AIX support.

Additionally, a patch should not break what might be working for some (via 
google I read of suggestions for other languages, e.g., java, where .so members 
are extracted or installed side-by-side with the .a archive) - as the IBM 
run-rime loader also accepts/looks for both .a and .so. And/or for when 
/sbin/ldconfig is available. And when available, is this used as last case, or 
preferred?

Lastly, we cannot assume we will know the name of the member based on the name 
of the library. In many cases, e.g., libiconv.a the IBM names are shr.o, shr4.o 
and shr4_64.o - and these are the only member names unless GNU libiconv.a has 
been added (and then libconv.a also contains libiconv.so.2 (so version support 
is also desired!)) in both 32 and 64 bit mode.

One technical detail I have not been able to discover yet: how to determine 
whether in 32-bit or 64-bit mode. This will be important for libraries that 
have shr4.o and shr4_64.o as the members that need to be dlopened.

Lastly: about adding the "stanza": python is a new language for me. Any 
assistance with a patch - to keep it properly 'pythonized'

In short, I shall continue my studies/learning. Assistance from anyone wiser 
than me is welcome!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[David Edelsohn]

David Edelsohn added the comment:

ctypes util.py "simply" needs support for AIX.  There already is special 
support for Windows, Darwin, BSDs, Solaris.  Is the question about the 
technical details for equivalent functionality on AIX or about adding a stanza 
to Lib/ctpyes/util.py?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Ned Deily]

Changes by Ned Deily :


--
nosy: +David.Edelsohn

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

The _dlopen call in __init__.py I have been able to fix (hack) with the 
following:

root@x064:[/data/prj/aixtools/python/python-2.7.10/Lib/ctypes]diff -u 
__init__.py /opt/lib/python2.7/ctypes/__init__.py
--- __init__.py 2015-05-23 16:09:01 +
+++ /opt/lib/python2.7/ctypes/__init__.py   2016-02-26 08:40:19 +
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -32,6 +33,11 @@
 if int(_os.uname()[2].split('.')[0]) < 8:
 DEFAULT_MODE = RTLD_GLOBAL
 
+if _os.name == "posix" and _sys.platform.startswith("aix"):
+RTLD_NOW= 0x0002
+RTLD_MEMBER = 0x0004
+DEFAULT_MODE |= (RTLD_NOW | RTLD_MEMBER)
+
 from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \
  FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI, \
  FUNCFLAG_USE_ERRNO as _FUNCFLAG_USE_ERRNO, \

I have an additional hack in util.py so that, e.g.,
find_Library("crypto")

returns:
 '/usr/lib/libcrypto.a(libcrypto.so)'

When that is passed to dlopen (plus RTLD_MEMBER) the dlopen succeeds.



With some help in util.py, to do some sensible searching, being able to 
establish if in 32 or 64-bit mode, etc. somehing nice can be made here (imho).

p.s. can any verify whether this is limited to python 2.7? Or should python 3.X 
be added as well?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Changes by Michael Felt :


--
title: ctypes.util.find_library fails ALWAYS when gcc is not used -> 
ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com