Stuart Henderson writes: > I suggest using ${MODPY_EGG_VERSION} in setup.py and instead of cp, > use "${SUBST_CMD} -m $mode -c $file1 $file2"
>> RCS file: files/setup.py >> diff -N files/setup.py >> --- /dev/null 1 Jan 1970 00:00:00 -0000 >> +++ files/setup.py 5 Apr 2021 05:36:16 -0000 >> @@ -0,0 +1,195 @@ >> +#!/usr/bin/env python3 > > ${MODPY_BIN} here perhaps? Here is a fresh diff. It does the following: - Uses ${MODPY_EGG_VERSION} and SUBST_CMD in setup.py to only have to update version in one place (from sthen@) - Uses ${MODPY_BIN} in ${FILESDIR}/setup.py (from sthen@) - Removes -L{LOCALBASE}/lib in ${FILESDIR}/setup.py. Python bindings incorrectly linked -ltorrent-rasterbar with /usr/local/lib/libtorrent-rasterbar.so.3.0 instead of the freshly built ${WRKSRC}/src/.libs/libtorrent-rasterbar.so.4.0. To do this I modified ${FILESDIR}/setup.py: cfg_vars[key] = value.replace('-L${LOCALBASE}/lib/', '') mariani@ still OK? Looking for another OK in addition to commit. Feedback and tests are welcome. Details ======= Steps to reproduce: 1. pkg_add libtorrent-rasterbar (1.2.10p0) 2. /usr/local/lib/libtorrent-rasterbar.so.3.0 is picked up. 3. install this version 1.2.13 4. import libtorrent in python interpreter $ python3 Python 3.8.8 (default, Apr 28 2021, 15:17:34) [Clang 11.1.0 ] on openbsd6 Type "help", "copyright", "credits" or "license" for more information. >>> import libtorrent python3:/usr/local/lib/python3.8/site-packages/libtorrent.cpython-38.so: undefined symbol '_ZN10libtorrent14session_handle6pausedE' Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: Cannot load specified object Set a breakpoint in ${FILESDIR}/setup.py before ext = [Extension( import pdb; pdb.set_trace() Note "-L/usr/local/lib" in BLDSHARED and LDSHARED. (Pdb) print(cfg_vars) {'ABIFLAGS': '', 'AC_APPLE_UNIVERSAL_BUILD': 0, 'AIX_GENUINE_CPLUSPLUS': 0, 'ALT_SOABI': 0, 'ANDROID_API_LEVEL': 0, 'AR': 'ar', 'ARFLAGS': 'rcs', 'BASECFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code', 'BASECPPFLAGS': '', 'BASEMODLIBS': '', 'BINDIR': '/usr/local/bin', 'BINLIBDEST': '/usr/local/lib/python3.8', 'BLDLIBRARY': '-L. -lpython3.8', 'BLDSHARED': 'cc -pthread -shared -fPIC -L/usr/local/lib/ -L/usr/obj/ports/Python-3.8.8/Python-3.8.8 -L/usr/local/lib/', ... 'LDSHARED': 'cc -pthread -shared -fPIC -L/usr/local/lib/ -L/usr/obj/ports/Python-3.8.8/Python-3.8.8 -L/usr/local/lib/' This is the problematic line with the first two -L/usr/local/lib messing up the search path for -ltorrent-rasterbar. With the fix, these two go away: c++ -pthread -shared -fPIC -L/usr/local/lib/ -L/usr/obj/ports/Python-3.8.8/Python-3.8.8 -L/usr/local/lib/ -O2 -pipe bindings/python/src/alert.o ... -L../../src/.libs -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -ltorrent-rasterbar -lboost_system-mt -lpthread -lboost_system-mt -lboost_python38-mt -lpthread -lssl -lcrypto -o build/lib.openbsd-6.9-amd64-3.8/libtorrent.cpython-38.so -fvisibility=hidden -fvisibility-inlines-hidden /usr/local/lib/libiconv.so.7.0 > On 2021/04/24 01:47, Nam Nguyen wrote: >> "Elias M. Mariani" writes: >> >> > Tested build on amd64. >> > >> > OK mariani@ >> >> Ping for 1.2.13 update. The diff is pasted inline for convenience. I >> would like another OK in addition to mariani@'s, focusing on my proposal >> that carrying an old 1.2.11 ${FILESDIR}/setup.py is maintainable (with >> backup plan going back to b2 and new setup.py if it ever stops >> working). rsadowski@ gave an OK on an older 1.2.11 diff. >> >> > >> > Cheers. >> > >> > >> > On Mon, Apr 5, 2021 at 3:28 AM Nam Nguyen <n...@berkeley.edu> wrote: >> >> >> Here is a fresh diff with some tweaks. Need some additional feedback on >> >> carrying the old 1.2.11 ${FILESDIR}/setup.py. >> >> >> >> Please find a fresh diff that additionally: >> >> - remove autotools and use CONFIGURE_STYLE = gnu now that my pull >> >> request was accepted for 1.2.13. configure now correctly detects the >> >> default C++ standard. see: >> >> https://github.com/arvidn/libtorrent/pull/5026 >> >> - removes patches/patch-configure_ac >> >> >> >> as before: >> >> - major bump because check_sym reports removed symbols. one such removed >> >> symbol is parameter change for add_read_buffer: >> >> see: >> >> https://github.com/arvidn/libtorrent/commit/6522fc46f599c49f96d184498e2ce2e4d95ed0ea >> >> check_sym: https://namtsui.com/public/check_sym_libtorrent.txt >> >> - carry ${FILESDIR}/setup.py from 1.2.11 because 1.2.12 relies on >> >> boost-build (see justification inline). >> >> >> >> Carrying an old setup.py is relatively safe. I have a hacky WIP that >> >> actually successfully builds libtorrent-rasterbar using the new setup.py >> >> and boost-build (b2). There are unresolved issues: >> >> - I had to create /usr/local/lib/libtorrent-rasterbar.so because the new >> >> setup.py cannot cope with openbsd's versioning (e.g., >> >> libtorrent-rasterbar.so.4.0) >> >> - I needed to specify BOOST_ROOT as boost's ${WRKSRC}. This means that >> >> the boost source, including jamfiles, must be provided. >> >> - boost must provide b2 >> >> >> >> It is possible to use b2, but, for now, just carry the old >> >> setup.py. This can be fleshed out if setup.py ever stops working. >> >> >> >> I tested with deluge and qbittorrent. The test suite in a similar state >> >> with 2 skipped. Index: Makefile =================================================================== RCS file: /cvs/ports/net/libtorrent-rasterbar/Makefile,v retrieving revision 1.15 diff -u -p -u -p -r1.15 Makefile --- Makefile 23 Feb 2021 19:39:32 -0000 1.15 +++ Makefile 16 May 2021 09:23:46 -0000 @@ -2,11 +2,10 @@ COMMENT = C++ library implementing a BitTorrent client -MODPY_EGG_VERSION = 1.2.10 +MODPY_EGG_VERSION = 1.2.13 DISTNAME = libtorrent-rasterbar-${MODPY_EGG_VERSION} -REVISION = 0 -SHARED_LIBS += torrent-rasterbar 3.0 # 10.0.0 +SHARED_LIBS += torrent-rasterbar 4.0 # 10.0.0 CATEGORIES = net devel @@ -18,7 +17,7 @@ PERMIT_PACKAGE = Yes WANTLIB += ${COMPILER_LIBCXX} boost_python${MODPY_VERSION:C/\.//g}-mt WANTLIB += boost_system-mt crypto iconv m ssl -MASTER_SITES = https://github.com/arvidn/libtorrent/releases/download/libtorrent-${MODPY_EGG_VERSION}/ +MASTER_SITES = https://github.com/arvidn/libtorrent/releases/download/v${MODPY_EGG_VERSION}/ MODULES = lang/python @@ -30,10 +29,7 @@ LIB_DEPENDS = converters/libiconv \ # boost COMPILER = base-clang ports-gcc -CONFIGURE_STYLE = autoreconf - -AUTOCONF_VERSION = 2.69 -AUTOMAKE_VERSION = 1.16 +CONFIGURE_STYLE = gnu CONFIGURE_ARGS = --enable-python-binding \ --enable-tests \ @@ -53,6 +49,9 @@ CONFIGURE_ARGS += --enable-debug pre-configure: sed -i 's,-Os,,g' ${WRKSRC}/configure +# use setup.py from 1.2.11 because >=1.2.12 introduced dependency on boost-build + ${SUBST_CMD} -m 644 -c ${FILESDIR}/setup.py \ + ${WRKSRC}/bindings/python/setup.py pre-test: ln -sf ${MODPY_BIN} ${WRKDIR}/bin/python Index: distinfo =================================================================== RCS file: /cvs/ports/net/libtorrent-rasterbar/distinfo,v retrieving revision 1.8 diff -u -p -u -p -r1.8 distinfo --- distinfo 7 Sep 2020 04:24:17 -0000 1.8 +++ distinfo 16 May 2021 09:23:46 -0000 @@ -1,2 +1,2 @@ -SHA256 (libtorrent-rasterbar-1.2.10.tar.gz) = 0N0wvcOSZYfEJB9AaNjjliimwfn2z1MZXw6byQAXvvs= -SIZE (libtorrent-rasterbar-1.2.10.tar.gz) = 4128498 +SHA256 (libtorrent-rasterbar-1.2.13.tar.gz) = l20ncf/NVk8IpjNR6cIuhCqqjNKfb3/iXRacA4qEToU= +SIZE (libtorrent-rasterbar-1.2.13.tar.gz) = 4160116 Index: files/setup.py =================================================================== RCS file: files/setup.py diff -N files/setup.py --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ files/setup.py 16 May 2021 09:23:46 -0000 @@ -0,0 +1,196 @@ +#!${MODPY_BIN} + + +from distutils.core import setup, Extension +from distutils.sysconfig import get_config_vars +import os +import platform +import sys +import shutil +import multiprocessing + + +class flags_parser: + def __init__(self): + self.include_dirs = [] + self.library_dirs = [] + self.libraries = [] + + def parse(self, args): + """Parse out the -I -L -l directives + + Returns: + list: All other arguments + """ + ret = [] + for token in args.split(): + prefix = token[:2] + if prefix == '-I': + self.include_dirs.append(token[2:]) + elif prefix == '-L': + self.library_dirs.append(token[2:]) + elif prefix == '-l': + self.libraries.append(token[2:]) + else: + ret.append(token) + return ret + + +def arch(): + if platform.system() == 'Darwin': + __, __, machine = platform.mac_ver() + if machine.startswith('ppc'): + return ['-arch', machine] + return [] + + +def target_specific(): + if platform.system() == 'Darwin': + # On mavericks, clang will fail when unknown arguments are passed in. + # python distutils will pass in arguments it doesn't know about. + return ['-Wno-error=unused-command-line-argument-hard-error-in-future'] + return [] + + +try: + with open('compile_flags') as _file: + extra_cmd = _file.read() +except Exception: + extra_cmd = None + +try: + with open('link_flags') as _file: + ldflags = _file.read() +except Exception: + ldflags = None + +# this is to pull out compiler arguments from the CXX flags set up by the +# configure script. Specifically, the -std=c++11 flag is added to CXX and here +# we pull out everything starting from the first flag (i.e. something starting +# with a '-'). The actual command to call the compiler may be more than one +# word, for instance "ccache g++". +try: + with open('compile_cmd') as _file: + cmd = _file.read().split(' ') + while len(cmd) > 0 and not cmd[0].startswith('-'): + cmd = cmd[1:] + extra_cmd += ' '.join(cmd) +except Exception: + pass + +ext = None +packages = None + +if '--bjam' in sys.argv: + del sys.argv[sys.argv.index('--bjam')] + + if '--help' not in sys.argv \ + and '--help-commands' not in sys.argv: + + toolset = '' + file_ext = '.so' + + if platform.system() == 'Windows': + file_ext = '.pyd' + # See https://wiki.python.org/moin/WindowsCompilers for a table of msvc versions + # used for each python version + # Specify the full version number for 9.0 and 10.0 because apparently + # older versions of boost don't support only specifying the major number and + # there was only one version of msvc with those majors. + # Only specify the major for msvc-14 so that 14.1, 14.11, etc can be used. + # Hopefully people building with msvc-14 are using a new enough version of boost + # for this to work. + if sys.version_info[0:2] in ((2, 6), (2, 7), (3, 0), (3, 1), (3, 2)): + toolset = ' toolset=msvc-9.0' + elif sys.version_info[0:2] in ((3, 3), (3, 4)): + toolset = ' toolset=msvc-10.0' + elif sys.version_info[0:2] in ((3, 5), (3, 6)): + toolset = ' toolset=msvc-14' + else: + # unknown python version, lets hope the user has the right version of msvc configured + toolset = ' toolset=msvc' + + parallel_builds = ' -j%d' % multiprocessing.cpu_count() + if sys.maxsize > 2**32: + address_model = ' address-model=64' + else: + address_model = ' address-model=32' + + # add extra quoting around the path to prevent bjam from parsing it as a list + # if the path has spaces + os.environ['LIBTORRENT_PYTHON_INTERPRETER'] = '"' + sys.executable + '"' + + # build libtorrent using bjam and build the installer with distutils + cmdline = ('b2 libtorrent-link=static boost-link=static release ' + 'optimization=space stage_module --abbreviate-paths' + + address_model + toolset + parallel_builds) + print(cmdline) + if os.system(cmdline) != 0: + print('build failed') + sys.exit(1) + + try: + os.mkdir('build') + except Exception: + pass + try: + shutil.rmtree('build/lib') + except Exception: + pass + try: + os.mkdir('build/lib') + except Exception: + pass + try: + os.mkdir('libtorrent') + except Exception: + pass + shutil.copyfile('libtorrent' + file_ext, + 'build/lib/libtorrent' + file_ext) + + packages = ['libtorrent'] + +else: + # Remove '-Wstrict-prototypes' compiler option, which isn't valid for C++. + cfg_vars = get_config_vars() + for key, value in list(cfg_vars.items()): + if isinstance(value, str): + cfg_vars[key] = value.replace('-Wstrict-prototypes', '') + cfg_vars[key] = value.replace('-L${LOCALBASE}/lib/', '') + + src_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "src")) + source_list = [os.path.join(src_dir, s) for s in os.listdir(src_dir) if s.endswith(".cpp")] + + flags = flags_parser() + ext_extra = {} + + if ldflags: + # ldflags parsed first to ensure the correct library search path order + ext_extra["extra_link_args"] = flags.parse(ldflags) + arch() + + if extra_cmd: + ext_extra["extra_compile_args"] = flags.parse(extra_cmd) + arch() + target_specific() + + ext = [Extension( + 'libtorrent', + sources=sorted(source_list), + language='c++', + include_dirs=flags.include_dirs, + library_dirs=flags.library_dirs, + libraries=['torrent-rasterbar'] + flags.libraries, + **ext_extra) + ] + +setup( + name='python-libtorrent', + version='${MODPY_EGG_VERSION}', + author='Arvid Norberg', + author_email='ar...@libtorrent.org', + description='Python bindings for libtorrent-rasterbar', + long_description='Python bindings for libtorrent-rasterbar', + url='http://libtorrent.org', + platforms=[platform.system() + '-' + platform.machine()], + license='BSD', + packages=packages, + ext_modules=ext +) Index: patches/patch-configure_ac =================================================================== RCS file: patches/patch-configure_ac diff -N patches/patch-configure_ac --- patches/patch-configure_ac 4 Sep 2020 04:24:28 -0000 1.1 +++ /dev/null 1 Jan 1970 00:00:00 -0000 @@ -1,17 +0,0 @@ -$OpenBSD: patch-configure_ac,v 1.1 2020/09/04 04:24:28 rsadowski Exp $ - -use c++14 instead of c++11. see: -https://github.com/arvidn/libtorrent/pull/5026 - -Index: configure.ac ---- configure.ac.orig -+++ configure.ac -@@ -144,7 +144,7 @@ AS_ECHO "Checking for boost libraries:" - - AX_BOOST_BASE([1.58]) - --AX_CXX_COMPILE_STDCXX_11([noext], [mandatory]) -+AX_CXX_COMPILE_STDCXX(14, [noext], [mandatory]) - - AX_BOOST_SYSTEM() - AS_IF([test -z "$BOOST_SYSTEM_LIB"], Index: patches/patch-include_libtorrent_config_hpp =================================================================== RCS file: /cvs/ports/net/libtorrent-rasterbar/patches/patch-include_libtorrent_config_hpp,v retrieving revision 1.4 diff -u -p -u -p -r1.4 patch-include_libtorrent_config_hpp --- patches/patch-include_libtorrent_config_hpp 4 Sep 2020 04:24:28 -0000 1.4 +++ patches/patch-include_libtorrent_config_hpp 16 May 2021 09:23:46 -0000 @@ -2,7 +2,7 @@ $OpenBSD: patch-include_libtorrent_confi Index: include/libtorrent/config.hpp --- include/libtorrent/config.hpp.orig +++ include/libtorrent/config.hpp -@@ -414,6 +414,10 @@ POSSIBILITY OF SUCH DAMAGE. +@@ -429,6 +429,10 @@ POSSIBILITY OF SUCH DAMAGE. #define TORRENT_USE_UNC_PATHS 0 #endif Index: pkg/PLIST =================================================================== RCS file: /cvs/ports/net/libtorrent-rasterbar/pkg/PLIST,v retrieving revision 1.6 diff -u -p -u -p -r1.6 PLIST --- pkg/PLIST 4 Jan 2021 14:06:35 -0000 1.6 +++ pkg/PLIST 16 May 2021 09:23:46 -0000 @@ -43,6 +43,7 @@ include/libtorrent/aux_/noexcept_movable include/libtorrent/aux_/numeric_cast.hpp include/libtorrent/aux_/openssl.hpp include/libtorrent/aux_/path.hpp +include/libtorrent/aux_/pool.hpp include/libtorrent/aux_/portmap.hpp include/libtorrent/aux_/proxy_settings.hpp include/libtorrent/aux_/range.hpp