Re: [scl.org] rh-python36 SPEC file question (circular dependency)
On 9 June 2018 at 02:10, Charalampos Stratakis wrote: > I hope that helps (and I hope I didn't forget any steps :) ). > There's also https://github.com/ncoghlan/pyscl-devel/ from back when I was looking at making an sclo-python rolling-release that tracked the latest released version (even across major version bumps). My working notes for that process: https://github.com/ncoghlan/pyscl-devel/issues/1 While I never ended up following through on that project (since I switched jobs and aren't using SCLs for anything any more), I did get as far as having a fully automated bootstrap build in a previously empty chroot. (The current approach unfortunately doesn't work with COPR or Koji though, as those don't offer a way to override the dist tag for the bootstrap builds the way that mock does: https://github.com/ncoghlan/pyscl-devel/blob/master/rpmlb/sclorg-distgit-download.yml#L59 ) Cheers, Nick. P.S. If anyone would like to take over pyscl-devel development with a view to making maintenance of the Python SCLs more automated (and easier to contribute to), just let me know and I'll unarchive the repo. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] (Lack of) progress on a rolling release Python SCL
Hi folks, A while back I started working on a rolling release Python SCL to track the latest stable release of CPython: https://github.com/ncoghlan/pyscl-devel While I got a fully automated build running locally in mock, the release field modification trick I used to handle bootstrapping new feature releases doesn't work in COPR or koji, so the bootstrapping needs to be redesigned to mutate the bootstrap package names and add appropriate Obsoletes definitions to the real packages in order to work with those backends (I'd prefer the bootstrapping to be idempotent if you rerun it with unchanged sources, so I don't like any approaches based on actually bumping the release number). However, for a variety of reasons, I'm unlikely to complete that work myself anytime soon, so this post is mainly a notice for anyone else interested in the topic that I'm no longer working on it, but my work so far can be found at the repo link above. Cheers, Nick. ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python 3.6 SCL availability
On Thu, Oct 5, 2017 at 3:39 PM, Remi Collet <fed...@famillecollet.com> wrote: > Le 05/10/2017 à 06:25, Nick Coghlan a écrit : >> P.S. I'm not sure when it's expected that the CentOS rebuilds of the >> RHSCL 3.0 collections will be available through >> softwarecollections.org > > AFAIK most of the RHSCL 3.0 Beta packages are already rebuilt in CBS and > are available in the centos-sclo-rh-testing repository > > Ex: https://buildlogs.centos.org/centos/7/sclo/x86_64/rh/rh-python36/ Nice! Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] Python 3.6 SCL availability
Figuring out how best to manage building a rolling sclo-python release in COPR and/or Koji is still a work in progress, but I wanted to note that a pre-built Python 3.6.2 SCL is now available as part of the just-released RHSCL 3.0 beta: https://developers.redhat.com/blog/2017/10/04/red-hat-updates-python-php-node-js-supports-new-arches/ The source code for that SCL is available through sclorg-distgit: https://github.com/sclorg-distgit/python/tree/sig-sclo7-rh-python36-rh If you want to build that locally, then adapting the work I already did for 3.5 in https://github.com/ncoghlan/pyscl-devel/ to work with 3.6 instead would probably be the most reliable way to include everything needed. Cheers, Nick. P.S. I'm not sure when it's expected that the CentOS rebuilds of the RHSCL 3.0 collections will be available through softwarecollections.org -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Setting SCL RPM build options in COPR?
On Wed, Sep 20, 2017 at 12:39 AM, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> wrote: > Nick, > > Now that your question has been answered, let me ask mine.The > instructions to build pyscl-devel separate from Copr are not clear enough for > me. Are you suggesting that I run pipsi in a virtual environment? pipsi creates an auto-activated virtual environment, so the call to `rpmlb` implicitly runs the command in an environment with all the right Python dependencies. I'll add a note about how pipsi works to the README for the benefit of folks that aren't already familiar with it, as simply installing rpmlb into a separately managed virtual environment is indeed a valid alternative. > How is scrlo-python involved? It's the first and last item in the build recipe (the first time to get the relevant scl macros and other helpers defined, the second time to actually define the metapackage that depends on the default package set for the SCL). > I'm going to try later today, and want to be clear about what is involved... > I've built multi-package rpms before, sometimes from scripts and makefiles, > but not multi-package repositories from scripts, and I've never used Copr and > friends. I've never used a chroot to build rpms, but I think I've installed > mock even though I didn't really need it. If you're on Fedora, you'll probably want to do "dnf install fedora-packager" to ensure you have the necessary pieces (since the script uses `fedpkg` in addition to `mock`). If you're on RHEL or CentOS, you'll need to get `fedpkg` from EPEL instead. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Setting SCL RPM build options in COPR?
On Tue, Sep 19, 2017 at 5:31 PM, Petr Kubat <pku...@redhat.com> wrote: > On 09/19/2017 08:16 AM, Nick Coghlan wrote: >> I couldn't find anything in sclorg-distgit >> that actually *sets* them for the rh-python35 case. >> >> >> https://github.com/sclorg-distgit/rh-python35/blob/sig-sclo7-rh-python35-rh/macros.additional.rh-python35 >> has the comment "the @scl@* macros are defined in >> macros.python3.python33 in python33-python-devel" >> >> That's presumably referring to >> >> https://github.com/sclorg-distgit/python/blob/sig-sclo7-rh-python35-rh/macros.python3, >> which still doesn't *set* "@scl@" or "@vendorscl@", it assumes they're >> set somewhere else. > > > The "@scl@" and "@vendorscl@" symbols are replaced by proper values during > the build of the metapackage [1], and the resulting macros get installed > using the *-build sub-package [2] as Honza mentioned. > > [1] > https://github.com/sclorg-distgit/rh-python35/blob/sig-sclo7-rh-python35-rh/rh-python35.spec#L113 > [2] > https://github.com/sclorg-distgit/rh-python35/blob/sig-sclo7-rh-python35-rh/rh-python35.spec#L137 Ah, thanks - that's the step I was missing :) I'll amend the sclo-python version of the macro files to explain that more clearly, and probably put something in the pyscl-devel README as well. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Setting SCL RPM build options in COPR?
On Mon, Sep 18, 2017 at 9:29 PM, Honza Horak <hho...@redhat.com> wrote: > On 09/18/2017 07:29 AM, Nick Coghlan wrote: >> 2. Assuming I haven't missed anything, how do the *default* values for >> "scl" and "vendorscl" actually get set? > > We can kinda control what packages are part of the minimal buildroot for > every copr or cbs tag -- for SCL case there is the -build package that > sets this 'scl' macro. You can add more macros if you like and they will be > defined at a time a particular SRPM/RPM is built. By that trick and changing > what macros are defined in -build, you may basically build more > variants of a single SRPM and produce different output. That's what I thought, but I couldn't find anything in sclorg-distgit that actually *sets* them for the rh-python35 case. https://github.com/sclorg-distgit/rh-python35/blob/sig-sclo7-rh-python35-rh/macros.additional.rh-python35 has the comment "the @scl@* macros are defined in macros.python3.python33 in python33-python-devel" That's presumably referring to https://github.com/sclorg-distgit/python/blob/sig-sclo7-rh-python35-rh/macros.python3, which still doesn't *set* "@scl@" or "@vendorscl@", it assumes they're set somewhere else. So I'm still confused as to: - what's up with the "@" symbols in "@scl@" and "@vendorscl@"? - where can I find an example package that shows how to define them via a package in the buildroot so I can set up sclo-python-preview COPR builds? - where can I find documentation that explains how to do this when you *can't* pass arbitrary RPM build options? Cheers, Nick. P.S. https://www.softwarecollections.org currently appears to be down (I noticed when attempting to find docs about this) -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] Setting SCL RPM build options in COPR?
Using RPM List Builder, I have a recipe for bootstrapping the initial set of sclo-python RPMs locally in mock: https://github.com/ncoghlan/pyscl-devel/ Before building that in the CentOS build system, I'm aiming to first do a preview build in COPR: https://copr.fedorainfracloud.org/coprs/ncoghlan/sclo-python-preview/ However, I've hit a problem in translating the local mock build command into a copr-cli build command, which is that the local builds are relying on the ability to pass in arbitrary RPM build options to specify the right settings for "scl" and "vendorscl". As far as I can tell, that isn't going to work for COPR or CBS, since they expect to be able to just build the component as is, and don't offer the ability to configure the build step. So I have two questions: 1. Have I missed something, and it's actually possible to pass extra RPM build options for COPR builds? (it would make sense to me that I can't - the build isn't going to very repeatable if I can run it with different non-default settings each time) 2. Assuming I haven't missed anything, how do the *default* values for "scl" and "vendorscl" actually get set? Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python "latest" SCLo
On Thu, Jul 13, 2017 at 5:18 PM, Nick Coghlan <ncogh...@redhat.com> wrote: > On Wed, Jul 12, 2017 at 4:22 PM, Nick Coghlan <ncogh...@redhat.com> wrote: >> I wasn't planning to publish the COPR builds anywhere other than COPR, >> they'd just be a place for me to tinker with things before pushing >> them into CBS as real builds. > > I realised I'm going to want a public development repo to hold any > notes I make about the process, as well as any custom tools I build, > so I've created that here: https://github.com/ncoghlan/pyscl-devel/ Status update on this: - with some adjustments to the relevant spec files, I came up with an rpm-list-builder recipe that can bootstrap the whole rh-python35 SCL stack in a clean mock chroot in about an hour (depending on your system specs) - the README at https://github.com/ncoghlan/pyscl-devel/ covers how to run such a build yourself if you're so inclined As noted at https://github.com/ncoghlan/pyscl-devel/issues/1, the next steps will be to switch the built SCL name over to "sclo-python", and then get it building in COPR using actual bootstrap component releases rather than private RPM builds with tweaked release tags. I still have a PyCon Australia talk to finish, though, so I'm planning to take a break from working on this until after that's done. I'm definitely open to PRs if anyone wanted to take a look at what's involved in the rename in the meantime :) Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] RPM list builder & RHSCL rebuild recipes
On Sat, Jul 15, 2017 at 1:12 AM, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> wrote: > It might be nice to provide an sclo-python35 and sclo-python36 so that users > such as we have an easier time not updating. Aye, I think that would be sensible, even though it's not something I'd personally want to work on in my own time. However, I think the most suitable model would be something similar to the way the community addons for PHP work, where the rh-pythonXY packages are still used as a base, with community managed addon streams being built around those, similar to the community addons for PHP that Remi publishes. That is, the way I'd see that working in terms of the way the underlying distros work: - sclo-python: Fedora equivalent (community defined) - rh-pythonXY (in RHSCL): RHEL equivalent (Red Hat defined) - rh-pythonXY (from SCLo): CentOS equivalent (still Red Hat defined) - sclo-pythonXY: EPEL equivalent (community defined) That approach *does* leave open the question around Red Hat's security policy for rh-pythonXY updates (which is currently backport-based, the same as RHEL, with rebasing to new upstream maintenance releases being the exception rather than the rule), but sclo-python will be able to fill the "always rebase" role for 3.6 until 3.7 comes out, which means we can postpone coming up with a different answer until we get closer to having an upstream 3.7.0 release to worry about (and even there we have options - we may decide that sclo-python shouldn't update until after 3.7.1 is out, for example). > I hope we will not for a long time be in the situation between python 2.7 > and python 3, but it could happen between 36 and 37 for any of a number of > reasons. However, that implies more work to get it set up.It becomes > non-automated as 37 comes out. Yep, which is why I think we're going to need Red Hat's paid maintenance staff somewhere in the sustaining engineering path for the stable versions, similar to the way that CentOS is a RHEL rebuild that Fedora's EPEL can build on top of. That kind of work makes for a decent job, but a fairly lousy volunteer activity :) Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] RPM list builder & RHSCL rebuild recipes
In putting together the todo list for the sclo-python SCL, Red Hat's Python maintenance team pointed me towards some of the tools used to maintain the Red Hat SCLs: https://github.com/ncoghlan/pyscl-devel/issues/1#issuecomment-315037884 In particular, RPM List Builder automates running through a series of packages builds based on a YAML recipe file: https://github.com/sclorg/rpm-list-builder/blob/master/docs/users_guide.md That then allows the build sequence for an SCL (including any required bootstrapping builds) to be automated. For example, the Python 3.5 SCL build sequence: https://github.com/sclorg/rhscl-rebuild-recipes/blob/master/python.yml#L87 These aren't currently mentioned in the SCL guide or on the SCLo-SIG wiki, so I figured I'd start by posting a message about them here, and then after I have the sclo-python SCL up and running try to come up with some suitable documentation updates. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python "latest" SCLo
On Wed, Jul 12, 2017 at 4:22 PM, Nick Coghlan <ncogh...@redhat.com> wrote: > On Tue, Jul 11, 2017 at 11:39 PM, Honza Horak <hho...@redhat.com> wrote: >> This guide is actually not complete, as I've just realized -- copr is only >> one way to build SCLs for www.softwarecollections.org, but we can include >> SCLs from CBS (cbs.centos.org, build by SCLo SIG) as well. We should fix >> this part of documentation, thanks for pointing to it.. > > I wasn't planning to publish the COPR builds anywhere other than COPR, > they'd just be a place for me to tinker with things before pushing > them into CBS as real builds. I realised I'm going to want a public development repo to hold any notes I make about the process, as well as any custom tools I build, so I've created that here: https://github.com/ncoghlan/pyscl-devel/ The identified steps for creating and publishing an initial version of sclo-python have been turned into an issue here: https://github.com/ncoghlan/pyscl-devel/issues/1 Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python "latest" SCLo
On Tue, Jul 11, 2017 at 11:39 PM, Honza Horak <hho...@redhat.com> wrote: > On 07/11/2017 10:44 AM, Nick Coghlan wrote: >> 1. Create a new sclo-python metapackage, using >> https://github.com/sclorg-distgit/rh-python35/tree/master as a >> starting point >> 2. For now, just create a `sig-sclo7-sclo-python` branch (we can look >> at an sclo6 branch once 7 is working) >> 3. Edit https://github.com/ncoghlan/sclo-python/tree/sig-sclo7-sclo-python >> for the rh-python35 -> sclo-python name change >> 4. Start doing some test builds in COPR as per >> https://www.softwarecollections.org/en/docs/ > > This guide is actually not complete, as I've just realized -- copr is only > one way to build SCLs for www.softwarecollections.org, but we can include > SCLs from CBS (cbs.centos.org, build by SCLo SIG) as well. We should fix > this part of documentation, thanks for pointing to it.. I wasn't planning to publish the COPR builds anywhere other than COPR, they'd just be a place for me to tinker with things before pushing them into CBS as real builds. >> In parallel with that: >> >> 4. Submit a buildsys request for an sclo-python tag that's similar to >> the existing rh-python35 one (akin to >> https://bugs.centos.org/view.php?id=9661 ) > > I can help with requesting CBS stuff, I think we can request both and build > el7 first.. existance of the tag el6 does not mean we need to ever release > it.. if we don't want to from some reason.. I'm happy to publish both, I just figured it made sense to focus on EL7 first, since I've never been through this process before :) So the two pieces here would be: - my sig-sclo membership request is currently still pending in https://accounts.centos.org - we'll need to file a tag request once we agree on the name > The only thing we need to make clear is the naming -- sclo-python3 or > sclo-python? From my PoV sclo-python3 better describes what is inside.. but > maybe there are other opinions.. If there's ever a Python 4.0, I'd update the rolling SCL to track it rather than sticking with the last 3.x release, so I think "sclo-python" conveys that intent more clearly. I'm also assuming that as long as there's customer demand for them, RH will continue to provide the version specific "rh-pythonXY" SCLs (or something functionally equivalent), so there isn't going to be any great need to restrict the rolling SCL updates - folks will be able to move onto sclo-python if they want to get ahead of the rh-pythonXY streams for some reason, and then potentially drop back to the Red Hat ones if they want to stick with an older release for a while even after a new one becomes available. In Fedora Modularity [1,2] terms, I'm essentially seeing sclo-python as corresponding to a "latest" stream label, while the various rh-pythonXY SCLs would correspond to "X.Y" stream labels. Cheers, Nick. [1] https://docs.pagure.org/modularity/ [2] https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/AKSJ67CS5G7WBQQKO6OCIISC3ARSUG7L/ -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python "latest" SCLo
On Mon, Jul 3, 2017 at 3:47 PM, Nick Coghlan <ncogh...@redhat.com> wrote: > On Fri, Jun 30, 2017 at 1:24 AM, Davis, Daniel (NIH/NLM) [C] > <daniel.da...@nih.gov> wrote: >> I’ve been lurking on this list for a while, and I wanted to bring myself up >> to date. I noticed some talk of a community SCL for a “latest” Python, >> which would be a non-patched pure build of Python that is kept up-to-date by >> the community. Where is that at? Who is leading it? > > That would be me, but it stalled while I was working on some Python > 3.7 changes. Happily though, I have some time to spend on > bootstrapping the "rolling Python" SCL this week, so I'm going to > start working through the list of tasks at > https://wiki.centos.org/SpecialInterestGroup/SCLo#head-b408f06ad89fd3a67686f755eafac7ce310ee081 > (using the rh-python35 collection as a starting point) So I finally started working on the technical aspects of this, and as near as I can figure out, what I'll need to do is: 1. Create a new sclo-python metapackage, using https://github.com/sclorg-distgit/rh-python35/tree/master as a starting point 2. For now, just create a `sig-sclo7-sclo-python` branch (we can look at an sclo6 branch once 7 is working) 3. Edit https://github.com/ncoghlan/sclo-python/tree/sig-sclo7-sclo-python for the rh-python35 -> sclo-python name change 4. Start doing some test builds in COPR as per https://www.softwarecollections.org/en/docs/ In parallel with that: 4. Submit a buildsys request for an sclo-python tag that's similar to the existing rh-python35 one (akin to https://bugs.centos.org/view.php?id=9661 ) 5. Figure out what, if anything, needs to be done in terms of adding relevant branches to the other Python repos on https://github.com/sclorg-distgit 6. Actually start building real SCLs in the CentOS build system for publication to softwarecollections.org Cheers, Nick. P.S. Additional next step: provide feedback to the Fedora Modularity group on which aspects of this strike me as being the most annoyingly tedious, and what could potentially be done to make distro-submodule-based SCL definitions and updates less time consuming than the current process based directly on RPMs and dist-git :) -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] CentOS build system sclo-sig application sent
Hi folks, As mentioned in another thread, I'm finally getting started on bootstrapping a "rolling Python" SCL that follows the latest upstream releases (so it will follow 3.6 until the 3.7.0 release, then switch to following 3.7, etc, etc). Part of that involves officially signing up for SCLo SIG membership in the build system, so this is a note to say that: 1. I've applied for sclo-sig membership via accounts.centos.org 2. My account name is "ncoghlan" (but using my personal address rather than my RH one) 3. The purpose of the application is to set up a "sclo-python" community SCL that tracks the latest upstream stable release as described above At some point we may want to add a "sclo-python-preview" stream that includes the release candidates issued prior to upstream maintenance releases, and switches to the next maintenance branch once the first beta comes out, but if I/we do do anything like that, it would most likely be driven by the start of the 3.7 beta cycle rather than being something that gets done immediately. I'm starting the repo for this at https://github.com/ncoghlan/sclo-python/ (to be transferred to https://github.com/sclorg-distgit at a later date), and will chat to Tomas Orsava about how best to go about seeding that. Cheers, Nick. P.S. I'm starting to think it may actually make sense to start out with the 3.5 Red Hat SCL as a template, and then do a 3.5 -> 3.6 transition within the sclo-python stream to make it absolutely clear that when we say "rolling release, including major version upgrades" we really do mean "rolling release, including major version upgrades". Otherwise, regardless of what we say in the docs, I can see folks getting upset when the 3.6 -> 3.7 switch comes around and they happened to be relying on a previously deprecated feature that had finally been removed. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python3 (always latest) community SCL
On Wed, May 3, 2017 at 10:35 PM, Nick Coghlan <ncogh...@redhat.com> wrote: > On Wed, May 3, 2017 at 9:44 PM, Tomas Orsava <tors...@redhat.com> wrote: >> I do see the benefits of the added rolling SCL, though in my mind the >> benefits are lessened by the pre-existence of Python 3 in EPEL, and thus I'm >> not sure it's worth the added maintenance. However, that's only my personal >> reasoning. >> >> If we can find volunteers for the effort I'll be glad to lend a hand with >> the creation of the SCL. > > I'd like to see this happen enough to volunteer to maintain it myself, > so I'll start working through the checklist that Honza posted :) To provide an indicative timeline for this: I'm currently trying to get PEP 538's locale coercion squared away for Python 3.7 before Fedora 26 hits code freeze, and then will be busy with proposal reviews for the PyCon Australia Education Seminar for the next couple of weeks. So that will mean I'll start figuring out the details of setting up this rolling community SCL some time in late May or early June. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python3 (always latest) community SCL
On Wed, May 3, 2017 at 9:44 PM, Tomas Orsava <tors...@redhat.com> wrote:
> On 05/03/2017 09:13 AM, Nick Coghlan wrote:
>>> On 04/11/2017 10:24 AM, Nick Coghlan wrote:
>> While I thought this sounded reasonable at the time, it turns out to
>> have a lot of problems in practice, as it makes it hard for developers
>> to say "I just want to run on a pre-built version of the latest
>> upstream Python". Instead, they have to choose between:
>>
>> * referring to python3.x in their own code, and having to update all
>> those references to switch to a new version
>> * keeping their "python3" references, and trusting that the originally
>> planned EPEL transitions will happen in a timely fashion
>
>
> I agree that it's not exactly seamless, you can make a single boolean macro
> in your spec file using which it will be switched to the alternative Python,
> so it shouldn't be a significant maintenance burden.
For the use cases I'm interested in, there's no application level spec
file - just a container definition or Ansible deployment script.
SCLs are really useful for those cases, since it's relatively
straightforward to ensure the SCL is enabled at the appropriate times,
while RPM macros aren't available.
>> And then regardless of the approach they choose, they have to rely on
>> either virtualenv or fiddling with the system level symlink to run
>> against an alternative Python 3 stack.
>
> If I may ask, what do you mean by fiddling with the system level symlink? I
> believe they can invoke `/usr/bin/python3.X` using the %{__python3_other}
> macro.
Choosing a stack other than the default one at runtime (rather than
when building an RPM). I'm not suggesting changing the symlink would
be a good idea, just pointing out it's basically the only option left
if you're not using either SCLs or a Python level venv and aren't
willing to qualify every Python 3 invocation with a specific minor
version.
>> That approach is then a lot closer to the traditional RPM update flow
>> from upstream->Fedora->RHEL->CentOS, just modified to be
>> upstream->SCLo->RHSCL->EPEL since there isn't a system Python 3 stack
>> in RHEL & CentOS.
>
> I do see the benefits of the added rolling SCL, though in my mind the
> benefits are lessened by the pre-existence of Python 3 in EPEL, and thus I'm
> not sure it's worth the added maintenance. However, that's only my personal
> reasoning.
>
> If we can find volunteers for the effort I'll be glad to lend a hand with
> the creation of the SCL.
I'd like to see this happen enough to volunteer to maintain it myself,
so I'll start working through the checklist that Honza posted :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Platform Engineering, Brisbane
___
SCLorg mailing list
SCLorg@redhat.com
https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python3 (always latest) community SCL
On Fri, Apr 14, 2017 at 12:27 AM, Tomas Orsava <tors...@redhat.com> wrote: > Hi! > > On 04/11/2017 10:24 AM, Nick Coghlan wrote: >> >> I've been mulling this idea over for the past couple of weeks, and I'm >> wondering if it might make sense to create a rolling "sclo-python3" >> SCL, that's initially forked from >> https://www.softwarecollections.org/en/scls/rhscl/rh-python35/, but >> explicitly promises to rebase to new Python feature releases when they >> come out. >> >> So if people were happy to always run on the leading edge (even for >> X.Y.0 releases), they could use "sclo-python3", but if they wanted to >> stay on a particular X.Y release for a while, they would need to >> switch to the downstream rh-pythonXY SCLs. >> >> Remi, if I wanted to do that, where would I start? >> https://github.com/sclorg-distgit is useful as a reference for >> submitting changes to existing community SCLs, but it doesn't provide >> any guidance on how to start a new one (and that info is also missing >> from the wiki). > > It would be great to have a state-of-the-art Python in Enterprise Linux, but > I think it would be better using the existing (though lacking maintenance) > Python 3 in EPEL [0] mechanism. > > While installing and using SCLs isn't hard, I think an RPM packaged version > is still easier for both maintenance and usage and people can have EPEL > packages built against it as well. In addition the transition mechanism is > smoother, as two Python versions are coexisting during the transition > period, whereas the rolling SCL would just switch and everyone would have to > immediately adjust. > > What would be the advantage of creating a rolling 'scl-python3' collection > over the EPEL mechanism [1]? The main practical issue I see with the EPEL parallel installation design is the part about updating "/usr/bin/python3" to new "stable" versions: == In a situation when python3X is in EPEL and 3.X+1 is released upstream, the following happens: * python3X+1 package is created for EPEL ASAP and all extension packages are built also for this new python3X+1 stack. * When all packages are rebuilt for python3X+1, the old python3X stack is retired after certain period. This period gives everyone enough time to rebuild their packages while being as short as possible. There is intentionally no hard limit here, we will approach this case by case. * "/usr/bin/python3" belongs to the "stable" python3X stack. Switching /usr/bin/python3 from python3X to python3X+1 happens shortly before the end of transitional period (== before obsoleting python3X) and it is announced on epel-devel. Usage of "/usr/bin/python3" is discouraged in favour of using /usr/bin/python3.X explicitly. == While I thought this sounded reasonable at the time, it turns out to have a lot of problems in practice, as it makes it hard for developers to say "I just want to run on a pre-built version of the latest upstream Python". Instead, they have to choose between: * referring to python3.x in their own code, and having to update all those references to switch to a new version * keeping their "python3" references, and trusting that the originally planned EPEL transitions will happen in a timely fashion And then regardless of the approach they choose, they have to rely on either virtualenv or fiddling with the system level symlink to run against an alternative Python 3 stack. Accordingly, the main benefit I see to the rolling community SCL is that it makes "python3" load the SCL version when the SCL is enabled, which *does* make it easy for developers to choose between running against either the latest Python 3.x (by building on top of either Fedora or the proposed community SCL), or against a specific Python 3.x version (by using the Red Hat SCLs). If the native EPEL builds are retained, then they would be a *downstream* of the Red Hat SCLs, so the complete maintenance flow would be: - upstream release happens - Fedora system Python and rolling Python 3 community SCL are updated - Python 3.x Red Hat SCLs are updated - EPEL Python 3.x stacks are updated That approach is then a lot closer to the traditional RPM update flow from upstream->Fedora->RHEL->CentOS, just modified to be upstream->SCLo->RHSCL->EPEL since there isn't a system Python 3 stack in RHEL & CentOS. Cheers, Nick. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Python 3.6 availability
On Sun, Mar 26, 2017 at 12:50 AM, meson <me...@posteo.de> wrote: > Hi, > > is there any estimate on when Python 3.6 will be available as SCL? > > Our devs are asking about it. If it's going to take a long time, I'll try my > hand at building the packages myself. It likely makes sense to have some community maintained sclo-* Python packages regardless, as the rh-* ones are officially Red Hat maintained, and hence: - are published in line with Red Hat's product schedules rather than necessarily being ASAP after the upstream release - have RHEL-style rules against rebasing components within the collection (so some updates have to wait for the next revision of the entire collection) - are restricted to components that Red Hat is currently willing to commercially support with security backports By contrast, community packages could adopt more permissive policies around package inclusion and rebasing, without having to take Red Hat's commercial support obligations into account the way the official SCLs do. > A general question: is the SCL development open? I did not find any public > git repositories apart from the CentOS source server. I'd love to > contribute. Remi, do you have access to edit the CentOS wiki? It would be good to provide a pointer to https://github.com/sclorg-distgit from https://wiki.centos.org/SpecialInterestGroup/SCLo and you have a much better understanding than I do of how the existing sclo-* SCLs are maintained. Cheers, Nick. P.S. Perhaps it would be worth adding an "sclo-admin" repo to the sclo-distgit org, and pinning it? Then the README for that could be used as a second entry point for info about the SCLo sig, and it could also provide an issue tracker for the SIG itself, rather than relying solely on the mailing list and the RHSCL component in Red Hat's bugzilla instance (which only covers the official SCLs anyway). -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Feedback on /usr/bin symlinks for python
On Thu, Mar 23, 2017 at 12:50 AM, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> wrote: > For the python SCLs, I would potentially love this, but one would need to be > careful to version the wrappers/symlinks, and/or worry about rpaths and ld > config. Aye, Python's going to have to be a special case as it isn't possible to uninstall the base "python" package in current versions of RHEL & CentOS. However, it should be possible to get system-wide Python SCLs to work similarly to the pythonXY packages in Fedora and EPEL, where they only provide the "/usr/bin/pythonX.Y" links, and not the more generic "python" or "pythonX" links. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
Re: [scl.org] Idea: Software Collections Daemons Made System-wide
On Wed, Mar 22, 2017 at 1:05 AM, Honza Horak <hho...@redhat.com> wrote: > This is basically a kick-off for getting more feedback for an idea shared at > http://www.themindiseverything.eu/2017/03/software-collections-daemons-made.html. > > Shortly, SCL has worked nicely for several years and people love them. But > even the beloved ones have some issues. And what we hear from users, the > issues with Software Collections concept currently are basically those: > > * we need to use scl enable, which changes $PATH and other environment > variables, so the binaries placed in different location are visible by the > system > * scripts originally written for "normal" MySQL use full paths (like > /usr/bin/mysql) which does not work when we only have the Software > Collection installed > * Data directory, config files and log files are on different location than > it is common > > The blog post tries to summarize possible solution, which I'm looking for > feedback now, ideally by replying to this mail.. If you change the exec line in the wrapper script to be: exec -a $0 `basename "$0"` "$@" I believe this may actually also deal with the Python sys.executable and hence setuptools and pipsi compatibility problems that I raised in https://bugzilla.redhat.com/show_bug.cgi?id=1385471 Otherwise sys.executable will still be set to the unwrapped binary, and any generated shebang lines would refer to that rather than the wrapper script. Cheers, Nick. -- Nick Coghlan Red Hat Platform Engineering, Brisbane ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
[scl.org] Problems with Python scripts that use SCL runtimes
Hi folks, The question of the explicit "scl enable" step for using SCLs is one that's come up a few times in different contexts, and I finally sat down to document a relatively straightforward scenario that illustrates the problem: https://github.com/ncoghlan/scl-pipsi-demo#demonstrating-the-problem What the demo covers doing is: 1. Enable the Python 3.5 SCL 2. Use pipsi within that SCL to install the pygmentize script for a particular user within a virtual environment configured to use that particular runtime 3. Disable the SCL 4. Show that the installed script doesn't work due to the runtime being unable to find its shared libraries Ideally, the fact I used an SCL to install (and subsequently run) pygmentize would be an invisible implementation detail. However, at the moment, it's visible, since Python's virtual environment machinery only fixes paths for Python level imports - it doesn't touch the paths for operating system level dynamic library loading. As far as I am aware, changing the SCL binaries to set RPATH and/or RUNPATH should be sufficient to resolve the problem, at least with pipsi managed script installation (plain "--user" installations are likely to still have problems, but I'm more comfortable with a restriction saying not to use --user installs in conjunction with SCLs) Regards, Nick. -- Nick Coghlan Fedora Environments & Stacks Red Hat Developer Experience, Brisbane Software Development Workflow Designer & Process Architect ___ SCLorg mailing list SCLorg@redhat.com https://www.redhat.com/mailman/listinfo/sclorg
