Re: F20 System Wide Change: Perl 5.18
On 08/08/13 12:50, Petr Pisar wrote:
On 2013-08-08, Paul Howarth wrote:
On 08/08/13 10:01, Phil Knirsch wrote:
Just like Seth's script it has a few obviously problems as for really
correct build ordering you need to flip-flop between srpms/specfiles and
binary rpms, but that will need a bit of a different approach. As it
stands right now some of the dependencies it can't detect and resolve
are build time generated ones, which might a showstopper for some uses
of it.
I've tested the version i have here with several complete past Fedora
trees, for texlive and KDE rebuilds and updates and for all the cases
the output look pretty sane.
Hope this helps,
We already know where it's beneficial to break build dependency cycles
and have built that into the spec files, triggered by the
%{perl_bootstrap} macro being set (which it is in the current Rawhide
perl). So a build ordering script for perl bootstrapping should take
that into account.
And therefore my slow script considers SRPM as well as binary packages
distinctively. Therefore it clones git repostories, creates SRPMs in mock
environment (because data from koji will differ from to-rebuild packages
due to the perl_bootstrap macro), retrieves just built binary packages,
extracts the requires/provides and solves them on its own. Of course it
submits whole groups of packages to rebuild and it waits for build root
rotation. It does prety much everything what is needed.
My own script does something similar but doesn't build the packages,
since there are hardly any for which the runtime dependencies of
bootstrapped packages are different from those of non-bootstrapped
packages - I have a separate override configuration file in which I
specify those differences. Needless to say, this saves a lot of time. So
the git checkout is used to generate SRPMs, and I get the buildreqs from
those, and using the existing Rawhide metadata for the runtime dependencies.
(Curriously, this year I could not build in 16 threads as last year,
becuse dist-git starts refusing on 10 threads. Also after ausil started
his mass rebuild, my koji client started getting host unreachable errors.)
However the really slow part is the dependency solver. I had no time to
improve it before this rebuild. I will do before next time. But now I'm
not going to change it becuse it's to complicated.
I did my depsolver in python, leveraging code from yum. It's actually
quite quick, much quicker than extracting the dependency data from git,
except in the presence of circular build dependencies, but it does a
decent job of identifying and prioritizing those, and I keep on top of
them by running the script periodically and raising bugs where needed.
Neverthless we are looming to the point where no new packages will be
possible to rebuild due to unsatisified depenencies. We are now at
95 % done. The rest will need fixing or dropping
. Once
we get there, I will publish list of failed and unfinished packages
together with dependency graph.
I have already done 169 changes in the F20 packages to achieve those
95 % and I'm quite tired and sick of all of that. (Mainly
because maintainers do not upgrade their packages or because they do not
declare all needed dependencies). (I'm thinking of running dummy
rebuilds during a year regularly, to find all these buggy packages before
next Perl rebuild.)
It's a good plan. There are people running through all modules on CPAN
and raising bugs where they break with bleadperl but if upstreams don't
fix these bugs there's not much we can do.
Paul.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 2013-08-08, Paul Howarth wrote:
> On 08/08/13 10:01, Phil Knirsch wrote:
>>
>> Just like Seth's script it has a few obviously problems as for really
>> correct build ordering you need to flip-flop between srpms/specfiles and
>> binary rpms, but that will need a bit of a different approach. As it
>> stands right now some of the dependencies it can't detect and resolve
>> are build time generated ones, which might a showstopper for some uses
>> of it.
>>
>> I've tested the version i have here with several complete past Fedora
>> trees, for texlive and KDE rebuilds and updates and for all the cases
>> the output look pretty sane.
>>
>> Hope this helps,
>
> We already know where it's beneficial to break build dependency cycles
> and have built that into the spec files, triggered by the
> %{perl_bootstrap} macro being set (which it is in the current Rawhide
> perl). So a build ordering script for perl bootstrapping should take
> that into account.
>
And therefore my slow script considers SRPM as well as binary packages
distinctively. Therefore it clones git repostories, creates SRPMs in mock
environment (because data from koji will differ from to-rebuild packages
due to the perl_bootstrap macro), retrieves just built binary packages,
extracts the requires/provides and solves them on its own. Of course it
submits whole groups of packages to rebuild and it waits for build root
rotation. It does prety much everything what is needed.
(Curriously, this year I could not build in 16 threads as last year,
becuse dist-git starts refusing on 10 threads. Also after ausil started
his mass rebuild, my koji client started getting host unreachable errors.)
However the really slow part is the dependency solver. I had no time to
improve it before this rebuild. I will do before next time. But now I'm
not going to change it becuse it's to complicated.
Neverthless we are looming to the point where no new packages will be
possible to rebuild due to unsatisified depenencies. We are now at
95 % done. The rest will need fixing or dropping
. Once
we get there, I will publish list of failed and unfinished packages
together with dependency graph.
I have already done 169 changes in the F20 packages to achieve those
95 % and I'm quite tired and sick of all of that. (Mainly
because maintainers do not upgrade their packages or because they do not
declare all needed dependencies). (I'm thinking of running dummy
rebuilds during a year regularly, to find all these buggy packages before
next Perl rebuild.)
-- Petr
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 08/08/13 10:01, Phil Knirsch wrote:
On 08/07/2013 10:56 AM, Marcela Mašláňová wrote:
On 08/07/2013 03:01 AM, Dennis Gilmore wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Tue, 06 Aug 2013 13:39:31 -0700
Adam Williamson wrote:
On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote:
On Thu, 13 Jun 2013 13:23:51 + (UTC)
Petr Pisar wrote:
On 2013-06-12, Kevin Fenzi wrote:
So, there's nothing preventing the side tag and rebuild anytime
now right? 5.18.0 is out, so we could start that work in
rawhide?=20
Currently 5.18.0 does not pass one test when running in mock and
koji. (It's because of the terminal usage in tested perl
debugger.) We think we could have solved this issue in a few days.
Cool.
Could you explain how the side tag inheritance works? It inherits
everything from rawhide, even builds made after the side tag
creation,
yes.
except packages whose builds have been already made in the side
tag. Am I right? That means we still get fresh third-party
dependencies from rawhide.
yes. However, there's are several downsides:
- Each side tag adds newrepo tasks which increases load a lot.
- If you rebuild perl-foo-1.0-1 in the side tag against the new
perl, then the maintainer has to fix something in rawhide, they
would build perl-foo-1.0-2 in rawhide and when the side tag was
merged back over either everyone would get the older one with the
bug, or the newer one against the old perl. So, it's really
important to not take a long time using a side tag to avoid this
problem as much as possible.
Seems like this one came true in practice. It seems like a 5.18
rebuild run was done in a side tag and then merged back into Rawhide.
Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done
prior to the general F20 mass rebuild - so the mass rebuild won out,
and effectively squelched the perl rebuild.
The f20-perl tag was merged back before the mass rebuild was started.
so everything in the mass rebuild was built against the new perl.
however because the perl rebuild was at a week there was quite a few
packages rebuilt against the old perl. we need to work out how to build
perl quicker. your analysis is not really correct.
Dennis
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
iEYEARECAAYFAlIBnHcACgkQkSxm47BaWfeWKgCeKSTmyV0Yrn9oulqe3UfCgEFH
ZxgAn3vU40FXlBwcxg7hRpyx40OeGdVN
=fMp/
-END PGP SIGNATURE-
If someone knows about a tool, which can give build order faster than
Petr's tool, then it would help ;-)
Marcela
I've been working on improving one of Seth's last scripts call
buildorder that he wrote to order a set or SRPMS[1].
The current version is far from complete and/or bugfree yet, but i've
attached the current version to this email.
It's really simple: You give it a set of srpms and it spits out the
buildorder for it as precisely as it can get it.
The improvements i made was to get rid of the topology sort module he
had been using and replaced it with my own implementation which also
automatically breaks build loops at spots that are likely to be most
beneficial to be broken. It does grouping as well, just like Seth's
original version, but the actual group output is commented out at the
moment as i'm using it for sequentially rebuilding things, not in
parallel, so groups don't matter for my use case. If you want groups
simply remove the comment from the
# print "Group %d (%d reqs)" % (group, minreqs)
line.
It also fixes an issue with the global macro definitions which didn't
get cleared.
I've also added the manual provides specified in the specfiles, that
gave a bit better dependency completion.
Last (and least) there is some commented out code it in where you can
potentially use this code to also find the full buildchain for a
specific package, basically allowing you to specify 1 or a few srpms
you'd like to build and the script would spit out a list of all srpms
you'll need to build for those packages in the necessary order.
Just like Seth's script it has a few obviously problems as for really
correct build ordering you need to flip-flop between srpms/specfiles and
binary rpms, but that will need a bit of a different approach. As it
stands right now some of the dependencies it can't detect and resolve
are build time generated ones, which might a showstopper for some uses
of it.
I've tested the version i have here with several complete past Fedora
trees, for texlive and KDE rebuilds and updates and for all the cases
the output look pretty sane.
Hope this helps,
We already know where it's beneficial to break build dependency cycles
and have built that into the spec files, triggered by the
%{perl_bootstrap} macro being set (which it is in the current Rawhide
perl). So a build ordering script for perl bootstrapping should take
that into account.
Paul.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-c
Re: F20 System Wide Change: Perl 5.18
On 08/07/2013 10:56 AM, Marcela Mašláňová wrote: On 08/07/2013 03:01 AM, Dennis Gilmore wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 06 Aug 2013 13:39:31 -0700 Adam Williamson wrote: On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: On Thu, 13 Jun 2013 13:23:51 + (UTC) Petr Pisar wrote: On 2013-06-12, Kevin Fenzi wrote: So, there's nothing preventing the side tag and rebuild anytime now right? 5.18.0 is out, so we could start that work in rawhide?=20 Currently 5.18.0 does not pass one test when running in mock and koji. (It's because of the terminal usage in tested perl debugger.) We think we could have solved this issue in a few days. Cool. Could you explain how the side tag inheritance works? It inherits everything from rawhide, even builds made after the side tag creation, yes. except packages whose builds have been already made in the side tag. Am I right? That means we still get fresh third-party dependencies from rawhide. yes. However, there's are several downsides: - Each side tag adds newrepo tasks which increases load a lot. - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, then the maintainer has to fix something in rawhide, they would build perl-foo-1.0-2 in rawhide and when the side tag was merged back over either everyone would get the older one with the bug, or the newer one against the old perl. So, it's really important to not take a long time using a side tag to avoid this problem as much as possible. Seems like this one came true in practice. It seems like a 5.18 rebuild run was done in a side tag and then merged back into Rawhide. Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done prior to the general F20 mass rebuild - so the mass rebuild won out, and effectively squelched the perl rebuild. The f20-perl tag was merged back before the mass rebuild was started. so everything in the mass rebuild was built against the new perl. however because the perl rebuild was at a week there was quite a few packages rebuilt against the old perl. we need to work out how to build perl quicker. your analysis is not really correct. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlIBnHcACgkQkSxm47BaWfeWKgCeKSTmyV0Yrn9oulqe3UfCgEFH ZxgAn3vU40FXlBwcxg7hRpyx40OeGdVN =fMp/ -END PGP SIGNATURE- If someone knows about a tool, which can give build order faster than Petr's tool, then it would help ;-) Marcela I've been working on improving one of Seth's last scripts call buildorder that he wrote to order a set or SRPMS[1]. The current version is far from complete and/or bugfree yet, but i've attached the current version to this email. It's really simple: You give it a set of srpms and it spits out the buildorder for it as precisely as it can get it. The improvements i made was to get rid of the topology sort module he had been using and replaced it with my own implementation which also automatically breaks build loops at spots that are likely to be most beneficial to be broken. It does grouping as well, just like Seth's original version, but the actual group output is commented out at the moment as i'm using it for sequentially rebuilding things, not in parallel, so groups don't matter for my use case. If you want groups simply remove the comment from the # print "Group %d (%d reqs)" % (group, minreqs) line. It also fixes an issue with the global macro definitions which didn't get cleared. I've also added the manual provides specified in the specfiles, that gave a bit better dependency completion. Last (and least) there is some commented out code it in where you can potentially use this code to also find the full buildchain for a specific package, basically allowing you to specify 1 or a few srpms you'd like to build and the script would spit out a list of all srpms you'll need to build for those packages in the necessary order. Just like Seth's script it has a few obviously problems as for really correct build ordering you need to flip-flop between srpms/specfiles and binary rpms, but that will need a bit of a different approach. As it stands right now some of the dependencies it can't detect and resolve are build time generated ones, which might a showstopper for some uses of it. I've tested the version i have here with several complete past Fedora trees, for texlive and KDE rebuilds and updates and for all the cases the output look pretty sane. Hope this helps, Thanks & regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany #!/usr/bin/python -tt # skvidal at fedoraproject.org & pknirsch at redhat.com # take a set of srpms # resolve their buildreqs (pkg name only and virtual build provides) # sort them # break them
Re: F20 System Wide Change: Perl 5.18
On 08/08/13 03:47, Adam Williamson wrote: On Thu, 2013-08-08 at 10:43 +0800, Christopher Meng wrote: Just a note, I find my package's deps have magic requires for 5.16.2(not 5.16.3), does it mean that this has not rebuilded since f19? Why? http://kojipkgs.fedoraproject.org//work/tasks/5588/5735588/root.log It means they failed or got superseded from the perl 5.18 rebuild for some reason. Instead of asking others to spend their time for you, why not find out for yourself? All you have to do is go to Koji, search for each package, and look through the recent builds to see what was going on. In this case it's because your package depends on various perl-Catalyst packages, which are in turn not buildable because one of their dependencies is incompatible with Perl 5.18. Many of these packages have now got blocks/depends on data included in their FTBFS bugzilla tickets, which make it easier to track why particular packages aren't built, and provide notifications when an underlying dependency has been fixed. Paul. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On Thu, 2013-08-08 at 10:43 +0800, Christopher Meng wrote: > Just a note, I find my package's deps have magic requires for > 5.16.2(not 5.16.3), does it mean that this has not rebuilded since > f19? Why? > > http://kojipkgs.fedoraproject.org//work/tasks/5588/5735588/root.log It means they failed or got superseded from the perl 5.18 rebuild for some reason. Instead of asking others to spend their time for you, why not find out for yourself? All you have to do is go to Koji, search for each package, and look through the recent builds to see what was going on. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
Just a note, I find my package's deps have magic requires for 5.16.2(not 5.16.3), does it mean that this has not rebuilded since f19? Why? http://kojipkgs.fedoraproject.org//work/tasks/5588/5735588/root.log -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 07/08/13 09:56, Marcela Mašláňová wrote: On 08/07/2013 03:01 AM, Dennis Gilmore wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 06 Aug 2013 13:39:31 -0700 Adam Williamson wrote: On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: On Thu, 13 Jun 2013 13:23:51 + (UTC) Petr Pisar wrote: On 2013-06-12, Kevin Fenzi wrote: So, there's nothing preventing the side tag and rebuild anytime now right? 5.18.0 is out, so we could start that work in rawhide?=20 Currently 5.18.0 does not pass one test when running in mock and koji. (It's because of the terminal usage in tested perl debugger.) We think we could have solved this issue in a few days. Cool. Could you explain how the side tag inheritance works? It inherits everything from rawhide, even builds made after the side tag creation, yes. except packages whose builds have been already made in the side tag. Am I right? That means we still get fresh third-party dependencies from rawhide. yes. However, there's are several downsides: - Each side tag adds newrepo tasks which increases load a lot. - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, then the maintainer has to fix something in rawhide, they would build perl-foo-1.0-2 in rawhide and when the side tag was merged back over either everyone would get the older one with the bug, or the newer one against the old perl. So, it's really important to not take a long time using a side tag to avoid this problem as much as possible. Seems like this one came true in practice. It seems like a 5.18 rebuild run was done in a side tag and then merged back into Rawhide. Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done prior to the general F20 mass rebuild - so the mass rebuild won out, and effectively squelched the perl rebuild. The f20-perl tag was merged back before the mass rebuild was started. so everything in the mass rebuild was built against the new perl. however because the perl rebuild was at a week there was quite a few packages rebuilt against the old perl. we need to work out how to build perl quicker. your analysis is not really correct. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlIBnHcACgkQkSxm47BaWfeWKgCeKSTmyV0Yrn9oulqe3UfCgEFH ZxgAn3vU40FXlBwcxg7hRpyx40OeGdVN =fMp/ -END PGP SIGNATURE- If someone knows about a tool, which can give build order faster than Petr's tool, then it would help ;-) How long does Petr's tool take? My script runs in a couple of hours, most of which is the time it takes to grok the bootstrap build dependencies from git. It also only considers building packages that provide perl modules (basically those that would need a MODULE_COMPAT_* dependency) rather than everything that merely might have the slightest dependency on perl. Paul. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 08/07/2013 03:01 AM, Dennis Gilmore wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 06 Aug 2013 13:39:31 -0700 Adam Williamson wrote: On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: On Thu, 13 Jun 2013 13:23:51 + (UTC) Petr Pisar wrote: On 2013-06-12, Kevin Fenzi wrote: So, there's nothing preventing the side tag and rebuild anytime now right? 5.18.0 is out, so we could start that work in rawhide?=20 Currently 5.18.0 does not pass one test when running in mock and koji. (It's because of the terminal usage in tested perl debugger.) We think we could have solved this issue in a few days. Cool. Could you explain how the side tag inheritance works? It inherits everything from rawhide, even builds made after the side tag creation, yes. except packages whose builds have been already made in the side tag. Am I right? That means we still get fresh third-party dependencies from rawhide. yes. However, there's are several downsides: - Each side tag adds newrepo tasks which increases load a lot. - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, then the maintainer has to fix something in rawhide, they would build perl-foo-1.0-2 in rawhide and when the side tag was merged back over either everyone would get the older one with the bug, or the newer one against the old perl. So, it's really important to not take a long time using a side tag to avoid this problem as much as possible. Seems like this one came true in practice. It seems like a 5.18 rebuild run was done in a side tag and then merged back into Rawhide. Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done prior to the general F20 mass rebuild - so the mass rebuild won out, and effectively squelched the perl rebuild. The f20-perl tag was merged back before the mass rebuild was started. so everything in the mass rebuild was built against the new perl. however because the perl rebuild was at a week there was quite a few packages rebuilt against the old perl. we need to work out how to build perl quicker. your analysis is not really correct. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlIBnHcACgkQkSxm47BaWfeWKgCeKSTmyV0Yrn9oulqe3UfCgEFH ZxgAn3vU40FXlBwcxg7hRpyx40OeGdVN =fMp/ -END PGP SIGNATURE- If someone knows about a tool, which can give build order faster than Petr's tool, then it would help ;-) Marcela -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 08/06/2013 10:57 PM, Adam Williamson wrote: On Tue, 2013-08-06 at 22:52 +0200, Emmanuel Seyman wrote: * Adam Williamson [06/08/2013 13:43] : perl-Test-Refcount-0:0.07-8.fc19.noarch This module doesn't build against perl 5.18.x (so please stop trying to do it, folks). This is https://rt.cpan.org/Public/Bug/Display.html?id=85998 Unfortunately, lots of other things appear to depend on it, so it not building has a rather sever knock-on effect. For instance, perl-Future has it as a buildrequire, and perl-IO-Async has both perl-Future and perl-Test-Refcount as buildrequires... surely this sort of mess should've been sorted out before 5.18 was ever shipped stable?! Why? Upstream released new version, they can't check all modules on cpan. Usually modules, which can't be built have dead or slower upstreams. Marcela -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 06 Aug 2013 13:39:31 -0700 Adam Williamson wrote: > On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: > > On Thu, 13 Jun 2013 13:23:51 + (UTC) > > Petr Pisar wrote: > > > > > On 2013-06-12, Kevin Fenzi wrote: > > > > So, there's nothing preventing the side tag and rebuild anytime > > > > now right? 5.18.0 is out, so we could start that work in > > > > rawhide?=20 > > > > > > > Currently 5.18.0 does not pass one test when running in mock and > > > koji. (It's because of the terminal usage in tested perl > > > debugger.) We think we could have solved this issue in a few days. > > > > Cool. > > > > > Could you explain how the side tag inheritance works? It inherits > > > everything from rawhide, even builds made after the side tag > > > creation, > > > > yes. > > > > > except packages whose builds have been already made in the side > > > tag. Am I right? That means we still get fresh third-party > > > dependencies from rawhide. > > > > yes. However, there's are several downsides: > > > > - Each side tag adds newrepo tasks which increases load a lot. > > - If you rebuild perl-foo-1.0-1 in the side tag against the new > > perl, then the maintainer has to fix something in rawhide, they > > would build perl-foo-1.0-2 in rawhide and when the side tag was > > merged back over either everyone would get the older one with the > > bug, or the newer one against the old perl. So, it's really > > important to not take a long time using a side tag to avoid this > > problem as much as possible. > > Seems like this one came true in practice. It seems like a 5.18 > rebuild run was done in a side tag and then merged back into Rawhide. > Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done > prior to the general F20 mass rebuild - so the mass rebuild won out, > and effectively squelched the perl rebuild. The f20-perl tag was merged back before the mass rebuild was started. so everything in the mass rebuild was built against the new perl. however because the perl rebuild was at a week there was quite a few packages rebuilt against the old perl. we need to work out how to build perl quicker. your analysis is not really correct. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlIBnHcACgkQkSxm47BaWfeWKgCeKSTmyV0Yrn9oulqe3UfCgEFH ZxgAn3vU40FXlBwcxg7hRpyx40OeGdVN =fMp/ -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On Tue, 06 Aug 2013 13:57:42 -0700 Adam Williamson wrote: > On Tue, 2013-08-06 at 22:52 +0200, Emmanuel Seyman wrote: > > * Adam Williamson [06/08/2013 13:43] : > > > > > > perl-Test-Refcount-0:0.07-8.fc19.noarch > > > > This module doesn't build against perl 5.18.x (so please stop > > trying to do it, folks). > > > > This is https://rt.cpan.org/Public/Bug/Display.html?id=85998 > > Unfortunately, lots of other things appear to depend on it, so it not > building has a rather sever knock-on effect. > > For instance, perl-Future has it as a buildrequire, and perl-IO-Async > has both perl-Future and perl-Test-Refcount as buildrequires... A temporary workaround might be to disable the tests that require Test::RefCount in the packages that use it, until such time as it's fixed. That would help get more of the stack done, at least until we hit something that depends on a broken module for core functionality rather than just its test suite. > surely this sort of mess should've been sorted out before 5.18 was > ever shipped stable?! Well yes it should, but we only find these things by trying to build the 2,000+ perl packages and it would have been nice if the rebuild-in-progress hadn't been tagged into rawhide before the job was done. Paul. -- Fedora Extras Perl SIG http://www.fedoraproject.org/wiki/Extras/SIGs/Perl perl-devel mailing list perl-de...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/perl-devel
Re: F20 System Wide Change: Perl 5.18
On Tue, 2013-08-06 at 22:52 +0200, Emmanuel Seyman wrote: > * Adam Williamson [06/08/2013 13:43] : > > > > perl-Test-Refcount-0:0.07-8.fc19.noarch > > This module doesn't build against perl 5.18.x (so please stop > trying to do it, folks). > > This is https://rt.cpan.org/Public/Bug/Display.html?id=85998 Unfortunately, lots of other things appear to depend on it, so it not building has a rather sever knock-on effect. For instance, perl-Future has it as a buildrequire, and perl-IO-Async has both perl-Future and perl-Test-Refcount as buildrequires... surely this sort of mess should've been sorted out before 5.18 was ever shipped stable?! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
* Adam Williamson [06/08/2013 13:43] : > > perl-Test-Refcount-0:0.07-8.fc19.noarch This module doesn't build against perl 5.18.x (so please stop trying to do it, folks). This is https://rt.cpan.org/Public/Bug/Display.html?id=85998 Emmanuel -- Fedora Extras Perl SIG http://www.fedoraproject.org/wiki/Extras/SIGs/Perl perl-devel mailing list perl-de...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/perl-devel
Re: F20 System Wide Change: Perl 5.18
On Tue, Aug 6, 2013 at 2:43 PM, Adam Williamson wrote: > repoquery turns up these things that require either > perl(:MODULE_COMPAT_5.16.0) , perl(:MODULE_COMPAT_5.16.1) , > perl(:MODULE_COMPAT_5.16.2) or perl(:MODULE_COMPAT_5.16.3) : > > polymake-0:2.12-7.fc20.x86_64 > Don't try to rebuild this one. Upstream apparently knows how to fix it for perl 5.18, but I haven't managed to tease that knowledge out of them yet. Real soon now -- Jerry James http://www.jamezone.org/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On Tue, 2013-08-06 at 13:39 -0700, Adam Williamson wrote: > On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: > > On Thu, 13 Jun 2013 13:23:51 + (UTC) > > Petr Pisar wrote: > > > > > On 2013-06-12, Kevin Fenzi wrote: > > > > So, there's nothing preventing the side tag and rebuild anytime now > > > > right? 5.18.0 is out, so we could start that work in rawhide?=20 > > > > > > > Currently 5.18.0 does not pass one test when running in mock and koji. > > > (It's because of the terminal usage in tested perl debugger.) We think > > > we could have solved this issue in a few days. > > > > Cool. > > > > > Could you explain how the side tag inheritance works? It inherits > > > everything from rawhide, even builds made after the side tag creation, > > > > yes. > > > > > except packages whose builds have been already made in the side tag. > > > Am I right? That means we still get fresh third-party dependencies > > > from rawhide. > > > > yes. However, there's are several downsides: > > > > - Each side tag adds newrepo tasks which increases load a lot. > > - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, > > then the maintainer has to fix something in rawhide, they would build > > perl-foo-1.0-2 in rawhide and when the side tag was merged back over > > either everyone would get the older one with the bug, or the newer > > one against the old perl. So, it's really important to not take a > > long time using a side tag to avoid this problem as much as possible. > > Seems like this one came true in practice. It seems like a 5.18 rebuild > run was done in a side tag and then merged back into Rawhide. > Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done > prior to the general F20 mass rebuild - so the mass rebuild won out, and > effectively squelched the perl rebuild. > > I'm rebuilding the packages I have installed locally that seem to have > been affected by this, but I'm sure there are others... repoquery turns up these things that require either perl(:MODULE_COMPAT_5.16.0) , perl(:MODULE_COMPAT_5.16.1) , perl(:MODULE_COMPAT_5.16.2) or perl(:MODULE_COMPAT_5.16.3) : perl-DBD-AnyData-0:0.110-6.fc18.noarch perl-MIME-Lite-HTML-0:1.24-4.fc18.noarch perl-Net-Amazon-S3-0:0.53-5.fc18.noarch perl-Padre-0:0.90-6.fc18.noarch perl-SQL-Abstract-Limit-0:0.141-12.fc18.noarch perl-PDL-Graphics-PLplot-0:0.59-3.fc19.x86_64 gpsdrive-0:2.11-15.fc19.x86_64 mojomojo-0:1.08-2.fc19.noarch perl-BackPAN-Index-0:0.42-2.fc19.noarch perl-Bio-SamTools-0:1.35-2.fc19.x86_64 perl-CGI-Application-Plugin-DBIC-Schema-0:0.3-12.fc19.noarch perl-CGI-Application-Plugin-DBIx-Class-0:1.000101-2.fc19.noarch perl-CGI-Application-Structured-0:0.007-8.fc19.noarch perl-CGI-Application-Structured-Tools-0:0.015-4.fc19.noarch perl-CGI-Ex-0:2.38-4.fc19.noarch perl-CGI-FormBuilder-0:3.0501-17.fc19.noarch perl-CHI-0:0.56-1.fc19.noarch perl-CPS-0:0.14-5.fc19.noarch perl-Catalyst-Action-REST-0:1.06-2.fc19.noarch perl-Catalyst-ActionRole-ACL-0:0.07-2.fc19.noarch perl-Catalyst-Authentication-Credential-HTTP-0:1.015-3.fc19.noarch perl-Catalyst-Authentication-Store-DBIx-Class-0:0.1082-13.fc19.noarch perl-Catalyst-Controller-ActionRole-0:0.15-8.fc19.noarch perl-Catalyst-Controller-FormBuilder-0:0.06-8.fc19.noarch perl-Catalyst-Controller-HTML-FormFu-0:0.09004-2.fc19.noarch perl-Catalyst-Devel-0:1.37-4.fc19.noarch perl-Catalyst-Engine-PSGI-0:0.13-7.fc19.noarch perl-Catalyst-Model-Adaptor-0:0.10-5.fc19.noarch perl-Catalyst-Model-DBIC-Schema-0:0.60-4.fc19.noarch perl-Catalyst-Model-XMLRPC-0:0.04-14.fc19.noarch perl-Catalyst-Plugin-Authentication-0:0.10022-1.fc19.noarch perl-Catalyst-Plugin-Authorization-ACL-0:0.15-10.fc19.noarch perl-Catalyst-Plugin-Authorization-Roles-0:0.09-6.fc19.noarch perl-Catalyst-Plugin-Cache-0:0.12-1.fc19.noarch perl-Catalyst-Plugin-PageCache-0:0.31-7.fc19.noarch perl-Catalyst-Plugin-Session-State-Cookie-0:0.17-11.fc19.noarch perl-Catalyst-Plugin-Session-State-URI-0:0.15-9.fc19.noarch perl-Catalyst-Plugin-Session-Store-Cache-0:0.01-10.fc19.noarch perl-Catalyst-Plugin-Session-Store-DBIC-0:0.12-5.fc19.noarch perl-Catalyst-Plugin-Session-Store-Delegate-0:0.06-5.fc19.noarch perl-Catalyst-Plugin-Session-Store-FastMmap-0:0.16-4.fc19.noarch perl-Catalyst-Plugin-Session-Store-File-0:0.18-11.fc19.noarch perl-Catalyst-Plugin-Static-Simple-0:0.30-4.fc19.noarch perl-Catalyst-Plugin-Unicode-0:0.93-9.fc19.noarch perl-Catalyst-Plugin-Unicode-Encoding-0:1.8-2.fc19.noarch perl-Catalyst-View-Component-SubInclude-0:0.10-8.fc19.noarch perl-Catalyst-View-Email-0:0.33-1.fc19.noarch perl-Catalyst-View-Mason-0:0.18-11.fc19.noarch perl-Catalyst-View-PDF-Reuse-0:0.04-6.fc19.noarch perl-CatalystX-Profile-0:0.02-5.fc19.noarch perl-CatalystX-SimpleLogin-0:0.18-3.fc19.noarch perl-Class-DBI-AbstractSearch-0:0.07-19.fc19.noarch perl-DBICx-TestDatabase-0:0.04-6.fc19.noarch perl-DBIx-Class-0:0.08206-1.fc19.noarch perl-DBIx-Class-Cursor-Cached-0:1.001002-5.fc19.noarch perl-DBIx-Class-DateTime-Epoch-0:0.10-2.fc19.noa
Re: F20 System Wide Change: Perl 5.18
On Thu, 2013-06-13 at 11:10 -0600, Kevin Fenzi wrote: > On Thu, 13 Jun 2013 13:23:51 + (UTC) > Petr Pisar wrote: > > > On 2013-06-12, Kevin Fenzi wrote: > > > So, there's nothing preventing the side tag and rebuild anytime now > > > right? 5.18.0 is out, so we could start that work in rawhide?=20 > > > > > Currently 5.18.0 does not pass one test when running in mock and koji. > > (It's because of the terminal usage in tested perl debugger.) We think > > we could have solved this issue in a few days. > > Cool. > > > Could you explain how the side tag inheritance works? It inherits > > everything from rawhide, even builds made after the side tag creation, > > yes. > > > except packages whose builds have been already made in the side tag. > > Am I right? That means we still get fresh third-party dependencies > > from rawhide. > > yes. However, there's are several downsides: > > - Each side tag adds newrepo tasks which increases load a lot. > - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, > then the maintainer has to fix something in rawhide, they would build > perl-foo-1.0-2 in rawhide and when the side tag was merged back over > either everyone would get the older one with the bug, or the newer > one against the old perl. So, it's really important to not take a > long time using a side tag to avoid this problem as much as possible. Seems like this one came true in practice. It seems like a 5.18 rebuild run was done in a side tag and then merged back into Rawhide. Unfortunately, quite a lot of the 5.18 rebuilds seem to have been done prior to the general F20 mass rebuild - so the mass rebuild won out, and effectively squelched the perl rebuild. I'm rebuilding the packages I have installed locally that seem to have been affected by this, but I'm sure there are others... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: F20 System Wide Change: Perl 5.18
On 06/13/2013 03:23 PM, Petr Pisar wrote: > AFAIK, ld-linux.so prevents from this by reusing the first libperl.so addreses. However I have no idea if this is true for dlopen(). Do we get two interpreters with two sets of global variables? I'm not sure if it's good—it tends to cause random crashes if code from one copy is run on data generated by the other copy. It's not even necessary to have inconsistent global state, even totally innocuous things like if (ptr != &global_constant) free(ptr); break horribly. That being said, I used to dlopen(RTLD_GLOBAL) the libperl DSO in a JNI library for Java, and it appeared to work. But that was on Debian, and there, XS modules didn't link against libperl.so for performance reasons. -- Florian Weimer / Red Hat Product Security Team -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: F20 System Wide Change: Perl 5.18
On Thu, 13 Jun 2013 13:23:51 + (UTC) Petr Pisar wrote: > On 2013-06-12, Kevin Fenzi wrote: > > So, there's nothing preventing the side tag and rebuild anytime now > > right? 5.18.0 is out, so we could start that work in rawhide?=20 > > > Currently 5.18.0 does not pass one test when running in mock and koji. > (It's because of the terminal usage in tested perl debugger.) We think > we could have solved this issue in a few days. Cool. > Could you explain how the side tag inheritance works? It inherits > everything from rawhide, even builds made after the side tag creation, yes. > except packages whose builds have been already made in the side tag. > Am I right? That means we still get fresh third-party dependencies > from rawhide. yes. However, there's are several downsides: - Each side tag adds newrepo tasks which increases load a lot. - If you rebuild perl-foo-1.0-1 in the side tag against the new perl, then the maintainer has to fix something in rawhide, they would build perl-foo-1.0-2 in rawhide and when the side tag was merged back over either everyone would get the older one with the bug, or the newer one against the old perl. So, it's really important to not take a long time using a side tag to avoid this problem as much as possible. > I'd like to solve the linking issue before the mass rebuild, because > mass rebuild triggers tests on all packages which allows to find > introduced problems very soon. I'd not like to see after bigger delay > that the explicit linkage caused some serious problem and we needed to > revert the linkage change, meaning to mass-rebuild again. Absolutely. ...snip good info about double open issue... Yeah, I don't have an answer there, but hopefully some smarter folks will chime in on it. I agree we should come up with a definite plan before moving forward with mass rebuilding. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: F20 System Wide Change: Perl 5.18
On 2013-06-12, Kevin Fenzi wrote: > So, there's nothing preventing the side tag and rebuild anytime now > right? 5.18.0 is out, so we could start that work in rawhide?=20 > Currently 5.18.0 does not pass one test when running in mock and koji. (It's because of the terminal usage in tested perl debugger.) We think we could have solved this issue in a few days. Could you explain how the side tag inheritance works? It inherits everything from rawhide, even builds made after the side tag creation, except packages whose builds have been already made in the side tag. Am I right? That means we still get fresh third-party dependencies from rawhide. I'd like to solve the linking issue before the mass rebuild, because mass rebuild triggers tests on all packages which allows to find introduced problems very soon. I'd not like to see after bigger delay that the explicit linkage caused some serious problem and we needed to revert the linkage change, meaning to mass-rebuild again. > On the dlopen thing, what happens if you hit double-linking there? > Thats just a runtime error, not a build time error? is there any tools > we have to detect this?=20 > Almost each Perl package runs tests at build time. I do not say we check the double dlopen() with them, but we can be pretty sure it does not break at least the standard single dlopen() in the whole Perl ecosystem. Does anybody have experience with the multiple dlopen()? As far as I can see none distrubutor yet tried to solve this problem. E.g. Debian patches third-party consumers (like the slapd) to dlopen() with RTLD_GLOBAL. On the other hand this is not recommended due to possible symbol clashes as most of plug-ins have poor name-space discipline. (Do you rembember the libdb-5 overhaul in Fedora forced by random crashes if a different DSOs expected different libdb?) I will try to explain the problem with perl here: We have libperl.so. This is the Perl interpreter. Perl allows to write Perl modules in native code. This is implemented as DSOs dlopen-ed by libperl.so. These modules call functions defined in libperl.so. So far everything is good. However perl allows to be embedded into another application. That means libperl.so gets linked at compilation to your application. This is still Ok. However some bigger applications, like slapd or snmpd, wrap the embedded perl into yet another plug-in. A plug-in from point of view of the application. E.g. slapd dlopen-s its back_perl.so plug-in with RTLD_LOCAL. back_perl.so is linked to libperl.so, and libperl.so dlopen-s other Perl modules again as I described above. But because of RTLD_LOCAL in the slapd, libperl.so symbols are not visible to Perl modules thus the perl interpreter receives linking error while dlopen-ing the module. Deemed solution is to explicitly link all Perl modules to libperl.so at compilation (this is not true yet). This should ensure that dlopen() in the interpreter makes libperl.so symbols available to the module because it will link libperl.so at run-time again. But now have second instance of libperl.so getting into the address space. AFAIK, ld-linux.so prevents from this by reusing the first libperl.so addreses. However I have no idea if this is true for dlopen(). Do we get two interpreters with two sets of global variables? That would be good. And what about thread-safety? Frankly, I have to admid I haven't probed it yet. I will. But before that I'd like to know whether this is feasible way or I'm completly wrong. I belive that somebody had to try this path before me. -- Petr -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: F20 System Wide Change: Perl 5.18
On Wed, 12 Jun 2013 19:48:16 +0200 Miloslav Trmač wrote: > On Wed, Jun 12, 2013 at 7:38 PM, Kevin Fenzi wrote: > > So, there's nothing preventing the side tag and rebuild anytime now > > right? 5.18.0 is out, so we could start that work in rawhide? > > Is there such a rush? I don't think one week to discuss this and > figure out various corner cases is too much. For example... To clarify I don't mean to rush things, but if everything is ready the sooner the better, IMHO. Otherwise you could push up against branching date. The sooner large changes land, the sooner they can get stablized. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: F20 System Wide Change: Perl 5.18
On Wed, Jun 12, 2013 at 7:38 PM, Kevin Fenzi wrote: > So, there's nothing preventing the side tag and rebuild anytime now > right? 5.18.0 is out, so we could start that work in rawhide? Is there such a rush? I don't think one week to discuss this and figure out various corner cases is too much. For example... > On the dlopen thing, what happens if you hit double-linking there? Could you describe in more detail what is the "double-linking" you are concerned about, please? Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: F20 System Wide Change: Perl 5.18
On Tue, 11 Jun 2013 10:56:32 +0200 Jaroslav Reznik wrote: > = Proposed System Wide Change: Perl 5.18 = > https://fedoraproject.org/wiki/Changes/perl5.18 > > Change owner(s): Petr Písař , Jitka Plesníková > > > A new perl 5.18 version brings a lot of changes done over a year of > development. See [1] for more details. > > == Detailed description == > New perl is released every year and updates containing mainly bug > fixes follow during the year. The 5.18.0 version is stable release > this year and 5.18.1 and subsequent revisions will follow during the > year to fix issues introduced in 5.18.0. > > Because of change in dlopen(3) from RTLD_GLOBAL to RTLD_LOCAL (bug > #960048) [2], we will try to link all native perl module DSOs to > libperl.so explicitly. This item needs discussion. Current status > does not allow to use perl interpreter via dlopen() from other > programs (like slapd, snmpd), but proposed change can bring other > unforeseen difficulties (like double-linking). So, there's nothing preventing the side tag and rebuild anytime now right? 5.18.0 is out, so we could start that work in rawhide? On the dlopen thing, what happens if you hit double-linking there? Thats just a runtime error, not a build time error? is there any tools we have to detect this? Otherwise looks good to me. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
F20 System Wide Change: Perl 5.18
= Proposed System Wide Change: Perl 5.18 = https://fedoraproject.org/wiki/Changes/perl5.18 Change owner(s): Petr Písař , Jitka Plesníková A new perl 5.18 version brings a lot of changes done over a year of development. See [1] for more details. == Detailed description == New perl is released every year and updates containing mainly bug fixes follow during the year. The 5.18.0 version is stable release this year and 5.18.1 and subsequent revisions will follow during the year to fix issues introduced in 5.18.0. Because of change in dlopen(3) from RTLD_GLOBAL to RTLD_LOCAL (bug #960048) [2], we will try to link all native perl module DSOs to libperl.so explicitly. This item needs discussion. Current status does not allow to use perl interpreter via dlopen() from other programs (like slapd, snmpd), but proposed change can bring other unforeseen difficulties (like double-linking). [1] http://search.cpan.org/dist/perl-5.18.0/pod/perldelta.pod [2] https://bugzilla.redhat.com/show_bug.cgi?id=960048 ___ devel-announce mailing list devel-annou...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
