Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Tue, 03 Jun 2008, Brendan O'Dea wrote:
> > I tried this but it doesn't work. The variable needs to be set before perl
> > is executed apparently. So you'd need some trick so that the script exec
> > itself a second time but with the environment variable set.
>
> Really? Something like this fails?
>
> BEGIN {
> local $ENV{PERL_DL_NONLAZY} = 1;
> eval {
> require "Locale::Gettext";
> Locale::Gettext->import;
> };
> if ($@) {
> # handle
> }
> }
I wanted to fix it for all scripts that use Dpkg::Gettext (there's dpkg-divert
too) and as such I tried to add the "$ENV{PERL_DL_NONLAZY} = 1;" in
/usr/share/perl5/Dpkg/Gettext.pm:
--- a/scripts/Dpkg/Gettext.pm
+++ b/scripts/Dpkg/Gettext.pm
@@ -7,6 +7,7 @@ use strict;
use warnings;
BEGIN {
+local $ENV{PERL_DL_NONLAZY} = 1;
eval 'use Locale::gettext';
if ($@) {
eval q{
This doesn't work and I don't understand why.
However, now I tried to add the line directly to update-alternatives and it
worked:
--- a/scripts/update-alternatives.pl
+++ b/scripts/update-alternatives.pl
@@ -1,5 +1,9 @@
#!/usr/bin/perl --
+BEGIN {
+$ENV{PERL_DL_NONLAZY} = 1;
+}
+
use strict;
use warnings;
Can anyone explain me why?
To test I simply dpkg --force-depends --unpack
liblocale-gettext-perl_1.05-1_i386.deb (the package from etch) on a sid machine.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
[-doughera]
On Tue, Jun 3, 2008 at 12:11 AM, Raphael Hertzog <[EMAIL PROTECTED]> wrote:
> On Mon, 02 Jun 2008, Brendan O'Dea wrote:
>> Modifying update-alternatives to set $ENV{PERL_DL_NONLAZY} would
>> appear to be the most appropriate solution here.
>
> I tried this but it doesn't work. The variable needs to be set before perl
> is executed apparently. So you'd need some trick so that the script exec
> itself a second time but with the environment variable set.
Really? Something like this fails?
BEGIN {
local $ENV{PERL_DL_NONLAZY} = 1;
eval {
require "Locale::Gettext";
Locale::Gettext->import;
};
if ($@) {
# handle
}
}
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Mon, 02 Jun 2008, Brendan O'Dea wrote:
> On Sat, May 31, 2008 at 4:41 AM, Raphael Hertzog <[EMAIL PROTECTED]> wrote:
> > That's wrong. The "use Locale::Gettext" is protected by eval {} and
> > update-alternatives works well when the eval fails... as it subsitutes
> > gettext() with a simple sub return { [EMAIL PROTECTED] }.
> > That's why the initial request was to disable lazy symbol loading inside
> > eval.
>
> I was aware of the optional loading of Locale::gettext, but don't
> believe that changing the dlopen behaviour upstream is the right
> solution.
>
> This is a Debian-specific problem, and moreover one which occurs in a
> very narrow set of circumstances: programs/modules which
> opportunistically load a module (hence circumventing the dependency
> model) across upgrades to perl which change the binary ABI (which are
> rare).
>
> Modifying update-alternatives to set $ENV{PERL_DL_NONLAZY} would
> appear to be the most appropriate solution here.
I tried this but it doesn't work. The variable needs to be set before perl
is executed apparently. So you'd need some trick so that the script exec
itself a second time but with the environment variable set.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Sat, May 31, 2008 at 4:41 AM, Raphael Hertzog <[EMAIL PROTECTED]> wrote:
> On Fri, 30 May 2008, Andy Dougherty wrote:
>> On Sat, 31 May 2008, Brendan O'Dea wrote:
>> > Not really. Adding a version into the path simply changes the failure
>> > from an issue loading the .so (module is unusable) to one where the
>> > module is not found at all (module is unusable).
> [...]
> That's wrong. The "use Locale::Gettext" is protected by eval {} and
> update-alternatives works well when the eval fails... as it subsitutes
> gettext() with a simple sub return { [EMAIL PROTECTED] }.
> That's why the initial request was to disable lazy symbol loading inside
> eval.
I was aware of the optional loading of Locale::gettext, but don't
believe that changing the dlopen behaviour upstream is the right
solution.
This is a Debian-specific problem, and moreover one which occurs in a
very narrow set of circumstances: programs/modules which
opportunistically load a module (hence circumventing the dependency
model) across upgrades to perl which change the binary ABI (which are
rare).
Modifying update-alternatives to set $ENV{PERL_DL_NONLAZY} would
appear to be the most appropriate solution here.
--bod
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Fri, 30 May 2008, Andy Dougherty wrote:
> On Sat, 31 May 2008, Brendan O'Dea wrote:
> > On Fri, May 16, 2008 at 10:31 PM, Andy Dougherty <[EMAIL PROTECTED]> wrote:
> > > On Thu, 15 May 2008, Niko Tyni wrote:
> > >> In a nutshell, it's possible during the upgrade to temporarily end up
> > >> in a state where perl is still 5.8.8, but some XS modules are already
> > >> the new ones (ie. built for 5.10.0).
>
> > > Changing 5.10.0's $Config{vendorarch} to include the string "5.10" (much
> > > as is done for $Config{archlib} aleady) might solve this problem. Or at
> > > least it might cause the failure to appear earlier and in a way that is
> > > easier to trap gracefully.
> >
> > Not really. Adding a version into the path simply changes the failure
> > from an issue loading the .so (module is unusable) to one where the
> > module is not found at all (module is unusable).
>
> Agreed. In neither case does it actually work.
That's wrong. The "use Locale::Gettext" is protected by eval {} and
update-alternatives works well when the eval fails... as it subsitutes
gettext() with a simple sub return { [EMAIL PROTECTED] }.
So if the module is missing, update-alternatives works.
If the module is for the wrong perl version, then the problem is not catched
by eval {} due to the lazy symbol loading and the script is killed as soon
as it tries to use a symbol from the library.
That's why the initial request was to disable lazy symbol loading inside
eval.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Sat, 31 May 2008, Brendan O'Dea wrote:
> On Fri, May 16, 2008 at 10:31 PM, Andy Dougherty <[EMAIL PROTECTED]> wrote:
> > On Thu, 15 May 2008, Niko Tyni wrote:
> >> we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
> >> binary incompatibility in the XS module interface is biting us in an
> >> unexpected way.
> >>
> >> In a nutshell, it's possible during the upgrade to temporarily end up
> >> in a state where perl is still 5.8.8, but some XS modules are already
> >> the new ones (ie. built for 5.10.0).
> > Changing 5.10.0's $Config{vendorarch} to include the string "5.10" (much
> > as is done for $Config{archlib} aleady) might solve this problem. Or at
> > least it might cause the failure to appear earlier and in a way that is
> > easier to trap gracefully.
>
> Not really. Adding a version into the path simply changes the failure
> from an issue loading the .so (module is unusable) to one where the
> module is not found at all (module is unusable).
Agreed. In neither case does it actually work. However, the case of not
finding the module at all is probably easier to trap at compile time.
Because of lazy symbol loading, loading the wrong version of the module
sometimes appears to work -- until you actually try to use it.
--
Andy Dougherty [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Fri, May 16, 2008 at 10:31 PM, Andy Dougherty <[EMAIL PROTECTED]> wrote:
> On Thu, 15 May 2008, Niko Tyni wrote:
>> we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
>> binary incompatibility in the XS module interface is biting us in an
>> unexpected way.
>>
>> In a nutshell, it's possible during the upgrade to temporarily end up
>> in a state where perl is still 5.8.8, but some XS modules are already
>> the new ones (ie. built for 5.10.0).
>
> The default perl installation was deliberately designed to avoid just this
> problem by including a version-specific directory-path in
> $Config{vendorarch}. That way perl-5.8.8 would not try to load
> incompatible perl-5.10.0 modules. Debian's build script deliberately does
> not include a version-specific $Config{vendorarch}. It might be worth
> reviewing that decision.
I chose many releases ago (5.6.0) not to include a version in the
vendor path, since the packaging system is capable of ensuring that
the packages in those directories are correct for the current perl
version (more detail in the Debian Perl Policy). This has worked
pretty well.
One issue which should be highlighted is that no such guarantees are
made *during* the upgrade process. In fact Debian policy is pretty
explicit as to what is valid for maintainer scripts to invoke: either
the package providing the binary must be flagged as "Essential" (in
which case there are additional requirements placed on the package
such that it is functional even when not configured) or a
Pre-Dependency must be declared (ensuring that dpkg with not only
unpack, but will configure said dependency package before attempting
to configure the dependant).
So... after that legalese the actual problem in this case is that dpkg
(an Essential program) has a component: update-alternatives which is
not functioning when un-configured. Bad. But a problem with dpkg,
not perl.
> Changing 5.10.0's $Config{vendorarch} to include the string "5.10" (much
> as is done for $Config{archlib} aleady) might solve this problem. Or at
> least it might cause the failure to appear earlier and in a way that is
> easier to trap gracefully.
Not really. Adding a version into the path simply changes the failure
from an issue loading the .so (module is unusable) to one where the
module is not found at all (module is unusable).
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
(cutting off P5P from the cc: list, since that's getting off topic) 2008/5/18 Raphael Hertzog <[EMAIL PROTECTED]>: > As a current maintainer of dpkg, I can tell you that dpkg2 has never been > anything else than vaporware. I know of nobody working on a dpkg rewrite. Ah, pity, there was some interesting ideas there IIRC. But it's a long time since I've ever dabbled in dependency solvers. (I used to maintain rpm and urpmi for Mandriva.) >> > Hm, one option could be to have the packaging system (dpkg) set >> > PERL_DL_NONLAZY=1 for all preinst script invocations... >> >> I'm not 100% sure that's a good idea. > > Can you elaborate on the reasons? Are there stuff which could break? It's mere intuition. Perl modules are loaded, and often only one or two functions are used, so .so loading may not need to take place. You want to make sure they work even in strange installation circumstances. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Sun, 18 May 2008, Rafael Garcia-Suarez wrote: > > - the circumstances where this shows up in the Debian context are > > somewhat a corner case where 'preinst upgrade' scripts that need XS > > modules may get run with a new XS module but the old perl. The > > package dependencies are not yet guaranteed to be met at the > > 'preinst' time, hence the 'eval "require Foo"' construct. > > I've heard that dpkg2 was going to support pre-inst and post-uninst > dependencies. (Just like rpm does, which creates approximately the same > number of problems than it solves) As a current maintainer of dpkg, I can tell you that dpkg2 has never been anything else than vaporware. I know of nobody working on a dpkg rewrite. > > Hm, one option could be to have the packaging system (dpkg) set > > PERL_DL_NONLAZY=1 for all preinst script invocations... > > I'm not 100% sure that's a good idea. Can you elaborate on the reasons? Are there stuff which could break? Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
2008/5/17 Niko Tyni <[EMAIL PROTECTED]>: > How about just documenting this a bit more with something like the > attached patch? I still think it's unexpected that 'eval "require Foo"' > isn't enough to trap the error. Yes. Thanks, applied as change #33848. > Some clarifications: > > - I'm not familiar with the reasons for choosing > -Dvendorarch=/usr/lib/perl5, but I'll try to find out. > > - the circumstances where this shows up in the Debian context are > somewhat a corner case where 'preinst upgrade' scripts that need XS > modules may get run with a new XS module but the old perl. The > package dependencies are not yet guaranteed to be met at the > 'preinst' time, hence the 'eval "require Foo"' construct. I've heard that dpkg2 was going to support pre-inst and post-uninst dependencies. (Just like rpm does, which creates approximately the same number of problems than it solves) > Hm, one option could be to have the packaging system (dpkg) set > PERL_DL_NONLAZY=1 for all preinst script invocations... I'm not 100% sure that's a good idea. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Sat, May 17, 2008 at 12:02:31AM +0100, Nicholas Clark wrote:
> Changing this doesn't feel right. I suspect that Gisle's reasoning is part of
> my gut feeling on this. Lazy has been the default since 5.002 (1996), and
> prior to that was the only option. I never remember this being a problem.
Thanks to everybody for the comments. I get the point about loading a
library with some unresolved symbols and only using those that resolve,
so the change could break working code.
How about just documenting this a bit more with something like the
attached patch? I still think it's unexpected that 'eval "require Foo"'
isn't enough to trap the error.
Some clarifications:
- I'm not familiar with the reasons for choosing
-Dvendorarch=/usr/lib/perl5, but I'll try to find out.
- the circumstances where this shows up in the Debian context are
somewhat a corner case where 'preinst upgrade' scripts that need XS
modules may get run with a new XS module but the old perl. The
package dependencies are not yet guaranteed to be met at the
'preinst' time, hence the 'eval "require Foo"' construct.
- the only time this has actually showed up so far is in the
Locale::gettext case, and that was solved in the same way as in the 5.6
-> 5.8 transition, namely making the Locale::gettext package pre-depend
on the new perl. However, there may be other similar cases that haven't
been discovered yet.
- the fatal error in this case is "undefined symbol: Perl_Istack_sp_ptr" and
happens in DynaLoader::bootstrap(), when calling the "${module}::bootstrap"
function installed by dl_install_xsub(). This is somewhat different to
the usual "call a function that doesn't resolve" case, but having both
raise an exception in Perl instead of finishing off the process would
certainly be good.
- the difference to the 5.6 -> 5.8 transition is that we have lots more
packages and preinst scripts now, but we don't have a good way to
identify which XS modules are needed by preinst scripts. It may well
be that Locale::gettext is the only one (like it apparently was in the
last transition), but we don't really know that yet.
Hm, one option could be to have the packaging system (dpkg) set
PERL_DL_NONLAZY=1 for all preinst script invocations...
Cheers,
--
Niko Tyni [EMAIL PROTECTED]
diff --git a/pod/perlfunc.pod b/pod/perlfunc.pod
index 7e7692e..66dc1ae 100644
--- a/pod/perlfunc.pod
+++ b/pod/perlfunc.pod
@@ -1562,6 +1562,10 @@ determining whether a particular feature (such as C or C)
is implemented. It is also Perl's exception trapping mechanism, where
the die operator is used to raise exceptions.
+If you want to trap errors when loading an XS module, some problems with
+the binary interface (such as Perl version skew) may be fatal even with
+C unless C<$ENV{PERL_DL_NONLAZY}> is set. See L.
+
If the code to be executed doesn't vary, you may use the eval-BLOCK
form to trap run-time errors without incurring the penalty of
recompiling each time. The error, if any, is still returned in C<$@>.
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Sat, May 17, 2008 at 12:02:31AM +0100, Nicholas Clark wrote: > Perl 5.6.x and Perl 5.8.x are not binary compatible either, yet Debian > upgraded from 5.6.x to 5.8.x without hitting this issue. What changed? > DynaLoader certainly didn't, hence why I'm highly suspicious that changing > it is not the right solution here. I think that there's one "not" too many in there. Oops. Nicholas Clark -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Fri, May 16, 2008 at 03:04:21PM +0200, Rafael Garcia-Suarez wrote:
> 2008/5/16 Gisle Aas <[EMAIL PROTECTED]>:
> > Another objection is that you might simply want to load the module and use
> > some function that does resolve. Making require always fail if some
> > functions doesn't resolve prevent this for working at all. I think the user
> > better set PERL_DL_NONLAZY themselves if that's what they want.
>
> Good point.
Changing this doesn't feel right. I suspect that Gisle's reasoning is part of
my gut feeling on this. Lazy has been the default since 5.002 (1996), and
prior to that was the only option. I never remember this being a problem.
On Fri, May 16, 2008 at 08:31:24AM -0400, Andy Dougherty wrote:
> On Thu, 15 May 2008, Niko Tyni wrote:
>
> > Hi p5p,
> >
> > we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
> > binary incompatibility in the XS module interface is biting us in an
> > unexpected way.
> >
> > In a nutshell, it's possible during the upgrade to temporarily end up
> > in a state where perl is still 5.8.8, but some XS modules are already
> > the new ones (ie. built for 5.10.0).
>
> The default perl installation was deliberately designed to avoid just this
> problem by including a version-specific directory-path in
> $Config{vendorarch}. That way perl-5.8.8 would not try to load
> incompatible perl-5.10.0 modules. Debian's build script deliberately does
> not include a version-specific $Config{vendorarch}. It might be worth
> reviewing that decision.
Perl 5.6.x and Perl 5.8.x are not binary compatible either, yet Debian
upgraded from 5.6.x to 5.8.x without hitting this issue. What changed?
DynaLoader certainly didn't, hence why I'm highly suspicious that changing
it is not the right solution here.
Nicholas Clark
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
On Thu, 15 May 2008, Niko Tyni wrote:
> Hi p5p,
>
> we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
> binary incompatibility in the XS module interface is biting us in an
> unexpected way.
>
> In a nutshell, it's possible during the upgrade to temporarily end up
> in a state where perl is still 5.8.8, but some XS modules are already
> the new ones (ie. built for 5.10.0).
The default perl installation was deliberately designed to avoid just this
problem by including a version-specific directory-path in
$Config{vendorarch}. That way perl-5.8.8 would not try to load
incompatible perl-5.10.0 modules. Debian's build script deliberately does
not include a version-specific $Config{vendorarch}. It might be worth
reviewing that decision.
Changing 5.10.0's $Config{vendorarch} to include the string "5.10" (much
as is done for $Config{archlib} aleady) might solve this problem. Or at
least it might cause the failure to appear earlier and in a way that is
easier to trap gracefully.
> Details on the Debian issue can be found at ,
--
Andy Dougherty [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#479711: eval "require Foo" with binary-incompatible XS modules
Hi p5p,
we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
binary incompatibility in the XS module interface is biting us in an
unexpected way.
In a nutshell, it's possible during the upgrade to temporarily end up
in a state where perl is still 5.8.8, but some XS modules are already
the new ones (ie. built for 5.10.0).
The package scripts try to fail gracefully in situations like this by
doing effectively an 'eval "require Locale::gettext"' or the like.
Unfortunately this doesn't work: the require succeeds but the dynamic
linker kills the script off immediately afterwards because of a
symbol lookup error.
This can be circumvented by setting $ENV{PERL_DL_NONLAZY}=1 before
the eval, so it's arguably a feature. I'm wondering if it would make
sense for Perl to do this automatically inside all eval statements.
After all, the user is using the eval because he wants to catch
exceptions immediately.
If this is obviously too intrusive or otherwise broken, please hit me
with a cluebat. In that case, I think the eval entry in perlfunc.pod
could mention the issue.
I can take a shot at a patch for either case, I'd just like to know
which is the right thing to do.
Details on the Debian issue can be found at ,
cc'd as [EMAIL PROTECTED] .
# perl -e 'local $ENV{PERL_DL_NONLAZY}=0; eval "require Locale::gettext"; print
"got: $@" if $@; exit 0'; echo $?
perl: symbol lookup error: /usr/lib/perl5/auto/Locale/gettext/gettext.so:
undefined symbol: Perl_Istack_sp_ptr
127
# perl -e 'local $ENV{PERL_DL_NONLAZY}=1; eval "require Locale::gettext"; print
"got: $@" if $@; exit 0'; echo $?
got: Can't load '/usr/lib/perl5/auto/Locale/gettext/gettext.so' for module
Locale::gettext: /usr/lib/perl5/auto/Locale/gettext/gettext.so: undefined
symbol: Perl_Imarkstack_ptr_ptr at /usr/lib/perl/5.8/DynaLoader.pm line 225.
at (eval 1) line 3
Compilation failed in require at (eval 1) line 3.
0
Cheers,
--
Niko Tyni [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
