Re: Old, bundled cruft in PAR builds
On 9/20/06, Chris Dolan <[EMAIL PROTECTED]> wrote: On Sep 18, 2006, at 5:09 AM, Steffen Mueller wrote: > a) link in C libzip to do the extraction. That way, parl wouldn't > need any Perl at all just for the extraction of the .par Just a small note: this would have the positive side effect of making PAR binaries a little smaller, since the bootstrapper would be smaller and more of the Perl content would be inside the .zip container. Definitely, typically compression is better than 50% for the perl executable (in case of a statically linked perl exe) or the perl DLL (eg 960K -> 370K on Solaris or 790K -> 400K on Windows). Also it would make it easier just to see what has been packed into the pp'ed executable - "unzip -l" would tell the whole story. Cheers, Roderich
Re: Old, bundled cruft in PAR builds
On Sep 18, 2006, at 5:09 AM, Steffen Mueller wrote: a) link in C libzip to do the extraction. That way, parl wouldn't need any Perl at all just for the extraction of the .par Just a small note: this would have the positive side effect of making PAR binaries a little smaller, since the bootstrapper would be smaller and more of the Perl content would be inside the .zip container. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: Old, bundled cruft in PAR builds
Hi Steffen,
Roderich, you showed me a proof-of-concept reimplementation using self
extracting libzip. Would you think you could modify the parl building
process to use libzip instead of the corresponding Perl XS modules? Or
I certainly would like to expand on my proof-of-concept, at least to
the point that others can toy with it. But I definitely won't have time to do so
for at least three weeks. For those who like to comment,
I've outlined the idea below.
Creating an executable foo.exe from foo.pl would go like this:
(1) create foo.par as we do now with
pp -B -p -a "/usr/bin/perl;bin/perl" -o foo.par foo.pl
i.e also add the perl executable (as bin/perl)
(and perl shared library if necessary);
of course, script/main.pl (that gets implicitly
generated by pp, packed into foo.par and finally executed)
would have to be different, see below
(2) create a self extracting zip archive with a hacked version of unzipsfx.exe
(get it from www.info-zip.org) with simply
cat hacked_unzipsfx.exe foo.par > foo.exe
The hacked unzipsfx.exe would need to do the following
(the unmodified unzipsfx.exe is actually a stripped down version of unzip):
(0) keep original @argv and dont't process it (unmodified unzipsfx.exe
accepts "-d dir" to extract into dir instead of cwd, etc)
(1) determine name of cache directory (PAR_TEMP);
if it already exists, continue with (4)
(2) create cache directory and set it as the directory to extract the archive
into
(3) run the unmodified main part of unzipsfx.exe which extracts _everything_
in foo.par into PAR_TEMP
(4) set up some environment variables (e.g PAR_TEMP, LD_LIBRARY_PATH)
(5) finally exec($PAR_TEMP/bin/perl, $PAR_TEMP/script/main.pl, @argv)
script/main.pl would look like:
BEGIN { @INC = ("$ENV{PAR_TEMP}/lib"); }
do "$ENV{PAR_TEMP}/script/foo.pl";
This doesn't even need PAR.pm (and modules it depends on) at runtime,
because we extract everything. A possible refinement would only extract
perl.exe and the core modules needed to get PAR.pm going, the rest of the
modules would then be loaded via PAR from the executable (which is
also a PAR).
Cheers, Roderich
Re: Old, bundled cruft in PAR builds
Hi Roderich, hi list, I'm sorry that this reply took so long. I wanted to get Audrey's input on this and I was away for a few days. Roderich Schupp schrieb: [...] Another observation point: when you install PAR on Debian (package libpar-perl), parl doesn't have any modules packed in (that's because the executable gets stripped during Debian packaging which discards everything simply appended to the executable). But you can pp scripts just fine, the necessary modules are taken from the environment at pp time. (Of course, this Debian parl is no longer a self-contained perl loader, it only works when all necessary modules are present on the system.) Audrey and I can see two ways to improve the situation. a) link in C libzip to do the extraction. That way, parl wouldn't need any Perl at all just for the extraction of the .par b) add an additional loader, say "pp-parl" which is akin to debian's stripped parl. That pp-parl would be used by pp. Both solutions would solve the binary incompatibility issue which is bound to crop up a lot in the future still. Solution a) might involve some performance benefit and just feels "Right". My problem is that my C is weak and I doubt I could implement any of the two solutions. Roderich, you showed me a proof-of-concept reimplementation using self extracting libzip. Would you think you could modify the parl building process to use libzip instead of the corresponding Perl XS modules? Or if not, would somebody else like to jump in and help? Of course, if you (plural) think the libzip approach is too difficult to implement or just not worth the effort, I'd also be thrilled to see the problem solved using b). Steffen
Re: Old, bundled cruft in PAR builds
As most of you probably know, various modules (some of them using XS) are built into parl(.exe), the PAR binary loader which is prepended to any binary executables produced by pp, at build time. I've just conducted a litle experiment and now I'm not so sure that this description of the workings of pp is correct... Anyway I used "strings -a parl" to determine what module versions had been packed into parl. For Compress::Zlib I found 1.41 for both the Perl module (Compress/Zlib.pm) and the XS part (Compress/Zlib.so). That's what was installed at the time I build PAR. Then I packed a one-liner: pp -o foo.exe -e 'print $Compress::Zlib::VERSION;' Running foo.exe prints "1.41" and inspection of foo.exe (again with strings -a) shows both parts of Compress::Zlib at 1.41. Next I upgraded Compress::Zlib to 1.42, but didn't rebuild PAR, pp'ed the one liner again and ran it: Compress::Zlib object version 1.41 does not match bootstrap parameter 1.42 at /homes/schuppr/lib/sun4-solaris/Compress/Zlib.pm line 100. Compilation failed in require at /homes/schuppr/lib/Archive/Zip.pm line 24. BEGIN failed--compilation aborted at /homes/schuppr/lib/Archive/Zip.pm line 24. Compilation failed in require at -e line 318. Inspection shows that the contained Compress/Zlib.pm is now at 1.42, but Compress/Zlib.so is still at 1.41, hence the "bootstrap version mismatch". This shows that some stuff in a pp'ed exe gets taken from the Perl environment at pp time while other stuff is taken from parl. Note: I used the PAR version at hand: 0.941. Another observation point: when you install PAR on Debian (package libpar-perl), parl doesn't have any modules packed in (that's because the executable gets stripped during Debian packaging which discards everything simply appended to the executable). But you can pp scripts just fine, the necessary modules are taken from the environment at pp time. (Of course, this Debian parl is no longer a self-contained perl loader, it only works when all necessary modules are present on the system.) Cheers, Roderich
Re: Old, bundled cruft in PAR builds
Hi! I am using PAR extensively, but I do not know anything about the internals (I always meant to, but I haven't found the time yet...). I always make sure to re-compile PAR after I've updated the modules I know it depends on (this is only an option because I compile eeverything myself). I think one thing which might help would be to add an exhaustive list of the modules PAR really depends on (I mean, something like 'pp --list-dependency' or anything smarter which would write out such a list). I would say, having PAR actively check the modules version when it's called, and issue a warning if it finds out that there is a version mismatch would be quite handy, but... These are just some random thoughts on the topic (and very self-centered at that)... Philippe Steffen Mueller wrote: > > Hi list, > > please take thirty seconds to read this: > http://rt.cpan.org/Public/Bug/Display.html?id=21459 > > As most of you probably know, various modules (some of them using XS) > are built into parl(.exe), the PAR binary loader which is prepended to > any binary executables produced by pp, at build time. > > This is a problem for several reasons. One is that if I install a new > version of some module, I either have to recompile PAR or expect a > random version to be loaded. (Not really random, but random from a user > point of view.) > > Another, possibly bigger problem is that people install PAR via ppm or > as .par binaries. Those binaries have the versions of modules that the > producer of the binary happened to have installed. An installed Perl and > the Perl PAR was built with might be incompatible or any bundled modules > might be. (As is the case in the above ticket.) > > What can we do about this? If we cannot "fix" this, how can we make > users more aware of this? Where in the documentation does this caveat > belong? Where on par.perl.org? > > Thanks for your input, > Steffen
