Re: cygwin patches integrating back into standard gnu
See, to build a shared lib, you really really need to use libtool-devel, which is libtool-1.5, and which requires automake 1.5.0 and autoconf 2.50. However, those packages are just now -- after 1.5 years -- coming into widespread use, because 1) autoconf 2.5x is in some ways incompatible with autoconf-2.13 -- which means that an upstream package maintainer has to decide Okay, everybody who hacks on my package now must install autoconf-2.5x on their system. But then each developer also must make a decision -- Hmm. I can only install autoconf-2.13 OR 2.5x but not both. These 5 packages I like to hack on require 2.13. Those 2 require 2.5x. Shall I switch to ac-2.5x and stop hacking on the 5 old packages? So that's why many (upstream) maintainers have been loathe to 'make the switch' -- and why some of our patches are large. A two-line change to configure.in may lead to many thousands of changes in configure after re-autoconfing with 2.5x. that's just silly. Gnu builds are uniformly prefix-friendly, and there was a simple way to 'make a development environment' for any given platform, then you could have your autoconf-2.13 (old) environment, and your autoconf-2.5 (new) environment based on path. And you would hack on one or the other based on what environment you were in. It'd probably be easier to port between the two, too. Which - I might add, cygwin does *not* lend itself to given that each build.sh script that I saw had hard-coded paths in it for building to /usr/bin, non-standard steps for install, etc. If it built clean in whatever prefix you desired, you could do a little substitution trick, to get binary builds for *any* path: 1) substitute (in binary files) any prefix with a different prefix, and pad it with null characters ('/tmp/very/long/prefix' becomes '/my/prefix...' where . is \0) 2) substitute (in text files) any prefix with a different prefix minus the null padding. and hence have more than one cygwin environment per cygwin.dll. It'd probably be wise to integrate this with setup.exe so you would have the option to install packages in a non-standard place. 2) Now, multiply that by automake-1.4p5 vs automake-1.7.5, and libtool.m4/ltmain.sh from libtool-1.5 vs. libtool.m4/ltconfig/ltmain.sh from libtool-1.4.3. Ok, one question... If I'm going to temporarily maintain these patches, it sounds like I'm going to need to only apply them with cygwin builds, right? And that they may not fit with older gnu builds? 3) Things are slowly getting better. Some platforms are now finally providing mechanisms where both autoconf-2.13 and autoconf-2.5x can coexist. (Cygwin has been doing this for years, but Red Hat et al took a little longer) Plus, every week that goes by, another upstream maintainer takes the plunge -- opening the way for our patches to go back. This trend is now (finally) accelerating. well, its nice to see that there are formal mechanisms, but the method for doing this (two environments based on multiple paths) has been around for a while.. Anyways, I could (or someone could) modify it so that, as an option, the patches within are sent to the appropriate mailing list for inclusion. I would think that such a matrix would be helpful in general, as well as a centralized user which could be a conduit for submitting patches to the right place. (which to me is a lot better idea than everyone using the script to send the same patch over and over) But 400k of patches seems just a bit high. Oh god no. Automated patch-spam to mailing lists? I can't think of a better way to ensure that our patches are rejected. I think you misunderstand me. You would: a) have a centralized account, call it 'cygwin-gnu-patches' or somesuch. b) There would be a maintainer for that account, who would be a go-between that doesn't have an agenda in accepting or rejecting patches, but is more of a spam-filter. Sort of a 'patch pumpkin' to accept perl5's terminology, but whose only goal is to make sure the patches are in acceptable form before sending them off to the right destination. c) There would be a matrix of mailing lists based per package, cygwin maintainers per package, a cross-reference that tells which patch goes where. There would be a database which shows the status of each patch (whether its at maintainer submission stage, maintainer acceptance stage, gnu submission stage, gnu acceptance stage) d) Users would compose a description for the patch, as well as the patch itself and send it to this mailing list. e) a mechanism for showing acceptance of the patch would be given to to both cygwin or gnu (or other mailing list) maintainers so that you could track the status of the patches' submission. f) any discussion on the patch would
Re: cygwin patches integrating back into standard gnu
Edward S. Peschko wrote: See, to build a shared lib, you really really need to use libtool-devel, which is libtool-1.5, and which requires automake 1.5.0 and autoconf 2.50. However, those packages are just now -- after 1.5 years -- coming into widespread use, because that's just silly. Gnu builds are uniformly prefix-friendly, and there was a simple way to 'make a development environment' for any given platform, then you could have your autoconf-2.13 (old) environment, and your autoconf-2.5 (new) environment based on path. And you would hack on one or the other based on what environment you were in. It'd probably be easier to port between the two, too. HA HA HA HA HA! You're living in a dream world. Sure, I can build autoconf-2.13 to install into, say, /usr/autotool/stable, and automake-2.5x to install into, for instance, /usr/autotool/devel. No problem. [Taken a look at your cygwin installation lately? Hmm?] And then, those packages will always and forever look in /usr/autotool/*/share/aclocal[-VER]/ for .m4 and .am macros. (btw, the [-VER] thing is a fairly recent [~ 1yr] put in place to help automake coexist with itself) But other packages -- gettext, libiconv, etc -- don't put their m4 macros there. No, they put their macros into their OWN ${prefix}/share/aclocal/ directory -- usually /usr/share/aclocal. But my two autoconf's (extend to automake, libtool) won't look there. Unless I then, for every package that I develop, set my PATH appropriately AND remember to aclocal/autoconf with the -I/usr/share/aclocal flag whenever I re-autotool. OR, I could add a feature to standard autoconf/automake/etc so that it looks in a site-specifiable list of directories, in addition to /usr/autotool/*/share/aclocal[-VER]. Oh wait. I did that. Two years ago. And got the patch accepted into official automake-1.5/6.x. What were you doing then? Basking in how well all GNU packages are prefix-friendly? Oh, and one other thing: autoconf, automake, and libtool -- in the past -- all needed to be installed with the same prefix. So if you had two intallations of autoconf with prefixA and prefixB, then you also need to install automake into both prefixA and prefixB, ditto libtool. This restriction has been slightly relieved recently...but if you've got old enough versions then you still have to deal with this issue. That's why cygwin provides autoconf-2.13 + automake-1.4p6 + libtool-1.4.3 and autoconf-2.5x + automake-1.7.x + libtool-1.5 Which - I might add, cygwin does *not* lend itself to given that each build.sh script that I saw had hard-coded paths in it for building to /usr/bin, non-standard steps for install, etc. And your favorite RPM .spec files do what, exactly? the build scripts are cygwin's version of .spec files; their JOB is to specify, according to some policy, those things that GNU packages allow one to specify. Like --prefix. At some point, somebody somewhere has to specify those values, or nothing actually gets compiled. We do it in the build script, and follow a policy we have here hashed out. Debian does it in their rules files, following their social contract policy. Red Hat does it in .spec files, following some internal corporate [now external Fedora-project] policy. Don't like the policy? Then manually modify the spec file. Or rules file. Or build.sh file. If it built clean in whatever prefix you desired, you could do a little substitution trick, to get binary builds for *any* path: 1) substitute (in binary files) any prefix with a different prefix, and pad it with null characters ('/tmp/very/long/prefix' becomes '/my/prefix...' where . is \0) 2) substitute (in text files) any prefix with a different prefix minus the null padding. and hence have more than one cygwin environment per cygwin.dll. This is so fraught with error and misconception that I don't even know where to begin. Plus, it's not even related to the original point of this thread: getting patches back upstream, and why are some cygwin patches so big. So I'm not going there. It'd probably be wise to integrate this with setup.exe so you would have the option to install packages in a non-standard place. Oh god no. Click here to automatically fsck-up all the binaries I just installed THAT's a GREAT idea! I nominate you to support that nightmare. Ok, one question... If I'm going to temporarily maintain these patches, Huh? Who said *you* were going to maintain them? Aren't the cygwin package owners doing that already? it sounds like I'm going to need to only apply them with cygwin builds, right? And that they may not fit with older gnu builds? No, not at all. There's nothing very special about the cygwin versions of the new autotools. They're just NEW(er). (automake-devel and autoconf-devel are stock. So is libtool-devel as of the 1.5 release) Thus, they are by definition cross-platform. The thing is, popt
Re: cygwin patches integrating back into standard gnu
cwilson wrote: So that YOU don't have to? (And wait. What are the current cygwin maintainers doing so badly that you want to take over and redo their jobs for them?) Look. Patches should go back to the upstream package. There should not be a bunch of extant, uncommitted patches laying about -- for ANY platform. IF there are, it *means* something: 3) The patch has been rejected by the current maintainers. Ditto. Funny, I *just today* recieved a reply concerning a patch I submitted upstream to the zlib developers -- who are preparing the 1.2.1 release. With apologies to the zlib maintainer, I quote the relevant portion and my response here: - ZLIB correspondence Though I tried, I could not bring myself to apply the patch. It does too much violence to what's there and working. Perhaps a patch that does not try to build both static and shared at the same time could be simpler. Yeah, that's fair. I did put in a wee bit of the patch though--appending .exe as appropriate, and the minigzip.c patch. Yes, this is actually a great help. EXE-related changes are always the most tedious part of cygwin/mingw patches, as with each new release those changes must be replicated by hand. (Not that it's an issue with zlib, which has only had 1.5 releases while I've been working with it and cygwin : 1.1.3 - 1.1.4, and (not yet) 1.1.4 - 1.2.x ) But with other packages, for some reason, other mods just seem to easily migrate up to each new release -- patch spews a few warnings about fuzz offset, and that's it. But the exe stuff almost always must be redone by hand. So, accepting just that portion of the patch is more helpful than you'd think. - ZLIB correspondence Here's a SUGGESTION Instead of proposing grand new layers of bureaucracy requiring time, dedication and server resources and which PLACE BARRIERS between the patch originator and the people who might actually integrate them into the official dists, simply contact a cygwin maintainer via the preferred mechanism for contacting ANY cygwin package maintainer -- this mailing list. about a package you're concerned about, -- Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: cygwin patches integrating back into standard gnu
On Sat, 2003-11-08 at 14:28, cwilson wrote: It'd probably be wise to integrate this with setup.exe so you would have the option to install packages in a non-standard place. Oh god no. Click here to automatically fsck-up all the binaries I just installed THAT's a GREAT idea! I nominate you to support that nightmare. No need for anyone to support it - it ain't going in setup, no how, no way. Someone wants a non-standard location, they roll their own binary. Finito. But they didn't really pursue this too strongly -- instead, they focused on attempting to make the transition smooth ('autoupdate', etc). They ignored the network-stasis effects (essentially, a classic 'deadlock' problem: you first, no you first...) Yes, and IMO a nutso one. Coexistence is more important than smooth transition in the absence of a dictator forcing concurrent upgrades. Rob -- GPG key available at: http://members.aardvark.net.au/lifeless/keys.txt. signature.asc Description: This is a digitally signed message part
Re: cygwin patches integrating back into standard gnu
Robert Collins wrote: But they didn't really pursue this too strongly -- instead, they focused on attempting to make the transition smooth ('autoupdate', etc). They ignored the network-stasis effects (essentially, a classic 'deadlock' problem: you first, no you first...) Yes, and IMO a nutso one. Coexistence is more important than smooth transition in the absence of a dictator forcing concurrent upgrades. Yeah. You know, every time this subject comes up I'm reminded of an essay I saw online a few years back. It had to do with that bug-a-boo we all like to criticize MS for: their insistence that old programs work on new OS's. We whine that this restricts modern platforms by subjecting them to the bad decisions of the past. The essay on the other hand said that coexistence -- I can keep my old programs even after I upgrade my OS -- is what led to MS's unprecedented dominance. It counterintuitively allowed people to upgrade *their APPS* faster! (What? Because I can run OldApp on NewOS, I'll buy NewApp sooner? Yep.) Without coexistance, you're faced with an all-or-nothing proposition. Like Apple's uphill switch campaign. Get the better OS -- and simultaneously ditch all of your old apps. Replace them all at the same time -- with this month's paycheck. Or go without for months. Or keep two computers up and running (Honey, I've got to do Quicken... Oh, that only works on the kid's computer now...) So, you stick with OldOS and OldApp until it just gets too damn painful. Which may take years. With coexistence, you can replace the OS today. Then next month replace one or two apps. etc etc. Pretty soon, you've transitioned completely (or almost so; it's asymptotic) -- which means (a) better penetration by MS's upgraded products, and (b) lots more dough leaving your wallet and going to MS/Adobe/Intuit/etc sooner. This is good for the software companies [tertiary effects like MS domination == bad ignored] and good for software users [one wouldn't buy the new apps at all if their perceived value was less than the cost; so obviously the purchaser thinks it was a good]. That was a really good essay. I wish I could remember who wrote it and where I saw it... Now, apply the same logic to autoconf-2.13 and 2.5x. We'd have been THRU this transition completely if the autoconf developers had done the following: 1) spend six months developing 2.14 -- a 2.13-compatible version of autoconf, that provided the additional capability of coexisting with the new feature-branch autoconf (what we call 2.5x). 2) then spend six months NOT adding features to 2.5x, but making sure that it seamlessly could coexist side-by-side with 2.14. Release 2.15, 2.16 as needed while working out these kinks, but always maintain 2.13-compatibility. 3) Spend six more months developing the spiffy new- incompatible features you wanted all along in the 2.5x branch. 4) Spring it on the world and watch everybody install it as soon as they can. What's the danger? They still have 2.1x and can keep using it, but they can also play with the shiny new toy. Mmmm... Toys Everbody knows how geeks are with new gadgets. In 30 days, you'd've seen dozens of major packages make the switch. Six months after that, everything important would have switched. Total time elapsed: 24 months. And no bad feelings (those [EMAIL PROTECTED] autoconf bastards...) Instead, two years ago they said Here ya go: autoconf-2.5x. It's been rearchitected internally and we really like it. It doesn't add a whole lot of new features, and it will choke on some files that the old version accepted. But trust us, it does things The Right Way. Never mind that these benefits are invisible to the end-user. Oh yeah -- and you can't easily use it and 2.13 on the same computer thanks to a variety of bad decisions on our part. And the latest automake and libtool releases require that you use this new incompatible autoconf. Have fun. Where are we now? Two years later, we kinda sorta can have coexistence on some platforms, by using 3rd party wrappers, but it still isn't seamless. Some packages have made the switch, others are toying with it, but it's always painful, acrimonious, and slow. Gcc/binutils are still playing at it. Worse, it's slowed autoconf's own development and acceptance, as well as that of automake, libtool, m4... Oh, and it's made MY life harder. :-) -- Chuck [Boy, am I in a rant-y mood lately or what.] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
cygwin patches integrating back into standard gnu
I was curious - exactly what is the process to submit cygwin patches to the respective projects that support cygwin as a target? I've been integrating cygwin into the build for the OSes I support, and I find that there are hundreds of thousands of lines of patches for cygwin (around 400k). Some are trivial, some are fairly substantial (ex: popt's patch is approx 1/3 the size of the regular distribution!) I'm loathe to have to support these patches, esepcially because some seem to be cross-platform unfriendly, so I was hoping that some sort of concerted effort is being made or could be made to make the ported cygwin packages 'build clean from the box', so that ./configure --prefix=..; make; make install would work for 99% of them without patching. To that end, I've put together - attached to this message - a small perl script that goes off, fetches all of the latest cygwin project source code, and extracts all the patches and README files from the tarred packages.. It saves the source files in './build', and the patches in './patches' It should run as-is under cygwin, but if it doesn't I wouldn't mind putting together a small PAR file to make it self-contained. Anyways, I could (or someone could) modify it so that, as an option, the patches within are sent to the appropriate mailing list for inclusion. I would think that such a matrix would be helpful in general, as well as a centralized user which could be a conduit for submitting patches to the right place. (which to me is a lot better idea than everyone using the script to send the same patch over and over) But 400k of patches seems just a bit high. Ed extract_cygwin_patches.p use strict; use File::Path; use FileHandle; use Data::Dumper; mkdir(build); system(rm -rf patches); mkdir(patches); chdir(build); my $cygwin_entries = cache_cygwin(); my $cygwin_projects = cygwin_projects( $cygwin_entries); my $project; foreach $project (@$cygwin_projects) { print STDERR \n\nGETTING PROJECT :$project:\n\n; my $latest_project = get_latest_project($project, $cygwin_entries); my ($shortpath) = ($latest_project =~ m(?:.*)/(.*)sg); system(rm -f $shortpath) if (-e $shortpath); ftp(ftp://planetmirror.com/pub/cygwin/$latest_project;) if (!-e $shortpath); extract_patch($latest_project, $project ); } cleanup(); sub cleanup { chdir(../patches); my $file; foreach $file (glob(*)) { if ( $file !~ m\.patch$ $file !~ mREADME) # { system(rm -f $file) if (-e $file); } } } chdir(..); # -- start subs --- sub shell { my $fh = new FileHandle($_[0]); my $line = $fh; close($fh); return(1) if ($line =~ m^#! || $line =~ m#.*shell); return(0); } sub cache_cygwin { if (-e ls-lR.gz) { system(rm ls-lR.gz); } ftp('ftp://planetmirror.com/pub/cygwin/ls-lR.gz'); my $cygwin = _ls_lr(ls-lR); my @return = grep(mrelease m-src m\.tar\.gz|\.tar\.bz2, @$cygwin); return([EMAIL PROTECTED]); } sub cygwin_projects { my ($entries) = @_; my %MATCH; grep { m^./release(/[^/]+/) $MATCH{$1}++ } @$entries; my @projects = sort keys(%MATCH); return([EMAIL PROTECTED]); } sub get_latest_project { my ($project, $cygwin_projects) = @_; my @releases = grep(m$project, @$cygwin_projects); @releases = sort { _cygwin_tuple($b, $a) } @releases; return($releases[0]); } sub extract_patch { my ($latest_project, $project) = @_; $project =~ s/sg; my ($shortpath) = ($latest_project =~ m.*/(.*)); my $filenames = tar($shortpath, { list = 1 }); my @patches_or_hints = grep(m(?:patch|cygwin)i, @$filenames); # @patches_or_hints = grep(!m\.c$, @patches_or_hints); # if (@patches_or_hints) { tar($shortpath, { extract = 1, filenames = [EMAIL PROTECTED], outflat = ../patches/${project}_file, outfull = 0 }); } else { print STDERR Package $shortpath does not have any patches!!\n; } } sub tar { my ($shortpath, $config) = @_; if ($config-{list}) { my @files = ($shortpath =~ m\.bz2$)? `/bin/tar tjf $shortpath` : # ($shortpath =~ m\.tar.gz$)? `/bin/tar tzf $shortpath` : # ($shortpath =~ m\.tar$)? `/bin/tar tf $shortpath` : `/bin/tar tf $shortpath`; # chomp(@files); return([EMAIL PROTECTED]); } elsif ($config-{extract}) { my $flags = ($shortpath =~ m\.bz2$)? xjf : # ($shortpath =~ m\.tar.gz$)? tzf : # ($shortpath =~ m\.tar$)? tf : tf; # my @filenames = ($config-{filenames})? @{$config-{filenames}} : (); if ($config-{filenames} [EMAIL PROTECTED]{filenames}}) { print STDERR you asked not to extract any files from $shortpath!Skipping.\n; } if ($config-{outflat}) { mkdir(tmpdir.$$); chdir(tmpdir.$$); system(/bin/tar $flags ../$shortpath @filenames); @filenames = grep( -f $_, @filenames); my $file; foreach $file (@filenames) { my $sub = $config-{outflat};
Re: cygwin patches integrating back into standard gnu
Edward S. Peschko wrote: I was curious - exactly what is the process to submit cygwin patches to the respective projects that support cygwin as a target? I've been integrating cygwin into the build for the OSes I support, and I find that there are hundreds of thousands of lines of patches for cygwin (around 400k). Some are trivial, some are fairly substantial (ex: popt's patch is approx 1/3 the size of the regular distribution!) Most upstream maintainers will accept modest patches to help support the cygwin platform. By and large, the packages with huge packages are simply autoconf-generated stuff. See, to build a shared lib, you really really need to use libtool-devel, which is libtool-1.5, and which requires automake 1.5.0 and autoconf 2.50. However, those packages are just now -- after 1.5 years -- coming into widespread use, because 1) autoconf 2.5x is in some ways incompatible with autoconf-2.13 -- which means that an upstream package maintainer has to decide Okay, everybody who hacks on my package now must install autoconf-2.5x on their system. But then each developer also must make a decision -- Hmm. I can only install autoconf-2.13 OR 2.5x but not both. These 5 packages I like to hack on require 2.13. Those 2 require 2.5x. Shall I switch to ac-2.5x and stop hacking on the 5 old packages? So that's why many (upstream) maintainers have been loathe to 'make the switch' -- and why some of our patches are large. A two-line change to configure.in may lead to many thousands of changes in configure after re-autoconfing with 2.5x. 2) Now, multiply that by automake-1.4p5 vs automake-1.7.5, and libtool.m4/ltmain.sh from libtool-1.5 vs. libtool.m4/ltconfig/ltmain.sh from libtool-1.4.3. 3) Things are slowly getting better. Some platforms are now finally providing mechanisms where both autoconf-2.13 and autoconf-2.5x can coexist. (Cygwin has been doing this for years, but Red Hat et al took a little longer) Plus, every week that goes by, another upstream maintainer takes the plunge -- opening the way for our patches to go back. This trend is now (finally) accelerating. I'm loathe to have to support these patches, esepcially because some seem to be cross-platform unfriendly, Okay. Most (all?) of mine are cross-platform-friendly, but after all, cygwin maintainers are maintaining **cygwin** ports -- so you can hardly blame them for not doing *extra* work. so I was hoping that some sort of concerted effort is being made or could be made to make the ported cygwin packages 'build clean from the box', so that ./configure --prefix=..; make; make install would work for 99% of them without patching. AFAIK, every cygwin maintainer has the goal of pushing patches back upstream. However, some upstream maintainers are more resistant than others. To that end, I've put together - attached to this message - a small perl script that goes off, fetches all of the latest cygwin project source code, and extracts all the patches and README files from the tarred packages.. It saves the source files in './build', and the patches in './patches' It should run as-is under cygwin, but if it doesn't I wouldn't mind putting together a small PAR file to make it self-contained. Anyways, I could (or someone could) modify it so that, as an option, the patches within are sent to the appropriate mailing list for inclusion. I would think that such a matrix would be helpful in general, as well as a centralized user which could be a conduit for submitting patches to the right place. (which to me is a lot better idea than everyone using the script to send the same patch over and over) But 400k of patches seems just a bit high. Oh god no. Automated patch-spam to mailing lists? I can't think of a better way to ensure that our patches are rejected. The Right Way To Do This is for each (cygwin) maintainer to handle it -- after all, they are the most informed on the subject. Plus, in many cases you don't WANT to send all 400k of patches: 1) most (upstream) maintainers want small, easily digestible patches -- so mega-patches must be split up into functional units. 2) the autotool-generated portion of each patch shouldn't go back; only the changes to the source files (configure.in, Makefile.am) with the note that This assumes you reautoconf with autoconf-2.5x or higher, re-automake with automake-1.7.5 or higher, re-libtoolize with libtool-1.5 or higher. etc. A dumb patch-spammer is NOT the way to go, here. -- Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: cygwin patches integrating back into standard gnu
Charles Wilson wrote: 1) most (upstream) maintainers want small, easily digestible patches -- so mega-patches must be split up into functional units. See, for instance, this week's libtool-patches mailing list. http://mail.gnu.org/archive/html/libtool-patches/2003-11/index.html -- Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/