Re: new tar behavior and --wildcards
Bdale Garbee wrote: The following table summarizes pattern-matching default values: MembersDefault settings -- Inclusion `--no-wildcards --anchored --no-wildcards-match-slash' Exclusion `--wildcards --no-anchored --wildcards-match-slash' -- Footnotes -- Will this break my Amanda config? I'm not sure what flags Amanda passes to tar, but I'm pretty syre I have no control (absent changing the source, of course) over them. Do I need to worry now which version of tar is installed on each machine being backed up by Amanda? That'll be fun :-( -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
On Wed, Jun 28, 2006 at 10:36:20AM +0200, Bill Allombert wrote: On Wed, Jun 28, 2006 at 07:02:15AM +0200, Christian Perrier wrote: Debian still has to provide an upgrade path for users upgrading from Sarge. We cannot blindly break users scripts. Here, the only way seems to be putting an entry in NEWS.Debian (for users script, ie things not under our control). In addition, I would suggest we reinstate the previous behaviour, but display a warning when wildcards are used but --wildcards is not set. The warning would tell people about the migration and explains they must fix their scripts to use --wildcards before upgrading to etch+1. I can only second that - we really should support our users at least that much (see point 4 in the social contract). Breaking users' scripts in these ways needs more than a simple note in NEWS.Debian (you'll get dozens of them when upgrading from sarge to etch). Cheers, -- Guido signature.asc Description: Digital signature
Re: new tar behavior and --wildcards
On Wed, Jun 28, 2006 at 10:32:30PM -0400, Bdale Garbee wrote: On Wed, 2006-06-28 at 10:36 +0200, Bill Allombert wrote: In addition, I would suggest we reinstate the previous behaviour, but display a warning when wildcards are used but --wildcards is not set. The problem with this is that generating a new warning can also cause people to need to update scripts, since lots of people seem to parse the output of commands like tar in wrapper scripts. So, I'm not convinced that this is really a good idea. I'm also always hesitant to deviate Debian default behavior for utilities like tar from upstream. Well, but the new tar is already generating warning anyway and people parsing tar stderr output get what they deserve. It is Debian role to provide transition plan when upstream break backward compatibility, and we do that on a large scale. All in all, I'm not yet convinced that reverting to the old wildcard behavior is the right thing to do. I've only heard about problems in a few (four?) packages so far, and all of them are Debian-specific programs that should be easy for us to update. I see no need for panic, though it's obviously and clearly regrettable that the packages in questions are ones that affect processes like building and testing Debian packages. But we have not yet even started to check for breakage. Someone could start to rebuild the whole archive with the new tar and report problem, check every maintainer script in sarge and etch, etc. I have to say I am a bit surprised you did not even test if the basic packaging tools still worked before uploading the package, especially given some previous incidents. In such case, it is better to upload the package to experimental, report bugs found in other packages, and then upload to unstable when the scope of the breakage is better understood. Cheers, -- Bill. [EMAIL PROTECTED] Imagine a large red swirl here. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards (proposed middle ground)
[EMAIL PROTECTED] (Barak A. Pearlmutter) writes: As a compromise that addresses some of the issues I would suggest the following: go with upstream, but add some convenience code, to whit: (1) Hot-wire tar to check an environment variable TAR_WILDCARD_DEFAULT and activate the --wildcard option if set. I'm not sure I see the point of this. The warnings/errors generated seem reasonably obvious, and inventing an additional, less-obvious mechanism instead of just assuming people will try --wildcard when tar suggests it seems like a step in the wrong direction. (2) Hot-wire tar to print a warning message to stderr if it (a) is defaulting to the --no-wildcard behaviour and, (b) it notices a filename that, had tar instead been in --wildcards mode, would have been expanded. Actually, tar already does this. I realize now that you must not have actually seen the warning in question, which helps explain to me why you were suggesting (1) above. Bdale -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards (proposed middle ground)
Oops, guess I should have checked if it was already done. My bad. (Given that the warning is working, why were people making such a fuss? Well, never mind.) -- Barak A. Pearlmutter [EMAIL PROTECTED] Hamilton Institute Dept Comp Sci, NUI Maynooth, Co. Kildare, Ireland http://www.bcl.hamilton.ie/~barak/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
Debian still has to provide an upgrade path for users upgrading from Sarge. We cannot blindly break users scripts. Here, the only way seems to be putting an entry in NEWS.Debian (for users script, ie things not under our control). We did something similar with su but we did it earlier in the release cycle, and I think we have already lot of work to do to release etch on time. At first reading of that thread, I tend to agree with Bill here. It seems pretty late in the release cycle to make that change. What about reimplementing the old behaviour in tar, make it the default for etch, with allowing the new behaviour to be voluntarily enforced (through an env variable or so)and revert this *after* the release of Etch. Involving the release team (if not already done, which I doubt for something managed by Bdale) could be good if that's likely to delay the release. signature.asc Description: Digital signature
Re: new tar behavior and --wildcards
On Wed, Jun 28, 2006 at 07:02:15AM +0200, Christian Perrier wrote: Debian still has to provide an upgrade path for users upgrading from Sarge. We cannot blindly break users scripts. Here, the only way seems to be putting an entry in NEWS.Debian (for users script, ie things not under our control). In addition, I would suggest we reinstate the previous behaviour, but display a warning when wildcards are used but --wildcards is not set. The warning would tell people about the migration and explains they must fix their scripts to use --wildcards before upgrading to etch+1. This way it is easy to find problematic scripts. Cheers, -- Bill. [EMAIL PROTECTED] Imagine a large red swirl here. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards (proposed middle ground)
Both sides in this discussion seem to have valid concerns: FOR making --wildcard the default - compatibility with upstream - compatibility with standards - compatibility with other distributions - whatever reasons POSIX had for this were probably sensible - upstream's judgement on this is likely to be correct AGAINST making --wildcard the default - very difficult to figure out what scrips are affected - mysterious breakage far into the future - requires grubbing around and inserting --wildcard in many places As a compromise that addresses some of the issues I would suggest the following: go with upstream, but add some convenience code, to whit: (1) Hot-wire tar to check an environment variable TAR_WILDCARD_DEFAULT and activate the --wildcard option if set. (2) Hot-wire tar to print a warning message to stderr if it (a) is defaulting to the --no-wildcard behaviour and, (b) it notices a filename that, had tar instead been in --wildcards mode, would have been expanded. If stderr is not hooked up, the warning could reasonably be sent using syslog() instead. I wouldn't bother adding any mechanism to shut these warnings off; if one really wants them shut off, use either --wildcards or --no-wildcards. Point (1) would allow people to easily check if some breakage is caused by this change, or to use old scripts/sources without contortions; and point (2) would serve to catch problems in scripts that might otherwise elude us. Point (2) would be especially helpful given that we're coming up on a release, so it would be nice to find and correct all affected scripts as rapidly as possible. -- Barak A. Pearlmutter Hamilton Institute Dept Comp Sci, NUI Maynooth, Co. Kildare, Ireland http://www.bcl.hamilton.ie/~barak/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
On Tue, Jun 27, 2006 at 03:40:49PM +0200, Thijs Kinkhorst wrote: In that case just reverting the Debian change isn't the right way. If you think that the change is wrong, then you should make upstream fix it: changing behaviour of tar is one thing, but having different behaviour of a basic tool like tar on different distros is very undesirable. Sarge and Etch are such different distros. Cheers, -- Bill. [EMAIL PROTECTED] Imagine a large red swirl here. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
On Wed, 2006-06-28 at 10:36 +0200, Bill Allombert wrote: Here, the only way seems to be putting an entry in NEWS.Debian (for users script, ie things not under our control). Good idea, Christian. In addition, I would suggest we reinstate the previous behaviour, but display a warning when wildcards are used but --wildcards is not set. The problem with this is that generating a new warning can also cause people to need to update scripts, since lots of people seem to parse the output of commands like tar in wrapper scripts. So, I'm not convinced that this is really a good idea. I'm also always hesitant to deviate Debian default behavior for utilities like tar from upstream. All in all, I'm not yet convinced that reverting to the old wildcard behavior is the right thing to do. I've only heard about problems in a few (four?) packages so far, and all of them are Debian-specific programs that should be easy for us to update. I see no need for panic, though it's obviously and clearly regrettable that the packages in questions are ones that affect processes like building and testing Debian packages. Bdale -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
Le lun 26 juin 2006 21:53, Petr Vandrovec a écrit : Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. I second that. it's now completely unpossible to do basic packaging work, because such a change wasn't planned. I also don't find it wise, if we still want to release this year, to introduce such a change *now*. Bdale, I *really* beg you to postpone that default change post-etch. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpztJ1RxD8lx.pgp Description: PGP signature
Re: new tar behavior and --wildcards
On Tue, 2006-06-27 at 10:02 +0200, Pierre Habouzit wrote: Le lun 26 juin 2006 21:53, Petr Vandrovec a écrit : Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. I second that. it's now completely unpossible to do basic packaging work, because such a change wasn't planned. I also don't find it wise, if we still want to release this year, to introduce such a change *now*. Bdale, I *really* beg you to postpone that default change post-etch. Is there any idea of the number of packages actually affected by this? I've harly seen an RC bug flood arise out of this; I've only seen two, one of which is already pending upload. Probably a few more will arise, but the fix is trivial. So I wonder if it would be useful to revert the change, since we have to change at some point and at this point the effects do not seem to be quite dramatic. Maybe you have signs indicating otherwise? Thijs signature.asc Description: This is a digitally signed message part
Re: new tar behavior and --wildcards
Le mar 27 juin 2006 13:37, Thijs Kinkhorst a écrit : On Tue, 2006-06-27 at 10:02 +0200, Pierre Habouzit wrote: Le lun 26 juin 2006 21:53, Petr Vandrovec a écrit : Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. I second that. it's now completely unpossible to do basic packaging work, because such a change wasn't planned. I also don't find it wise, if we still want to release this year, to introduce such a change *now*. Bdale, I *really* beg you to postpone that default change post-etch. Is there any idea of the number of packages actually affected by this? I've harly seen an RC bug flood arise out of this; I've only seen two, one of which is already pending upload. Probably a few more will arise, but the fix is trivial. So I wonder if it would be useful to revert the change, since we have to change at some point and at this point the effects do not seem to be quite dramatic. Maybe you have signs indicating otherwise? I fear there will be a lot. for me lintian failed, and I had some curious behaviour with one package I sponsored recently. To avoid problems atm, I force my pbuilder to use the tar from testing. As tar is a core component of the distro, I'm quite afraid of that change, that may have quite well hidden side effects, and be quite desastrous :/ -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpE5YY18rCGa.pgp Description: PGP signature
Re: new tar behavior and --wildcards
Thijs Kinkhorst wrote: On Tue, 2006-06-27 at 10:02 +0200, Pierre Habouzit wrote: Le lun 26 juin 2006 21:53, Petr Vandrovec a écrit : Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. I second that. it's now completely unpossible to do basic packaging work, because such a change wasn't planned. I also don't find it wise, if we still want to release this year, to introduce such a change *now*. Bdale, I *really* beg you to postpone that default change post-etch. Is there any idea of the number of packages actually affected by this? It's not so much packages already in the archive, it's every package that is being prepared to be uploaded. Lintian *always* fails for all packages that I build on a system with the updated tar. None of those packages failed prior to the tar update. Linda just crashes, every single time. Again, never did before. Without those two, how can new packages be uploaded safely? dpkg-buildpackage generates an error message but does continue to operate. I've harly seen an RC bug flood arise out of this; I've only seen two, I'm not a DD (in the NM process) but couldn't that simply be because the Debian machines themselves (like p.d.o and qa.d.o) have not upgraded to this release of tar? If those machines become unable to run lintian or linda without errors, isn't that going to be a flood of a different kind? one of which is already pending upload. Probably a few more will arise, but the fix is trivial. You mean downgrading to the version of tar in testing? So I wonder if it would be useful to revert the change, since we have to change at some point and at this point the effects do not seem to be quite dramatic. Maybe you have signs indicating otherwise? PLEASE can tar be reverted to 1.15.1dfsg-3 in unstable? Maybe tar 1.15.91-1 should go into experimental until lintian, dpkg-buildpackage and linda can all be prepared for the new behaviour. If lintian functionality isn't restored soon, how can new packages be uploaded safely? -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ signature.asc Description: OpenPGP digital signature
Re: new tar behavior and --wildcards
On Tue, 2006-06-27 at 13:00 +0100, Neil Williams wrote: It's not so much packages already in the archive, it's every package that is being prepared to be uploaded. Lintian *always* fails for all packages that I build on a system with the updated tar. None of those packages failed prior to the tar update. Your argument revolves around the fact that lintian is now broken. As a matter of fact, lintian has already been fixed in CVS for that, so after the next upload of lintian, the argument that this affects *all* packages is not valid anymore. I'm confident that the lintian maintainers won't wait long with their new version. Thijs signature.asc Description: This is a digitally signed message part
Re: new tar behavior and --wildcards
On Tue, 27 Jun 2006 13:37:21 +0200 Thijs Kinkhorst [EMAIL PROTECTED] wrote: On Tue, 2006-06-27 at 10:02 +0200, Pierre Habouzit wrote: Le lun 26 juin 2006 21:53, Petr Vandrovec a écrit : Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. I second that. it's now completely unpossible to do basic packaging work, because such a change wasn't planned. I also don't find it wise, if we still want to release this year, to introduce such a change *now*. Bdale, I *really* beg you to postpone that default change post-etch. Is there any idea of the number of packages actually affected by this? I've harly seen an RC bug flood arise out of this; I've only seen two, one of which is already pending upload. Probably a few more will arise, but the fix is trivial. So I wonder if it would be useful to revert the change, since we have to change at some point and at this point the effects do not seem to be quite dramatic. Maybe you have signs indicating otherwise? It is also bound to break numerous private scripts on peoples systems. And for no good reason, the default has been like this for years now, why change that? For the POSIX-pedantic people there is always the POSIX mode. grts Tim -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
On Tue, 2006-06-27 at 15:31 +0200, Tim Dijkstra wrote: It is also bound to break numerous private scripts on peoples systems. And for no good reason, the default has been like this for years now, why change that? For the POSIX-pedantic people there is always the POSIX mode. In that case just reverting the Debian change isn't the right way. If you think that the change is wrong, then you should make upstream fix it: changing behaviour of tar is one thing, but having different behaviour of a basic tool like tar on different distros is very undesirable. Thijs signature.asc Description: This is a digitally signed message part
Re: new tar behavior and --wildcards
Pierre Habouzit [EMAIL PROTECTED] writes: I fear there will be a lot. for me lintian failed, and I had some curious behaviour with one package I sponsored recently. To avoid problems atm, I force my pbuilder to use the tar from testing. A fixed version of lintian will be uploaded today. I was waiting a little to see if anything came out of the other tar oddity that I reported last night that also needed to be changed, but I'll probably upload it soon just because this is affecting a lot of people. We can always upload another fix later. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
On Mon, Jun 26, 2006 at 01:22:12PM -0600, Bdale Garbee wrote: Since this seems to have been an intentional behavior change by upstream to better align with a published standard, I'm uninclined to fight it, and think our best response is to update our utilities to include the --wildcards option, with a suitable versioned dependency on tar. Debian still has to provide an upgrade path for users upgrading from Sarge. We cannot blindly break users scripts. Also, before doing such change we need to audit: 1) all build scripts (i.e. rebuild the whole archive with the new tar) and see how many packages FTBFS. 2) all Sarge maintainer scripts, init script and cron scripts 3) all Etch maintainer scripts, init script and cron scripts 4) all Sarge packages for tar usage to allow for partial upgrade. 5) all Etch packages for tar usage We did something similar with su but we did it earlier in the release cycle, and I think we have already lot of work to do to release etch on time. Cheers, -- Bill. [EMAIL PROTECTED] Imagine a large red swirl here. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
new tar behavior and --wildcards
The new tar behavior with respect to wildcards is not a change I introduced just for Debian, it's a new upstream change that appears to be quite intentional and well documented, as per this text from the tar info docs: The following table summarizes pattern-matching default values: MembersDefault settings -- Inclusion `--no-wildcards --anchored --no-wildcards-match-slash' Exclusion `--wildcards --no-anchored --wildcards-match-slash' -- Footnotes -- (1) Notice that earlier GNU `tar' versions used globbing for inclusion members, which contradicted to UNIX98 specification and was not documented. Obviously, the problems reported with various Debian utilities are due to the default now being --no-wildcards for inclusion combined with a dependency on the footnoted feature of previous versions of GNU tar. Since this seems to have been an intentional behavior change by upstream to better align with a published standard, I'm uninclined to fight it, and think our best response is to update our utilities to include the --wildcards option, with a suitable versioned dependency on tar. Bdale -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
[I'm not on Debian-devel, so please CC me] Bdale Garbee wrote: The new tar behavior with respect to wildcards is not a change I introduced just for Debian, it's a new upstream change that appears to be quite intentional and well documented, as per this text from the tar info docs: The following table summarizes pattern-matching default values: MembersDefault settings -- Inclusion `--no-wildcards --anchored --no-wildcards-match-slash' Exclusion `--wildcards --no-anchored --wildcards-match-slash' -- Footnotes -- (1) Notice that earlier GNU `tar' versions used globbing for inclusion members, which contradicted to UNIX98 specification and was not documented. Although maybe that it was not documented, it was widely used, and it exists for at least 6 years. So upstream should fix documentation instead of tar behavior. As documentation is not part of Debian, it does not matter for Debian anyway. Obviously, the problems reported with various Debian utilities are due to the default now being --no-wildcards for inclusion combined with a dependency on the footnoted feature of previous versions of GNU tar. Since this seems to have been an intentional behavior change by upstream to better align with a published standard, I'm uninclined to fight it, and think our best response is to update our utilities to include the --wildcards option, with a suitable versioned dependency on tar. This decision makes tar completely incompatible. Programs which worked fine with tar for 6 years are suddenly broken, and now you have to have two versions - one for 'tar' before this brokeness, which do not pass --wildcards, and one for this broken 'tar', which passes --wildcards. And older version on newer 'tar' extracts nothing, while new version on older 'tar' fails with an unknown option error. Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. Petr Vandrovec -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
Petr Vandrovec wrote in message news:[EMAIL PROTECTED] Since this seems to have been an intentional behavior change by upstream to better align with a published standard, I'm uninclined to fight it, and think our best response is to update our utilities to include the --wildcards option, with a suitable versioned dependency on tar. This decision makes tar completely incompatible. Programs which worked fine with tar for 6 years are suddenly broken, and now you have to have two versions - one for 'tar' before this brokeness, which do not pass --wildcards, and one for this broken 'tar', which passes --wildcards. And older version on newer 'tar' extracts nothing, while new version on older 'tar' fails with an unknown option error. Maybe it could be default for tar's POSIX mode, but I have no idea why GNU mode behavior should be changed in any way. Petr Vandrovec Indeed. If POSIX compatability is important to the GNU tar maintainer, then why has he not updated GNU pax yet? UNIX03 has no tar, but has pax. So reviving the paxutils package seems more imporant than fixing an incompatibility with an outdated version of the Unix spec. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: new tar behavior and --wildcards
Petr Vandrovec [EMAIL PROTECTED] writes: This decision makes tar completely incompatible. Programs which worked fine with tar for 6 years are suddenly broken, and now you have to have two versions - one for 'tar' before this brokeness, which do not pass --wildcards, and one for this broken 'tar', which passes --wildcards. And older version on newer 'tar' extracts nothing, while new version on older 'tar' fails with an unknown option error. The --wildcards flag was added in tar 1.13.20 in 2001-06-27, so passing --wildcards is safe even for oldstable. Only the default changed. -- Russ Allbery ([EMAIL PROTECTED]) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
