A NOTE has been added to this issue. ====================================================================== https://www.austingroupbugs.net/view.php?id=251 ====================================================================== Reported By: dwheeler Assigned To: ajosey ====================================================================== Project: 1003.1(2008)/Issue 7 Issue ID: 251 Category: Base Definitions and Headers Type: Enhancement Request Severity: Objection Priority: normal Status: Under Review Name: David A. Wheeler Organization: User Reference: Section: XBD 3.170 Filename Page Number: 60 Line Number: 1781 Interp Status: --- Final Accepted Text: ====================================================================== Date Submitted: 2010-05-03 18:49 UTC Last Modified: 2023-02-10 18:14 UTC ====================================================================== Summary: Forbid newline, or even bytes 1 through 31 (inclusive), in filenames ====================================================================== Relationships ID Summary ---------------------------------------------------------------------- related to 0000293 restricting readdir normalization of fi... related to 0000573 Please add '+' to the portable filename... has duplicate 0000545 Escape Filenames as per Uniform Resourc... related to 0000291 filenames need not contain characters ======================================================================
---------------------------------------------------------------------- (0006146) dwheeler (reporter) - 2023-02-10 18:14 https://www.austingroupbugs.net/view.php?id=251#c6146 ---------------------------------------------------------------------- I was fully aware when I proposed this that this was a big change, and might not be accepted. Still, there's been a problem that POSIX *allows* control characters in filenames (particularly newline), yet hasn't provided standard mechanisms to deal with them, so it's often impractical to process filenames securely (a sad state of affairs). The latest efforts to allow \0 (NUL) terminated filename lists is a big step forward. It's now at least possible to portably process filename lists, and using a mechanism that is already widely implemented. mirabilos : > Honestly, I fully agree with the need to escape some kinds of filenames. Widely deployed, and easily later-added-to-existing-systems, measures like “print -0” and “read -d ''” exist. Actually, make that ... > On that note, please do *not* standardise on GNU sort (-z for NUL-terminated), but on BSD’s: > mirbsd$ printf 'foo\0bar\0' | sort -R '' | hd 00000000 62 61 72 00 66 6F 6F 00 - |bar.foo.| > ... > is much more flexible than GNU’s… > -z, --zero-terminated > line delimiter is NUL, not newline > > for the same reason the shell read built-in command has -d for the delimiter (first octet of $OPTARG is used) instead of -0. Sorting on filenames is useful, especially when you're trying to create reproducible builds. That said, don't use "-R" in the standard for sort to specify a record delimiter, as GNU sort already uses -R for "random-sort". I think it should be possible to find another option letter or two that is either already used for its purpose *or* doesn't conflict with an existing implementation. I would prefer that sort support both a an option meaning "use NUL terminator" option as well as a more general option of "use this delimiter (empty means NUL)". The resulting scripts are a little easier to read when there's a simple option, especially when the script is already quoted. In addition, different implementations have seen value in each, so it makes sense to support them both. I believe the sort specification could add "-z" for saying "use NUL terminators for input/output record separators"; the "-z" option is used in GNU sort, MacOS sort, and probably others. I think you could use "-D" as the option for an arbitrary delimiter, similar to read -d DELIMITER where an empty delimiter means NUL. As with other cases, in at LEAST the case where NUL is used, it SHOULD be an error if there are 1+ bytes at the end that are not terminated by the terminator, as this suggests partial data. Issue History Date Modified Username Field Change ====================================================================== 2010-05-03 18:49 dwheeler New Issue 2010-05-03 18:49 dwheeler Status New => Under Review 2010-05-03 18:49 dwheeler Assigned To => ajosey 2010-05-03 18:49 dwheeler Name => David A. Wheeler 2010-05-03 18:49 dwheeler Section => XBD 3.170 Filename 2010-05-03 18:49 dwheeler Page Number => 60 2010-05-03 18:49 dwheeler Line Number => 1781 2010-05-03 19:37 dwheeler Note Added: 0000412 2011-03-10 16:04 msbrown Note Added: 0000689 2011-04-11 20:28 eblake Interp Status => --- 2011-04-11 20:28 eblake Note Added: 0000739 2011-04-11 20:28 eblake Summary Forbid bytes 1 through 31 (inclusive) in filenames => Forbid newline, or even bytes 1 through 31 (inclusive), in filenames 2011-04-11 21:17 dwheeler Note Added: 0000740 2011-07-07 15:31 user27 Note Added: 0000887 2011-12-17 21:43 dwheeler Issue Monitored: dwheeler 2012-02-22 06:48 oiaohm Note Added: 0001140 2012-02-22 19:25 dwheeler Note Added: 0001141 2012-02-22 19:38 wlerch Note Added: 0001142 2012-02-23 00:55 oiaohm Note Added: 0001143 2012-02-25 02:25 oiaohm Note Added: 0001147 2012-02-25 02:40 eblake Relationship added related to 0000545 2012-03-29 23:10 Don Cragun Relationship replaced has duplicate 0000545 2012-08-03 19:35 eblake Relationship added related to 0000291 2012-08-03 19:36 eblake Relationship added related to 0000293 2012-08-03 19:36 eblake Relationship added related to 0000573 2012-12-23 05:12 user229 Note Added: 0001437 2013-01-07 23:29 oiaohm Note Added: 0001438 2013-01-07 23:47 oiaohm Note Added: 0001439 2013-08-14 02:43 jrincayc Note Added: 0001711 2013-08-14 03:23 dwheeler Note Added: 0001712 2013-08-14 05:41 dalias Note Added: 0001713 2013-08-14 07:34 oiaohm Note Added: 0001714 2013-08-15 02:33 jrincayc Note Added: 0001715 2013-08-15 02:48 dalias Note Added: 0001716 2013-08-15 03:31 jrincayc Note Added: 0001717 2013-08-16 02:43 jrincayc Note Added: 0001721 2013-08-16 05:03 oiaohm Note Added: 0001722 2013-08-16 10:46 a brouwer Note Added: 0001723 2013-08-17 15:03 dwheeler Note Added: 0001724 2013-08-17 16:13 dalias Note Added: 0001725 2013-08-18 02:01 dwheeler Note Added: 0001726 2013-08-18 05:22 oiaohm Note Added: 0001727 2013-08-18 06:10 oiaohm Note Added: 0001728 2013-08-18 13:12 jrincayc Note Added: 0001730 2013-08-18 19:09 dwheeler Note Added: 0001731 2013-08-19 09:25 geoffclare Note Added: 0001732 2013-08-19 10:48 a brouwer Note Added: 0001733 2013-08-19 12:34 jrincayc Note Added: 0001734 2013-08-19 14:55 oiaohm Note Added: 0001735 2013-08-19 22:40 oiaohm Note Added: 0001736 2013-08-19 22:57 user229 Note Added: 0001737 2013-08-19 23:21 dalias Note Added: 0001738 2013-08-19 23:48 shware_systems Note Added: 0001739 2013-08-20 02:40 user229 Note Added: 0001740 2013-08-20 03:23 dalias Note Added: 0001741 2013-08-20 06:38 shware_systems Note Added: 0001742 2013-08-20 07:02 dalias Note Added: 0001743 2013-08-20 14:52 dwheeler Note Added: 0001744 2013-08-21 12:23 oiaohm Note Added: 0001745 2013-08-21 12:25 oiaohm Note Added: 0001746 2013-08-21 12:49 oiaohm Note Added: 0001747 2013-08-21 13:12 steffen Note Added: 0001748 2013-08-21 14:02 oiaohm Note Added: 0001749 2013-08-30 04:44 oiaohm Note Added: 0001779 2014-12-05 14:45 safinaskar Note Added: 0002480 2015-06-22 16:48 mirabilos Note Added: 0002730 2016-03-27 10:43 dannyniu Issue Monitored: dannyniu 2023-02-10 18:14 dwheeler Note Added: 0006146 ======================================================================
