On 2013-08-12 02:51:52 +0200, Adam Borowski wrote: > Detecting non-UTF files is easy: > * false positives are impossible > * false negatives are extremely unlikely: combinations of letters that would > happen to match a valid utf character don't happen naturally, and even if > they did, every single combination in the file tested would need to match > valid utf.
Not that unlikely, and it is rather annoying that Firefox (and therefore Iceweasel) gets this wrong due to an ambiguity with TIS-620. IMHO, in case of ambiguity, UTF-8 should always be preferred by default (applications could have options to change the preferences). Bug reports: https://bugzilla.mozilla.org/show_bug.cgi?id=760050 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719481 > On the other hand, detecting text files is hard. Deciding whether a file is a text file may be hard even for a human. What about text files with ANSI control sequences? > The best tool so far, "file", makes so many errors it's useless for > this purpose. Yes. > One could use location: like, declaring stuff in /etc/ and > /usr/share/doc/ to be text unless proven otherwise, but that's an > incomplete hack. Only hashbangs can be considered reliable, but > scripts are not where most documentation goes. > > Also, should HTML be considered text or not? Updating http-equiv is not > rocket surgery, detecting HTML with fancy extensions can be. I think better questions could be: why do you want to regard a file as text? For what purpose(s)? For the "all shipped text files in UTF-8" rule only? What about examples whose purpose is to have a file in a charset different from UTF-8? > 4a. perl and pod > > Considering perl to be text raises one more issue: pod. By perl's design, > pod without a specified encoding is considered to be ISO-8859-1, even if > the file contains "use utf8;". This is surprising, and many authors use > UTF-8 like everywhere else, leading to obvious results ("man gdm3" for one > example). Thus, there should be a tool (preferably the one mentioned > above) that checks perl files for pod with undeclared encoding, and raises > alarm if the file contains any bytes with high bit set. If a conversion > encoding is specified, such a declaration could be added automatically. Yes, undeclared encoding when not ASCII should be regarded as a bug. -- Vincent Lefèvre <vinc...@vinc17.net> - Web: <http://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130812105035.ga28...@xvii.vinc17.org