Thanks for the info. I have tried your experiments and found them very interesting. But, in all of your examples, perl and HexEdit see exactly the same thing; whereas, in my problem, perl and HexEdit were seeing things differently. Perl seemed to have double vision.
Now I seem to have resolved the problem--sort of. I believe it's a bug in BBEdit. Perhaps you could check it out with the following two file/scripts (Mac Roman).
=== first file/script ===
#!/usr/bin/perl -w
# file0.pl - The data in "file0.pl" is a real bullet,
# namely A5. But the script "file0.pl" can't
# find it when run from BBEdit.use strict;
while (<DATA>) {
print "found A5\n" if /\xa5/;
} __DATA__
*=== second file/script ===
#!/usr/bin/perl -w
# file1.pl - On the other hand, this script has no trouble
# finding the bullet in the data of "file0.pl",
# whether or not it is run from BBEdit. use strict;
use FindBin qw($Bin); open FILE, "<$Bin/file0.pl";
while (<FILE>) {
print "found A5\n" if /\xa5/;
}
close FILE;If you create the two files in the same folder and make then both executable, then the commands
$ ./file1.pl
$ ./file2.plproduce the same result, "found A5". If you run "file1.pl" from BBEdit, you get the "found A5" result as well. However, if you open "file0.pl" in BBEdit and run it either directly or "in Terminal", no output is produced. As far as BBEdit is concerned, that 1 byte A5 bullet is now 3 bytes long. You can find it with the pattern "\xe2\x80\xa2". Apparently BBEdit is reinterpreting the single byte of data as 3 bytes of a "UTF8 encoding with no byte-order mark," to quote you, Sherm.
Regards,
Vic
At 12:14 PM +0000 1/11/04, John Delacour wrote:
At 9:26 pm -0500 10/1/04, Vic Norton wrote:
I'm sorry, John. I was talking figuratively. I didn't mean real bullets.
FIguratively or no, you were right on target with your choice. The bullet is a character in the 'macintosh' character set (referred to wrongly by the Perl people "MacRoman") which does not exist in the widely used (or at least declared) charset Latin-1 and has the same 8-bit codepoint as the i with diaeresis � � � in the Windows-1252 charset. It is to rebuild this Tower of Babel that Unicode was conceived and, far too slowly, brought into the computer world first in Windows NT and finally in Mac OS X. Unicode is a 'good thing' but it requires to be learned about and you'll come unstuck pretty often if you don't put aside a bit of time to do so.
<http://www.unicode.org/standard/WhatIsUnicode.html>
% perldoc -X Encode | more
SEE ALSO Encode::Encoding, Encode::Supported, Encode::PerlIO, encoding, perlebcdic, "open" in perlfunc, perlunicode, utf8, the Perl Unicode Mailing List <[EMAIL PROTECTED]>
How come Perl sees "C2 A0" whenever HexEdit sees "CA" and visa versa? I don't care what kind of characters we are talking here. To paraphrase Gertrude Stein, "a byte is a byte is a byte." At least that's what I thought until now.
Gertrude Stein was a character. Some characters are a byte. Some are not, and you have to care.
use strict ; my $file = "/tmp/file"; # open FILE, ">:utf8", $file ; print FILE "\xF0" ; close FILE ; # open FILE, $file ; print "UTF-8:\t" . <FILE> . $/ ; close FILE; # open FILE, ">$file" ; print FILE "\xF0" ; close FILE ; # open FILE, $file ; print "MacRoman:\t" . <FILE> . $/ ; close FILE; exit ;
JD
At 1:45 AM -0500 1/11/04, Sherm Pendley wrote:
On Jan 10, 2004, at 9:26 PM, Vic Norton wrote:
How come Perl sees "C2 A0" whenever HexEdit sees "CA" and visa versa? I don't care what kind of characters we are talking here. To paraphrase Gertrude Stein, "a byte is a byte is a byte." At least that's what I thought until now.
Like John said - text encoding.
The file you're viewing with HexEdit is most likely encoded using MacRoman, or possibly ISO 8859-1. Internally, Perl uses UTF8 encoding.
Try this: Create a new text file in BBEdit, and enter a bullet (opt-8). Save it using the default text encoding. HexEdit shows a single byte in the file: A5. Now, open the file again, and save a copy of it using UTF8 encoding with no byte-order mark. HexEdit now shows *three* bytes: E2 80 A2. And, you have to tell BBEdit what encoding the file uses when you open it - without the byte-order mark, BBEdit can't tell it's UTF8.
Just for grins, save it again, this time *with* the byte-order mark. HexEdit now reports *six* bytes in the file: EF BB BF E2 80 A2.
In other words, yes - a byte is a byte is a byte. But you're not working with bytes, you're working with text. A character is not always a byte. It can be several bytes, depending on how it's encoded.
sherm--
