Hi everyone!
I think I stumbled over a bug in the latest R 2.9.2 patched for OS X:
R version 2.9.2 Patched (2009-09-24 r49861)
i386-apple-darwin9.8.0
When I try to sort latin1-encoded character vectors, R sometimes
crashes with a segmentation fault. I'm running OS X 10.5.8 and have
observed this behaviour both with the i386 and x86_64 builds, in the
R.app GUI as well as on the command line.
Here's a minimal example that reliably triggers the crash on my machine:
=====
print(sessionInfo())
words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc", "\xe4\xfc")
str(words)
print(table(Encoding(words)))
Encoding(words) <- "latin1" # this is the correct encoding!
print(table(Encoding(words)))
N <- 1000
words <- rep(words, length.out=N)
print(N)
for (i in 1:N) {
x <- words[1:i]
# the following line will crash for some i, depending on the particular
# strings in <words> and the subset selected for <x> above
order(x)
}
=====
The output I get from this code is appended at the end of the mail.
Note that R incorrectly declares the latin1 strings in <word> to have
UTF-8 encoding (this seems wrong to me because the \x escapes insert
raw bytes into the string). The crash only occurs if the correct
"latin1" encoding (or "unknown") is explicitly specified. Otherwise
the string handling code appears to ignore everything after the first
invalid multibyte character.
I haven't been able to trigger the bug without some kind of loop. The
crash always occurs at the same iteration, but this changes depending
on the contents of <words> and the specific subset selected in each
loop iteration. Also note that the 64-bit version of R gives a
different error message. If I omit the unrelated statement
"print(N)", the 64-bit version segfaults and the 32-bit version just
hangs with high CPU load. All this suggests to me that there must be
some insidious memory corruption or stack/range overflow in the
internal ordering code.
Can other people reproduce this problem on different platforms and
possibly with different versions of R?
BTW, I ran into the crash when trying to read.delim() a file in latin1
encoding, using either encoding="latin1" or fileEncoding="latin1", and
then converting it back and forth between a character vector and a
factor. I still don't understand what's going on there. The
behaviour of read.delim() seems to depend very much on my locale
settings when running R, which is rather unpleasant. Is there a way
to find out how strings are stored internally (i.e. getting the exact
byte representation) and whether R believes them to be in UTF-8 or
latin1 encoding?
Best regards,
Stefan Evert
[ stefan.ev...@uos.de | http://purl.org/stefan.evert ]
Output of sample code on my machine:
> print(sessionInfo())
R version 2.9.2 Patched (2009-09-24 r49861)
i386-apple-darwin9.8.0
locale:
en_GB/en_GB/C/C/en_GB/en_GB
attached base packages:
[1] stats graphics grDevices utils datasets methods base
>
> words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc",
"\xe4\xfc")
> str(words)
chr [1:7] "aa" "ab" "a\xfc" "a\xe4" "b\xe4" "b\xfc" ...
> print(table(Encoding(words)))
unknown UTF-8
2 5
>
> Encoding(words) <- "latin1" # this is the correct encoding!
> print(table(Encoding(words)))
latin1 unknown
5 2
>
> N <- 1000
> words <- rep(words, length.out=N)
>
> print(N)
[1] 1000
> for (i in 1:N) {
+ x <- words[1:i]
+ # the following line will crash for some i, depending on the
particular
+ # strings in <words> and the subset selected for <x> above
+ order(x)
+ }
*** caught bus error ***
address 0x86, cause 'non-existent physical address'
Traceback:
1: order(x)
aborting ...
Bus error
64-bit version:
> print(sessionInfo())
R version 2.9.2 Patched (2009-09-24 r49861)
x86_64-apple-darwin9.8.0
locale:
en_GB/en_GB/C/C/en_GB/en_GB
attached base packages:
[1] stats graphics grDevices utils datasets methods base
>
> words <- c("aa", "ab", "a\xfc", "a\xe4", "b\xe4", "b\xfc",
"\xe4\xfc")
> str(words)
chr [1:7] "aa" "ab" "a\xfc" "a\xe4" "b\xe4" "b\xfc" ...
> print(table(Encoding(words)))
unknown UTF-8
2 5
>
> Encoding(words) <- "latin1" # this is the correct encoding!
> print(table(Encoding(words)))
latin1 unknown
5 2
>
> N <- 1000
> words <- rep(words, length.out=N)
>
> print(N)
[1] 1000
> for (i in 1:N) {
+ x <- words[1:i]
+ # the following line will crash for some i, depending on the
particular
+ # strings in <words> and the subset selected for <x> above
+ order(x)
+ }
Error in order(x) : 'translateCharUTF8' must be called on a CHARSXP
Execution halted
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
