Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 22/09/2015 9:22 AM, Vinh Nguyen wrote: > On Tue, Sep 22, 2015 at 1:46 AM, Duncan Murdoch > wrote: >> However, Herve's second example does illustrate a bug I can reproduce: >> one character out of 1 was read incorrectly. Could you please try >> his final patch and see if it fixes your AIX problems? > > Just tried the latest dcf.c from R-dev and install.packages no longer > segfaults from readDCF. Thank you both Duncan and Murdoch for your > help. You're welcome. (Just wanted to point out: I'm only one person. But I suspect you meant to thank Hervé. I'll do it for you: thanks!) Duncan Murdoch Can finally make use of this AIX box now (fyi placed request to > get gdb installed but who knows when that would've been installed). > > Thanks again. > > -- Vinh > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On Tue, Sep 22, 2015 at 1:46 AM, Duncan Murdoch wrote: > However, Herve's second example does illustrate a bug I can reproduce: > one character out of 1 was read incorrectly. Could you please try > his final patch and see if it fixes your AIX problems? Just tried the latest dcf.c from R-dev and install.packages no longer segfaults from readDCF. Thank you both Duncan and Murdoch for your help. Can finally make use of this AIX box now (fyi placed request to get gdb installed but who knows when that would've been installed). Thanks again. -- Vinh __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 21/09/2015 4:50 PM, Hervé Pagès wrote:
> Hi,
>
> Note that one significant change to read.dcf() that happened since R
> 3.0.2 is the addition of support for arbitrary long lines (commit
> 63281), which never worked:
>
>dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
>writeLines(dcf, "test.dcf")
>nchar(read.dcf("test.dcf"))
>#aa
># [1,] 8186
>
> The culprit being line 53 in src/main/dcf.c where the author of the
> Rconn_getline2() function only copies 'nbuf' chars from 'buf' to 'buf2'
> when in fact 'nbuf + 1' chars have been stored in 'buf' so far.
>
> Quickest fix:
>
> Index: src/main/dcf.c
> ===
> --- src/main/dcf.c(revision 69404)
> +++ src/main/dcf.c(working copy)
> @@ -50,7 +50,7 @@
> if(nbuf+2 >= bufsize) { // allow for terminator below
> bufsize *= 2;
> char *buf2 = R_alloc(bufsize, sizeof(char));
> - memcpy(buf2, buf, nbuf);
> + memcpy(buf2, buf, nbuf + 1);
> buf = buf2;
> }
> if(c != '\n'){
>
> However a better fix would be to have 'nbuf' actually contain the nb
> of chars that was stored in 'buf' so far (as it name suggests):
>
> Index: src/main/dcf.c
> ===
> --- src/main/dcf.c(revision 69404)
> +++ src/main/dcf.c(working copy)
> @@ -42,12 +42,12 @@
> /* Use R_alloc as this might get interrupted */
> static char *Rconn_getline2(Rconnection con)
> {
> -int c, bufsize = MAXELTSIZE, nbuf = -1;
> +int c, bufsize = MAXELTSIZE, nbuf = 0;
> char *buf;
>
> buf = R_alloc(bufsize, sizeof(char));
> while((c = Rconn_fgetc(con)) != R_EOF) {
> - if(nbuf+2 >= bufsize) { // allow for terminator below
> + if(nbuf+1 >= bufsize) { // allow for terminator below
> bufsize *= 2;
> char *buf2 = R_alloc(bufsize, sizeof(char));
> memcpy(buf2, buf, nbuf);
> @@ -54,17 +54,19 @@
> buf = buf2;
> }
> if(c != '\n'){
> - buf[++nbuf] = (char) c;
> + buf[nbuf++] = (char) c;
> } else {
> - buf[++nbuf] = '\0';
> + buf[nbuf++] = '\0';
> break;
> }
> }
> +if (nbuf == 0)
> +return NULL;
> /* Make sure it is null-terminated even if file did not end with
>* newline.
>*/
> -if(nbuf >= 0 && buf[nbuf]) buf[++nbuf] = '\0';
> -return (nbuf == -1) ? NULL: buf;
> +buf[nbuf-1] = '\0';
> +return buf;
> }
>
> That improves readability and reduces the risk of bugs.
>
> Also note that Rconn_getline2() allocates a new buffer for each line in
> the DCF file. So we got support for arbitrary long lines (a rare
> situation) at the price of a slow down and increased memory usage for
> all DCF files. Sounds less than optimal :-/
I'll put in your patch plus a partial fix for the efficiency issue. (It
will allocate the first buffer on the stack.)
Should show up in R-devel and R-patched in due course.
Duncan Murdoch
>
> Cheers,
> H.
>
>
> On 09/21/2015 11:01 AM, Duncan Murdoch wrote:
>> On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
>>> Here's an update:
>>>
>>> I checked the ChangeLog for R, and it seems like readDCF was changed
>>> in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
>>> over to 3.2.2, and R compiled fine and install.packages now work for
>>> me.
>>>
>>> This is probably not ideal, but it at least makes R usable on AIX for
>>> me. Would definitely like to help figure out what's wrong with the
>>> new dcf.c on AIX.
>>
>> I don't know if anyone on the core team has access to AIX, so you're
>> likely on your own for this.
>>
>> I'd suggest running R in a debugger (gdb or whatever you have), and
>> identifying exactly which line in dcf.c fails, and why. If you tell us
>> that, we might be able to spot what is going wrong.
>>
>> Duncan Murdoch
>>
>>>
>>> Thanks.
>>>
>>> -- Vinh
>>>
>>>
>>> On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
> Dear list (re-posting from r-help as r-devel is probably more
> appropriate),
>
> I was able to successfully compile R on our AIX box at work using the
> GNU compilers following the instructions on the R Administration
> guide. The output can be seen at here
> (https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
> no errors .
>
> However, I get a segfault whenever I try to use the in
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 21/09/2015 10:20 PM, Vinh Nguyen wrote:
> On Mon, Sep 21, 2015 at 2:48 PM, Duncan Murdoch
> wrote:
>> On 21/09/2015 4:50 PM, Hervé Pagès wrote:
>>> Hi,
>>>
>>> Note that one significant change to read.dcf() that happened since R
>>> 3.0.2 is the addition of support for arbitrary long lines (commit
>>> 63281), which never worked:
>>>
>>>dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
>>>writeLines(dcf, "test.dcf")
>>>nchar(read.dcf("test.dcf"))
>>>#aa
>>># [1,] 8186
>>>
>>
>> I don't see that in R 3.2.2 on OSX or 3.2.2 patched on Windows:
>>
>>>nchar(read.dcf("test.dcf"))
>> aa
>> [1,] 1
>
> I'm at home now and getting the following from Revolution R Open 3.2.2 on a
> Mac:
You should post bug reports about that build to Revolution. We have no
idea what changes it includes, and (being part of Microsoft) they have
tens of thousands more employees to deal with their bugs than we (being
all volunteers) do.
However, Herve's second example does illustrate a bug I can reproduce:
one character out of 1 was read incorrectly. Could you please try
his final patch and see if it fixes your AIX problems?
>
>> dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
>> writeLines(dcf, "test.dcf")
>> nchar(read.dcf("test.dcf"))
>aa
> [1,] 8186
>
> Here's what I see in the file:
>
> $ wc -c test.dcf
>
>10005 test.dcf
>
> Same results using Revolution R Open on Windows. Also same results on
> regular R 3.2.0 on Windows.
Version 3.2.0 is not current.
Duncan Murdoch
>
> -- Vinh
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On Mon, Sep 21, 2015 at 2:48 PM, Duncan Murdoch
wrote:
> On 21/09/2015 4:50 PM, Hervé Pagès wrote:
>> Hi,
>>
>> Note that one significant change to read.dcf() that happened since R
>> 3.0.2 is the addition of support for arbitrary long lines (commit
>> 63281), which never worked:
>>
>>dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
>>writeLines(dcf, "test.dcf")
>>nchar(read.dcf("test.dcf"))
>>#aa
>># [1,] 8186
>>
>
> I don't see that in R 3.2.2 on OSX or 3.2.2 patched on Windows:
>
>>nchar(read.dcf("test.dcf"))
> aa
> [1,] 1
I'm at home now and getting the following from Revolution R Open 3.2.2 on a Mac:
> dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
> writeLines(dcf, "test.dcf")
> nchar(read.dcf("test.dcf"))
aa
[1,] 8186
Here's what I see in the file:
$ wc -c test.dcf
10005 test.dcf
Same results using Revolution R Open on Windows. Also same results on
regular R 3.2.0 on Windows.
-- Vinh
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 09/21/2015 02:48 PM, Duncan Murdoch wrote:
On 21/09/2015 4:50 PM, Hervé Pagès wrote:
Hi,at
Note that one significant change to read.dcf() that happened since R
3.0.2 is the addition of support for arbitrary long lines (commit
63281), which never worked:
dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
writeLines(dcf, "test.dcf")
nchar(read.dcf("test.dcf"))
#aa
# [1,] 8186
I don't see that in R 3.2.2 on OSX or 3.2.2 patched on Windows:
nchar(read.dcf("test.dcf"))
aa
[1,] 1
You're just being lucky that 'buf2[nbuf + 1]' is not '\0' after
allocation by R_alloc(). Try this:
val <- paste(rep(letters, length.out=1), collapse="")
writeLines(paste("aa:", val), "test.dcf")
identical(val, read.dcf("test.dcf")[1])
Maybe you won't be so lucky with this one.
H.
Duncan Murdoch
The culprit being line 53 in src/main/dcf.c where the author of the
Rconn_getline2() function only copies 'nbuf' chars from 'buf' to 'buf2'
when in fact 'nbuf + 1' chars have been stored in 'buf' so far.
Quickest fix:
Index: src/main/dcf.c
===
--- src/main/dcf.c (revision 69404)
+++ src/main/dcf.c (working copy)
@@ -50,7 +50,7 @@
if(nbuf+2 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
- memcpy(buf2, buf, nbuf);
+ memcpy(buf2, buf, nbuf + 1);
buf = buf2;
}
if(c != '\n'){
However a better fix would be to have 'nbuf' actually contain the nb
of chars that was stored in 'buf' so far (as it name suggests):
Index: src/main/dcf.c
===
--- src/main/dcf.c (revision 69404)
+++ src/main/dcf.c (working copy)
@@ -42,12 +42,12 @@
/* Use R_alloc as this might get interrupted */
static char *Rconn_getline2(Rconnection con)
{
-int c, bufsize = MAXELTSIZE, nbuf = -1;
+int c, bufsize = MAXELTSIZE, nbuf = 0;
char *buf;
buf = R_alloc(bufsize, sizeof(char));
while((c = Rconn_fgetc(con)) != R_EOF) {
- if(nbuf+2 >= bufsize) { // allow for terminator below
+ if(nbuf+1 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
memcpy(buf2, buf, nbuf);
@@ -54,17 +54,19 @@
buf = buf2;
}
if(c != '\n'){
- buf[++nbuf] = (char) c;
+ buf[nbuf++] = (char) c;
} else {
- buf[++nbuf] = '\0';
+ buf[nbuf++] = '\0';
break;
}
}
+if (nbuf == 0)
+return NULL;
/* Make sure it is null-terminated even if file did not end with
* newline.
*/
-if(nbuf >= 0 && buf[nbuf]) buf[++nbuf] = '\0';
-return (nbuf == -1) ? NULL: buf;
+buf[nbuf-1] = '\0';
+return buf;
}
That improves readability and reduces the risk of bugs.
Also note that Rconn_getline2() allocates a new buffer for each line in
the DCF file. So we got support for arbitrary long lines (a rare
situation) at the price of a slow down and increased memory usage for
all DCF files. Sounds less than optimal :-/
Cheers,
H.
On 09/21/2015 11:01 AM, Duncan Murdoch wrote:
On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
Here's an update:
I checked the ChangeLog for R, and it seems like readDCF was changed
in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
over to 3.2.2, and R compiled fine and install.packages now work for
me.
This is probably not ideal, but it at least makes R usable on AIX for
me. Would definitely like to help figure out what's wrong with the
new dcf.c on AIX.
I don't know if anyone on the core team has access to AIX, so you're
likely on your own for this.
I'd suggest running R in a debugger (gdb or whatever you have), and
identifying exactly which line in dcf.c fails, and why. If you tell us
that, we might be able to spot what is going wrong.
Duncan Murdoch
Thanks.
-- Vinh
On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
Dear list (re-posting from r-help as r-devel is probably more appropriate),
I was able to successfully compile R on our AIX box at work using the
GNU compilers following the instructions on the R Administration
guide. The output can be seen at here
(https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
no errors .
However, I get a segfault whenever I try to use the install.packages
fun
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 21/09/2015 4:50 PM, Hervé Pagès wrote:
> Hi,
>
> Note that one significant change to read.dcf() that happened since R
> 3.0.2 is the addition of support for arbitrary long lines (commit
> 63281), which never worked:
>
>dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
>writeLines(dcf, "test.dcf")
>nchar(read.dcf("test.dcf"))
>#aa
># [1,] 8186
>
I don't see that in R 3.2.2 on OSX or 3.2.2 patched on Windows:
>nchar(read.dcf("test.dcf"))
aa
[1,] 1
Duncan Murdoch
> The culprit being line 53 in src/main/dcf.c where the author of the
> Rconn_getline2() function only copies 'nbuf' chars from 'buf' to 'buf2'
> when in fact 'nbuf + 1' chars have been stored in 'buf' so far.
>
> Quickest fix:
>
> Index: src/main/dcf.c
> ===
> --- src/main/dcf.c(revision 69404)
> +++ src/main/dcf.c(working copy)
> @@ -50,7 +50,7 @@
> if(nbuf+2 >= bufsize) { // allow for terminator below
> bufsize *= 2;
> char *buf2 = R_alloc(bufsize, sizeof(char));
> - memcpy(buf2, buf, nbuf);
> + memcpy(buf2, buf, nbuf + 1);
> buf = buf2;
> }
> if(c != '\n'){
>
> However a better fix would be to have 'nbuf' actually contain the nb
> of chars that was stored in 'buf' so far (as it name suggests):
>
> Index: src/main/dcf.c
> ===
> --- src/main/dcf.c(revision 69404)
> +++ src/main/dcf.c(working copy)
> @@ -42,12 +42,12 @@
> /* Use R_alloc as this might get interrupted */
> static char *Rconn_getline2(Rconnection con)
> {
> -int c, bufsize = MAXELTSIZE, nbuf = -1;
> +int c, bufsize = MAXELTSIZE, nbuf = 0;
> char *buf;
>
> buf = R_alloc(bufsize, sizeof(char));
> while((c = Rconn_fgetc(con)) != R_EOF) {
> - if(nbuf+2 >= bufsize) { // allow for terminator below
> + if(nbuf+1 >= bufsize) { // allow for terminator below
> bufsize *= 2;
> char *buf2 = R_alloc(bufsize, sizeof(char));
> memcpy(buf2, buf, nbuf);
> @@ -54,17 +54,19 @@
> buf = buf2;
> }
> if(c != '\n'){
> - buf[++nbuf] = (char) c;
> + buf[nbuf++] = (char) c;
> } else {
> - buf[++nbuf] = '\0';
> + buf[nbuf++] = '\0';
> break;
> }
> }
> +if (nbuf == 0)
> +return NULL;
> /* Make sure it is null-terminated even if file did not end with
>* newline.
>*/
> -if(nbuf >= 0 && buf[nbuf]) buf[++nbuf] = '\0';
> -return (nbuf == -1) ? NULL: buf;
> +buf[nbuf-1] = '\0';
> +return buf;
> }
>
> That improves readability and reduces the risk of bugs.
>
> Also note that Rconn_getline2() allocates a new buffer for each line in
> the DCF file. So we got support for arbitrary long lines (a rare
> situation) at the price of a slow down and increased memory usage for
> all DCF files. Sounds less than optimal :-/
>
> Cheers,
> H.
>
>
> On 09/21/2015 11:01 AM, Duncan Murdoch wrote:
>> On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
>>> Here's an update:
>>>
>>> I checked the ChangeLog for R, and it seems like readDCF was changed
>>> in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
>>> over to 3.2.2, and R compiled fine and install.packages now work for
>>> me.
>>>
>>> This is probably not ideal, but it at least makes R usable on AIX for
>>> me. Would definitely like to help figure out what's wrong with the
>>> new dcf.c on AIX.
>>
>> I don't know if anyone on the core team has access to AIX, so you're
>> likely on your own for this.
>>
>> I'd suggest running R in a debugger (gdb or whatever you have), and
>> identifying exactly which line in dcf.c fails, and why. If you tell us
>> that, we might be able to spot what is going wrong.
>>
>> Duncan Murdoch
>>
>>>
>>> Thanks.
>>>
>>> -- Vinh
>>>
>>>
>>> On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
> Dear list (re-posting from r-help as r-devel is probably more
> appropriate),
>
> I was able to successfully compile R on our AIX box at work using the
> GNU compilers following the instructions on the R Administration
> guide. The output can be seen at here
> (https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
> no errors .
>
> However, I get a segfault whenever I try to use the install.packages
> function to install packages. Usi
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 09/21/2015 01:50 PM, Hervé Pagès wrote:
Hi,
Note that one significant change to read.dcf() that happened since R
3.0.2 is the addition of support for arbitrary long lines (commit
63281), which never worked:
dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
writeLines(dcf, "test.dcf")
nchar(read.dcf("test.dcf"))
#aa
# [1,] 8186
The culprit being line 53 in src/main/dcf.c where the author of the
Rconn_getline2() function only copies 'nbuf' chars from 'buf' to 'buf2'
when in fact 'nbuf + 1' chars have been stored in 'buf' so far.
Quickest fix:
Index: src/main/dcf.c
===
--- src/main/dcf.c(revision 69404)
+++ src/main/dcf.c(working copy)
@@ -50,7 +50,7 @@
if(nbuf+2 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
-memcpy(buf2, buf, nbuf);
+memcpy(buf2, buf, nbuf + 1);
buf = buf2;
}
if(c != '\n'){
However a better fix would be to have 'nbuf' actually contain the nb
of chars that was stored in 'buf' so far (as it name suggests):
Index: src/main/dcf.c
===
--- src/main/dcf.c(revision 69404)
+++ src/main/dcf.c(working copy)
@@ -42,12 +42,12 @@
/* Use R_alloc as this might get interrupted */
static char *Rconn_getline2(Rconnection con)
{
-int c, bufsize = MAXELTSIZE, nbuf = -1;
+int c, bufsize = MAXELTSIZE, nbuf = 0;
char *buf;
buf = R_alloc(bufsize, sizeof(char));
while((c = Rconn_fgetc(con)) != R_EOF) {
-if(nbuf+2 >= bufsize) { // allow for terminator below
+if(nbuf+1 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
memcpy(buf2, buf, nbuf);
@@ -54,17 +54,19 @@
buf = buf2;
}
if(c != '\n'){
-buf[++nbuf] = (char) c;
+buf[nbuf++] = (char) c;
} else {
-buf[++nbuf] = '\0';
+buf[nbuf++] = '\0';
break;
}
}
+if (nbuf == 0)
+return NULL;
/* Make sure it is null-terminated even if file did not end with
* newline.
*/
-if(nbuf >= 0 && buf[nbuf]) buf[++nbuf] = '\0';
-return (nbuf == -1) ? NULL: buf;
+buf[nbuf-1] = '\0';
^^
oops... need to be:
buf[nbuf] = '\0';
Cheers,
H.
+return buf;
}
That improves readability and reduces the risk of bugs.
Also note that Rconn_getline2() allocates a new buffer for each line in
the DCF file. So we got support for arbitrary long lines (a rare
situation) at the price of a slow down and increased memory usage for
all DCF files. Sounds less than optimal :-/
Cheers,
H.
On 09/21/2015 11:01 AM, Duncan Murdoch wrote:
On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
Here's an update:
I checked the ChangeLog for R, and it seems like readDCF was changed
in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
over to 3.2.2, and R compiled fine and install.packages now work for
me.
This is probably not ideal, but it at least makes R usable on AIX for
me. Would definitely like to help figure out what's wrong with the
new dcf.c on AIX.
I don't know if anyone on the core team has access to AIX, so you're
likely on your own for this.
I'd suggest running R in a debugger (gdb or whatever you have), and
identifying exactly which line in dcf.c fails, and why. If you tell us
that, we might be able to spot what is going wrong.
Duncan Murdoch
Thanks.
-- Vinh
On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen
wrote:
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen
wrote:
Dear list (re-posting from r-help as r-devel is probably more
appropriate),
I was able to successfully compile R on our AIX box at work using the
GNU compilers following the instructions on the R Administration
guide. The output can be seen at here
(https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
no errors .
However, I get a segfault whenever I try to use the install.packages
function to install packages. Using debug, I was able to trace it to
the readDCF function:
Browse[2]>
debug: if (!all) return(.Internal(readDCF(file, fields, keep.white)))
Browse[2]>
debug: return(.Internal(readDCF(file, fields, keep.white)))
Browse[2]>
*** caught segfault ***
address 4, cause 'invalid permissions'
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:
Was curious if anyone
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
Hi,
Note that one significant change to read.dcf() that happened since R
3.0.2 is the addition of support for arbitrary long lines (commit
63281), which never worked:
dcf <- paste(c("aa: ", rep(letters, length.out=1)), collapse="")
writeLines(dcf, "test.dcf")
nchar(read.dcf("test.dcf"))
#aa
# [1,] 8186
The culprit being line 53 in src/main/dcf.c where the author of the
Rconn_getline2() function only copies 'nbuf' chars from 'buf' to 'buf2'
when in fact 'nbuf + 1' chars have been stored in 'buf' so far.
Quickest fix:
Index: src/main/dcf.c
===
--- src/main/dcf.c (revision 69404)
+++ src/main/dcf.c (working copy)
@@ -50,7 +50,7 @@
if(nbuf+2 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
- memcpy(buf2, buf, nbuf);
+ memcpy(buf2, buf, nbuf + 1);
buf = buf2;
}
if(c != '\n'){
However a better fix would be to have 'nbuf' actually contain the nb
of chars that was stored in 'buf' so far (as it name suggests):
Index: src/main/dcf.c
===
--- src/main/dcf.c (revision 69404)
+++ src/main/dcf.c (working copy)
@@ -42,12 +42,12 @@
/* Use R_alloc as this might get interrupted */
static char *Rconn_getline2(Rconnection con)
{
-int c, bufsize = MAXELTSIZE, nbuf = -1;
+int c, bufsize = MAXELTSIZE, nbuf = 0;
char *buf;
buf = R_alloc(bufsize, sizeof(char));
while((c = Rconn_fgetc(con)) != R_EOF) {
- if(nbuf+2 >= bufsize) { // allow for terminator below
+ if(nbuf+1 >= bufsize) { // allow for terminator below
bufsize *= 2;
char *buf2 = R_alloc(bufsize, sizeof(char));
memcpy(buf2, buf, nbuf);
@@ -54,17 +54,19 @@
buf = buf2;
}
if(c != '\n'){
- buf[++nbuf] = (char) c;
+ buf[nbuf++] = (char) c;
} else {
- buf[++nbuf] = '\0';
+ buf[nbuf++] = '\0';
break;
}
}
+if (nbuf == 0)
+return NULL;
/* Make sure it is null-terminated even if file did not end with
* newline.
*/
-if(nbuf >= 0 && buf[nbuf]) buf[++nbuf] = '\0';
-return (nbuf == -1) ? NULL: buf;
+buf[nbuf-1] = '\0';
+return buf;
}
That improves readability and reduces the risk of bugs.
Also note that Rconn_getline2() allocates a new buffer for each line in
the DCF file. So we got support for arbitrary long lines (a rare
situation) at the price of a slow down and increased memory usage for
all DCF files. Sounds less than optimal :-/
Cheers,
H.
On 09/21/2015 11:01 AM, Duncan Murdoch wrote:
On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
Here's an update:
I checked the ChangeLog for R, and it seems like readDCF was changed
in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
over to 3.2.2, and R compiled fine and install.packages now work for
me.
This is probably not ideal, but it at least makes R usable on AIX for
me. Would definitely like to help figure out what's wrong with the
new dcf.c on AIX.
I don't know if anyone on the core team has access to AIX, so you're
likely on your own for this.
I'd suggest running R in a debugger (gdb or whatever you have), and
identifying exactly which line in dcf.c fails, and why. If you tell us
that, we might be able to spot what is going wrong.
Duncan Murdoch
Thanks.
-- Vinh
On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
Dear list (re-posting from r-help as r-devel is probably more appropriate),
I was able to successfully compile R on our AIX box at work using the
GNU compilers following the instructions on the R Administration
guide. The output can be seen at here
(https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
no errors .
However, I get a segfault whenever I try to use the install.packages
function to install packages. Using debug, I was able to trace it to
the readDCF function:
Browse[2]>
debug: if (!all) return(.Internal(readDCF(file, fields, keep.white)))
Browse[2]>
debug: return(.Internal(readDCF(file, fields, keep.white)))
Browse[2]>
*** caught segfault ***
address 4, cause 'invalid permissions'
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:
Was curious if anyone has a clue on why such error exists or what I
could do to fix it? I'm able
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
On 21/09/2015 1:49 PM, Vinh Nguyen wrote:
> Here's an update:
>
> I checked the ChangeLog for R, and it seems like readDCF was changed
> in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
> over to 3.2.2, and R compiled fine and install.packages now work for
> me.
>
> This is probably not ideal, but it at least makes R usable on AIX for
> me. Would definitely like to help figure out what's wrong with the
> new dcf.c on AIX.
I don't know if anyone on the core team has access to AIX, so you're
likely on your own for this.
I'd suggest running R in a debugger (gdb or whatever you have), and
identifying exactly which line in dcf.c fails, and why. If you tell us
that, we might be able to spot what is going wrong.
Duncan Murdoch
>
> Thanks.
>
> -- Vinh
>
>
> On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
>> Hi there,
>>
>> I just wanted to follow up on this readDCF issue with install.packages
>> on AIX on R 3.*. I'm happy to help try potential solutions or debug
>> if anyone could point me in the right direction.
>>
>> To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
>> This was not the case up until R 2.15.3. This makes install.packages
>> not usable. Thanks.
>>
>> -- Vinh
>>
>>
>> On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
>>> Dear list (re-posting from r-help as r-devel is probably more appropriate),
>>>
>>> I was able to successfully compile R on our AIX box at work using the
>>> GNU compilers following the instructions on the R Administration
>>> guide. The output can be seen at here
>>> (https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
>>> no errors .
>>>
>>> However, I get a segfault whenever I try to use the install.packages
>>> function to install packages. Using debug, I was able to trace it to
>>> the readDCF function:
>>>
>>> Browse[2]>
>>> debug: if (!all) return(.Internal(readDCF(file, fields, keep.white)))
>>> Browse[2]>
>>> debug: return(.Internal(readDCF(file, fields, keep.white)))
>>> Browse[2]>
>>>
>>> *** caught segfault ***
>>> address 4, cause 'invalid permissions'
>>>
>>> Possible actions:
>>> 1: abort (with core dump, if enabled)
>>> 2: normal R exit
>>> 3: exit R without saving workspace
>>> 4: exit R saving workspace
>>> Selection:
>>>
>>> Was curious if anyone has a clue on why such error exists or what I
>>> could do to fix it? I'm able to install packages via R CMD INSTALL,
>>> but I would hate to have to manually determine dependencies, download
>>> the source for each package, and install them "by hand" via R CMD
>>> INSTALL.
>>>
>>> I went back and compiled older versions of R to see if this error
>>> exists. On R 3.0.3, I get:
>>>
>>> debug(available.packages)
>>> install.packages('ggplot2', dep=TRUE, repo='http://cran.stat.ucla.edu')
>>> ...
>>> Browse[2]>
>>> debug: z <- res0 <- tryCatch(read.dcf(file = tmpf), error = identity)
>>> Browse[2]>
>>> Error: segfault from C stack overflow
>>>
>>> On R 2.15.3, I do not see the error.
>>>
>>> Would be great to get this resolved. Thank you for your help.
>>>
>>> -- Vinh
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
Here's an update:
I checked the ChangeLog for R, and it seems like readDCF was changed
in 3.0.2. I went on a whim and copied src/main/dcf.c from R 2.15.3
over to 3.2.2, and R compiled fine and install.packages now work for
me.
This is probably not ideal, but it at least makes R usable on AIX for
me. Would definitely like to help figure out what's wrong with the
new dcf.c on AIX.
Thanks.
-- Vinh
On Mon, Sep 21, 2015 at 10:01 AM, Vinh Nguyen wrote:
> Hi there,
>
> I just wanted to follow up on this readDCF issue with install.packages
> on AIX on R 3.*. I'm happy to help try potential solutions or debug
> if anyone could point me in the right direction.
>
> To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
> This was not the case up until R 2.15.3. This makes install.packages
> not usable. Thanks.
>
> -- Vinh
>
>
> On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
>> Dear list (re-posting from r-help as r-devel is probably more appropriate),
>>
>> I was able to successfully compile R on our AIX box at work using the
>> GNU compilers following the instructions on the R Administration
>> guide. The output can be seen at here
>> (https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
>> no errors .
>>
>> However, I get a segfault whenever I try to use the install.packages
>> function to install packages. Using debug, I was able to trace it to
>> the readDCF function:
>>
>> Browse[2]>
>> debug: if (!all) return(.Internal(readDCF(file, fields, keep.white)))
>> Browse[2]>
>> debug: return(.Internal(readDCF(file, fields, keep.white)))
>> Browse[2]>
>>
>> *** caught segfault ***
>> address 4, cause 'invalid permissions'
>>
>> Possible actions:
>> 1: abort (with core dump, if enabled)
>> 2: normal R exit
>> 3: exit R without saving workspace
>> 4: exit R saving workspace
>> Selection:
>>
>> Was curious if anyone has a clue on why such error exists or what I
>> could do to fix it? I'm able to install packages via R CMD INSTALL,
>> but I would hate to have to manually determine dependencies, download
>> the source for each package, and install them "by hand" via R CMD
>> INSTALL.
>>
>> I went back and compiled older versions of R to see if this error
>> exists. On R 3.0.3, I get:
>>
>> debug(available.packages)
>> install.packages('ggplot2', dep=TRUE, repo='http://cran.stat.ucla.edu')
>> ...
>> Browse[2]>
>> debug: z <- res0 <- tryCatch(read.dcf(file = tmpf), error = identity)
>> Browse[2]>
>> Error: segfault from C stack overflow
>>
>> On R 2.15.3, I do not see the error.
>>
>> Would be great to get this resolved. Thank you for your help.
>>
>> -- Vinh
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] segfault with readDCF on R 3.1.2 on AIX 6.1 when using install.packages
Hi there,
I just wanted to follow up on this readDCF issue with install.packages
on AIX on R 3.*. I'm happy to help try potential solutions or debug
if anyone could point me in the right direction.
To re-cap, it appears readDCF is segfault'ing since R 3.* on AIX.
This was not the case up until R 2.15.3. This makes install.packages
not usable. Thanks.
-- Vinh
On Tue, Nov 11, 2014 at 10:23 AM, Vinh Nguyen wrote:
> Dear list (re-posting from r-help as r-devel is probably more appropriate),
>
> I was able to successfully compile R on our AIX box at work using the
> GNU compilers following the instructions on the R Administration
> guide. The output can be seen at here
> (https://gist.github.com/nguyenvinh/504321ea9c89d8919bef) and yields
> no errors .
>
> However, I get a segfault whenever I try to use the install.packages
> function to install packages. Using debug, I was able to trace it to
> the readDCF function:
>
> Browse[2]>
> debug: if (!all) return(.Internal(readDCF(file, fields, keep.white)))
> Browse[2]>
> debug: return(.Internal(readDCF(file, fields, keep.white)))
> Browse[2]>
>
> *** caught segfault ***
> address 4, cause 'invalid permissions'
>
> Possible actions:
> 1: abort (with core dump, if enabled)
> 2: normal R exit
> 3: exit R without saving workspace
> 4: exit R saving workspace
> Selection:
>
> Was curious if anyone has a clue on why such error exists or what I
> could do to fix it? I'm able to install packages via R CMD INSTALL,
> but I would hate to have to manually determine dependencies, download
> the source for each package, and install them "by hand" via R CMD
> INSTALL.
>
> I went back and compiled older versions of R to see if this error
> exists. On R 3.0.3, I get:
>
> debug(available.packages)
> install.packages('ggplot2', dep=TRUE, repo='http://cran.stat.ucla.edu')
> ...
> Browse[2]>
> debug: z <- res0 <- tryCatch(read.dcf(file = tmpf), error = identity)
> Browse[2]>
> Error: segfault from C stack overflow
>
> On R 2.15.3, I do not see the error.
>
> Would be great to get this resolved. Thank you for your help.
>
> -- Vinh
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
