[Rcpp-devel] Rolling ADF test
Dear Rcpp-devel list, Wasn’t sure if this got sent last time as I didn’t get a response. I am looking to carry out a fixed window length rolling ADF test (with no intercept), over some time series data currently in xts format. To do this I need to first fit a regression to the data, then use the residuals as an input into the ADF test, from which I can get a p-value to see if I can reject the idea of the data having a unit root or not. I know I can use fastLm() from RcppArmadillo for the regression, and I know that the ADF test also runs an OLS regression as part of it too, I would have assumed finding unit roots to be a reasonably common use, but currently can’t find any “out of the box” functions that carry out C++ optimised ADF tests, are there any that can easily be sourced/included into Rcpp? Secondly, in terms of implementation, originally I was using rollapply(), to carry out my rolling window ADF test, with all lm() calls in both the original regression and the adf test being converted to fastLm() and basically using the adfTest2() function created here (http://blog.quanttrader.org/2012/04/adftest-function-enhanced-with-rcpp-armadillo/) and put it into a rollapply, but since rollapply() can sometimes be slow, I was wondering if it would make sense to use a for loop to carry out the rolling ADF test instead, and thus was hoping for a Rcpp/RcppArmadillo implementation of the ADF test? is there one? I am relatively new to C++, so just to confirm for the fixed window length rolling function would I just be creating a new matrix at each row, which is basically just a subset of the original data that has been put in? and the using the matrix in my function? and would I be overwriting the matrix each time i get to a new row? I think there might be some memory management issues here, so some memory efficient suggested examples would be great! Any advice/links on carrying out any part of this would be much appreciated. Thanks in advance, HLM___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
Ok I was able to find the code causing the bug. So it looks like the
pointers you get from an Rcpp::Vector using .begin() become invalid
after that the Rcpp::Vector goes out of scope (and this makes sense),
what I do not understand is that this Rcpp::Vector was allocated in R
and should still be living during the execution of the Rcpp call
(that's why I wasn't expecting the pointer to be invalid).
This is the exact code (the one above is probably fine):
@@ in CPP @@i
struct GapMat {
int* ptr;
int* colset;
int nrow;
int ncol;
inline int* colptr(int col){
return ptr + colset[col];
}
GapMat(){}
GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
};
GapMat getGapMat(Rcpp::List gapmat){
IntegerVector vec = gapmat[vec];
IntegerVector pos = gapmat[colset];
int nrow = gapmat[nrow];
return GapMat(vec.begin(), pos.begin(), nrow, pos.length());
}
// [[Rcpp::export]]
IntegerVector colSumsGapMat(Rcpp::List gapmat){
GapMat mat = getGapMat(gapmat);
IntegerVector res(mat.ncol);
for (int i = 0; i mat.ncol; ++i){
for (int j = 0; j mat.nrow; ++j){
res[i] += mat.colptr(i)[j];
}
}
return res;
}
@@ in R (with gdb debugger as suggested by Dirk) @@i
library(Rcpp)
sourceCpp(scratchpad.cpp)
vec - rnbinom(3e7, mu=0.1, size=1); storage.mode(vec) - integer
nr - 80
colset - sample(3e7-nr, 1e7)
foo - vec[colset] #this is only to trigger some obscure garbage
collection mechanisms...
for (i in 1:10){
colset - sample(3e7-nr, 1e7)
gapmat - list(vec=vec, nrow=nr, colset=colset-1)
cs - colSumsGapMat(gapmat)
print(sum(cs))
}
[1] 8000
[1] 8000
[1] 80016890
[1] 80008144
[1] 80016022
[1] 80021609
Program received signal SIGSEGV, Segmentation fault.
0x718a5455 in GapMat::colptr (this=0x7fffc120, col=0) at
scratchpad.cpp:295
295return ptr + colset[col];
@@@
Why did it happen? What should I do to make sure that my pointers
remain valid? My goal is to convert safely some vectors or matrices
that exist in R to some pointers, how can I do that?
Thanks a lot for your help
Ale
On Tue, Feb 11, 2014 at 3:44 PM, Dirk Eddelbuettel e...@debian.org wrote:
In essence: Yes
On 11 February 2014 at 15:18, Alessandro Mammana wrote:
| Hi all,
| I got another segfault using Rcpp. It is very difficult to understand
| where it happens and to reduce it to a minimal example, so for now I
| am not posting very precise code here, but I have a suspicion, maybe
| you could help me saying if my suspect is right.
Use a debugger:
R -d gdb
and then proceed as normal.
Make sure your compiler flags include -g as well.
Dirk
| I am doing something similar:
|
| in a .cpp file:
| @@@
| struct GapMat {
| int* ptr;
| int* colset;
| int nrow;
| int ncol;
|
|
| inline int* colptr(int col){
| return ptr + colset[col];
| }
|
| GapMat(){}
|
| GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
| ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
| };
|
|
| // [[Rcpp::export]]
| IntegerVector colSumsGapMat(Rcpp::IntegerVector vec,
| Rcpp::IntegerVector pos, int nrow){
|GapMat mat(vec.begin(), pos.begin(), nrow, pos.length());
|IntegerVector res(pos.length());
|
| for (int i = 0; i pos.length(); ++i){
| for (int j = 0; j nrow; ++j){
| res[i] += mat.colptr(i)[j];
| }
| }
|
| return res;
| }
| @
|
| from R:
|
| vec - a very big integer vector
| nrow - 80
| pos - a very big subset of positions, such that max(pos) + nrow
length(vec)
| colsums - colSumsGapMat(vec, pos, nrow)
|
|
| from time to time I get a segfault.
| Note: this is not exactly the code that produces the segfault (because
| that one is very complicated), so it might be that this code is
| totally fine.
|
| My suspicion:
|
| I am using the pointer vec.begin(), but then I am allocating new
| memory in the R area of memory with IntegerVector res(pos.length())
| and R decides to move the original values of vec to some other
| place, making the pointer invalid.
|
| Is that possible
|
| Sorry for being very vague and thx in advance!!!
| Ale
|
| --
| Alessandro Mammana, PhD Student
| Max Planck Institute for Molecular Genetics
| Ihnestraße 63-73
| D-14195 Berlin, Germany
| ___
| Rcpp-devel mailing list
| Rcpp-devel@lists.r-forge.r-project.org
| https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
--
Alessandro Mammana, PhD Student
Max Planck Institute for Molecular Genetics
Ihnestraße 63-73
D-14195 Berlin, Germany
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
On 12 February 2014 at 11:47, Alessandro Mammana wrote:
| Ok I was able to find the code causing the bug. So it looks like the
Thanks for the added detail.
| pointers you get from an Rcpp::Vector using .begin() become invalid
| after that the Rcpp::Vector goes out of scope (and this makes sense),
| what I do not understand is that this Rcpp::Vector was allocated in R
| and should still be living during the execution of the Rcpp call
| (that's why I wasn't expecting the pointer to be invalid).
|
| This is the exact code (the one above is probably fine):
| @@ in CPP @@i
|
| struct GapMat {
| int* ptr;
| int* colset;
| int nrow;
| int ncol;
|
|
| inline int* colptr(int col){
| return ptr + colset[col];
| }
|
| GapMat(){}
|
| GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
| ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
| };
|
|
| GapMat getGapMat(Rcpp::List gapmat){
| IntegerVector vec = gapmat[vec];
| IntegerVector pos = gapmat[colset];
| int nrow = gapmat[nrow];
|
| return GapMat(vec.begin(), pos.begin(), nrow, pos.length());
| }
|
| // [[Rcpp::export]]
| IntegerVector colSumsGapMat(Rcpp::List gapmat){
|
| GapMat mat = getGapMat(gapmat);
| IntegerVector res(mat.ncol);
|
| for (int i = 0; i mat.ncol; ++i){
| for (int j = 0; j mat.nrow; ++j){
| res[i] += mat.colptr(i)[j];
| }
| }
|
| return res;
| }
|
| @@ in R (with gdb debugger as suggested by Dirk) @@i
| library(Rcpp)
| sourceCpp(scratchpad.cpp)
|
| vec - rnbinom(3e7, mu=0.1, size=1); storage.mode(vec) - integer
| nr - 80
|
| colset - sample(3e7-nr, 1e7)
| foo - vec[colset] #this is only to trigger some obscure garbage
| collection mechanisms...
|
| for (i in 1:10){
| colset - sample(3e7-nr, 1e7)
| gapmat - list(vec=vec, nrow=nr, colset=colset-1)
| cs - colSumsGapMat(gapmat)
| print(sum(cs))
| }
|
| [1] 8000
| [1] 8000
| [1] 80016890
| [1] 80008144
| [1] 80016022
| [1] 80021609
|
| Program received signal SIGSEGV, Segmentation fault.
| 0x718a5455 in GapMat::colptr (this=0x7fffc120, col=0) at
| scratchpad.cpp:295
| 295return ptr + colset[col];
|
| @@@
|
| Why did it happen? What should I do to make sure that my pointers
| remain valid? My goal is to convert safely some vectors or matrices
| that exist in R to some pointers, how can I do that?
Not sure. It looks fine at first instance. But then it's early in the morning
and I had very little coffee yet...
Maybe the fact that you tickle the gc() via vec[colset] has something to do
with it, maybe it has not. Maybe I would try the decomposition of the List
object inside the colSumsGapMat() function to keep it simpler. Or if you
_really_ want an external object to iterate over, memcpy it out.
With really large object, you may be stressing parts of the code that have
not been stressed the same way. If it breaks, you do get to keep both pieces.
Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
Ah wait, my bad (as always T.T), I found a much simpler explanation:
colset - sample(3e7-nr, 1e7)
storage.mode(colset)
[1] integer
storage.mode(colset-1)
[1] double
So when I was unwrapping colset I allocated new memory in Rcpp to
convert from double to integer, which was no longer valid when I went
out of scope.
I think it is a bit dangerous that you never know if you are
allocating memory or just wrapping R objects when parsing arguments in
Rcpp.
Is there a way of ensuring that NOTHING gets copied when parsing
arguments? Can you throw an exception if the type you try to cast to
is not the one you expect?
You might imagine that with large datasets this is important.
Sorry for bothering and thanks again,
Ale
On Wed, Feb 12, 2014 at 1:10 PM, Dirk Eddelbuettel e...@debian.org wrote:
On 12 February 2014 at 11:47, Alessandro Mammana wrote:
| Ok I was able to find the code causing the bug. So it looks like the
Thanks for the added detail.
| pointers you get from an Rcpp::Vector using .begin() become invalid
| after that the Rcpp::Vector goes out of scope (and this makes sense),
| what I do not understand is that this Rcpp::Vector was allocated in R
| and should still be living during the execution of the Rcpp call
| (that's why I wasn't expecting the pointer to be invalid).
|
| This is the exact code (the one above is probably fine):
| @@ in CPP @@i
|
| struct GapMat {
| int* ptr;
| int* colset;
| int nrow;
| int ncol;
|
|
| inline int* colptr(int col){
| return ptr + colset[col];
| }
|
| GapMat(){}
|
| GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
| ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
| };
|
|
| GapMat getGapMat(Rcpp::List gapmat){
| IntegerVector vec = gapmat[vec];
| IntegerVector pos = gapmat[colset];
| int nrow = gapmat[nrow];
|
| return GapMat(vec.begin(), pos.begin(), nrow, pos.length());
| }
|
| // [[Rcpp::export]]
| IntegerVector colSumsGapMat(Rcpp::List gapmat){
|
| GapMat mat = getGapMat(gapmat);
| IntegerVector res(mat.ncol);
|
| for (int i = 0; i mat.ncol; ++i){
| for (int j = 0; j mat.nrow; ++j){
| res[i] += mat.colptr(i)[j];
| }
| }
|
| return res;
| }
|
| @@ in R (with gdb debugger as suggested by Dirk) @@i
| library(Rcpp)
| sourceCpp(scratchpad.cpp)
|
| vec - rnbinom(3e7, mu=0.1, size=1); storage.mode(vec) - integer
| nr - 80
|
| colset - sample(3e7-nr, 1e7)
| foo - vec[colset] #this is only to trigger some obscure garbage
| collection mechanisms...
|
| for (i in 1:10){
| colset - sample(3e7-nr, 1e7)
| gapmat - list(vec=vec, nrow=nr, colset=colset-1)
| cs - colSumsGapMat(gapmat)
| print(sum(cs))
| }
|
| [1] 8000
| [1] 8000
| [1] 80016890
| [1] 80008144
| [1] 80016022
| [1] 80021609
|
| Program received signal SIGSEGV, Segmentation fault.
| 0x718a5455 in GapMat::colptr (this=0x7fffc120, col=0) at
| scratchpad.cpp:295
| 295return ptr + colset[col];
|
| @@@
|
| Why did it happen? What should I do to make sure that my pointers
| remain valid? My goal is to convert safely some vectors or matrices
| that exist in R to some pointers, how can I do that?
Not sure. It looks fine at first instance. But then it's early in the morning
and I had very little coffee yet...
Maybe the fact that you tickle the gc() via vec[colset] has something to do
with it, maybe it has not. Maybe I would try the decomposition of the List
object inside the colSumsGapMat() function to keep it simpler. Or if you
_really_ want an external object to iterate over, memcpy it out.
With really large object, you may be stressing parts of the code that have
not been stressed the same way. If it breaks, you do get to keep both pieces.
Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
--
Alessandro Mammana, PhD Student
Max Planck Institute for Molecular Genetics
Ihnestraße 63-73
D-14195 Berlin, Germany
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
On 12 February 2014 at 13:36, Alessandro Mammana wrote: | Ah wait, my bad (as always T.T), I found a much simpler explanation: Isn't it lovely when persistence pays off? ;-) | colset - sample(3e7-nr, 1e7) | storage.mode(colset) | [1] integer | storage.mode(colset-1) | [1] double | | So when I was unwrapping colset I allocated new memory in Rcpp to | convert from double to integer, which was no longer valid when I went | out of scope. Well that is sort-of a known issue. Look for discussions of clone() in the archive. | I think it is a bit dangerous that you never know if you are | allocating memory or just wrapping R objects when parsing arguments in | Rcpp. | Is there a way of ensuring that NOTHING gets copied when parsing | arguments? Can you throw an exception if the type you try to cast to | is not the one you expect? If you don't require an (implicit) cast and you don't use clone(), nothing gets copied. That;s how proxy objects around SEXP work. | You might imagine that with large datasets this is important. You can also use XPtr, and XPtr in combination with bigmemory's big.matrix, to keep data away from R. Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
Le 12 févr. 2014 à 13:36, Alessandro Mammana mamm...@molgen.mpg.de a écrit :
Ah wait, my bad (as always T.T), I found a much simpler explanation:
colset - sample(3e7-nr, 1e7)
storage.mode(colset)
[1] integer
storage.mode(colset-1)
[1] double
So when I was unwrapping colset I allocated new memory in Rcpp to
convert from double to integer, which was no longer valid when I went
out of scope.
I think it is a bit dangerous that you never know if you are
allocating memory or just wrapping R objects when parsing arguments in
Rcpp.
Is there a way of ensuring that NOTHING gets copied when parsing
arguments? Can you throw an exception if the type you try to cast to
is not the one you expect?
You might imagine that with large datasets this is important.
Silent coercion was added by design. Rcpp does not give you a « strict » mode.
One thing you can do is something like this:
#include Rcpp.h
using namespace Rcpp ;
template typename T
class Strict : public T {
public:
Strict( SEXP x ) {
if( TYPEOF(x) != T::r_type::value )
stop( not compatible ) ;
T::Storage::set__(x) ;
}
} ;
// [[Rcpp::export]]
int foo( StrictNumericVector v ){
return v.size() ;
}
You’d get e.g.
foo(rnorm(10))
[1] 10
foo(1:10)
Error in eval(expr, envir, enclos) : not compatible
Calls: sourceCpp ... source - withVisible - eval - eval - foo - Anonymous
Execution halted
Sorry for bothering and thanks again,
Ale
On Wed, Feb 12, 2014 at 1:10 PM, Dirk Eddelbuettel e...@debian.org wrote:
On 12 February 2014 at 11:47, Alessandro Mammana wrote:
| Ok I was able to find the code causing the bug. So it looks like the
Thanks for the added detail.
| pointers you get from an Rcpp::Vector using .begin() become invalid
| after that the Rcpp::Vector goes out of scope (and this makes sense),
| what I do not understand is that this Rcpp::Vector was allocated in R
| and should still be living during the execution of the Rcpp call
| (that's why I wasn't expecting the pointer to be invalid).
|
| This is the exact code (the one above is probably fine):
| @@ in CPP @@i
|
| struct GapMat {
| int* ptr;
| int* colset;
| int nrow;
| int ncol;
|
|
| inline int* colptr(int col){
| return ptr + colset[col];
| }
|
| GapMat(){}
|
| GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
| ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
| };
|
|
| GapMat getGapMat(Rcpp::List gapmat){
| IntegerVector vec = gapmat[vec];
| IntegerVector pos = gapmat[colset];
| int nrow = gapmat[nrow];
|
| return GapMat(vec.begin(), pos.begin(), nrow, pos.length());
| }
|
| // [[Rcpp::export]]
| IntegerVector colSumsGapMat(Rcpp::List gapmat){
|
| GapMat mat = getGapMat(gapmat);
| IntegerVector res(mat.ncol);
|
| for (int i = 0; i mat.ncol; ++i){
| for (int j = 0; j mat.nrow; ++j){
| res[i] += mat.colptr(i)[j];
| }
| }
|
| return res;
| }
|
| @@ in R (with gdb debugger as suggested by Dirk)
@@i
| library(Rcpp)
| sourceCpp(scratchpad.cpp)
|
| vec - rnbinom(3e7, mu=0.1, size=1); storage.mode(vec) - integer
| nr - 80
|
| colset - sample(3e7-nr, 1e7)
| foo - vec[colset] #this is only to trigger some obscure garbage
| collection mechanisms...
|
| for (i in 1:10){
| colset - sample(3e7-nr, 1e7)
| gapmat - list(vec=vec, nrow=nr, colset=colset-1)
| cs - colSumsGapMat(gapmat)
| print(sum(cs))
| }
|
| [1] 8000
| [1] 8000
| [1] 80016890
| [1] 80008144
| [1] 80016022
| [1] 80021609
|
| Program received signal SIGSEGV, Segmentation fault.
| 0x718a5455 in GapMat::colptr (this=0x7fffc120, col=0) at
| scratchpad.cpp:295
| 295return ptr + colset[col];
|
| @@@
|
| Why did it happen? What should I do to make sure that my pointers
| remain valid? My goal is to convert safely some vectors or matrices
| that exist in R to some pointers, how can I do that?
Not sure. It looks fine at first instance. But then it's early in the morning
and I had very little coffee yet...
Maybe the fact that you tickle the gc() via vec[colset] has something to do
with it, maybe it has not. Maybe I would try the decomposition of the List
object inside the colSumsGapMat() function to keep it simpler. Or if you
_really_ want an external object to iterate over, memcpy it out.
With really large object, you may be stressing parts of the code that have
not been stressed the same way. If it breaks, you do get to keep both
pieces.
Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
--
Alessandro Mammana, PhD Student
Max Planck Institute for Molecular Genetics
Ihnestraße 63-73
D-14195 Berlin, Germany
___
Rcpp-devel mailing list
Re: [Rcpp-devel] Array into C++ with cppFunction
Hi Dirk, thanks for pointing me in the right direction.
My solution below for a 3D array (documenting for rubes like myself). N
library(Rcpp)
library(RcppArmadillo)
cppFunction('
double NewFunc( NumericVector ArrayR ) {
IntegerVector DimsA = ArrayR.attr(dim);
arma::cubeArrayC(ArrayR.begin(),DimsA[0],DimsA[1],DimsA[2],false);
double Output = ArrayC(1,2,3) + ArrayC(2,3,4);
return Output;
}', depends=RcppArmadillo)
NewFunc(array(1,dim=c(5,5,5))) # Test, works
On Tue, Feb 11, 2014 at 3:34 PM, Dirk Eddelbuettel e...@debian.org wrote:
(Resending, this time with CC to list. Sorry for the extra first post.
--Dirk)
On 11 February 2014 at 11:49, Nick Menzies wrote:
| Is there a way to pass an array into c++ when using cppFunction? I
realize
| arrays have been discussed a number of times, but the solutions I have
found
| (e.g. http://markovjumps.blogspot.com/2011/12/
| r-array-to-rcpparmadillo-cube.html, or https://www.mail-archive.com/
| rcpp-devel@lists.r-forge.r-project.org/msg05897.html) are described in
the
| context of cxxfunction, and I haven't been able to work how to implement
this
| with cppFunction.
An array is just a vector with a dimension attribute. Treat it the same at
the C++ level.
There are several posts at the Rcpp Gallery dealing with reading / setting
attributes, and you should know how to pass a single vector through.
Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
--
Nick Menzies
nick.menz...@gmail.com
404 217 1076
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Array into C++ with cppFunction
hi nick,
On 12 February 2014 at 09:23, Nick Menzies wrote:
| Hi Dirk, thanks for pointing me in the right direction.
|
| My solution below for a 3D array (documenting for rubes like myself). N
|
| library(Rcpp)
| library(RcppArmadillo)
|
| cppFunction('
| double NewFunc( NumericVector ArrayR ) {
| IntegerVector DimsA = ArrayR.attr(dim);
| arma::cube ArrayC(ArrayR.begin(),DimsA[0],DimsA[1],DimsA
| [2],false);
| double Output = ArrayC(1,2,3) + ArrayC(2,3,4);
| return Output;
| }', depends=RcppArmadillo)
|
| NewFunc(array(1,dim=c(5,5,5))) # Test, works
Looks good -- do you want to roll this up with a paragraph or two of
motivation and explanation and submit to the Rcpp Gallery?
Dirk
| On Tue, Feb 11, 2014 at 3:34 PM, Dirk Eddelbuettel e...@debian.org wrote:
|
|
| (Resending, this time with CC to list. Sorry for the extra first post.
| --Dirk)
|
| On 11 February 2014 at 11:49, Nick Menzies wrote:
| | Is there a way to pass an array into c++ when using cppFunction? I
| realize
| | arrays have been discussed a number of times, but the solutions I have
| found
| | (e.g. http://markovjumps.blogspot.com/2011/12/
| | r-array-to-rcpparmadillo-cube.html, or https://www.mail-archive.com/
| | rcpp-devel@lists.r-forge.r-project.org/msg05897.html) are described in
| the
| | context of cxxfunction, and I haven't been able to work how to implement
| this
| | with cppFunction.
|
| An array is just a vector with a dimension attribute. Treat it the same at
| the C++ level.
|
| There are several posts at the Rcpp Gallery dealing with reading / setting
| attributes, and you should know how to pass a single vector through.
|
| Dirk
|
| --
| Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
|
|
|
|
| --
|
| Nick Menzies
| nick.menz...@gmail.com
| 404 217 1076
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
___
Rcpp-devel mailing list
Rcpp-devel@lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Rolling ADF test
On Wed, Feb 12, 2014 at 2:58 AM, Hideyoshi Maeda hideyoshi.ma...@gmail.comwrote: Dear Rcpp-devel list, Wasn't sure if this got sent last time as I didn't get a response. For next time, you can check the archives if you are not sure if your message was sent to the list. http://lists.r-forge.r-project.org/pipermail/rcpp-devel/2014-February/thread.html I am looking to carry out a fixed window length rolling ADF test (with no intercept), over some time series data currently in xts format. To do this I need to first fit a regression to the data, then use the residuals as an input into the ADF test, from which I can get a p-value to see if I can reject the idea of the data having a unit root or not. I know I can use fastLm() from RcppArmadillo for the regression, and I know that the ADF test also runs an OLS regression as part of it too, I would have assumed finding unit roots to be a reasonably common use, but currently can't find any out of the box functions that carry out C++ optimised ADF tests, are there any that can easily be sourced/included into Rcpp? I haven't come across any out of the box optimized rolling regression functions. For now I have been using the TTR package to do simple efficient rolling regressions. See my answer to this question http://stackoverflow.com/questions/11873123/rolling-regression-over-multiple-columns-in-r I'd be interested in contributing some using Rcpp/Armadillo if someone can tell me which package they would fit in. If we had a versatile rolling regression function it could be made to do efficient rolling ADF tests as well. HTH Regards Sameer ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Segfault, is it because of iterators/pointers?
I like the Strict mode idea, I will use it, thanks!
On Wed, Feb 12, 2014 at 2:34 PM, Romain Francois
rom...@r-enthusiasts.com wrote:
Le 12 févr. 2014 à 13:36, Alessandro Mammana mamm...@molgen.mpg.de a écrit :
Ah wait, my bad (as always T.T), I found a much simpler explanation:
colset - sample(3e7-nr, 1e7)
storage.mode(colset)
[1] integer
storage.mode(colset-1)
[1] double
So when I was unwrapping colset I allocated new memory in Rcpp to
convert from double to integer, which was no longer valid when I went
out of scope.
I think it is a bit dangerous that you never know if you are
allocating memory or just wrapping R objects when parsing arguments in
Rcpp.
Is there a way of ensuring that NOTHING gets copied when parsing
arguments? Can you throw an exception if the type you try to cast to
is not the one you expect?
You might imagine that with large datasets this is important.
Silent coercion was added by design. Rcpp does not give you a strict
mode.
One thing you can do is something like this:
#include Rcpp.h
using namespace Rcpp ;
template typename T
class Strict : public T {
public:
Strict( SEXP x ) {
if( TYPEOF(x) != T::r_type::value )
stop( not compatible ) ;
T::Storage::set__(x) ;
}
} ;
// [[Rcpp::export]]
int foo( StrictNumericVector v ){
return v.size() ;
}
You'd get e.g.
foo(rnorm(10))
[1] 10
foo(1:10)
Error in eval(expr, envir, enclos) : not compatible
Calls: sourceCpp ... source - withVisible - eval - eval - foo -
Anonymous
Execution halted
Sorry for bothering and thanks again,
Ale
On Wed, Feb 12, 2014 at 1:10 PM, Dirk Eddelbuettel e...@debian.org wrote:
On 12 February 2014 at 11:47, Alessandro Mammana wrote:
| Ok I was able to find the code causing the bug. So it looks like the
Thanks for the added detail.
| pointers you get from an Rcpp::Vector using .begin() become invalid
| after that the Rcpp::Vector goes out of scope (and this makes sense),
| what I do not understand is that this Rcpp::Vector was allocated in R
| and should still be living during the execution of the Rcpp call
| (that's why I wasn't expecting the pointer to be invalid).
|
| This is the exact code (the one above is probably fine):
| @@ in CPP @@i
|
| struct GapMat {
| int* ptr;
| int* colset;
| int nrow;
| int ncol;
|
|
| inline int* colptr(int col){
| return ptr + colset[col];
| }
|
| GapMat(){}
|
| GapMat(int* _ptr, int* _colset, int _nrow, int _ncol):
| ptr(_ptr), colset(_colset), nrow(_nrow), ncol(_ncol){}
| };
|
|
| GapMat getGapMat(Rcpp::List gapmat){
| IntegerVector vec = gapmat[vec];
| IntegerVector pos = gapmat[colset];
| int nrow = gapmat[nrow];
|
| return GapMat(vec.begin(), pos.begin(), nrow, pos.length());
| }
|
| // [[Rcpp::export]]
| IntegerVector colSumsGapMat(Rcpp::List gapmat){
|
| GapMat mat = getGapMat(gapmat);
| IntegerVector res(mat.ncol);
|
| for (int i = 0; i mat.ncol; ++i){
| for (int j = 0; j mat.nrow; ++j){
| res[i] += mat.colptr(i)[j];
| }
| }
|
| return res;
| }
|
| @@ in R (with gdb debugger as suggested by Dirk)
@@i
| library(Rcpp)
| sourceCpp(scratchpad.cpp)
|
| vec - rnbinom(3e7, mu=0.1, size=1); storage.mode(vec) - integer
| nr - 80
|
| colset - sample(3e7-nr, 1e7)
| foo - vec[colset] #this is only to trigger some obscure garbage
| collection mechanisms...
|
| for (i in 1:10){
| colset - sample(3e7-nr, 1e7)
| gapmat - list(vec=vec, nrow=nr, colset=colset-1)
| cs - colSumsGapMat(gapmat)
| print(sum(cs))
| }
|
| [1] 8000
| [1] 8000
| [1] 80016890
| [1] 80008144
| [1] 80016022
| [1] 80021609
|
| Program received signal SIGSEGV, Segmentation fault.
| 0x718a5455 in GapMat::colptr (this=0x7fffc120, col=0) at
| scratchpad.cpp:295
| 295return ptr + colset[col];
|
| @@@
|
| Why did it happen? What should I do to make sure that my pointers
| remain valid? My goal is to convert safely some vectors or matrices
| that exist in R to some pointers, how can I do that?
Not sure. It looks fine at first instance. But then it's early in the
morning
and I had very little coffee yet...
Maybe the fact that you tickle the gc() via vec[colset] has something to do
with it, maybe it has not. Maybe I would try the decomposition of the List
object inside the colSumsGapMat() function to keep it simpler. Or if you
_really_ want an external object to iterate over, memcpy it out.
With really large object, you may be stressing parts of the code that have
not been stressed the same way. If it breaks, you do get to keep both
pieces.
Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
--
Alessandro Mammana, PhD Student
Max Planck Institute for
[Rcpp-devel] Wrapper generation, keep source untouched
Hi All, I knew I wouldn't be able avoid Rcpp forever. :) I am about create R wrappers to a C++ library, old style, not templating. Rcpp attributes, and custom as and wrap() seem to be the way to go, and they are indeed very appealing. (If only they would have existed when I started with igraph!) My question is, is there a way you keep the original sources of the wrapped library intact? I would be updating the sources regularly, and I figured the best would be not to touch them at all. According to the docs, compileAttributes does not support this. Do you have anything against implementing this in Rcpp? If not, and you can help me coming up with some syntax that you like, I can code it up. Best, Gabor ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Wrapper generation, keep source untouched
On 12 February 2014 at 10:57, Gábor Csárdi wrote: | Hi All, I knew I wouldn't be able avoid Rcpp forever. :) We're happy to have you now :) I'm sure good things will come of this. | I am about create R wrappers to a C++ library, old style, not templating. Rcpp | attributes, and custom as and wrap() seem to be the way to go, and they are | indeed very appealing. (If only they would have existed when I started with | igraph!) | | My question is, is there a way you keep the original sources of the wrapped | library intact? I would be updating the sources regularly, and I figured the | best would be not to touch them at all. Did you see the 'Rcpp-extending.pdf' vignette about intrusive vs non-intrusive as and wrap? Of course, this was all written in the dark ages before Rcpp Attributes -- but latter uses when doing the magic. You may also want custom plugins and all that, for which you need the automagic conversion. | According to the docs, compileAttributes does not support this. Do you have | anything against implementing this in Rcpp? If not, and you can help me coming | up with some syntax that you like, I can code it up. Attributes can surely be extended, but some discussion or prototyping may not hurt. Cheers, Dirk -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Wrapper generation, keep source untouched
On Wed, Feb 12, 2014 at 11:41 AM, Dirk Eddelbuettel e...@debian.org wrote: [...] | My question is, is there a way you keep the original sources of the wrapped | library intact? I would be updating the sources regularly, and I figured the | best would be not to touch them at all. Did you see the 'Rcpp-extending.pdf' vignette about intrusive vs non-intrusive as and wrap? Sure, I have read that. This helps with the conversions, if I want to write the wrapper myself. Ideally I would have the wrapper generated by Rcpp attributes. Essentially what I am asking for is the possibility to avoid adding the // [[Rcpp::export]] lines (and other attributes configuration) to the original sources, but having these lines in a separate file (or files?). This file would be essentially the description of the wrapper to be generated by Rcpp attributes. I have to admit that I have only tried the Rcpp examples in the manuals so far, so please forgive me if what I am asking for is unreasonable, or is already solved. [...] | According to the docs, compileAttributes does not support this. Do you have | anything against implementing this in Rcpp? If not, and you can help me coming | up with some syntax that you like, I can code it up. Attributes can surely be extended, but some discussion or prototyping may not hurt. OK, so I'll open an issue for this in your github issue tracker, and fork the repo as well. G. [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] Wrapper generation, keep source untouched
OK, actually there is a simple way to achieve this, with minimal modifications to the Rcpp codebase: https://github.com/gaborcsardi/Rcpp/commit/8b160547d50d668099dff3802c01001baaf415b6 With this, I can put the functions I want to wrap to a separate file, that is essentially a header file, with contents like this: #include Rcpp.h using namespace Rcpp; // [[Rcpp::export]] List rcpp_hello_world(); The other reason why this is great is that I can now define potentially different conversions to arguments of the same C/C++ type. E.g. if I have (hypothetical) functions like this double mean(MyNumericVector* vector); void permute(MyNumericVector *vector); then I can write the different as conversion for the first vector, where I don't actually need to copy the SEXP, by simply putting this in the header file that defines the wrapping: double mean(in_MyNumericVector *vector); void permute(inout_MyNumericVector *vector); and then defining conversions for in_MyNumericVector and inout_MyNumericVector. All good. Best, Gabor On Wed, Feb 12, 2014 at 12:05 PM, Gábor Csárdi csardi.ga...@gmail.comwrote: On Wed, Feb 12, 2014 at 11:41 AM, Dirk Eddelbuettel e...@debian.orgwrote: [...] | My question is, is there a way you keep the original sources of the wrapped | library intact? I would be updating the sources regularly, and I figured the | best would be not to touch them at all. Did you see the 'Rcpp-extending.pdf' vignette about intrusive vs non-intrusive as and wrap? Sure, I have read that. This helps with the conversions, if I want to write the wrapper myself. Ideally I would have the wrapper generated by Rcpp attributes. Essentially what I am asking for is the possibility to avoid adding the // [[Rcpp::export]] lines (and other attributes configuration) to the original sources, but having these lines in a separate file (or files?). This file would be essentially the description of the wrapper to be generated by Rcpp attributes. I have to admit that I have only tried the Rcpp examples in the manuals so far, so please forgive me if what I am asking for is unreasonable, or is already solved. [...] | According to the docs, compileAttributes does not support this. Do you have | anything against implementing this in Rcpp? If not, and you can help me coming | up with some syntax that you like, I can code it up. Attributes can surely be extended, but some discussion or prototyping may not hurt. OK, so I'll open an issue for this in your github issue tracker, and fork the repo as well. G. [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
[Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
Hi, The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 which is the default compiler on the CRAN build machine (see http://cran.us.r-project.org/web/checks/check_flavors.html, the row r-release-macosx-x86_64). This has not been a problem so far because the package has not been rebuilt since November 5. But the next time the package is updated, it will fail to build on the CRAN mac build machine, unless the issue that causes the failure is addressed. The error is: [...] /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: note: Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = Rcpp::PreserveStorage] make: *** [Modules.o] Error 1 ERROR: compilation failed for package ‘ConConPiWiFun’ * removing ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ * restoring previous ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ This is the same error that has been discussed before on this list in the context of the Bioconductor package mzR. Apparently the problem is caused by the fact that g++ 4.2.1 does not create a default constructor; clang++ and newer versions of g++ can deal with this. This same problem occurs with other CRAN packages, among them: lm.br RSofia I didn't test exhaustively; there could be more. More concerning, installing the binary versions of these packages and then loading them causes errors that may be related to Rcpp: library(ConConPiWiFun) Loading required package: Rcpp Error in .doLoadActions(where, attach) : error in load action .__A__.1 for package ConConPiWiFun: loadModule(module = mod_cplfunction, what = TRUE, env = ns, : Unable to load module mod_cplfunction: unimplemented type 'any' in 'eval' Error: package or namespace load failed for ‘ConConPiWiFun’ library(lm.br) Loading required package: Rcpp *** caught segfault *** address 0x1d0031, cause 'memory not mapped' Error in .doLoadActions(where, attach) : error in load action .__A__.1 for package lm.br: loadModule(module = Clmbr, what = TRUE, env = ns, loadNow = TRUE): Unable to load module Clmbr: 'getCharCE' must be called on a CHARSXP Error: package or namespace load failed for ‘lm.br’ library(RSofia) Loading required package: Rcpp *** caught segfault *** address 0x0, cause 'unknown' [...] This suggests that these packages need to be rebuilt against Rcpp 0.11.0, and either patched to deal with the compiler/constructor issue, or perhaps that issue can be addressed in Rcpp itself. Thanks, Dan sessionInfo() R version 3.0.2 Patched (2013-12-18 r64488) Platform: x86_64-apple-darwin10.8.0 (64-bit) locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
Dan, [ Impossible Subject:, there is no Rcpp 0.11.1 amywhere yet. ] On 12 February 2014 at 17:19, Dan Tenenbaum wrote: | The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 which is the default compiler on the CRAN build machine (see http://cran.us.r-project.org/web/checks/check_flavors.html, the row r-release-macosx-x86_64). | | This has not been a problem so far because the package has not been rebuilt since November 5. But the next time the package is updated, it will fail to build on the CRAN mac build machine, unless the issue that causes the failure is addressed. | | The error is: | | [...] | /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: note: Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = Rcpp::PreserveStorage] | make: *** [Modules.o] Error 1 | ERROR: compilation failed for package ‘ConConPiWiFun’ | * removing ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | * restoring previous ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | | This is the same error that has been discussed before on this list in the context of the Bioconductor package mzR. | | Apparently the problem is caused by the fact that g++ 4.2.1 does not create a default constructor; clang++ and newer versions of g++ can deal with this. | | This same problem occurs with other CRAN packages, among them: | | lm.br | RSofia They all passed when I did my checks -- Ubuntu 13.10, g++-4.8. I do not claim to have the resources to test all package against all compilers. If you want to retest all CRAN package, I'll happily walk you through my scripts. Preferably off-list. | I didn't test exhaustively; there could be more. | | More concerning, installing the binary versions of these packages and then loading them causes errors that may be related to Rcpp: I was under the impression that you knew you needed to _recompile_. | library(ConConPiWiFun) | Loading required package: Rcpp | Error in .doLoadActions(where, attach) : | error in load action .__A__.1 for package ConConPiWiFun: loadModule(module = mod_cplfunction, what = TRUE, env = ns, : Unable to load module mod_cplfunction: unimplemented type 'any' in 'eval' | | Error: package or namespace load failed for ‘ConConPiWiFun’ | | | library(lm.br) | Loading required package: Rcpp | | *** caught segfault *** | address 0x1d0031, cause 'memory not mapped' | Error in .doLoadActions(where, attach) : | error in load action .__A__.1 for package lm.br: loadModule(module = Clmbr, what = TRUE, env = ns, loadNow = TRUE): Unable to load module Clmbr: 'getCharCE' must be called on a CHARSXP | Error: package or namespace load failed for ‘lm.br’ | | library(RSofia) | Loading required package: Rcpp | | *** caught segfault *** | address 0x0, cause 'unknown' | | [...] | | | This suggests that these packages need to be rebuilt against Rcpp 0.11.0, Yes, we knew. That is from the release announcement dated Feb 3 where I say [...] This does however mean that your current packages using Rcpp will break, so you need rebuild all packages using Rcpp this one time. [...] | and either patched to deal with the compiler/constructor issue, or perhaps | that issue can be addressed in Rcpp itself. Kevin has taken care of the mzR / constructor issue. May I once again suggest that you take advantage of the fact he works in your building / institution and try to sort this out locally? Thanks so much. Dirk | | Thanks, | Dan | | sessionInfo() | R version 3.0.2 Patched (2013-12-18 r64488) | Platform: x86_64-apple-darwin10.8.0 (64-bit) | | locale: | [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 | | attached base packages: | [1] stats graphics grDevices utils datasets methods base | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel -- Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
Hi Dirk, Sorry for the typo in my subject. Very briefly: There are three CRAN packages (probably more) that can't be run from binaries and they can't be recompiled (with CRAN's default compiler for the platform). Therefore anyone who wants to use these 3 packages on a Mac is SOL. They could compile them if they downloaded Xcode and clang. That's asking a lot more than we typically ask of CRAN users on a Mac. I'm talking about ordinary end users, not developers. All three packages can be fixed with a patch like the one Kevin made for mzR. But that patch belongs in Rcpp. If the patch is not applied to Rcpp, it does no good to other packages which we still haven't discovered won't build with g++ 4.2.1. If you knew that all packages with dependencies on Rcpp would need to be rebuilt, why did you not ask the maintainers of all these packages to bump their version numbers? That would solve the problem in many cases (or at least expose cases where there were g++ 4.2.1 compilation problems). Instead CRAN is pushing out unusable binaries to the world. Seems to me it's a lot simpler to do the rebuilding once -- on CRAN -- than to ask Mac and Windows non-developer users to do it, over and over, using software not installed by default on those platforms. This means that all the automated building that CRAN does every day is for naught, as far as these packages/platforms are concerned. I am trying to help, here. This is not about solving problems at Bioconductor--I'm trying to point out problems lurking on CRAN. It's wonderful that Kevin is in my building but that's not germane here. I think the patch required for Rcpp is trivial--Kevin can correct me if I'm wrong. Dan - Original Message - From: Dirk Eddelbuettel e...@debian.org To: Dan Tenenbaum dtene...@fhcrc.org Cc: rcpp-devel rcpp-devel@lists.r-forge.r-project.org Sent: Wednesday, February 12, 2014 6:09:38 PM Subject: Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++4.2.1 Dan, [ Impossible Subject:, there is no Rcpp 0.11.1 amywhere yet. ] On 12 February 2014 at 17:19, Dan Tenenbaum wrote: | The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 | which is the default compiler on the CRAN build machine (see | http://cran.us.r-project.org/web/checks/check_flavors.html, the | row r-release-macosx-x86_64). | | This has not been a problem so far because the package has not been | rebuilt since November 5. But the next time the package is | updated, it will fail to build on the CRAN mac build machine, | unless the issue that causes the failure is addressed. | | The error is: | | [...] | /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: | note: | Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const | Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = | Rcpp::PreserveStorage] | make: *** [Modules.o] Error 1 | ERROR: compilation failed for package ‘ConConPiWiFun’ | * removing | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | * restoring previous | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | | This is the same error that has been discussed before on this list | in the context of the Bioconductor package mzR. | | Apparently the problem is caused by the fact that g++ 4.2.1 does | not create a default constructor; clang++ and newer versions of | g++ can deal with this. | | This same problem occurs with other CRAN packages, among them: | | lm.br | RSofia They all passed when I did my checks -- Ubuntu 13.10, g++-4.8. I do not claim to have the resources to test all package against all compilers. If you want to retest all CRAN package, I'll happily walk you through my scripts. Preferably off-list. | I didn't test exhaustively; there could be more. | | More concerning, installing the binary versions of these packages | and then loading them causes errors that may be related to Rcpp: I was under the impression that you knew you needed to _recompile_. | library(ConConPiWiFun) | Loading required package: Rcpp | Error in .doLoadActions(where, attach) : | error in load action .__A__.1 for package ConConPiWiFun: | loadModule(module = mod_cplfunction, what = TRUE, env = ns, : | Unable to load module mod_cplfunction: unimplemented type | 'any' in 'eval' | | Error: package or namespace load failed for ‘ConConPiWiFun’ | | | library(lm.br) | Loading required package: Rcpp | | *** caught segfault *** | address 0x1d0031, cause 'memory not mapped' | Error in .doLoadActions(where, attach) : | error in load action .__A__.1 for package lm.br: | loadModule(module = Clmbr, what = TRUE, env = ns, loadNow = | TRUE): Unable to load module Clmbr: 'getCharCE' must be called | on a CHARSXP | Error: package or namespace load failed for ‘lm.br’ | |
Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
Yes, the patch is trivial. Dan, we can try installing the latest version of Rcpp on GitHub on the BioC Mac build machine and confirm that these packages do build on Mac with llvm-g++4.2. And I agree that either 1. CRAN needs to rebuild the binaries for packages linking to Rcpp, or 2. these package maintainers need to send a maintenance bump. Of course, they can do neither until we submit a fixed version of Rcpp to CRAN, because essentially the bug means, IIUC, that any package using Rcpp Modules cannot be built with llvm-g++4.2. My opinion is that we should submit a maintenance release after issues on everyone's favourite OS, Solaris, are confirmed to be worked out. Cheers, Kevin On Wed, Feb 12, 2014 at 6:44 PM, Dan Tenenbaum dtene...@fhcrc.org wrote: Hi Dirk, Sorry for the typo in my subject. Very briefly: There are three CRAN packages (probably more) that can't be run from binaries and they can't be recompiled (with CRAN's default compiler for the platform). Therefore anyone who wants to use these 3 packages on a Mac is SOL. They could compile them if they downloaded Xcode and clang. That's asking a lot more than we typically ask of CRAN users on a Mac. I'm talking about ordinary end users, not developers. All three packages can be fixed with a patch like the one Kevin made for mzR. But that patch belongs in Rcpp. If the patch is not applied to Rcpp, it does no good to other packages which we still haven't discovered won't build with g++ 4.2.1. If you knew that all packages with dependencies on Rcpp would need to be rebuilt, why did you not ask the maintainers of all these packages to bump their version numbers? That would solve the problem in many cases (or at least expose cases where there were g++ 4.2.1 compilation problems). Instead CRAN is pushing out unusable binaries to the world. Seems to me it's a lot simpler to do the rebuilding once -- on CRAN -- than to ask Mac and Windows non-developer users to do it, over and over, using software not installed by default on those platforms. This means that all the automated building that CRAN does every day is for naught, as far as these packages/platforms are concerned. I am trying to help, here. This is not about solving problems at Bioconductor--I'm trying to point out problems lurking on CRAN. It's wonderful that Kevin is in my building but that's not germane here. I think the patch required for Rcpp is trivial--Kevin can correct me if I'm wrong. Dan - Original Message - From: Dirk Eddelbuettel e...@debian.org To: Dan Tenenbaum dtene...@fhcrc.org Cc: rcpp-devel rcpp-devel@lists.r-forge.r-project.org Sent: Wednesday, February 12, 2014 6:09:38 PM Subject: Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++4.2.1 Dan, [ Impossible Subject:, there is no Rcpp 0.11.1 amywhere yet. ] On 12 February 2014 at 17:19, Dan Tenenbaum wrote: | The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 | which is the default compiler on the CRAN build machine (see | http://cran.us.r-project.org/web/checks/check_flavors.html, the | row r-release-macosx-x86_64). | | This has not been a problem so far because the package has not been | rebuilt since November 5. But the next time the package is | updated, it will fail to build on the CRAN mac build machine, | unless the issue that causes the failure is addressed. | | The error is: | | [...] | /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: | note: | Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const | Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = | Rcpp::PreserveStorage] | make: *** [Modules.o] Error 1 | ERROR: compilation failed for package 'ConConPiWiFun' | * removing | '/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun' | * restoring previous | '/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun' | | This is the same error that has been discussed before on this list | in the context of the Bioconductor package mzR. | | Apparently the problem is caused by the fact that g++ 4.2.1 does | not create a default constructor; clang++ and newer versions of | g++ can deal with this. | | This same problem occurs with other CRAN packages, among them: | | lm.br | RSofia They all passed when I did my checks -- Ubuntu 13.10, g++-4.8. I do not claim to have the resources to test all package against all compilers. If you want to retest all CRAN package, I'll happily walk you through my scripts. Preferably off-list. | I didn't test exhaustively; there could be more. | | More concerning, installing the binary versions of these packages | and then loading them causes errors that may be related to Rcpp: I was under the impression that you knew you needed to _recompile_. | library(ConConPiWiFun) |
Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
On 12 February 2014 at 18:44, Dan Tenenbaum wrote: | Hi Dirk, | | Sorry for the typo in my subject. | | Very briefly: | | There are three CRAN packages (probably more) that can't be run from binaries and they can't be recompiled (with CRAN's default compiler for the platform). | | Therefore anyone who wants to use these 3 packages on a Mac is SOL. They could compile them if they downloaded Xcode and clang. That's asking a lot more than we typically ask of CRAN users on a Mac. I'm talking about ordinary end users, not developers. | | All three packages can be fixed with a patch like the one Kevin made for mzR. But that patch belongs in Rcpp. If the patch is not applied to Rcpp, it does no good to other packages which we still haven't discovered won't build with g++ 4.2.1. | | If you knew that all packages with dependencies on Rcpp would need to be rebuilt, why did you not ask the maintainers of all these packages to bump their version numbers? Look at this repo, and its commits: https://github.com/RcppCore/rcpp-logs See eg the directory https://github.com/RcppCore/rcpp-logs/tree/master/patches/2014-01 Who do you think I wrote these patches for? I have no bone in the dogfight between Apple and the FSF, and it is not my fault that those users are stuck at g++-4.2.* -- for now, or forever. Romain works on a Mac, he possibly could test this but didn't care to. I don't own a mac. Exactly what do you want me to do? | That would solve the problem in many cases (or at least expose cases where there were g++ 4.2.1 compilation problems). Instead CRAN is pushing out unusable binaries to the world. Seems to me it's a lot simpler to do the rebuilding once -- on CRAN -- than to ask Mac and Windows non-developer users to do it, over and over, using software not installed by default on those platforms. This means that all the automated building that CRAN does every day is for naught, as far as these packages/platforms are concerned. | | I am trying to help, here. This is not about solving problems at Bioconductor--I'm trying to point out problems lurking on CRAN. It's wonderful that Kevin is in my building but that's not germane here. | | I think the patch required for Rcpp is trivial--Kevin can correct me if I'm wrong. Making Rcpp release is not trivial. As of today, we have 174 dependencies on CRAN alone. You on BioC have about another 17. If *you* need an interim release, make one locally has I have suggested before. With this, I will try to no longer respond to your emails. They soak up a fair amount of time and seem to achieve little. I'll cut my losses here. Dirk | | Dan | | | | | | - Original Message - | From: Dirk Eddelbuettel e...@debian.org | To: Dan Tenenbaum dtene...@fhcrc.org | Cc: rcpp-devel rcpp-devel@lists.r-forge.r-project.org | Sent: Wednesday, February 12, 2014 6:09:38 PM | Subject: Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1 | | | Dan, | | [ Impossible Subject:, there is no Rcpp 0.11.1 amywhere yet. ] | | On 12 February 2014 at 17:19, Dan Tenenbaum wrote: | | The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 | | which is the default compiler on the CRAN build machine (see | | http://cran.us.r-project.org/web/checks/check_flavors.html, the | | row r-release-macosx-x86_64). | | | | This has not been a problem so far because the package has not been | | rebuilt since November 5. But the next time the package is | | updated, it will fail to build on the CRAN mac build machine, | | unless the issue that causes the failure is addressed. | | | | The error is: | | | | [...] | | /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: | | note: | | Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const | | Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = | | Rcpp::PreserveStorage] | | make: *** [Modules.o] Error 1 | | ERROR: compilation failed for package ‘ConConPiWiFun’ | | * removing | | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | | * restoring previous | | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | | | | This is the same error that has been discussed before on this list | | in the context of the Bioconductor package mzR. | | | | Apparently the problem is caused by the fact that g++ 4.2.1 does | | not create a default constructor; clang++ and newer versions of | | g++ can deal with this. | | | | This same problem occurs with other CRAN packages, among them: | | | | lm.br | | RSofia | | They all passed when I did my checks -- Ubuntu 13.10, g++-4.8. I do | not | claim to have the resources to test all package against all | compilers. | | If you want to retest all CRAN package, I'll happily walk you through | my | scripts. Preferably off-list. | | | I didn't test exhaustively;
Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1
- Original Message - From: Kevin Ushey kevinus...@gmail.com To: Dan Tenenbaum dtene...@fhcrc.org Cc: Dirk Eddelbuettel e...@debian.org, rcpp-devel rcpp-devel@lists.r-forge.r-project.org Sent: Wednesday, February 12, 2014 6:53:01 PM Subject: Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1 Yes, the patch is trivial. Dan, we can try installing the latest version of Rcpp on GitHub on the BioC Mac build machine and confirm that these packages do build on Mac with llvm-g++4.2. OK, I will do that and get back to you. And I agree that either 1. CRAN needs to rebuild the binaries for packages linking to Rcpp, or 2. these package maintainers need to send a maintenance bump. Of course, they can do neither until we submit a fixed version of Rcpp to CRAN, because essentially the bug means, IIUC, that any package using Rcpp Modules cannot be built with llvm-g++4.2. My opinion is that we should submit a maintenance release after issues on everyone's favourite OS, Solaris, are confirmed to be worked out. That's great. Thanks, Dan Cheers, Kevin On Wed, Feb 12, 2014 at 6:44 PM, Dan Tenenbaum dtene...@fhcrc.org wrote: Hi Dirk, Sorry for the typo in my subject. Very briefly: There are three CRAN packages (probably more) that can't be run from binaries and they can't be recompiled (with CRAN's default compiler for the platform). Therefore anyone who wants to use these 3 packages on a Mac is SOL. They could compile them if they downloaded Xcode and clang. That's asking a lot more than we typically ask of CRAN users on a Mac. I'm talking about ordinary end users, not developers. All three packages can be fixed with a patch like the one Kevin made for mzR. But that patch belongs in Rcpp. If the patch is not applied to Rcpp, it does no good to other packages which we still haven't discovered won't build with g++ 4.2.1. If you knew that all packages with dependencies on Rcpp would need to be rebuilt, why did you not ask the maintainers of all these packages to bump their version numbers? That would solve the problem in many cases (or at least expose cases where there were g++ 4.2.1 compilation problems). Instead CRAN is pushing out unusable binaries to the world. Seems to me it's a lot simpler to do the rebuilding once -- on CRAN -- than to ask Mac and Windows non-developer users to do it, over and over, using software not installed by default on those platforms. This means that all the automated building that CRAN does every day is for naught, as far as these packages/platforms are concerned. I am trying to help, here. This is not about solving problems at Bioconductor--I'm trying to point out problems lurking on CRAN. It's wonderful that Kevin is in my building but that's not germane here. I think the patch required for Rcpp is trivial--Kevin can correct me if I'm wrong. Dan - Original Message - From: Dirk Eddelbuettel e...@debian.org To: Dan Tenenbaum dtene...@fhcrc.org Cc: rcpp-devel rcpp-devel@lists.r-forge.r-project.org Sent: Wednesday, February 12, 2014 6:09:38 PM Subject: Re: [Rcpp-devel] CRAN package does not build with Rcpp 0.11.1 and g++ 4.2.1 Dan, [ Impossible Subject:, there is no Rcpp 0.11.1 amywhere yet. ] On 12 February 2014 at 17:19, Dan Tenenbaum wrote: | The CRAN package ConConPiWiFun does not compile under g++ 4.2.1 | which is the default compiler on the CRAN build machine (see | http://cran.us.r-project.org/web/checks/check_flavors.html , the | row r-release-macosx-x86_64). | | This has not been a problem so far because the package has not | been | rebuilt since November 5. But the next time the package is | updated, it will fail to build on the CRAN mac build machine, | unless the issue that causes the failure is addressed. | | The error is: | | [...] | /Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include/Rcpp/Reference.h:34: | note: | Rcpp::Reference_ImplStoragePolicy::Reference_Impl(const | Rcpp::Reference_ImplStoragePolicy) [with StoragePolicy = | Rcpp::PreserveStorage] | make: *** [Modules.o] Error 1 | ERROR: compilation failed for package ‘ConConPiWiFun’ | * removing | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | * restoring previous | ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library/ConConPiWiFun’ | | This is the same error that has been discussed before on this | list | in the context of the Bioconductor package mzR. | | Apparently the problem is caused by the fact that g++ 4.2.1 does | not create a default constructor; clang++ and newer versions of | g++ can deal with this. | | This same problem occurs with other CRAN packages, among them: | | lm.br | RSofia They all passed when I did my checks -- Ubuntu 13.10, g++-4.8. I do not
