Re: [R] How to specify ff object filepaths when reading a CSV file into a ff data frame.
Really old subject…, so, all my apologizes for digging up
but, since I also ran into this… maybe this hack can be useful to someone
I propose monkey patching here:
library(ff)
my.as.ffdf.data.frame <- function (x, vmode = NULL, col_args = list(), ...)
{
rnam <- attr(x, "row.names")
if (is.integer(rnam)) {
if (all(rnam == seq_along(rnam)))
rnam <- NULL
else rnam <- as.character(rnam)
}
x <- as.list(x)
vmodes <- vector("list", length(x))
if (!is.null(vmode)) {
nam <- names(x)
if (is.list(vmode)) {
vnam <- names(vmode)
i <- match(vnam, .vmode[.vimplemented])
if (any(is.na(i)))
stop("vmodes not implemented: ", paste("'", vnam[is.na(i)],
"'", collapse = ",", sep = ""))
names(vmodes) <- nam
for (v in vnam) vmodes[vmode[[v]]] <- v
}
else {
vnam <- names(vmode)
if (is.null(vnam)) {
vmodes <- as.list(rep(vmode, length.out = length(x)))
}
else {
i <- match(vnam, nam)
if (any(is.na(i)))
stop("names not matched for vmode specification: ",
paste("'", vnam[is.na(i)], "'", collapse = ",",
sep = ""))
vmodes[i] <- as.list(vmode)
}
}
}
# <<< patching <<
# new to R, there are certainly better checks, or butter type than nested
list...
if(!is.null(names(col_args))){
if (is.null(col_args$pattern))
col_args$pattern <- "ffdf"
}else{
if(length(col_args)>0){
for(i in 1:length(col_args)){
tmp_col_args=col_args[[i]]
if (is.null(tmp_col_args$pattern))
tmp_col_args$pattern <- "ffdf"
}
}
}
# patching >
ret <- lapply(seq_along(x), function(i, ...) {
# <<< patching <<
if(is.null(names(col_args)) & length(col_args)>0){
col_args=col_args[[i]]
}
# patching >
xi <- x[[i]]
AsIs <- inherits(xi, "AsIs")
if (AsIs) {
oldClass(xi) <- oldClass(xi)[-match("AsIs", oldClass(xi))]
ret <- do.call("as.ff", c(list(xi, vmode = vmodes[[i]]),
col_args))
oldClass(ret) <- c("AsIs", oldClass(ret))
ret
}
else {
do.call("as.ff", c(list(xi, vmode = vmodes[[i]]),
col_args))
}
}, ...)
names(ret) <- names(x)
do.call("ffdf", c(ret, list(row.names = rnam), l))
}
# hook |
library(methods)
unlockBinding("as.ffdf.data.frame", as.environment("package:ff"))
assignInNamespace("as.ffdf.data.frame",
my.as.ffdf.data.frame,
ns="ff", envir=as.environment("package:ff"))
assign("as.ffdf.data.frame", my.as.ffdf.data.frame,
as.environment("package:ff"))
lockBinding("as.ffdf.data.frame", as.environment("package:ff"))
asffdf_args=list(col_args=list(
list(filename = "tmp/a.ff", overwrite=TRUE,
readonly=TRUE),
list(filename = "tmp/b.ff", overwrite=TRUE,
readonly=TRUE)))
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] How to specify ff object filepaths when reading a CSV file into a ff data frame.
Hi, I have done another simple test, I test the two syntext against a
CSV file with only one column, both success,
> fdf <- read.csv.ffdf(file="D:/rtemp/fftest2.csv",asffdf_args = list( col_args
> = list(filename=c("F:/a.f"
> fdf
ffdf (all open) dim=c(2,1), dimorder=c(1,2) row.names=NULL
ffdf virtual mapping
PhysicalName VirtualVmode PhysicalVmode AsIs VirtualIsMatrix
PhysicalIsMatrix PhysicalElementNo PhysicalFirstCol PhysicalLastCol
PhysicalIsOpen
col1 col1 integer integer FALSE FALSE
FALSE 11 1
TRUE
ffdf data
col1
11
22
> fdf <- read.csv.ffdf(file="D:/rtemp/fftest2.csv",asffdf_args = list( col_args
> = c(list(filename="D:/a2.f"
> fdf
ffdf (all open) dim=c(2,1), dimorder=c(1,2) row.names=NULL
ffdf virtual mapping
PhysicalName VirtualVmode PhysicalVmode AsIs VirtualIsMatrix
PhysicalIsMatrix PhysicalElementNo PhysicalFirstCol PhysicalLastCol
PhysicalIsOpen
col1 col1 integer integer FALSE FALSE
FALSE 11 1
TRUE
ffdf data
col1
11
22
>
Regards,
Xiaobo Gu
On Fri, Dec 24, 2010 at 11:27 PM, Xiaobo Gu wrote:
> Hi,
> The read.csv.ffdf function in package ff will create the ff object
> physical file in the default directories, I am trying to let the files
> created in the paths users specify, I think the point is to make use
> of the asffdf_args parameter,
> I have a test CSV file named D:\rtemp\fftest.csv, the content of the
> file is as following:
>
> col1,col2,col3
> 1,"amber",2.4
> 2,"linda",4.5
>
> I tried the following code, hoping ff will create the physical files
> for col1,col2 and col3 to D:/a.f,D:/b.f,D:/c.f respectively
>
> fdf <- read.csv.ffdf(file="D:/rtemp/fftest.csv",asffdf_args = list(
> col_args = c(list(filename="D:/a.f"), list(filename="D:/b.f"),
> list(filename="D:/c.f"
> and the error message is :
> Error in as.ff.default(1:2, vmode = NULL, filename = "D:/a.f",
> filename = "D:/b.f", :
> formal argument "filename" matched by multiple actual arguments
>
> I also tried the following:
>
>> fdf <- read.csv.ffdf(file="D:/rtemp/fftest.csv",asffdf_args = list( col_args
>> = list(filename=c("D:/a.f","D:/b.f","D:/c.f"
> Error in ff(initdata = initdata, length = length, levels = levels,
> ordered = ordered, :
> bad argument initdata for existing file; initializing existing file is
> invalid
> In addition: Warning messages:
> 1: In if (file.exists(filename)) { :
> the condition has length > 1 and only the first element will be used
> 2: In if (file.exists(filename)) { :
> the condition has length > 1 and only the first element will be used
> 3: In if (file.access(filename, 4) == -1) { :
> the condition has length > 1 and only the first element will be used
> 4: In if (file.access(filename, 2) == -1) { :
> the condition has length > 1 and only the first element will be used
> 5: In if (is.na(filesize)) stop("unable to open file") :
> the condition has length > 1 and only the first element will be used
>
> My questions are:
> 1. What's the datatype of the col_args parameter of the as.ffdf function
> 2. If I can make layout of the asffdf_args parameter correct, how can
> I set the exact filenames for each column of the ff data frame.
>
> Regards,
>
> Xiaobo Gu
>
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] How to specify ff object filepaths when reading a CSV file into a ff data frame.
Hi,
The read.csv.ffdf function in package ff will create the ff object
physical file in the default directories, I am trying to let the files
created in the paths users specify, I think the point is to make use
of the asffdf_args parameter,
I have a test CSV file named D:\rtemp\fftest.csv, the content of the
file is as following:
col1,col2,col3
1,"amber",2.4
2,"linda",4.5
I tried the following code, hoping ff will create the physical files
for col1,col2 and col3 to D:/a.f,D:/b.f,D:/c.f respectively
fdf <- read.csv.ffdf(file="D:/rtemp/fftest.csv",asffdf_args = list(
col_args = c(list(filename="D:/a.f"), list(filename="D:/b.f"),
list(filename="D:/c.f"
and the error message is :
Error in as.ff.default(1:2, vmode = NULL, filename = "D:/a.f",
filename = "D:/b.f", :
formal argument "filename" matched by multiple actual arguments
I also tried the following:
> fdf <- read.csv.ffdf(file="D:/rtemp/fftest.csv",asffdf_args = list( col_args
> = list(filename=c("D:/a.f","D:/b.f","D:/c.f"
Error in ff(initdata = initdata, length = length, levels = levels,
ordered = ordered, :
bad argument initdata for existing file; initializing existing file is invalid
In addition: Warning messages:
1: In if (file.exists(filename)) { :
the condition has length > 1 and only the first element will be used
2: In if (file.exists(filename)) { :
the condition has length > 1 and only the first element will be used
3: In if (file.access(filename, 4) == -1) { :
the condition has length > 1 and only the first element will be used
4: In if (file.access(filename, 2) == -1) { :
the condition has length > 1 and only the first element will be used
5: In if (is.na(filesize)) stop("unable to open file") :
the condition has length > 1 and only the first element will be used
My questions are:
1. What's the datatype of the col_args parameter of the as.ffdf function
2. If I can make layout of the asffdf_args parameter correct, how can
I set the exact filenames for each column of the ff data frame.
Regards,
Xiaobo Gu
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
