redirected to r-devel, because there are implementational details of [.data.frame discussed here. spoiler: at the bottom there is a fairly interesting performance result.
Romain Francois wrote: > > Hi, > > This is a bug I think. [.data.frame treats its arguments differently > depending on the number of arguments. you might want to hesitate a bit before you say that something in r is a bug, if only because it drives certain people mad. r is a carefully tested software, and [.data.frame is such a basic function that if what you talk about were a bug, it wouldn't have persisted until now. treating the arguments differently depending on their number is actually (if clearly...) documented: if there is one index (the 'i'), it selects columns. if there are two, 'i' selects rows. however, not all seems fine, there might be a design flaw: # dummy data frame d = structure(names=paste('col', 1:3, sep='.'), data.frame(row.names=paste('row', 1:3, sep='.'), matrix(1:9, 3, 3))) d[1:2] # correctly selects two first columns # 1:2 passed to [.data.frame as i, no j given d[,1:2] # correctly selects two first columns # 1:2 passed to [.data.frame as j, i given the missing argument value (note the comma) d[,i=1:2] # correctly selects two first rows # 1:2 passed to [.data.frame as i, j given the missing argument value (note the comma) d[j=1:2,] # correctly selects two first columns # 1:2 passed to [.data.frame as j, i given the missing argument value (note the comma) d[i=1:2] # correctly (arguably) selects the first two columns # 1:2 passed to [.data.frame as i, no j given d[j=1:2] # wrong: returns the whole data frame # does not recognize the index as i because it is explicitly named 'j' # does not recognize the index as j because there is only one index i say this *might* be a design flaw because it's hard to judge what the design really is. the r language definition (!) [1, sec. 3.4.3 p. 18] says: " The most important example of a class method for [ is that used for data frames. It is not be described in detail here (see the help page for [.data.frame, but in broad terms, if two indices are supplied (even if one is empty) it creates matrix-like indexing for a structure that is basically a list of vectors of the same length. If a single index is supplied, it is interpreted as indexing the list of columns—in that case the drop argument is ignored, with a warning." it does not say what happens when only one *named* index argument is given. from the above, it would indeed seem that there is a *bug* here: in the last example above only one index is given, and yet columns are not selected, even though the *language definition* says they should. (so it's not a documented feature, it's a contra-definitional misfeature -- a bug?) somewhat on the side, the 'matrix-like indexing' above is fairly misleading; just try the same patterns of indexing -- one index, two indices, named indices -- on a data frame and a matrix of the same shape: m = matrix(1:9, 3, 3) md = data.frame(m) md[1] # the first column m[1] # the first element (i.e., m[1,1]) md[,i=3] # third row m[,i=3] # third column the quote above refers to the ?'[.data.frame' for details. unfortunately, it the help page a lump of explanations for various '['-like operators, and it is *not* a definition of any sort. it does not provide much more detail on '[.data.frame' -- it is hardly as a design specification. in particular, it does not explain the issue of named arguments to '[.data.frame' at all. `[.data.frame` only is called with two arguments in the second case, > so > the following condition is true: > > if(Narg < 3L) { # list-like indexing or matrix indexing > > And then, the function assumes the argument it has been passed is i, > and > eventually calls NextMethod("[") which I think calls > `[.listof`(x,i,...), since i is missing in `[.data.frame` it is not > passed to `[.listof`, so you have something equivalent to as.list(d) > []. > > I think we can replace the condition with this one: > > if(Narg < 3L && !has.j) { # list-like indexing or matrix indexing > > or this: > > if(Narg < 3L) { # list-like indexing or matrix indexing > if(has.j) i <- j > indeed, for a moment i thought a trivial fix somewhere there would suffice. unfortunately, the code for [.data.frame [2, lines 500-641] is so clean and readable that i had to give up reading it, forget fixing. instead, i wrote an new version of '[.data.frame' from scratch. it fixes (or at least seems to fix, as far as my quick assessment goes) the problem. the function subdf (see the attached dataframe.r) is the new version of '[.data.frame': # dummy data frame d = structure(names=paste('col', 1:3, sep='.'), data.frame(row.names=paste('row', 1:3, sep='.'), matrix(1:9, 3, 3))) d[j=1:2] # incorrect: the whole data frame subdf(d, j=1:2) # correct, only the first two columns otherwise, subdf returns results equivalent (sensu all.equal; see below) to those returned by [.data.frame on the same input, modulo some more or less minor details. for example, i think the dropped-drop warnings go wrong in the original: d[1, drop=FALSE] # warning: drop argument will be ignored which suggests that dimensions will be dropped, while the intention is that the actual argument will be ignored and the value will be FALSE instead (while the default is TRUE, since i is specified). well, it's just one more confusing bit in r. the rewritten version warns about dropped drop only if it is explicitly TRUE: subdf(d, 1, drop=FALSE) # no warning subdf(d, 1, drop=TRUE) # warning another issue the differs in my version is that i don't see much sense in being able to select rows by indexing with NA: d[NA,1] # one row filled with NAs d[NA,] # data frame of the shape of d, filled with NAs which is incoherent with how NA are treated in columns indices (i.e., raise an error). the rewritten version raises an error if any element of any index is an NA. these minor differences are easily modifiable should compliance with the original 'design' be desirable. interestingly, there is a reduction in code by some 40 lines (~30%) wrt. the original, even though the new code is quite redundant (but thus were the original, too). with a little effort, it can be compressed further, but i felt it would become more convoluted and less readable, and also less efficient. procedural abstraction could help, but would also negatively impact performance. (presumably, an implementation in c would run faster.) incidentally (here's the best part!), my version seems to perform much better than the original, at least in a limited set of naive benchmarks. here are some results, which you can (hopefully) reproduce using the code in the attached test.r. the data is a dummy df with 1k rows and 1k columns, filled with rnorm; each indexing was repeated 1000 times for both the original and the modified version: original patched ratio test 1 0.002 0.001 2.00 d[] 2 0.027 0.001 27.00 d[drop = FALSE] 3 0.025 0.002 12.50 d[drop = TRUE] 4 0.026 0.002 13.00 d[, drop = FALSE] 5 0.026 0.003 8.67 d[, drop = TRUE] 6 1.274 0.002 637.00 d[, ] 7 1.255 0.001 1255.00 d[, , ] 8 1.183 0.001 1183.00 d[, , drop = FALSE] 9 1.183 0.003 394.33 d[, , drop = TRUE] 10 0.013 0.011 1.18 d[r] 11 0.040 0.034 1.18 d[r, drop = TRUE] 12 0.037 0.010 3.70 d[r, drop = FALSE] 13 0.012 0.011 1.09 d[i = r] 14 0.036 0.034 1.06 d[i = r, drop = TRUE] 15 0.037 0.011 3.36 d[i = r, drop = FALSE] 16 0.222 0.163 1.36 d[rr] 17 0.247 0.112 2.21 d[rr, drop = FALSE] 18 0.204 0.144 1.42 d[rr, drop = TRUE] 19 0.174 0.120 1.45 d[i = rr] 20 0.201 0.125 1.61 d[i = rr, drop = FALSE] 21 0.215 0.147 1.46 d[i = rr, drop = TRUE] 22 2.266 1.159 1.96 d[rr, ] 23 2.236 1.164 1.92 d[rr, , drop = FALSE] 24 2.275 1.171 1.94 d[rr, , drop = TRUE] 25 2.269 1.165 1.95 d[i = rr, ] 26 2.264 1.155 1.96 d[i = rr, , drop = FALSE] 27 2.290 1.189 1.93 d[i = rr, , drop = TRUE] 28 2.301 1.198 1.92 d[, i = rr] 29 2.239 1.158 1.93 d[, i = rr, drop = FALSE] 30 2.310 1.161 1.99 d[, i = rr, drop = TRUE] 31 0.002 0.003 0.67 d[j = c] 32 0.026 0.011 2.36 d[j = c, drop = FALSE] 33 0.026 0.003 8.67 d[j = c, drop = TRUE] 34 0.001 0.111 0.01 d[j = cc] 35 0.025 0.110 0.23 d[j = cc, drop = FALSE] 36 0.025 0.111 0.23 d[j = cc, drop = TRUE] 37 0.243 0.051 4.76 d[rr, cc] 38 0.243 0.051 4.76 d[rr, cc, drop = FALSE] 39 0.244 0.050 4.88 d[rr, cc, drop = TRUE] 40 0.244 0.051 4.78 d[i = rr, cc] 41 0.243 0.050 4.86 d[i = rr, cc, drop = FALSE] 42 0.244 0.051 4.78 d[i = rr, cc, drop = TRUE] 43 0.243 0.052 4.67 d[cc, i = rr] 44 0.244 0.050 4.88 d[cc, i = rr, drop = FALSE] 45 0.247 0.052 4.75 d[cc, i = rr, drop = TRUE] 46 0.244 0.050 4.88 d[i = rr, j = cc] 47 0.244 0.051 4.78 d[i = rr, j = cc, drop = FALSE] 48 0.244 0.051 4.78 d[i = rr, j = cc, drop = TRUE] 49 0.244 0.051 4.78 d[j = cc, i = rr] 50 0.243 0.051 4.76 d[j = cc, i = rr, drop = FALSE] 51 0.245 0.051 4.80 d[j = cc, i = rr, drop = TRUE] 52 0.002 0.155 0.01 d[j = cn] 53 0.429 0.139 3.09 d[i = rn, j = cn] 54 1.791 0.690 2.60 d[i = c(TRUE, FALSE), j = c(FALSE, TRUE)] (note: the benchmark relies on a feature of rbenchmark that i have just added, so you may need to download/update the package before trying.) in some tests, the difference is two orders of magnitude; in some it's a factor of 2-5; in some there's no significant difference. in only a few cases, the original is way faster (e.g., tests 34 and 52), but this is because the original is wrong there (it simply ignores the index, so no wonder). all the expressions above used in benchmarking were also used to test the equivalence of output from the original and the new version (see test.r again), and all of them were negative (no difference) -- except for the cases where the original was wrong. i'd consider making a patch for src/library/base/R/dataframe.R, but there's a hack here: it seems that some code relies on some part of the 'design' that differs between the rewrite and the original, and the new code does not make (dataframe.R does, but then other sources fail). anyway, sourcing the attached dataframe.R suffices for testing. i will be happy to learn where my implementation, benchmarking, and/or result checking are naive or wrong in any way, as they surely are. vQ [1] http://cran.r-project.org/doc/manuals/R-lang.pdf [2] http://svn.r-project.org/R/trunk/src/library/base/R/dataframe.R
subdf = function(x, i, j, drop = TRUE) { # preliminary checks etc. if (!is.data.frame(x)) stop("the argument 'x' must be a data frame") has.i = !missing(i) has.j = !missing(j) has.drop = !missing(drop) nij = nargs() - has.drop - 1L if (has.i && any(is.na(i))) stop(sprintf('undefined %s selected', if (nij == 2L) 'rows' else 'columns')) if (has.j && any(is.na(j))) stop('undefined columns selected') # both i and j specified: select rows by i and columns by j, optional dropping if (has.i && has.j) { rownames = .row_names_info(x, 0L) nrows = length(rownames) if (is.logical(i)) { if (is.na(max(i))) stop('undefined rows selected') else i = seq(nrows)[i] } else if (is.character(i)) i = pmatch(i, rownames, duplicates.ok=TRUE) maxi = max(i) if (is.na(maxi) || maxi > nrows) stop('undefined rows selected') colnames = attr(x, 'names') ncols = length(colnames) if(is.logical(j)) j = seq(ncols)[j] else if (is.character(j)) j = pmatch(j, colnames, duplicates.ok=TRUE) maxj = max(j) if (is.na(maxj) || maxj > ncols) stop('undefined columns selected') if (drop) { if (length(j) == 1L) return (.subset2(x, j)[i]) if (has.drop && length(i) == 1L) return (lapply(.subset(x, j), '[', i)) } return (structure(lapply(.subset(x, j), '[', i), row.names={ if (any(duplicated(i))) make.unique(as.character(rownames[i])) else rownames[i] }, names={ if (any(duplicated(j))) make.unique(colnames[j]) else colnames[j] }, class='data.frame')) } # two-index form, only i specified: select rows by i, keep all columns, optional dropping if (has.i && nij == 2L) { rownames = .row_names_info(x, 0L) nrows = length(rownames) if (is.logical(i)) { if (is.na(max(i))) stop('undefined rows selected') else i = seq(nrows)[i] } else if (is.character(i)) i = pmatch(i, rownames, duplicates.ok=TRUE) maxi = max(i) if (is.na(maxi) || maxi > nrows) stop('undefined rows selected') if (drop) { if (length(x) == 1L) return (.subset2(x, 1)[i]) if (has.drop && length(i) == 1L) return (lapply(x, '[', i)) } return (structure(lapply(x, '[', i), row.names={ if (any(duplicated(i))) make.unique(as.character(rownames[i])) else rownames[i] }, names=attr(x, 'names'), class='data.frame')) } # one-index form, only i specified: select columns by i, no dropping if (has.i) { colnames = attr(x, 'names') ncols = length(colnames) if(is.logical(i)) i = seq(ncols)[i] else if (is.character(i)) i = pmatch(i, colnames, duplicates.ok=TRUE) maxi = max(i) if (is.na(maxi) || maxi > ncols) stop('undefined columns selected') if (has.drop && drop) warning('drop argument will be ignored') return (structure(.subset(x, i), row.names=attr(x, 'row.names'), names={ if (any(duplicated(i))) make.unique(colnames[i]) else colnames[i] }, class='data.frame')) } # one- or two-index form, only j specified: select columns by j, optional droping if (has.j) { colnames = attr(x, 'names') ncols = length(colnames) if(is.logical(j)) j = seq(ncols)[j] else if (is.character(j)) j = pmatch(j, colnames, duplicates.ok=TRUE) maxj = max(j) if (is.na(maxj) || maxj > ncols) stop('undefined columns selected') rownames = attr(x, 'row.names') if (drop) { if (length(j) == 1L) return (.subset2(x, j)) if (has.drop && length(rownames) == 1L) return (.subset(x, j)) } return (structure(.subset(x, j), row.names=rownames, names={ if (any(duplicated(j))) make.unique(colnames[j]) else colnames[j] }, class='data.frame')) } # neither i nor j specified: keep all columns and rows, optional or no dropping if (drop && has.drop) { if (length(attr(x, 'row.names')) == 1L) return (as.list(x)) if (length(x) == 1L) warning('drop argument will be ignored') } return (x) }
#!/usr/bin/r # prepare sample data nrow = 1000; ncol = 1000 d = structure(names=paste('col', 1:ncol, sep='.'), row.names=paste('row', 1:nrow, sep='.'), as.data.frame( matrix(rnorm(nrow*ncol), nrow=nrow, ncol=ncol))) r = nrow/2; rr = sample(nrow, nrow/10, replace=TRUE); rn = row.names(d)[rr] c = ncol/2; cc = sample(ncol, ncol/10, replace=TRUE); cn = names(d)[cc] # load subdf, the rewritten [.data.frame source('dataframe.r') # load a benchmarking routine library(rbenchmark) # compute the benchmark result = benchmark(replications=100, columns=c('test', 'elapsed'), order=NULL, # no indices d[], subdf(d), d[drop=FALSE], subdf(d, drop=FALSE), d[drop=TRUE], subdf(d, drop=TRUE), # empty i d[,drop=FALSE], subdf(d,,drop=FALSE), d[,drop=TRUE], subdf(d,,drop=TRUE), # empty i and j d[,], subdf(d, , ), d[,,], subdf(d, , , ), d[,,drop=FALSE], subdf(d, , , drop=FALSE), d[,,drop=TRUE], subdf(d, , , drop=TRUE), # scalar i, no j d[r], subdf(d, r), d[r, drop=TRUE], subdf(d, r, drop=TRUE), d[r, drop=FALSE], subdf(d, r, drop=FALSE), # explicit scalar i, no j d[i=r], subdf(d, i=r), d[i=r, drop=TRUE], subdf(d, i=r, drop=TRUE), d[i=r, drop=FALSE], subdf(d, i=r, drop=FALSE), # index i, no j d[rr], subdf(d, rr), d[rr, drop=FALSE], subdf(d, rr, drop=FALSE), d[rr, drop=TRUE], subdf(d, rr, drop=TRUE), # explicit index i, no j d[i=rr], subdf(d, rr), d[i=rr, drop=FALSE], subdf(d, rr, drop=FALSE), d[i=rr, drop=TRUE], subdf(d, rr, drop=TRUE), # index i, empty j d[rr, ], subdf(d, rr, ), d[rr, , drop=FALSE], subdf(d, rr, , drop=FALSE), d[rr, , drop=TRUE], subdf(d, rr, , drop=TRUE), # explicit index i, empty j d[i=rr, ], subdf(d, rr, ), d[i=rr, , drop=FALSE], subdf(d, rr, , drop=FALSE), d[i=rr, , drop=TRUE], subdf(d, rr, , drop=TRUE), d[, i=rr], subdf(d, , i=rr), d[, i=rr, drop=FALSE], subdf(d, , i=rr, drop=FALSE), d[, i=rr, drop=TRUE], subdf(d, , i=rr, drop=TRUE), # explicit scalar j, no i d[j=c], subdf(d, j=c), d[j=c, drop=FALSE], subdf(d, j=c, drop=FALSE), d[j=c, drop=TRUE], subdf(d, j=c, drop=TRUE), # explicit index j, no i d[j=cc], subdf(d, j=cc), d[j=cc, drop=FALSE], subdf(d, j=cc, drop=FALSE), d[j=cc, drop=TRUE], subdf(d, j=cc, drop=TRUE), # index i, index j d[rr, cc], subdf(d, rr, cc), d[rr, cc, drop=FALSE], subdf(d, rr, cc, drop=FALSE), d[rr, cc, drop=TRUE], subdf(d, rr, cc, drop=TRUE), # explicit index i, index j d[i=rr, cc], subdf(d, i=rr, cc), d[i=rr, cc, drop=FALSE], subdf(d, i=rr, cc, drop=FALSE), d[i=rr, cc, drop=TRUE], subdf(d, i=rr, cc, drop=TRUE), d[cc, i=rr], subdf(d, cc, i=rr), d[cc, i=rr, drop=FALSE], subdf(d, cc, i=rr, drop=FALSE), d[cc, i=rr, drop=TRUE], subdf(d, cc, i=rr, drop=TRUE), # explicit index i, explicit index j d[i=rr, j=cc], subdf(d, i=rr, j=cc), d[i=rr, j=cc, drop=FALSE], subdf(d, i=rr, j=cc, drop=FALSE), d[i=rr, j=cc, drop=TRUE], subdf(d, i=rr, j=cc, drop=TRUE), d[j=cc, i=rr], subdf(d, j=cc, i=rr), d[j=cc, i=rr, drop=FALSE], subdf(d, j=cc, i=rr, drop=FALSE), d[j=cc, i=rr, drop=TRUE], subdf(d, j=cc, i=rr, drop=TRUE), # arguably incorrect call # explicit name index i # d[i=rn], subdf(d, i=rn), # explicit name index j d[j=cn], subdf(d, j=cn), # explicit name index i, explicit name index j d[i=rn, j=cn], subdf(d, i=rn, j=cn), # explicit logical index i, explicit logical index j d[i=c(TRUE,FALSE), j=c(FALSE,TRUE)], subdf(d, i=c(TRUE,FALSE), j=c(FALSE,TRUE)) ) # benchmark # collect and print the results patched = 2*(1:(nrow(result)/2)); original= patched-1 output = with(result, data.frame( original=elapsed[original], patched=elapsed[patched], ratio=round(elapsed[original]/elapsed[patched], digits=2), test=test[original])) print(output, right=FALSE) # check equivalence of results check = function(original, patched) { args = list(original=deparse(substitute(original)), patched=deparse(substitute(patched))) if (!isTRUE(all.equal(original, patched, check.attributes=FALSE))) with(args, cat(sprintf('*** %s and %s differ\n', original, patched))) } # no indices check(d[], subdf(d)) check(d[drop=FALSE], subdf(d, drop=FALSE)) check(d[drop=TRUE], subdf(d, drop=TRUE)) # empty i check(d[,drop=FALSE], subdf(d,,drop=FALSE)) check(d[,drop=TRUE], subdf(d,,drop=TRUE)) # empty i and j check(d[,], subdf(d, , )) check(d[,,], subdf(d, , , )) check(d[,,drop=FALSE], subdf(d, , , drop=FALSE)) check(d[,,drop=TRUE], subdf(d, , , drop=TRUE)) # scalar i, no j check(d[r], subdf(d, r)) check(d[r, drop=TRUE], subdf(d, r, drop=TRUE)) check(d[r, drop=FALSE], subdf(d, r, drop=FALSE)) # explicit scalar i, no j check(d[i=r], subdf(d, i=r)) check(d[i=r, drop=TRUE], subdf(d, i=r, drop=TRUE)) check(d[i=r, drop=FALSE], subdf(d, i=r, drop=FALSE)) # index i, no j check(d[rr], subdf(d, rr)) check(d[rr, drop=FALSE], subdf(d, rr, drop=FALSE)) check(d[rr, drop=TRUE], subdf(d, rr, drop=TRUE)) # explicit index i, no j check(d[i=rr], subdf(d, rr)) check(d[i=rr, drop=FALSE], subdf(d, rr, drop=FALSE)) check(d[i=rr, drop=TRUE], subdf(d, rr, drop=TRUE)) # index i, empty j check(d[rr, ], subdf(d, rr, )) check(d[rr, , drop=FALSE], subdf(d, rr, , drop=FALSE)) check(d[rr, , drop=TRUE], subdf(d, rr, , drop=TRUE)) # explicit index i, empty j check(d[i=rr, ], subdf(d, rr, )) check(d[i=rr, , drop=FALSE], subdf(d, rr, , drop=FALSE)) check(d[i=rr, , drop=TRUE], subdf(d, rr, , drop=TRUE)) check(d[, i=rr], subdf(d, , i=rr)) check(d[, i=rr, drop=FALSE], subdf(d, , i=rr, drop=FALSE)) check(d[, i=rr, drop=TRUE], subdf(d, , i=rr, drop=TRUE)) # these three checks reveal difference # explicit scalar j, no i check(d[j=c], subdf(d, j=c)) check(d[j=c, drop=FALSE], subdf(d, j=c, drop=FALSE)) check(d[j=c, drop=TRUE], subdf(d, j=c, drop=TRUE)) # these three checks reveal difference # explicit index j, no i check(d[j=cc], subdf(d, j=cc)) check(d[j=cc, drop=FALSE], subdf(d, j=cc, drop=FALSE)) check(d[j=cc, drop=TRUE], subdf(d, j=cc, drop=TRUE)) # index i, index j check(d[rr, cc], subdf(d, rr, cc)) check(d[rr, cc, drop=FALSE], subdf(d, rr, cc, drop=FALSE)) check(d[rr, cc, drop=TRUE], subdf(d, rr, cc, drop=TRUE)) # explicit index i, index j check(d[i=rr, cc], subdf(d, i=rr, cc)) check(d[i=rr, cc, drop=FALSE], subdf(d, i=rr, cc, drop=FALSE)) check(d[i=rr, cc, drop=TRUE], subdf(d, i=rr, cc, drop=TRUE)) check(d[cc, i=rr], subdf(d, cc, i=rr)) check(d[cc, i=rr, drop=FALSE], subdf(d, cc, i=rr, drop=FALSE)) check(d[cc, i=rr, drop=TRUE], subdf(d, cc, i=rr, drop=TRUE)) # explicit index i, explicit index j check(d[i=rr, j=cc], subdf(d, i=rr, j=cc)) check(d[i=rr, j=cc, drop=FALSE], subdf(d, i=rr, j=cc, drop=FALSE)) check(d[i=rr, j=cc, drop=TRUE], subdf(d, i=rr, j=cc, drop=TRUE)) check(d[j=cc, i=rr], subdf(d, j=cc, i=rr)) check(d[j=cc, i=rr, drop=FALSE], subdf(d, j=cc, i=rr, drop=FALSE)) check(d[j=cc, i=rr, drop=TRUE], subdf(d, j=cc, i=rr, drop=TRUE)) # arguably incorrect call # explicit name index i # d[i=rn], subdf(d, i=rn), # this check reveals difference # explicit name index j check(d[j=cn], subdf(d, j=cn)) # explicit name index i, explicit name index j check(d[i=rn, j=cn], subdf(d, i=rn, j=cn)) # explicit logical index i, explicit logical index j check(d[i=c(TRUE,FALSE), j=c(FALSE,TRUE)], subdf(d, i=c(TRUE,FALSE), j=c(FALSE,TRUE)))
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.