Jeff, Clearly you (and others) have put a lot of work into xts -- and I'm the beneficiary. So I'll stop complaining.
Thanks for the class (both code and explanation). *-- Russ * On Sun, May 8, 2011 at 8:23 PM, Jeff Ryan <jeff.a.r...@gmail.com> wrote: > Hi Russ, > > We're of course getting into some incredibly fine-level detail on how all > of this works. I'll try and explain issues as I recall them over the > development of xts and cbind.xts > > xts started as an extension of zoo. zoo is an extension of 'ts' (greatly > simplified comparison of course, but stay with me) > > Achim and Gabor have put tremendous effort into the design of zoo - with a > primary focus on keeping it consistent with base R behavior. That is, try > not to introduce unnecessary changes to the interface an R user is > accustomed to. The logic being that this makes for a more consistent > interface as well as a easier learning curve and hence greater/faster > adoption rate. > > 'xts' extends this, though with a bit more flexibility in terms of > consistency. Why? Simply put - some things about R annoyed me coming from a > time-series background. Number one was the fact that lag() is backwards. > Backwards from expectation, nearly all literature, and all standard > definitions. So xts breaks with lag(, n=1) behavior. This is obviously > confusing to some - but was the gamble I was willing to take - consistency > (with R) be damned! ;-) > > So, now back to cbind. cbind and merge in zoo-land (and xts by extension) > are the same. This isn't the case for other classes that use these - but > that is 'allowable' and 'expected' under a class dispatch system. The docs > for ?cbind state: > > For cbind (rbind) the column (row) names are taken from the > colnames (rownames) of the arguments if these are matrix-like. > Otherwise from the names of the arguments or where those are not > supplied and deparse.level > 0, by deparsing the expressions > given, for deparse.level = 1 only if that gives a sensible name > (a symbol, see is.symbol). > > Based on that, I'd argue that xts does it "right". Of course I'll also > point out that this is incorrect thinking as well - since this is a > description for the generic - and not for xts. But again in a highly > configurable object/class system, where you start to make a distinction of > right and wrong is itself up for debate. > > At the other end of the argument spectrum is _why not_. That is, why can't > cbind.xts handle the names to replace the colnames of objects passed in. > Here is where I'll point out that I am really just going by memory. > > Three major items are involved in cbind. One is that dispatch is quite > unlike nearly every other dispatch in R. This is a fact - nothing to do > with xts. > > * cbind isn't a generic (it's an .Internal call) > * it uses ... > * cbind can be called in numerous ways (I'll list only the common ones - > but with R you can do even crazier things) > > do.call(cbind, > do.call(cbind.xts, > cbind, > cbind.xts, > merge, > merge.xts, > do.call(merge, > do.call(merge.xts > > The rules of dispatch on cbind are really at a level that R-help has no > business discussing. The second part is where things actually get tricky > though. They all behave differently with respect to how args are handled - > when eval'd, etc. > > I'm sure you have read how R strains itself on 'big data'. This is true > and false. Improper use (or just naive use) can cause object copies in > places you really don't want. Much of xts at this point is implemented in > custom C code. The gain here is that you can make it eas(ier) to avoid > copies until you need them by writing in C. Obvious, but needs to be said. > > To figure out what the columns have - and if names are attached to the > objects in the pairlist (the "..." in this context) - you have to be very > careful. Touch anything in the wrong place or wrong time and you lose a > figurative arm and leg to memory copies. So, in 99.9999% of cases - where > you aren't naming (which would be an extra feature above and beyond c(olumn) > binding [the reason for cbind] - you run a very real risk of getting nailed > for copies you don't want. On 10MM obs that is almost manageable. On 100's > of millions or billions - it is kill -9 time. > > To compound the issue - recall all of those different dispatch methods. > Yep - they all behave just a bit differently. How? Honestly - I don't > know or care. I simply know you can't easily make the behavior consistent > amongst those calls. I have tried. And tried. > > End of day, and a very long R-help email, xts is different than base R. It > is even different than it's 'parent' zoo behavior. But in exchange for this > difference (and bit of learning/adjustment) you get a class that is faster > than anything else. > > Period. > > > x <- .xts(1:1e7, 1:1e7) # our time series object > > m <- coredata(x) # a matrix > > > str(x) > An xts object from 1969-12-31 18:00:01 to 1970-04-26 12:46:40 containing: > Data: int [1:10000000, 1] 1 2 3 4 5 6 7 8 9 10 ... > Indexed by objects of class: [POSIXt,POSIXct] TZ: America/Chicago > xts Attributes: > NULL > > > str(m) > int [1:10000000, 1] 1 2 3 4 5 6 7 8 9 10 ... > > > system.time(x[,1]) # get the first column > user system elapsed > 0.017 0.000 0.017 > > system.time(m[,1]) # ditto > user system elapsed > 0.152 0.000 0.153 > > Yep, nearly 10x faster than a matrix op - AND you still have the time > index. To get there you need to sometimes make sacrifices. xts does, though > I like to think they are well thought out and consistent* > > *enough ;-) > > Best, > Jeff > > > On Sun, May 8, 2011 at 8:57 PM, Joshua Ulrich <josh.m.ulr...@gmail.com>wrote: > >> Russ, >> >> On May 8, 2011 6:29 PM, "Russ Abbott" <russ.abb...@gmail.com> wrote: >> > >> > Hi Jeff, >> > >> > The xts class has some very nice features, and you have done a valuable >> > service in developing it. >> > >> > My primary frustration is how difficult it seems to be to find out what >> went >> > wrong when my code doesn't work. I've been writing quite sophisticated >> code >> > for a fairly long time. It's not that I'm new to software development. >> > >> > The column name rule is a good example. I'm willing to live with the >> rule >> > that column names are not changed for efficiency sake. What's difficult >> for >> > me is that I never saw that rule anywhere before. Of course, I'm not an >> R >> > expect. I've been using it for only a couple of months. But still, I >> would >> > have expected to run into a rule like that. >> > >> > Worse, since the rule is in conflict with the explicit intent of >> cbind--one >> > can name columns when using cbind; in fact the examples illustrate how >> to do >> > it--it would really be nice of cbind would issue a warning when one >> attempts >> > to rename a column in violation of that rule. Instead, cbind is silent, >> > giving no hint about what went wrong. >> > >> Naming columns is not the explicit intent of cbind. The explicit >> intent is to combine objects by columns. Please don't overstate the >> case. >> >> While the examples for the generic show naming columns, neither >> ?cbind.zoo or ?cbind.xts have such examples. That's a hint. >> >> > It's those sorts of things that have caused me much frustration. And >> it's >> > these sorts of things that seem pervasive in R. One never knows what >> one is >> > dealing with. Did something not work because there is a special case >> rule >> > that I haven't heard of? Did it not work because a special convenience >> was >> > programmed into a function in a way that conflicted with normal use? >> Since >> > these sorts of things seem to come up so often, I find myself feeling >> that >> > there is no good way to track down problems, which leads to a sense of >> > helplessness and confusion. That's not what one wants in a programming >> > language. >> > >> If that's not what one wants, one can always write their own >> programming language. >> >> Seriously, it seems like you want to rant more than understand what's >> going on. You have the R and xts help pages and the source code. The >> "Note" section of help(cbind) tells you that the method dispatch is >> different. It even tells you what R source file to look at to see how >> dispatching is done. Compare the relevant source files from >> base::cbind and xts::cbind.xts, look at the "R Language Definition" >> manual to see how method dispatch is normally done. >> >> But you've been writing quite sophisticated code for a fairly long >> time, so I'm not telling you anything you don't know... you just don't >> think you should have to do the legwork. >> >> > -- Russ >> > >> > >> >> -- >> Joshua Ulrich | FOSS Trading: www.fosstrading.com >> >> >> >> > On Sun, May 8, 2011 at 2:42 PM, Jeff Ryan <jeff.a.r...@gmail.com> >> wrote: >> > >> > > Hi Russ, >> > > >> > > Colnames don't get rewritten if they already exist. The reason is due >> to >> > > performance and how cbind is written at the R level. >> > > >> > > It isn't perfect per se, but the complexity and variety of dispatch >> that >> > > can take place for cbind in R, as it isn't a generic, is quite >> challenging >> > > to get to behave as one may hope. After years of trying I'd say it is >> > > nearly impossible to do what you want without causing horrible memory >> issues >> > > on non trivial objects they are use in production systems **using** >> xts on >> > > objects with billions of rows. Your simple case that has a simple >> > > workaround would cost everyone using in the other 99.999% of cases to >> pay a >> > > recurring cost that isn't tolerable. >> > > >> > > If this is frustrating to you you should stop using the class. >> > > >> > > Jeff >> > > >> > > Jeffrey Ryan | Founder | <jeffrey.r...@lemnica.com> >> > > jeffrey.r...@lemnica.com >> > > >> > > www.lemnica.com >> > > >> > > On May 8, 2011, at 2:07 PM, Russ Abbott <russ.abb...@gmail.com> >> wrote: >> > > >> > > I'm having troubles with the names of columns. >> > > >> > > quantmod deal with stock quotes. I've created an array of the first 5 >> > > closing prices from Jan 2007. (Is there a problem that the name is the >> same >> > > as the variable name? There shouldn't be.) >> > > >> > > > close >> > > >> > > close >> > > >> > > 2007-01-03 1416.60 >> > > >> > > 2007-01-04 1418.34 >> > > >> > > 2007-01-05 1409.71 >> > > >> > > 2007-01-08 1412.84 >> > > >> > > 2007-01-09 1412.11 >> > > >> > > >> > > When I try to create a more complex array by adding columns, the names >> get >> > > fouled up. Here's a simple example. >> > > >> > > > cbind(changed.close = close+1, zero = 0, close) >> > > >> > > close zero close.1 >> > > >> > > 2007-01-03 1417.60 0 1416.60 >> > > >> > > 2007-01-04 1419.34 0 1418.34 >> > > >> > > 2007-01-05 1410.71 0 1409.71 >> > > >> > > 2007-01-08 1413.84 0 1412.84 >> > > >> > > 2007-01-09 1413.11 0 1412.11 >> > > >> > > >> > > The first column should be called "changed.close", but it's called >> "close". >> > > The second column has the right name. The third column should be >> called >> > > "close" but it's called "close.1". Why is that? Am I missing >> something? >> > > >> > > If I change the order of the columns and let close have its original >> name, >> > > there is still a problem. >> > > >> > > > cbind(close, zero = 0, changed.close = close+1) >> > > >> > > close zero close.1 >> > > >> > > 2007-01-03 1416.60 0 1417.60 >> > > >> > > 2007-01-04 1418.34 0 1419.34 >> > > >> > > 2007-01-05 1409.71 0 1410.71 >> > > >> > > 2007-01-08 1412.84 0 1413.84 >> > > >> > > 2007-01-09 1412.11 0 1413.11 >> > > >> > > >> > > Now the names on the first two columns are ok, but the third column is >> > > still wrong. Again, why is that? Apparently it's not letting me >> assign a >> > > name to a column that comes from something that already has a name. >> Is that >> > > the way it should be? >> > > >> > > I don't get that same problem on a simpler example. >> > > >> > > >> > > > IX <- cbind(I=0, X=(1:3)) >> > > >> > > > IX >> > > >> > > I X >> > > >> > > [1,] 0 1 >> > > >> > > [2,] 0 2 >> > > >> > > [3,] 0 3 >> > > >> > > > cbind(Y = 1, Z = IX[, "I"], W = IX[, "X"]) >> > > >> > > Y Z W >> > > >> > > [1,] 1 0 1 >> > > >> > > [2,] 1 0 2 >> > > >> > > [3,] 1 0 3 >> > > >> > > >> > > Is this a peculiarity to xts objects? >> > > >> > > Thanks. >> > > >> > > *-- Russ * >> > > * >> > > * >> > > P.S. Once again I feel frustrated because it's taken me far more time >> than >> > > it deserves to track down and characterize this problem. I can fix it >> by >> > > using the names function. But I shouldn't have to do that. >> > > >> > > >> > >> > [[alternative HTML version deleted]] >> > >> > ______________________________________________ >> > R-help@r-project.org mailing list >> > https://stat.ethz.ch/mailman/listinfo/r-help >> > PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> > and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Jeffrey Ryan > > jeffrey.r...@lemnica.com > > www.lemnica.com > [[alternative HTML version deleted]]
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
