As a purely R question though, the OP's suggested code is rather efficient, though I wouldn't be surprised if a hair more efficiency couldn't be achieved by pre-allocating the sensorReadings[,2] instead of calling it twice. Probably a speed/memory trade-off at that point.
If he has enough data that this one liner bumps up on these sorts of performance constraints, R might not be the best platform: it's a rather straightforward problem in C (or friends) and converting to a rolling/accumulating median algorithm would let this be a single loop in C and pretty efficient -- also nicely adaptable to real-time work. stats::runmed provides a starting point if you need code. It's tangential (and perhaps I'm reading too much of myself in the problem), but this is not an uncommon problem in finance: the OP might want to look at the TTR package for various ideas and good implementations. Michael On Tue, Apr 3, 2012 at 12:15 AM, Bert Gunter <gunter.ber...@gene.com> wrote: > I strongly suggest you consult with a local statistician. Your > description is far too vague (to me anyway) to make any sense of and > probably requires a good deal of back and forth between you and a > competent data analyst to pin down what the issues and constraints > are. For example, what constitutes "interesting" patterns? -- do the > data need to be analyzed in real time? -- Are visual displays > sufficient or is some kind of numeric indication needed? ... etc. etc. > > One tentative word of advice, though: with this much data, do not get > embroiled with P values or other measures of statistical > "significance": anything you can see will be "significant." > > -- Bert > > On Mon, Apr 2, 2012 at 8:42 PM, Hasan Diwan <hasan.di...@gmail.com> wrote: >> I have approximately 2.5 million rows from a number of sensor >> readings. Having plotted these, I can see a given pattern (say a spike >> in the amplitude away from the mean). I would now like to automate >> this procedure as we're expecting a great deal more data in the near >> future. Is there any package or function that will make this possible? >> Many thanks! I suppose, I could do something like: >> amplitude <- abs(sensorReadings[sensorReadings[,2] > 1.2 * >> median(sensorReadings[,2]),]) >> Is this the most efficient way to do what I want or not? >> -- H >> >> -- >> Sent from my mobile device >> Envoyait de mon portable >> >> ______________________________________________ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. > > > > -- > > Bert Gunter > Genentech Nonclinical Biostatistics > > Internal Contact Info: > Phone: 467-7374 > Website: > http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.