Re: [R] Speed up plotting to MSWindows graphics window
If you are plotting that many data points, you might want to look at 'hexbin' as a way of aggregating the values to a different presentation. It is especially nice if you are doing a scatter plot with a lot of data points and trying to make sense out of it. On Wed, Apr 27, 2011 at 5:16 AM, Jonathan Gabris wrote: > Hello, > > I am working on a project analysing the performance of motor-vehicles > through messages logged over a CAN bus. > > I am using R 2.12 on Windows XP and 7 > > I am currently plotting the data in R, overlaying 5 or more plots of data, > logged at 1kHz, (using plot.ts() and par(new = TRUE)). > The aim is to be able to pan, zoom in and out and get values from the > plotted graph using a custom Qt interface that is used as a front end to > R.exe (all this works). > The plot is drawn by R directly to the windows graphic device. > > The data is imported from a .csv file (typically around 100MB) to a matrix. > (timestamp, message ID, byte0, byte1, ..., byte7) > I then separate this matrix into several by message ID (dimensions are in > the order of 8cols, 10^6 rows) > > The panning is done by redrawing the plots, shifted by a small amount. So as > to view a window of data from a second to a minute long that can travel the > length of the logged data. > > My problem is that, the redrawing of the plots whilst panning is too slow > when dealing with this much data. > i.e.: I can see the last graphs being drawn to the screen in the half-second > following the view change. > I need a fluid change from one view to the next. > > My question is this: > Are there ways to speed up the plotting on the MSWindows display? > By reducing plotted point densities to *sensible* values? > Using something other than plot.ts() - is the lattice package faster? > I don't need publication quality plots, they can be rougher... > > I have tried: > -Using matrices instead of dataframes - (works for calculations but not > enough for plots) > -increasing the max usable memory (max-mem-size) - (no change) > -increasing the size of the pointer protection stack (max-ppsize) - (no > change) > -deleting the unnecessary leftover matrices - (no change) > -I can't use lines() instead of plot() because of the very different scales > (rpm-1, flags -1to3) > > I am going to do some resampling of the logged data to reduce the vector > sizes. > (removal of *less* important data and use of window.ts()) > > But I am currently running out of ideas... > So if sombody could point out something, I would be greatfull. > > Thanks, > > Jonathan Gabris > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up plotting to MSWindows graphics window
> Date: Wed, 27 Apr 2011 14:40:23 +0200 > From: jonat...@k-m-p.nl > To: r-help@r-project.org > Subject: Re: [R] Speed up plotting to MSWindows graphics window > > > On 27/04/2011 13:18, Mike Marchywka wrote: > > > >> > Date: Wed, 27 Apr 2011 11:16:26 +0200 > >> > From:jonat...@k-m-p.nl > >> > To:r-help@r-project.org > >> > Subject: [R] Speed up plotting to MSWindows graphics window > >> > > >> > Hello, > >> > > >> > I am working on a project analysing the performance of motor-vehicles > >> > through messages logged over a CAN bus. > >> > > >> > > >> > I am currently plotting the data in R, overlaying 5 or more plots of > >> > data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). > >> > The aim is to be able to pan, zoom in and out and get values from the > >> > plotted graph using a custom Qt interface that is used as a front end to > >> > R.exe (all this works). > >> > The plot is drawn by R directly to the windows graphic device. > >> > > >> > The data is imported from a .csv file (typically around 100MB) to a > >> > matrix. > >> > (timestamp, message ID, byte0, byte1, ..., byte7) > >> > I then separate this matrix into several by message ID (dimensions are > >> > in the order of 8cols, 10^6 rows) > >> > > >> > The panning is done by redrawing the plots, shifted by a small amount. > >> > So as to view a window of data from a second to a minute long that can > >> > travel the length of the logged data. > >> > > >> > My problem is that, the redrawing of the plots whilst panning is too > >> > slow when dealing with this much data. > >> > i.e.: I can see the last graphs being drawn to the screen in the > >> > half-second following the view change. > >> > I need a fluid change from one view to the next. > >> > > >> > My question is this: > >> > Are there ways to speed up the plotting on the MSWindows display? > >> > By reducing plotted point densities to*sensible* values? > > Well, hard to know but it would help to know where all the time is going. > > Usually people start complaining when VM thrashing is common but if you are > > CPU limited you could try restricting the range of data you want to plot > > rather than relying on the plot to just clip the largely irrelevant points > > when you are zoomed in. It should not be too expensive to find the > > limits either incrementally or with binary search on ordered time series. > > Presumably subsetting is fast using foo[a:b,] > > > > One thing you may want to try for change of scale is wavelet or > > multi-resolution analysis. You can make a tree ( increasing memory usage > > but even VM here may not be a big penalty if coherence is high ) and > > display the resolution appropriate for the current scale. > > > > > I forgot to add, for plotting I use a command similar to: > > plot.ts(timestampVector, dataVector, xlim=c(a,b)) > > a and b are timestamps from timestampVector > > Is the xlim parameter sufficient for limiting the scope of the plots? > Or should I subset the timeseries each time I do a plot? well, maybe time series knows the data to be ordered, I never use that, but in general it has to go check each point and clip the out of range ones. It could I suppose binary search for start/end points but I don't know.Based on what you said it sounds like it does not. > > > The multi-resolution analysis looks interesting. > I shall spend some time finding out how to use the wavelets package. > > Cheers! > > > > >> > Using something other than plot.ts() - is the lattice package faster? > >> > I don't need publication quality plots, they can be rougher... > >> > > >> > I have tried: > >> > -Using matrices instead of dataframes - (works for calculations but not > >> > enough for plots) > >> > -increasing the max usable memory (max-mem-size) - (no change) > >> > -increasing the size of the pointer protection stack (max-ppsize) - (no > >> > change) > >> > -deleting the unnecessary leftover matrices - (no change) > >> > -I can't use lines() instead of plot() because of the very different > >> > scales (rpm-1, flags -1to3) > >> > > >> > I am going to do some resampling of the logged data to reduce the vector > >>
Re: [R] Speed up plotting to MSWindows graphics window
On 27/04/2011 13:18, Mike Marchywka wrote: > >> > Date: Wed, 27 Apr 2011 11:16:26 +0200 >> > From:jonat...@k-m-p.nl >> > To:r-help@r-project.org >> > Subject: [R] Speed up plotting to MSWindows graphics window >> > >> > Hello, >> > >> > I am working on a project analysing the performance of motor-vehicles >> > through messages logged over a CAN bus. >> > >> > I am using R 2.12 on Windows XP and 7 >> > >> > I am currently plotting the data in R, overlaying 5 or more plots of >> > data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). >> > The aim is to be able to pan, zoom in and out and get values from the >> > plotted graph using a custom Qt interface that is used as a front end to >> > R.exe (all this works). >> > The plot is drawn by R directly to the windows graphic device. >> > >> > The data is imported from a .csv file (typically around 100MB) to a >> > matrix. >> > (timestamp, message ID, byte0, byte1, ..., byte7) >> > I then separate this matrix into several by message ID (dimensions are >> > in the order of 8cols, 10^6 rows) >> > >> > The panning is done by redrawing the plots, shifted by a small amount. >> > So as to view a window of data from a second to a minute long that can >> > travel the length of the logged data. >> > >> > My problem is that, the redrawing of the plots whilst panning is too >> > slow when dealing with this much data. >> > i.e.: I can see the last graphs being drawn to the screen in the >> > half-second following the view change. >> > I need a fluid change from one view to the next. >> > >> > My question is this: >> > Are there ways to speed up the plotting on the MSWindows display? >> > By reducing plotted point densities to*sensible* values? > Well, hard to know but it would help to know where all the time is going. > Usually people start complaining when VM thrashing is common but if you are > CPU limited you could try restricting the range of data you want to plot > rather than relying on the plot to just clip the largely irrelevant points > when you are zoomed in. It should not be too expensive to find the > limits either incrementally or with binary search on ordered time series. > Presumably subsetting is fast using foo[a:b,] > > One thing you may want to try for change of scale is wavelet or > multi-resolution analysis. You can make a tree ( increasing memory usage > but even VM here may not be a big penalty if coherence is high ) and > display the resolution appropriate for the current scale. > > I forgot to add, for plotting I use a command similar to: plot.ts(timestampVector, dataVector, xlim=c(a,b)) a and b are timestamps from timestampVector Is the xlim parameter sufficient for limiting the scope of the plots? Or should I subset the timeseries each time I do a plot? The multi-resolution analysis looks interesting. I shall spend some time finding out how to use the wavelets package. Cheers! > >> > Using something other than plot.ts() - is the lattice package faster? >> > I don't need publication quality plots, they can be rougher... >> > >> > I have tried: >> > -Using matrices instead of dataframes - (works for calculations but not >> > enough for plots) >> > -increasing the max usable memory (max-mem-size) - (no change) >> > -increasing the size of the pointer protection stack (max-ppsize) - (no >> > change) >> > -deleting the unnecessary leftover matrices - (no change) >> > -I can't use lines() instead of plot() because of the very different >> > scales (rpm-1, flags -1to3) >> > >> > I am going to do some resampling of the logged data to reduce the vector >> > sizes. >> > (removal of*less* important data and use of window.ts()) >> > >> > But I am currently running out of ideas... >> > So if sombody could point out something, I would be greatfull. >> > >> > Thanks, >> > >> > Jonathan Gabris >> > >> > __ >> > R-help@r-project.org mailing list >> > https://stat.ethz.ch/mailman/listinfo/r-help >> > PLEASE do read the posting >> > guidehttp://www.R-project.org/posting-guide.html >> > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up plotting to MSWindows graphics window
On 27.04.2011 12:56, Duncan Murdoch wrote: Jonathan Gabris wrote: Hello, I am working on a project analysing the performance of motor-vehicles through messages logged over a CAN bus. I am using R 2.12 on Windows XP and 7 I am currently plotting the data in R, overlaying 5 or more plots of data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). The aim is to be able to pan, zoom in and out and get values from the plotted graph using a custom Qt interface that is used as a front end to R.exe (all this works). The plot is drawn by R directly to the windows graphic device. The data is imported from a .csv file (typically around 100MB) to a matrix. (timestamp, message ID, byte0, byte1, ..., byte7) I then separate this matrix into several by message ID (dimensions are in the order of 8cols, 10^6 rows) The panning is done by redrawing the plots, shifted by a small amount. So as to view a window of data from a second to a minute long that can travel the length of the logged data. My problem is that, the redrawing of the plots whilst panning is too slow when dealing with this much data. i.e.: I can see the last graphs being drawn to the screen in the half-second following the view change. I need a fluid change from one view to the next. My question is this: Are there ways to speed up the plotting on the MSWindows display? By reducing plotted point densities to *sensible* values? Using something other than plot.ts() - is the lattice package faster? I don't need publication quality plots, they can be rougher... I don't think there are any ways to plot in the standard device that are significantly faster than what you are doing if you want to see the updates. (I think it would be substantially faster if you hid the graphics window during the updates, but that won't suit you.) I'd suggest plotting a subset of the data during the updates, then plot the full dataset when it stops moving. For example, only plot a few hundred points, even spaced through the time series. ... and it highly depends on the data what can be improved. Example: For signals essential consisting of sine functions (i.e. harmonic signals), I am using a little dirty trick in the tuneR package, but that makes the assumption of having a high frequency sample of a harmonic signal without too much noise. Uwe Ligges Duncan Murdoch I have tried: -Using matrices instead of dataframes - (works for calculations but not enough for plots) -increasing the max usable memory (max-mem-size) - (no change) -increasing the size of the pointer protection stack (max-ppsize) - (no change) -deleting the unnecessary leftover matrices - (no change) -I can't use lines() instead of plot() because of the very different scales (rpm-1, flags -1to3) I am going to do some resampling of the logged data to reduce the vector sizes. (removal of *less* important data and use of window.ts()) But I am currently running out of ideas... So if sombody could point out something, I would be greatfull. Thanks, Jonathan Gabris __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up plotting to MSWindows graphics window
> Date: Wed, 27 Apr 2011 11:16:26 +0200 > From: jonat...@k-m-p.nl > To: r-help@r-project.org > Subject: [R] Speed up plotting to MSWindows graphics window > > Hello, > > I am working on a project analysing the performance of motor-vehicles > through messages logged over a CAN bus. > > I am using R 2.12 on Windows XP and 7 > > I am currently plotting the data in R, overlaying 5 or more plots of > data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). > The aim is to be able to pan, zoom in and out and get values from the > plotted graph using a custom Qt interface that is used as a front end to > R.exe (all this works). > The plot is drawn by R directly to the windows graphic device. > > The data is imported from a .csv file (typically around 100MB) to a matrix. > (timestamp, message ID, byte0, byte1, ..., byte7) > I then separate this matrix into several by message ID (dimensions are > in the order of 8cols, 10^6 rows) > > The panning is done by redrawing the plots, shifted by a small amount. > So as to view a window of data from a second to a minute long that can > travel the length of the logged data. > > My problem is that, the redrawing of the plots whilst panning is too > slow when dealing with this much data. > i.e.: I can see the last graphs being drawn to the screen in the > half-second following the view change. > I need a fluid change from one view to the next. > > My question is this: > Are there ways to speed up the plotting on the MSWindows display? > By reducing plotted point densities to *sensible* values? Well, hard to know but it would help to know where all the time is going. Usually people start complaining when VM thrashing is common but if you are CPU limited you could try restricting the range of data you want to plot rather than relying on the plot to just clip the largely irrelevant points when you are zoomed in. It should not be too expensive to find the limits either incrementally or with binary search on ordered time series. Presumably subsetting is fast using foo[a:b,] One thing you may want to try for change of scale is wavelet or multi-resolution analysis. You can make a tree ( increasing memory usage but even VM here may not be a big penalty if coherence is high ) and display the resolution appropriate for the current scale. > Using something other than plot.ts() - is the lattice package faster? > I don't need publication quality plots, they can be rougher... > > I have tried: > -Using matrices instead of dataframes - (works for calculations but not > enough for plots) > -increasing the max usable memory (max-mem-size) - (no change) > -increasing the size of the pointer protection stack (max-ppsize) - (no > change) > -deleting the unnecessary leftover matrices - (no change) > -I can't use lines() instead of plot() because of the very different > scales (rpm-1, flags -1to3) > > I am going to do some resampling of the logged data to reduce the vector > sizes. > (removal of *less* important data and use of window.ts()) > > But I am currently running out of ideas... > So if sombody could point out something, I would be greatfull. > > Thanks, > > Jonathan Gabris > > __ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Speed up plotting to MSWindows graphics window
Jonathan Gabris wrote: Hello, I am working on a project analysing the performance of motor-vehicles through messages logged over a CAN bus. I am using R 2.12 on Windows XP and 7 I am currently plotting the data in R, overlaying 5 or more plots of data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). The aim is to be able to pan, zoom in and out and get values from the plotted graph using a custom Qt interface that is used as a front end to R.exe (all this works). The plot is drawn by R directly to the windows graphic device. The data is imported from a .csv file (typically around 100MB) to a matrix. (timestamp, message ID, byte0, byte1, ..., byte7) I then separate this matrix into several by message ID (dimensions are in the order of 8cols, 10^6 rows) The panning is done by redrawing the plots, shifted by a small amount. So as to view a window of data from a second to a minute long that can travel the length of the logged data. My problem is that, the redrawing of the plots whilst panning is too slow when dealing with this much data. i.e.: I can see the last graphs being drawn to the screen in the half-second following the view change. I need a fluid change from one view to the next. My question is this: Are there ways to speed up the plotting on the MSWindows display? By reducing plotted point densities to *sensible* values? Using something other than plot.ts() - is the lattice package faster? I don't need publication quality plots, they can be rougher... I don't think there are any ways to plot in the standard device that are significantly faster than what you are doing if you want to see the updates. (I think it would be substantially faster if you hid the graphics window during the updates, but that won't suit you.) I'd suggest plotting a subset of the data during the updates, then plot the full dataset when it stops moving. For example, only plot a few hundred points, even spaced through the time series. Duncan Murdoch I have tried: -Using matrices instead of dataframes - (works for calculations but not enough for plots) -increasing the max usable memory (max-mem-size) - (no change) -increasing the size of the pointer protection stack (max-ppsize) - (no change) -deleting the unnecessary leftover matrices - (no change) -I can't use lines() instead of plot() because of the very different scales (rpm-1, flags -1to3) I am going to do some resampling of the logged data to reduce the vector sizes. (removal of *less* important data and use of window.ts()) But I am currently running out of ideas... So if sombody could point out something, I would be greatfull. Thanks, Jonathan Gabris __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Speed up plotting to MSWindows graphics window
Hello, I am working on a project analysing the performance of motor-vehicles through messages logged over a CAN bus. I am using R 2.12 on Windows XP and 7 I am currently plotting the data in R, overlaying 5 or more plots of data, logged at 1kHz, (using plot.ts() and par(new = TRUE)). The aim is to be able to pan, zoom in and out and get values from the plotted graph using a custom Qt interface that is used as a front end to R.exe (all this works). The plot is drawn by R directly to the windows graphic device. The data is imported from a .csv file (typically around 100MB) to a matrix. (timestamp, message ID, byte0, byte1, ..., byte7) I then separate this matrix into several by message ID (dimensions are in the order of 8cols, 10^6 rows) The panning is done by redrawing the plots, shifted by a small amount. So as to view a window of data from a second to a minute long that can travel the length of the logged data. My problem is that, the redrawing of the plots whilst panning is too slow when dealing with this much data. i.e.: I can see the last graphs being drawn to the screen in the half-second following the view change. I need a fluid change from one view to the next. My question is this: Are there ways to speed up the plotting on the MSWindows display? By reducing plotted point densities to *sensible* values? Using something other than plot.ts() - is the lattice package faster? I don't need publication quality plots, they can be rougher... I have tried: -Using matrices instead of dataframes - (works for calculations but not enough for plots) -increasing the max usable memory (max-mem-size) - (no change) -increasing the size of the pointer protection stack (max-ppsize) - (no change) -deleting the unnecessary leftover matrices - (no change) -I can't use lines() instead of plot() because of the very different scales (rpm-1, flags -1to3) I am going to do some resampling of the logged data to reduce the vector sizes. (removal of *less* important data and use of window.ts()) But I am currently running out of ideas... So if sombody could point out something, I would be greatfull. Thanks, Jonathan Gabris __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
