plotting large datasets
I've got a need to plot "value vs time" data for 32 channels simultaneously. Gnuplot isn't up to the task (not enough uniqueness, even mixing lines and points, or control - the graph is just a mess). What else is there? Ideally, I'd like something I can interact with - enable/disable channels, highlight channels, change the time range, etc. The data comes from processed log files, so either file or API input is OK. Ideas? -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
Around 06:10pm on Friday, August 21, 2009 (UK time), DJ Delorie scrawled: > I've got a need to plot "value vs time" data for 32 channels > simultaneously. Gnuplot isn't up to the task (not enough uniqueness, > even mixing lines and points, or control - the graph is just a mess). > What else is there? I use RRDtool. Steve -- (o< www.stevesearle.com //\ Powered by Fedora V_/_No MS products were used in the creation of this message 18:16:12 up 1 day, 23:39, 1 user, load average: 0.02, 0.05, 0.05 pgpnkicHyeS8s.pgp Description: PGP signature -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
On Fri, Aug 21, 2009 at 7:16 PM, Steve Searle wrote: > Around 06:10pm on Friday, August 21, 2009 (UK time), DJ Delorie scrawled: > >> I've got a need to plot "value vs time" data for 32 channels >> simultaneously. Gnuplot isn't up to the task (not enough uniqueness, >> even mixing lines and points, or control - the graph is just a mess). >> What else is there? > > I use RRDtool. > > Steve > Have you tried with R (http://www.r-project.org)? It is very powerful and IMO easy to use. Also there are a lot of packages. See at http://cran.r-project.org Another powerful tool for charting is GLE (http://www.gle-graphics.org/) -- Marco -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
On 8/21/09, DJ Delorie wrote: > > I've got a need to plot "value vs time" data for 32 channels > simultaneously. Gnuplot isn't up to the task (not enough uniqueness, > even mixing lines and points, or control - the graph is just a mess). > What else is there? > > Ideally, I'd like something I can interact with - enable/disable > channels, highlight channels, change the time range, etc. The data > comes from processed log files, so either file or API input is OK. > > Ideas? > 32 channels is a LOT. Could you, an artist or a draftsman do it by hand? Do you need all 32 channels on one page? i.e. can you plot 4, 8, 16 to a page and just print more pages. Can you use R to run statistics to select groups or in some other way make sense of it? Can you put the data behind a script on a web server and let the viewer pick subsets. -- NiftyFedora T o m M i t c h e l l -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
NiftyFedora Mitch writes: > 32 channels is a LOT. Hence the problem :-) > Could you, an artist or a draftsman do it by hand? > Do you need all 32 channels on one page? > i.e. can you plot 4, 8, 16 to a page and just print more pages. Ideally, I'd have a GUI where I can select which channels to view, or show them all as a "background" color and select which ones to color/highlight. I don't need to see them *all* at the same time (at least, not in a way that each channel is uniquely identifiable) but the ones I do see should be together (same scale and axes). Now, if gnuplot had options for dashed or dotted lines, I might have squeaked by with it... -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
> > I've got a need to plot "value vs time" data for 32 channels > > simultaneously. Gnuplot isn't up to the task (not enough uniqueness, > > even mixing lines and points, or control - the graph is just a mess). > > What else is there? You might try the pdplot program at http://www.omnisterra.com/walker/linux/pdplot/intro.htm It takes ascii datafiles and plots them to X11, postscript or PNG. The file format is x,y ascii data interspersed with commands like: title my plot xscale 1 time yscale 1 voltage 0 0 1 1 3 3 nextygraph yscale 1 current 0 8 2 2 3 4 nextygraph yscale 1 pressure 0 -1 3 3.99 the pdplot program is a daemon associated with a persistant plot window. You communicate with the daemon by sending data to it with "pd". For instance, to plot the above file you'd save it as "myplot" and do one of the following: pd myplot cat myplot | pd If you want to add new data to an already existing plot you cat do it with "cat newdata | pd -n". The general style and format works extremely well with awk(1) or perl(1) for massaging ascii data files. It's got a few rough edges because it is a brand new program, but I promise to jump on any bug reports if you want to give it a try. > > Ideally, I'd like something I can interact with - enable/disable > > channels, highlight channels, change the time range, etc. The data > > comes from processed log files, so either file or API input is OK. Sounds like a good use for Tcl/Tk. Write a front end to pd() with lots of options. Each time you make a change, Tk runs through the data a replots it with the desired format. kind regards, -- Rick Walker -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
Rick Walker writes: > It's got a few rough edges because it is a brand new program, but > I promise to jump on any bug reports if you want to give it a try. I can send you a pdplot input if you want to see what I'm up against. I got it to work, and the results are somewhat more usable that gnuplot in some ways, but less in other ways. Some first impressions: 1. The Time axis should have an option for time(2) format, converting to day/hours/sec etc. Currently, it shows the ten digit values in scientific format, which overlap. 2. It doesn't label the individual graphs, so you can't tell which graph is which. 3. The zooming is different than everyone else - should be click-drag rectangle (with outline), release to zoom. The second point should be allowed to be just outside the plot so you can reliably grab the zero axis. 4. Dude, scrollbars :-) 5. The git clone command fails. 6. The "homepage" link opens the homepage within a frame, need to re-target the top window. 7. If it had the ability to enable/disable the various graphs through the GUI, it'd be nearly prefect for me, caveat the above. Thanks! -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
DJ Delorie wrote: > I've got a need to plot "value vs time" data for 32 channels > simultaneously. Gnuplot isn't up to the task (not enough uniqueness, > even mixing lines and points, or control - the graph is just a mess). > What else is there? > > Ideally, I'd like something I can interact with - enable/disable > channels, highlight channels, change the time range, etc. The data > comes from processed log files, so either file or API input is OK. > > Ideas? Try xmgrace. Old style GUI, unconventional controls, but incredibly powerful and exceptional output quality when printing (EPS export). Not immediately easy to use, but it is worth the time needed for learning. xmgrace : excel_graphs = latex : word -- Roberto Ragusamail at robertoragusa.it -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets
On Fri, 2009-08-21 at 20:21 -0400, DJ Delorie wrote: > NiftyFedora Mitch writes: > > 32 channels is a LOT. > > Hence the problem :-) > > > Could you, an artist or a draftsman do it by hand? > > Do you need all 32 channels on one page? > > i.e. can you plot 4, 8, 16 to a page and just print more pages. > > Ideally, I'd have a GUI where I can select which channels to view, or > show them all as a "background" color and select which ones to > color/highlight. I don't need to see them *all* at the same time (at > least, not in a way that each channel is uniquely identifiable) but > the ones I do see should be together (same scale and axes). > > Now, if gnuplot had options for dashed or dotted lines, I might have > squeaked by with it... > I have used the charting capability of Calc (the spreadsheet) to plot data before, and that will handle quite a few lines at a time. Each data can be added or deleted easily from the range selection by simply typing in the range to view. Not quite like having a button that shows it, but perhaps you could write a macro to implement that capability? I have viewed 10 columns with over 3 samples each with this tool. I don't know about 32 columns, but it would probably handle that as well. Your biggest issue is how to set them up. The chart tool in calc will let you display them in various formats, and colors. Regards, Les H -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
Re: plotting large datasets (DJ Delorie)
Hi DJ, > I can send you a pdplot input if you want to see what I'm up against. > I got it to work, and the results are somewhat more usable that > gnuplot in some ways, but less in other ways. Some first impressions: Please do, and then we can probably take this discussion into private email. > 1. The Time axis should have an option for time(2) format, converting >to day/hours/sec etc. Currently, it shows the ten digit values in >scientific format, which overlap. I usually send my data through some other program to manipulate the data. I have a 3 line idiom in awk to process UNIX time into decimal dates like 2009.345 for plotting purposes. > 2. It doesn't label the individual graphs, so you can't tell which >graph is which. You can label each graph with the yscale command. "nextygraph" takes you to the next graph and "yscale " gives it a yscale label. You can also put legends in the graph itself with "label ", taking care to make sure it fits. > 3. The zooming is different than everyone else - should be click-drag >rectangle (with outline), release to zoom. The second point should >be allowed to be just outside the plot so you can reliably grab the >zero axis. It is different than gnuplot, but it is the same as HP's post processor which is what the initial target audience was used to. However, it's not hard to change the zooming, and I think your suggestion is a good one. > 4. Dude, scrollbars :-) What did you want to scroll? > 5. The git clone command fails. Thanks. I just fixed that. > 6. The "homepage" link opens the homepage within a frame, need to > re-target the top window. Thanks. I'll fix that. > 7. If it had the ability to enable/disable the various graphs through >the GUI, it'd be nearly prefect for me, caveat the above. This feature can be done nicely with a page of Tcl/Tk, customized to your particular need. Sometimes I use pdplot for multiple graphs, sometimes as simply a vector plotter. It's mostly optimized for driving through stdin from a script. That's why I suggest a front-end wrapper to handle your application-specific data munging and formatting. I use awk() perl() or Tk(). You could even write a super simple bash or ksh script that let you do something like: plot 1 2 3 ; plot datapoints 1,2 and 3 plot 3-6; plot datasets 3 through 6 The plot script would read the arguments, parse the data file with grep or awk, label the plot and spit the ascii representation to pd which would update the pdplot window. If you want to send me the details of your data storage format, I can make you a simple tool to select your plots on the fly. Every application is different, so I usually build this kind of stuff as part of the problem setup for each different investigation. Thanks for taking a look, and for the nice suggestions. kind regards, -- Rick Walker -- fedora-list mailing list fedora-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
