On Fri, Jun 06, 2014 at 04:00:59PM +0000, Clark, Gregory W. wrote: > Everything works fine EXCEPT that when I use the "Statistical > Quantities" and look at the data in the window under "Parameters" the > precision of the quantities is often too small. Sometimes I get four > decimal places on all the Parameters, but sometimes I only get only > two decimal places. The quantities I'm working with are small (mV) > surface potentials.
The absolute scale does not matter, the format is chosen automatically according to the magnitude of the values. You should always get at least three digits, often four. Generally, if the value range is close to a power of 10 but smaller (e.g. 0.9) you will get 3 digits, otherwise 4. So the ratio between difference of 1 in the last digit and the value range is kept about 1/1000. If the range is [0.0437, 10.5106] the minimum and maximum are shown as 0.04 and 10.51, i.e. the precision is absolute. Is this the problem? Do your small values have much better absolute precision than large values? Rms and Ra are displayed with the same format. Here an additional digit might be useful occasionally as they are typically several times smaller than the value range. Is this the problem?? Regards, Yeti ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech _______________________________________________ Gwyddion-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gwyddion-users
