Re: [CF-metadata] Proposal for better handling vector quantities in CF
On 11/24/2011 2:53 PM, Thomas Lavergne wrote: Dear all, This email is a proposal to strenghthen the storage and exploitation of vector/tensor data in CF. Thanks to Jonathan for commenting an early version of this note. As far as I can tell, vectors are not handled as such by CF, only their components (via the standard names defining them, e.g. sea_ice_x_velocity, northward_sea_ice_velocity, eastern_wind, etc...). Life and some applications (e.g. plotting) would be easier if it was possible to group all components of a vector field into a single "vector" object. Here is my use case: I have an ice drift product, thus two datasets to define my vectors: sea_ice_x_displacement, and sea_ice_y_displacement. Note that it could be any combination of x/y, north/east, module/direction. It is moreover not limited to ice drift, but rather applies to any 2D (3D as well) variables such as vectors. As far as I know, the current CF does not provide me a way to "group" these two components an re-unite them into a vector. Two consequences: 1) I cannot define a third variable (say status_flag) that would apply to the vector object (thus to both its components). And 2) computer programmes (that for example want to draw vectors instead of colour contours) have to "guess" that my CF file contains a vector. The software has to skim through my variables, check that any two pairs of standard names define a vector, and propose a "vector plot" option to the user. This might work in simple files, but will fail if my CF files contains 2 sets of vectors, say on e from model, the other from satellite: X_model, Y_model, X_sat, Y_sat. Will a software be smart enough to avoid proposing a (X_model,Y_sat) vector plots when all the 4 share the same standard_names: sea_ice_(x|y)_displacements? Here, an approach could be that the X dataset defines its corresponding Y dataset as an "auxiliary variable" (and the Y dataset would do the same with X). This would probably work, but does not solve my concern number 1 to share a 3rd variable with both X and Y. The solution I propose for discussion is to allow an umbrella "dummy" dataset (like the proj/mapping ones: no dimension, no data, just attributes). This umbrella variable would have a valid standard name "sea_ice_displacement_vector" (definition of "vector"). We would then define a new standard attribute pattern: components =, e.g. "dX dY dir". The string values in the list are the name of the datasets containing the components of the vector. Note that even for a 2D vector, I could choose to have both x/y and speed/dir in the same CF file, hence the need to allow more than just 2 "components", even for a 2D vector. We must have at least 2. So in my case: The two X and Y datasets and the direction: float dX(time, yc, xc) ; dX:long_name = "component of the displacement along the x axis of the grid" ; dX:standard_name = "sea_ice_x_displacement" ; dX:units = "km" ; dX:_FillValue = -1.e+10f ; dX:coordinates = "lat lon" ; dX:grid_mapping = "Polar_Stereographic_Grid" ; float dY(time, yc, xc) ; dY:long_name = "component of the displacement along the y axis of the grid" ; dY:standard_name = "sea_ice_y_displacement" ; dY:units = "km" ; dY:_FillValue = -1.e+10f ; dX:coordinates = "lat lon" ; dX:grid_mapping = "Polar_Stereographic_Grid" ; float dir(time, yc, xc) ; dY:long_name = "direction of the displacement" ; dY:standard_name = "direction_of_sea_ice_displacement" ; dY:units = "degrees" ; dY:_FillValue = -1.e+10f ; dX:coordinates = "lat lon" ; dX:grid_mapping = "Polar_Stereographic_Grid" ; The new dummy umbrella: int ice_drift_vector; drift_vector:standard_name = "sea_ice_displacement" ; drift_vector:long_name = "sea ice drift vector" ; drift_vector:components = "dX dY dir" ; A status flag for the vector: byte status_flag(time, yc, xc) ; status_flag:standard_name = "sea_ice_displacement status_flag" ; status_flag:long_name = "rejection and quality level flag" ; status_flag:valid_min = 0b ; status_flag:valid_max = 30b ; status_flag:grid_mapping = "Polar_Stereographic_Grid" ; status_flag:coordinates = "lat lon" ; status_flag:flag_values = 0b, 1b,..., 22b, 30b ; status_flag:flag_meanings = "missing_input_data over_land ... interpolated nominal_quality" ; When browsing through the file, a software would immediately see that there are vectors available (e.g. for display) and which datasets hold the components. It still have to read the component datasets to know how to use them (by reading the standard_name). We could even imagine that tools are able to automatically compute (for display or comparison)the vector length although only the x/y components are available. Some new standard names would be needed, of course. As well as the "components" attribute. A revisit/cleanup of many definitions of existing standard names of component variables could also be envisaged. This proposal needs to be more t
Re: [CF-metadata] udunits corresponding to Forel-Ule, milliequivalent
Upendra, >From the perspective of the UDUNITS unit-packages, the unit "Forel-Ule" is the unit "Forel" multiplied by the unit "Ule" -- neither, of which, is known by the UDUNITS or UDUNITS2 packages. I suggest you contact the author. Regards, Steve Emmerson On 12/08/2011 09:05 AM, Upendra Dadi wrote: > Hi, > I have some data which has "Forel-Ule" for the units used. Is there a > udunits corresponding to this? Also, what should I use for > milliequivalent ? Thanks for the help. > Upendra > ___ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits corresponding to Forel-Ule, milliequivalent
Hi Upendra,
We had some similar data from the Secchi disc depth observations
and related data recording the observed sea water colour on the Forel-Ule
water colour scale. The Forel-Ule scale is a set of character codes
F1,F3 ...F11, U1, U3U11 corresponding to known colour standards
e.g F1 is a deep Blue, U1 green and U11 brown. The scale therefore
doesn't have physical units so I don't think its in udunits.
I did start some work about how I could encode our the Secchi disc/Forel-Ule
data in netCDF and I thought that an array of character string
would suit i.e:
netcdf tmp {
dimensions:
OBS = 74 ;
string8 = 8 ;
variables:
double TIME(OBS) ;
double LATITUDE(OBS) ;
double LONGITUDE(OBS) ;
char FOREL_ULE(OBS, string8) ;
double SECCHI_DEPTH(OBS) ;
data:
...
FOREL_ULE =
"U1",
"U11",
"U11",
As for 'milliequivalent' I am not sure.
HTH
Andrew Walsh
Oceanographic Data Manager
RAN-Metoc-ODS
- Original Message -
From: "Upendra Dadi"
To:
Sent: Friday, December 09, 2011 7:58 AM
Subject: Re: [CF-metadata] udunits corresponding to Forel-Ule,milliequivalent
Hi,
I have some data which has "Forel-Ule" for the units used. Is there
a udunits corresponding to this? Also, what should I be using for
milliequivalent ? Thanks for the help.
Upendra
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standard name for sea water ph without
Dear Upendra We had a long discussion about this in 2009. In particular, John Graybeal's posting of 25 Mar 2009 gives several different pH definitions. It appeared that these versions of pH are probably not the same geophysical quantity, and it is therefore necessary to specify which one it is. Although they all have the same general intention of measuring something like H+ concentration, they are more distinct than just being different measurement techniques. You are right, if they were all the same geophysical quantity measured in different ways, we wouldn't record the measurement technique in the standard name. Cheers Jonathan > Hi All, > The standard name table has an entry called > "sea_water_ph_reported_on_total_scale". I have some data which does > not mention the scale used for the measurement of ph. Should there > be an another entry which does not mention the scale? Most of the > standard names I have seen doesn't mention the scale used|. Is it > common to attach within standard name, the scale used for the > measurement? ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits corresponding to Forel-Ule, milliequivalent
Hi, I have some data which has "Forel-Ule" for the units used. Is there a udunits corresponding to this? Also, what should I be using for milliequivalent ? Thanks for the help. Upendra ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standard name for sea water ph without
Hi Upendra, The reason the "reporting scale" is attached to this name is that the fundamental measurement, or property, to which it refers produces numbers that are not comparable to pH derived using other techniques. (They are actually measuring different quantities, not just a different offset/scale value.) From what I (not a scientist!) understand, it is often the case that pH that doesn't mention its scale has been measured in a way that is not an effective indicator of pH in sea water. So it is very important to understand the way the pH was measured, in order that the values be reported compatibly with others. I am not knowledgeable enough to know the right answer to your two questions, but the above may be useful input. John On Dec 8, 2011, at 08:35, Upendra Dadi wrote: > Hi All, > The standard name table has an entry called > "sea_water_ph_reported_on_total_scale". I have some data which does not > mention the scale used for the measurement of ph. Should there be an another > entry which does not mention the scale? Most of the standard names I have > seen doesn't mention the scale used. Is it common to attach within standard > name, the scale used for the measurement? > > Upendra > ___ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] udunits corresponding to Forel-Ule, milliequivalent
Hi, I have some data which has "Forel-Ule" for the units used. Is there a udunits corresponding to this? Also, what should I use for milliequivalent ? Thanks for the help. Upendra ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] standard name for sea water ph without
Hi All, The standard name table has an entry called "sea_water_ph_reported_on_total_scale". I have some data which does not mention the scale used for the measurement of ph. Should there be an another entry which does not mention the scale? Most of the standard names I have seen doesn't mention the scale used|. Is it common to attach within standard name, the scale used for the measurement? Upendra | ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Dear Lorenzo Thank you for your email. > I think the cell-methods mechanism has a partial overlapping with netCDF-U, > in that it can account for (some of the) UncertML Summary Statistics > concepts. However, it does not currently address Distributions and Samples. > We could think of extending it, but we preferred to introduce a new > mechanism, based on the standard URI syntax and RDF semantics. > > On the other hand, the cell-methods mechanism is arguably more fine-grained > than netCDF-U, allowing to express different methods on multi-dimensional > variables, particular as far as the semantics of dimension intervals is > concerned. Yes, I agree with your last point. An important aspect of cell_methods is that it relates to particular axes. Describing a quantity just as a "variance", for instance, can be rather vague: it may be necessary to know if it's a variance over space, over time or over ensemble members, for example. Possibly you could consider including your URIs and some other extra information as comments in cell_methods. These would be legal but unstandardised as far as CF is concerned, but you could standardise them in your convention e.g. double biotemperature_variance(lat,lon); biotemperature_variance:units = "degC"; // shouldn't it be degC^2 for a variance? biotemperature_variance:cell_methods="realization: variance (ref http://www.uncertml.org/distributions/normal#variance)" The cell_methods here refers to realization as a standard name, which is allowed even though realization isn't a dimension. If you do have a dimension for realization, as in one of your examples, the coordinate variable for that dimension could have a standard_name="realization" attribute. If the variance was over an existing dimension, that could be used e.g. double biotemperature_mean(time,lat,lon): biotemperature_mean:units = "degC"; biotemperature_mean:cell_methods="time: mean (ref http://www.uncertml.org/distributions/normal#mean)" Of course, this will only work for those statistical methods which are allowed by cell_methods. However, you could propose others to include in Appendix E if they are ways of computing statistics like those. Looking at your examples, I wonder why you have, for instance lon:_CoordinateAxisType = "Lon"; What is the need for this new attribute? CF already offers these two methods to indicate such an axis: lon:axis="X"; lon:standard_name="longitude"; and in addition, the units of degrees_east imply that it is longitude. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
