Dear Jim, Jonathan,
OK, I accept your point that this would be a new meaning of "missing_value",
and that probably causes more serious problems than the one we are trying to solve.
I believe that Jonathan is also correct in saying that we can't use flag_values
for this purpose without a change in the convention ... but perhaps we should
explore whether such a change would lead to a good solution. Do you have a
specific proposal in mind?
If not, here is an idea adapted from my last post. In this example I have used a real valued
dimension coordinate to define the data bins, with an arbitrary value in the first bin. The new
feature is an auxiliary coordinate which is constructed to label the first bin as
"missing_data" and the remaining bins as "data".
I believe that it would be mis-leading to use the existing standard name "height" for the variable
"zbins" in this example, because the first bin is not a height range. Hence, I suggest introducing
a new standard name "height_bins" which would allow one or more bins to have special meanings which
should be indicated by a string valued auxiliary coordinate.
"zbin_flags" could also be encoded as a character array, with values ["missing_values",
"data", "data", ......]. This could be done without changing the convention text, since character
arrays are allowed auxiliary coordinate. However, I believe that it would be worth making an adjustment here, since the
use of flags seems more natural.
I've given the "zbin_flags" a new standard name "bin_status_flag", which is related to
the existing term "status_flag", but refers to the status of the data being counted in the
histogram rather than to the status of the histogram itself. This allows us to refer unambiguously to the
fact that we are counting missing data in the first bin.
Would this work?
float data(time,lat,lon,zbins);
data: standard_name =
"histogram_of_equivalent_reflectivity_factor_over_height_above_reference_ellipsoid";
data: coordinates="zbin_flags status";
float zbins(zbina);
zbins: long_name="Height ranges (with bin for missing data at first element)"
zbins: units="m";
zbins: bounds="zbin_bnds";
zbins: standard_name = "height_bins";
float zbin_bnds(zindex,2);
integer zbin_flags(zbins);
zbin_flags: long_name = "Flags indicating the status of data which is counted in
each bin";
zbin_flags:standard_name = "bin_status_flag";
zbin_flags:flag_values = 0,1;
zbin_flags:flag_meanings = "missing_values data";
character status(char_len);
status:standard_name = "status_flag";
status:long_name = "Flag indicating quality of histogram";
float lat(lat);
float lon(lon);
data:
zbins = -9999., 25., 100., ....;
zbin_bnds = -9999.,0., 0., 50., 50., 150., ...
zbin_flags = 0,1,1,1,...
Regards,
Martin
________________________________
From: CF-metadata <cf-metadata-boun...@cgd.ucar.edu> on behalf of Jim Biard
<jbi...@cicsnc.org>
Sent: 10 May 2019 15:58
To: cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] Missing data bins in histograms
Hi.
Sorry I have been so quiet lately. I've been caught up in other activities.
I have a strong aversion to the proposal to overload the missing_value
attribute with a wholly different meaning. Using missing_value in this way will
produce unexpected results in a number of existing software packages. If the
minor modification to CF to designate flag attributes to be used on coordinate
variables doesn't seem like an acceptable solution for one reason or another, I
think we should define a new convention that doesn't add contradictory
interpretations of existing attributes.
Grace and peace,
Jim
On 5/2/19 11:49 AM, Martin Juckes - UKRI STFC wrote:
Dear Jonathan, Jim,
I’m sorry to have dropped this conversation after starting it three years ago.
We ended up not fixing the problem for CMIP6, but I think it is worth taking
another look.
Coming back to it again, I think that a variation on Jim’s suggestion could
work: rather than using flags it should be possible to use a coordinate
variable, as is done for some CMIP variables that have region names along one
axis. The NetCDF dimension would be an index, and the array of values defining
the bins would be an auxiliary coordinate which, I believe, is not subject to
the rules on monotonicity and missing values which apply to NetCDF dimensions.
There may be a need for some clarifications, but I think this approach would be
much closer to the current convention that any change in the specification for
non-auxiliary coordinate variables.
We have a specific use case in CMIP6 for which the bins are height bins (height of
detected cloud), with one bin reserved for "retrieval error".
This might not need a change in the convention rules, but it would help, I
think, to at least add an example and a standard name for the coordinate
variable. For example:
float data(time,lat,lon,zindex);
data: standard_name =
"histogram_of_equivalent_reflectivity_factor_over_height_above_reference_ellipsoid";
data: coordinates="zbins";
float zbins(zindex);
zbins: long_name="Height ranges (with bin for missing data at first
element)";
zbins:missing_value= -9999.;
zbins: units="m";
zbins: bounds="zbin_bnds";
zbins: standard_name = "????";
float zbin_bnds(zindex,2);
zbin_bnds:missing_value= -9999.;
float lat(lat);
float lon(lon);
data:
zbins = -9999., 25., 100., ....;
zbin_bnds = -9999.,-9999., 0., 50., 50., 150., ...
The use of missing_value in the bounds variable appears to conflict with
conformance rules, but I'm not sure if this is really banned by the convention
in this context.
Using missing_value in this way appears to be acceptable to the convention, but I think
it conflicts with the spirit of the convention: it is not indicating that a value of
"zbins" is missing, but indicating that this index of the array relates to a
count of missing values. For this reason I have omitted _FillValue.
The "zbins" auxiliary coordinate here is a height-like variable, but I don't think we can use a standard name
"height": is it worth adding a standard name "height_bins" defined to be "Height ranges, as
used, for example in a histogram or frequency distribution. A variable with this standard name may include a special
bin for the count or frequency of missing data. This should be indicated by setting the value of that bin and its
bounds to equal the missing_value of the variable. If there is no missing value bin, it is recommended that the term
'height' be used instead."
regards,
Martin
CF-metadata] Missing data bins in histograms
Jonathan Gregoryj.m.gregory at reading.ac.uk
<mailto:cf-metadata%40cgd.ucar.edu?Subject=Re%3A%20%5BCF-metadata%5D%20Missing%20data%20bins%20in%20histograms&In-Reply-To=%3C20161013094247.GF6219%40met.reading.ac.uk%3E><mailto:cf-metadata%40cgd.ucar.edu?Subject=Re%3A%20%5BCF-metadata%5D%20Missing%20data%20bins%20in%20histograms&In-Reply-To=%3C20161013094247.GF6219%40met.reading.ac.uk%3E>
Thu Oct 13 03:42:47 MDT 2016
* Previous message (by thread): [CF-metadata] Missing data bins in histograms
<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html>
* Next message (by thread): [CF-metadata] Usage of histogram_of_X_over_Z
<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html>
* Messages sorted by: [ date
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984>
[ thread
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984>
[ subject
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984>
[ author
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984>
________________________________
Dear Jim
In Appendix A it does not say that the flag attributes are allowed for
coordinate variables - it has just "D" in the "Use" column. This is not an
argument why they shouldn't be if there is a need, but they weren't introduced
with that in mind. The use which you suggested for Martin's case is a good
idea, but I think it would need a change to the convention.
Best wishes
Jonathan
----- Forwarded message from Jim Biard <jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
-----
Date: Wed, 12 Oct 2016 14:58:11 -0400
From: Jim Biard <jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
To: cf-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
Subject: Re: [CF-metadata] Missing data bins in histograms
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
Gecko/20100101 Thunderbird/45.4.0
Jonathan,
Missing/fill values are not allowed, but I don't see any language
prohibiting flags. I'd appreciate it if you could expand on your
thoughts about why they aren't allowed.
Grace and peace,
Jim
On 10/12/16 1:30 PM, Jonathan Gregory wrote:
Dear Jim
That is an ingenious idea. I don't think the flag atts are currently allowed
for coord variables, but they could be, I agree.
Best wishes
Jonathan
----- Forwarded message from Jim Biard <jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
-----
Date: Tue, 11 Oct 2016 14:39:56 -0400
From: Jim Biard <jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
To: cf-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
Subject: Re: [CF-metadata] Missing data bins in histograms
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
Gecko/20100101 Thunderbird/45.4.0
Hi.
Another approach could be to use flag_values and flag_meanings on
the coordinate variable to indicate one or more special coordinate
values that correspond to any number of "missing data" or "out of
bounds" bins. These attributes aren't forbidden by CF, and
everything should be fine as long as the coordinate variable remains
monotonic.
Grace and peace,
Jim
On 10/11/16 8:41 AM, martin.juckes at
stfc.ac.uk<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
wrote:
Hello,
the CF standard name list has two "histogram_.... " entries, and in the CMIP6 data
request we may need to add a third, a histogram_of_cloud_top_height. Besides the standard name, we
also need, for this new variable, a method of encoding the "missing data" bin in the
histogram. That is, the histogram should record frequency in 16 data bins and one additional bin
for the frequency of missing data.
Can we define a "missing_data_index" attribute for histogram variables, and use
this to indicate that the first bin in the array has this special purpose. It might be
more pythonic to put the _FillValue in the coordinate value for the missing data bin, but
I suspect that this would cause substantial problems for many software packages.
regards,
Martin
_______________________________________________
CF-metadata mailing list
CF-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/><http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc><http://www.facebook.com/cicsnc>
*Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC
<http://cicsnc.org/><http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/><http://ncsu.edu/>
NOAA National Centers for Environmental Information
<http://ncdc.noaa.gov/><http://ncdc.noaa.gov/>
/formerly NOAA’s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
<mailto:jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
o: +1 828 271 4900
/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate><https://www.facebook.com/NOAANCEIclimate>
and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo><https://www.facebook.com/NOAANCEIoceangeo>
information, and follow
us on Twitter at @NOAANCEIclimate
<https://twitter.com/NOAANCEIclimate><https://twitter.com/NOAANCEIclimate> and
@NOAANCEIocngeo
<https://twitter.com/NOAANCEIocngeo><https://twitter.com/NOAANCEIocngeo>. /
_______________________________________________
CF-metadata mailing list
CF-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
----- End forwarded message -----
_______________________________________________
CF-metadata mailing list
CF-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
CICS-NC <http://www.cicsnc.org/><http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc><http://www.facebook.com/cicsnc> *Jim
Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC
<http://cicsnc.org/><http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/><http://ncsu.edu/>
NOAA National Centers for Environmental Information
<http://ncdc.noaa.gov/><http://ncdc.noaa.gov/>
/formerly NOAA’s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
<mailto:jbiard at
cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>>
o: +1 828 271 4900
/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate><https://www.facebook.com/NOAANCEIclimate>
and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo><https://www.facebook.com/NOAANCEIoceangeo>
information, and follow
us on Twitter at @NOAANCEIclimate
<https://twitter.com/NOAANCEIclimate><https://twitter.com/NOAANCEIclimate> and
@NOAANCEIocngeo
<https://twitter.com/NOAANCEIocngeo><https://twitter.com/NOAANCEIocngeo>. /
_______________________________________________
CF-metadata mailing list
CF-metadata at
cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
----- End forwarded message -----
________________________________
* Previous message (by thread): [CF-metadata] Missing data bins in histograms
<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html>
* Next message (by thread): [CF-metadata] Usage of histogram_of_X_over_Z
<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html>
* Messages sorted by: [ date
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984>
[ thread
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984>
[ subject
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984>
[ author
]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984>
________________________________
More information about the CF-metadata mailing
list<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>
--
[CICS-NC] <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
formerly NOAA’s National Climatic Data Center
151 Patton Ave, Asheville, NC 28801
e: jbi...@cicsnc.org<mailto:jbi...@cicsnc.org>
o: +1 828 271 4900
Connect with us on Facebook for climate<https://www.facebook.com/NOAANCEIclimate> and ocean and
geophysics<https://www.facebook.com/NOAANCEIoceangeo> information, and follow us on Twitter at
@NOAANCEIclimate<https://twitter.com/NOAANCEIclimate> and
@NOAANCEIocngeo<https://twitter.com/NOAANCEIocngeo>.
