[CF-metadata] standards for probabilities
Hi! I am trying to create a document containing various probability values for weather forecasts. But I do have some problems finding out how to express what I want to say using the cf metadata standard. I want to express such things as "25th percentile precipitation amount" (based on ensemble data), and probability that air temperature will be within 2.5 degrees of the forecast. How should I do this? For percentiles, may I do something like this? float precipitation_25(time, x, y) ; precipitation_25:standard_name = "precipitation_amount" ; precipitation_25:long_name = "precipitation_amount 25th percentile" ; ... Also, as far as I can tell, there is no standardized names like probability_of_x or probability_of_x_within_y. How can I express this? -- Vegard ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Dear Vegard > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? You are right, this case has not yet been dealt with, although the guidelines for construction of standard names foresee that needs like this might arise! If the quantity is a precipitation_amount, it's fine to use that standard name. The question is how to record that is the 25th percentile. Two possible ways to do this would be: * To extend the possible syntax of cell_methods so that it can describe percentiles. It is already possible to indicate a median in cell_methods, and that is a particular percentile. The advantage of this way of doing it would be that you would record whether the distribution of precipitation amounts being considered was for time-variation, or spatial variation, or some other kind of variation. Obviously you could have a probability distribution with percentiles for many different independent variables. * To use a size-1 or scalar coordinate variable to record the probability, with a new standard_name, perhaps cumulative_distribution_function_of_precipitation_amount. The value of this coordinate would be 0.25 for the 25th percentile. The advantage of this method would be that you could have several different percentiles in the same variable, by having a multivalued probability coord. If you wanted to be specific about what the independent variable was, that would have to be included in the standard name as well e.g. cumulative_distribution_function_of_precipitation_amount_over_time. What do you think? Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Dear Jonathan, I prefer your second alternative. It's not what I do, but it's what I wish I did!! Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory [j.m.greg...@reading.ac.uk] Sent: 15 November 2011 10:11 To: Vegard B??nes Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] standards for probabilities Dear Vegard > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? You are right, this case has not yet been dealt with, although the guidelines for construction of standard names foresee that needs like this might arise! If the quantity is a precipitation_amount, it's fine to use that standard name. The question is how to record that is the 25th percentile. Two possible ways to do this would be: * To extend the possible syntax of cell_methods so that it can describe percentiles. It is already possible to indicate a median in cell_methods, and that is a particular percentile. The advantage of this way of doing it would be that you would record whether the distribution of precipitation amounts being considered was for time-variation, or spatial variation, or some other kind of variation. Obviously you could have a probability distribution with percentiles for many different independent variables. * To use a size-1 or scalar coordinate variable to record the probability, with a new standard_name, perhaps cumulative_distribution_function_of_precipitation_amount. The value of this coordinate would be 0.25 for the 25th percentile. The advantage of this method would be that you could have several different percentiles in the same variable, by having a multivalued probability coord. If you wanted to be specific about what the independent variable was, that would have to be included in the standard name as well e.g. cumulative_distribution_function_of_precipitation_amount_over_time. What do you think? Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata-- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] Burned area standard names
Dear All, I have received an enquiry from Kevin Tansey regarding these standard names that were proposed and discussed in July under a number of different mailing list threads: 'Request for new standard name' (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2011/022117.html ), 'More background on the burned area' (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2011/022137.html )and 'More background on the burned area (Jonathan Gregory)'. Two new standard names were proposed: burned_area; m2 '"X_area" means the horizontal area occupied by X within the grid cell.' burned_area_fraction; 1 '"X_area_fraction" means the fraction of horizontal area occupied by X. "X_area" means the horizontal area occupied by X within the grid cell. Burned area fraction is area of the land surface occupied by burn scars.' Three people made comments in support of adding these names to the CF Standard Name Table and no one commented against. I note Jonathan Gregory's and Martin Schultz's comments regarding adding something to the explanations to make it clear that the burned area refers to vegetation, which seems reasonable. I will add a sentence to both explanations along the lines of '"Burned area" means the area of burned vegetation." As there has been no further discussion of these names and they match the syntax of existing sea_ice names, these names are accepted for inclusion in the standard name table. I am currently reviewing the recent discussions under the TEOS-10 thread and am in the process of preparing an update to the standard name table. I plan to make the next update in 2-3 weeks time. Best wishes, Alison -- Alison Pamment Tel: +44 1235 778065 NCAS/British Atmospheric Data CentreEmail: alison.pamm...@stfc.ac.uk STFC Rutherford Appleton Laboratory R25, 2.22 Harwell Oxford, Didcot, OX11 0QX, U.K. -- Scanned by iCritical. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Thank you, Jonathan! :) So, a bit more concrete, this is option 1: float rain_25(time, y, x); rain_25:standard_name = "precipitation_amount"; rain_25:cell_methods = "realization: percentile(25)"; The only problem I see with this is that in the resulting cdm realization is not used anywhere, apart from possibly in cell methods. But maybe this is ok? If I understand the second option correctly, this would lead to something like this: float precipitation_amount(time, percentile, y, x); ... float percentile(percentile); percentile:units = "1"; percentile:standard_name = "cumulative_distribution_function_of_precipitation_amount"; But what is the purpose of explicitly refering to precipitation_amount in the standard name? would not cumulative_distribution_function be better? Then the same dimension could be used for other data, such as air_temperature. Or, if we want to add something about the nature of the source data for the function, it could be called something like cumulative_distribution_function_due_to_realization? I am still a bit uncertain about what is the best, though. -- Vegard - Original Message - Fra: "Jonathan Gregory" Til: "Vegard B??nes" Kopi: cf-metadata@cgd.ucar.edu Sendt: 15. november 2011 11:11:52 Emne: Re: [CF-metadata] standards for probabilities Dear Vegard > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? You are right, this case has not yet been dealt with, although the guidelines for construction of standard names foresee that needs like this might arise! If the quantity is a precipitation_amount, it's fine to use that standard name. The question is how to record that is the 25th percentile. Two possible ways to do this would be: * To extend the possible syntax of cell_methods so that it can describe percentiles. It is already possible to indicate a median in cell_methods, and that is a particular percentile. The advantage of this way of doing it would be that you would record whether the distribution of precipitation amounts being considered was for time-variation, or spatial variation, or some other kind of variation. Obviously you could have a probability distribution with percentiles for many different independent variables. * To use a size-1 or scalar coordinate variable to record the probability, with a new standard_name, perhaps cumulative_distribution_function_of_precipitation_amount. The value of this coordinate would be 0.25 for the 25th percentile. The advantage of this method would be that you could have several different percentiles in the same variable, by having a multivalued probability coord. If you wanted to be specific about what the independent variable was, that would have to be included in the standard name as well e.g. cumulative_distribution_function_of_precipitation_amount_over_time. What do you think? Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Dear Roy, Can you be a bit more concrete about why you prefer the second alternative? -- Vegard - Original Message - Fra: "Roy K. Lowry" Til: "Jonathan Gregory" , "Vegard B??nes" Kopi: cf-metadata@cgd.ucar.edu Sendt: 15. november 2011 11:17:01 Emne: RE: [CF-metadata] standards for probabilities Dear Jonathan, I prefer your second alternative. It's not what I do, but it's what I wish I did!! Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory [j.m.greg...@reading.ac.uk] Sent: 15 November 2011 10:11 To: Vegard B??nes Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] standards for probabilities Dear Vegard > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? You are right, this case has not yet been dealt with, although the guidelines for construction of standard names foresee that needs like this might arise! If the quantity is a precipitation_amount, it's fine to use that standard name. The question is how to record that is the 25th percentile. Two possible ways to do this would be: * To extend the possible syntax of cell_methods so that it can describe percentiles. It is already possible to indicate a median in cell_methods, and that is a particular percentile. The advantage of this way of doing it would be that you would record whether the distribution of precipitation amounts being considered was for time-variation, or spatial variation, or some other kind of variation. Obviously you could have a probability distribution with percentiles for many different independent variables. * To use a size-1 or scalar coordinate variable to record the probability, with a new standard_name, perhaps cumulative_distribution_function_of_precipitation_amount. The value of this coordinate would be 0.25 for the 25th percentile. The advantage of this method would be that you could have several different percentiles in the same variable, by having a multivalued probability coord. If you wanted to be specific about what the independent variable was, that would have to be included in the standard name as well e.g. cumulative_distribution_function_of_precipitation_amount_over_time. What do you think? Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata-- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Hello Vergand, One of my jobs is running a parameter vocabulary that currently has over 27,000 entries. Much of its bulk is due to the assignment of multiple parameter names for each step in a numeric sequence - such as radiation wavelengths or sediment grain-size expressed as percentiles. Consider a scenario where you start with a small group of standard percentiles - say 5, 25, 50, 75, 95. You set up a parameter name for each of these in the first instance, which is easy. Then along comes another user who wants to describe data with percentiles at a resolution of 1 per cent. So another 95 parameter names need to be set up. Then along comes another user who wants a resolution of 0.1 per cent. I start drowning in names and nobody can find anything. However, had I followed Jonathan's second solution all I would need to do as a vocabulary manager is set up one concept to describe the percentile axis, which covers every user from those who use a handful of percentiles to those whose percentile resolution requirements are beyond the bounds of my imagination. I know Jonathan's first option was based on propogation of cell methods and not standard names. However, these still need managing and if they become excessively abundant they also become difficult to navigate. Cheers, Roy. From: Vegard Bønes [vegard.bo...@met.no] Sent: 15 November 2011 13:17 To: Lowry, Roy K. Cc: cf-metadata@cgd.ucar.edu; Jonathan Gregory Subject: Re: [CF-metadata] standards for probabilities Dear Roy, Can you be a bit more concrete about why you prefer the second alternative? -- Vegard - Original Message - Fra: "Roy K. Lowry" Til: "Jonathan Gregory" , "Vegard B??nes" Kopi: cf-metadata@cgd.ucar.edu Sendt: 15. november 2011 11:17:01 Emne: RE: [CF-metadata] standards for probabilities Dear Jonathan, I prefer your second alternative. It's not what I do, but it's what I wish I did!! Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory [j.m.greg...@reading.ac.uk] Sent: 15 November 2011 10:11 To: Vegard B??nes Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] standards for probabilities Dear Vegard > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? You are right, this case has not yet been dealt with, although the guidelines for construction of standard names foresee that needs like this might arise! If the quantity is a precipitation_amount, it's fine to use that standard name. The question is how to record that is the 25th percentile. Two possible ways to do this would be: * To extend the possible syntax of cell_methods so that it can describe percentiles. It is already possible to indicate a median in cell_methods, and that is a particular percentile. The advantage of this way of doing it would be that you would record whether the distribution of precipitation amounts being considered was for time-variation, or spatial variation, or some other kind of variation. Obviously you could have a probability distribution with percentiles for many different independent variables. * To use a size-1 or scalar coordinate variable to record the probability, with a new standard_name, perhaps cumulative_distribution_function_of_precipitation_amount. The value of this coordinate would be 0.25 for the 25th percentile. The advantage of this method would be that you could have several different percentiles in the same variable, by having a multivalued probability coord. If you wanted to be specific about what the independent variable was, that would have to be included in the standard name as well e.g. cumulative_distribution_function_of_precipitation_amount_over_time. What do you think? Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata-- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Hi Vegard: I see some of these kinds of things from NCEP, encoded in GRIB, and Im still trying to understand what they are. So, some questions from a non-modeler: On 11/15/2011 2:10 AM, Vegard Bønes wrote: Hi! I am trying to create a document containing various probability values for weather forecasts. But I do have some problems finding out how to express what I want to say using the cf metadata standard. I want to express such things as "25th percentile precipitation amount" (based on ensemble data), and probability that air temperature will be within 2.5 degrees of the forecast. How should I do this? these are 2 different things, i guess? 1) "25th percentile precipitation amount" (based on ensemble data) * so here the data values are precip amounts? calculated from the cumulative distribution function (cdf) from an ensemble? * do you typically have other percentile amounts in the same file, eg 50 and 75? * presumably this is some distillation of the cdf, used when the individual ensemble values are not in the file? * is there any special handling that a generic tool could do, or is it a matter a just making this data available to some specialized application that you write? 2) probability that air temperature will be within 2.5 degrees of the forecast * so here the data values are probabilities between 0 and 1 ? * do you typically have other probabilities in the same file, eg within 1 degree, or 5 degrees? * is there any special handling that a generic tool could do with such info? john ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Most of the discussion so far has centered on (1). But this raises some other issues: > 2) probability that air temperature will be within 2.5 degrees of the forecast This is clearly trying to get at something akin to what we do with the "standard_error" standard name modifier and the "standard_error_multiplier" ancillary variable. The differences are that the units are flipped and the assumption of a normal distribution is removed. Can we use something analogous? Maybe a new standard name modifier like "distribution" or "probability" or "confidence" which requires an ancillary variable? The best fit for this particular case is "confidence" with an ancillary variable "confidence_interval" with value 2.5. But something more like "distribution" is more general and could be stretched to handle > 1) "25th percentile precipitation amount" (based on ensemble data) Here we might use the "distribution" (or "cumulative_distribution"?) standard name modifier, ancillary variable "cumulative_probability", value 0.25. -- Evan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] standards for probabilities
Hi John, All the assumptions you state are correct. Regarding the usage of the data, there exists a specialized application that uses this data. The problem is that others are interested in the same data, and I have no control over how they will use it. Because of that, I want the generated file to follow any standards as closely as possible. -- Vegard - Original Message - Fra: "John Caron" Til: cf-metadata@cgd.ucar.edu Sendt: 15. november 2011 14:38:24 Emne: Re: [CF-metadata] standards for probabilities Hi Vegard: I see some of these kinds of things from NCEP, encoded in GRIB, and Im still trying to understand what they are. So, some questions from a non-modeler: On 11/15/2011 2:10 AM, Vegard Bønes wrote: > Hi! > > I am trying to create a document containing various probability values for > weather forecasts. But I do have some problems finding out how to express > what I want to say using the cf metadata standard. > > I want to express such things as "25th percentile precipitation amount" > (based on ensemble data), and probability that air temperature will be within > 2.5 degrees of the forecast. How should I do this? these are 2 different things, i guess? 1) "25th percentile precipitation amount" (based on ensemble data) * so here the data values are precip amounts? calculated from the cumulative distribution function (cdf) from an ensemble? * do you typically have other percentile amounts in the same file, eg 50 and 75? * presumably this is some distillation of the cdf, used when the individual ensemble values are not in the file? * is there any special handling that a generic tool could do, or is it a matter a just making this data available to some specialized application that you write? 2) probability that air temperature will be within 2.5 degrees of the forecast * so here the data values are probabilities between 0 and 1 ? * do you typically have other probabilities in the same file, eg within 1 degree, or 5 degrees? * is there any special handling that a generic tool could do with such info? john ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] 2. Re: [cf-satellite] Sharing quality flags among multiple variables (Jonathan Gregory)
Dear Jonathan et al., what is the difference between a mean value and an observation count? You may add the 25th percentile to this list as well. As far as I can tell, the cell_methods attribute should be best suited for all of these and I don't see a need to work with standard_name modifiers. Blank separated lists would indeed be useful for this case. float thetao(lat,lon); thetao:standard_name="sea_water_potential_temperature"; thetao:ancillary_variables="nobs flags"; thetao:units="degC"; float so(lat,lon); thetao:standard_name="sea_water_salinity"; thetao:ancillary_variables="nobs flags"; thetao:units="psu"; // not allowed currently---but that's a different story! int nobs(lat,lon); nobs:standard_name=" sea_water_potential_temperature sea_water_salinity"; nobs:cell_methods="count(time)" // syntax may be wrong - I didn't look it up int flags(lat,lon); flags:flag_values = 0, 1, 2; flags:flag_meanings = "accepted_value range_outlier failed_inversion_check"; Cheers, Martin > Message: 2 > Date: Sat, 5 Nov 2011 09:16:18 + > From: Jonathan Gregory > Subject: Re: [CF-metadata] [cf-satellite] Sharing quality flags > amongmultiple variables > To: upendra.d...@noaa.gov > Cc: cf-metadata@cgd.ucar.edu, cf-satell...@unidata.ucar.edu > Message-ID: <2005091618.ga16...@met.reading.ac.uk> > Content-Type: text/plain; charset=us-ascii > > Dear all > > Referring to Mike's comment. I agree that the ancillary_variables attribute > indicates that the status_flag variable is associated with its data variable, > and > that alone identifies it to some extent, but (a) it doesn't specifically > indicate > its purpose, since there could be more than one ancillary variable for a given > data variables (status flag, standard error, number of observations, ...); (b) > the status flag variable can be regarded as a data variable in its own right, > and > as such needs a standard_name to be self-describing; it is quite possible, for > example, that variables for status_flag or number_of_observations might be > stored in different netCDF files from the variables they describe, and then > the correspondence would depend on them being distinguishable e.g. by > standard_name="sea_water_salinity number_of_observations" > standard_name="sea_water_potential_temperature > number_of_observations". > However, when the standard_name modifiers were introduced, I don't think > we foresaw the possibility that several data variables might need to share > the same ancillary variable e.g. when the number of observations for salinity > and temperature are the same. > > > Referring to Upendra's comment. We could introduce both changes, but I > think we should do that only if really necessary for existing examples that > are > likely to represent common use-cases. We should not complicate the CF > standard more than we have to! I would say the same about Mike's > ingenious scheme for include-statements. > > If it would serve the purpose that began this discussion, personally I would > favour the first solution, and generalise the use of the standard_name att, > like Edward said e.g. > > float thetao(lat,lon); > thetao:standard_name="sea_water_potential_temperature"; > thetao:ancillary_variables="nobs flags"; > thetao:units="degC"; > float so(lat,lon); > thetao:standard_name="sea_water_salinity"; > thetao:ancillary_variables="nobs flags"; > thetao:units="psu"; // not allowed currently---but that's a different story! > int nobs(lat,lon); > nobs:standard_name="sea_water_potential_temperature > sea_water_salinity number_of_observations"; int flags(lat,lon); > flags:flag_values = 0, 1, 2; > flags:flag_meanings = "accepted_value range_outlier > failed_inversion_check"; > > That is, we would change the text in CF sect 3.3 that reads > > "A standard name is associated with a variable via the attribute > standard_name which takes a string value comprised of a standard name > optionally followed by one or more blanks and a standard name modifier (a > string value from Appendix C, Standard Name Modifiers)." > > to > > "A standard name is associated with a variable via the attribute > standard_name which takes a string value that can have either of two forms. > The first form is a standard name alone. The second form is a blank- > separated list beginning with one or more standard names and ending with a > single standard name modifier (a string value from Appendix C, Standard > Name Modifiers) i.e. standard_name [standard_name ...] > standard_name_modifier. This second form permits a single variable to > provide ancillary data (see Section 3.4) for several variables that have > various > standard names." > > Naturally this would require change to software, such as the CF_checker, > which analyses the standard_name att, but it would not require any change > to software which uses the complete att simply as an identifying string, to > label plots, etc. > > What does you think?
