Re: [CF-metadata] new standard name proposal for total ozone in DU
Hi Jonathan, Martin, et al., Although Mass-Moles and frequency-period are examples of pairs of physically different units that are trivially convertible, DU is subtly different because it is defined in two physically different but equivalent ways. My preference is to add an additional comment to the notes for both std_names that recommends using the new std_name for consistency with other std_names, but would also happily live with the other two options (ie, either deprecating the old std_name, or expressing no preference). Best wishes, Philip --- Dr Philip Cameron-Smith, p...@llnl.gov, Lawrence Livermore National Lab. --- -Original Message- From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory Sent: Friday, December 07, 2012 5:31 AM To: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] new standard name proposal for total ozone in DU Dear Martin > Philip also raises a good point with respect to alias names: has it been > stated clearly that they must refer to "exactly the same quantity"? I believe > they should, because if we allow "trivial" unit conversions to count as > aliases, then even "wavelength" and "frequency" could be considered of > aliases, which surely no one would want. It does not say explicitly in your terms, but the convention (Appendix B) implies that the alias has the same definition as the quantity of which it is an alias, which I would say means it is exactly the same quantity. As Philip says, these quantities cannot be the same because they have different physical dimensions, even though the values may be numerically equal. I tend to think it should be sufficient to point out the alternative in the definition of each of these quantities (the one in mm and the one in mol m-2). Deprecation of a quantity would be a step further than CF usually takes. CF provides metadata for things people want to describe, rather than prescribing which things they ought to describe. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Jonathan and all There are aspects of this decision that relate to data sets, models, model/data set alignment, tools, and the scope of the CF user base. I agree that changing the default is problematic. Tools (and people) would have a hard time with prior to change/post change datasets that used the default, in which the only calendar indication was the version of CF used. However, the current default is also problematic - reasons I see are: 1) it does send a message to modelers who find the default unusable, 2) because different calendar defaults are used on the modeling and data side, current tools are limited to particular calendars, affecting users, and 3) the mixed Julian-Gregorian calendar is an ugly beast to peg as the standard forevermore. It's probably possible to overcome 2) by updating tools to include additional calendars. The calendar definitions in CF are pretty clear and it does not seem necessary to change existing ones, but maybe it is a good idea to add another, strict Gregorian (error before 1582). If the calendar default was removed in a future version of CF, this would create some issues, but it would also have benefits that may outweigh them: Some data sets would be CF compliant only under particular versions of CF. Is this the first time that a change to CF would have such an impact? Some tools might be modified to expect all data sets to specify calendars. These would produce errors if no calendar was specified, or in some cases, as Jonathan suggested, the user could be asked to provide the calendar if none was found. However, I imagine most tools would continue to assume the current default. The way to talk about these tools could be that they would work on CF-compliant data sets that spanned versions of CF pre/post removal of the default calendar I would expect that most software based on CF would not have to change. This does not seem so bad to me. The removal of the default solution should not produce the nastier kinds of errors that you would get if you changed the default. People will weigh costs and benefits differently - and I may be missing some major consideration above - but for me, it seems like it's worth it to remove the weird default calendar and move on. Best, - Cecelia On 12/7/2012 6:53 AM, Jonathan Gregory wrote: Dear all I think that we should be very cautious about backwards-incompatibility, even when changing CF versions. I would be worried about making a change which means metadata would have a different meaning according to which version of the convention is used. I know in principle it is safe because the file states the version, but I don't think we can depend on the version to be coded correctly, or on software to make correct use of it. Although these are errors, we ought nonetheless to design the convention to be as robust as possible in an error-prone world. Therefore I don't think it is safe to change the the default calendar from gregorian to proleptic_gregorian, because that could change the date you get for a given time coordinate. However, I do think it would be safe to make it illegal to use the default calendar for reference dates earlier than 1582-10-15, or for negative time coordinates (to make sure that dates before 1582-10-15 cannot be specified). In that case, real-world data for which there is no problem would be unaffected, but software would reject time coordinates whose interpretation is problematic. Software could provide an option whereby the user could state what the calendar actually is, when not specified, in order to override the error if the user knows what it truly is. Alternatively, they could easily repair the file with ncatted or some such tool. It should still be possible to use the real-world calendar before 1582-10-15, because CF might be used for real-world historical data earlier than that date, but it would be necessary to state the calendar explicitly. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] inclusion of GridSpec in CF?
Hi Alex, thank you! The timeframe I am hoping for is the next couple months - then we would have clear versions for our next set of releases. I understand the appeal of correspondence on list, but maybe a call is warranted to come up with a proposal for a coordinated strategy for versioning, documenting, and evolving GridSpec, UGRID, and maybe discrete geometries? Then that proposal could be vetted on the list. That does not seem to exist right now ... but maybe Jeff has one in mind? Best, - Cecelia On 12/7/2012 9:12 AM, Alexander Pletzer wrote: Hi Cecelia, We can certainly add a version to the wiki. I'd like to wait for Jeff to come back from vacation and see what the typical procedure is in this case (if there is one), if that's ok with you. Otherwise I cannot make some version number, put it on the wiki some time next week Cheers, --Alex On 12/06/2012 11:43 AM, Cecelia DeLuca wrote: Hi Alex, Our current GridSpec input file implementation is minimal at best, because we just did a single tile and anything distinctly GridSpeccian comes in with multiple tiles. It's as much intention as implementation at this point. Still, we have had to worry about multiple GridSpec flavors already. ESMF also supports the Common Information Model (CIM) grids package, a slightly different version of GridSpec, as a metadata output format. This is part of our general CIM XML output support. That's a pain, but not unworkable: the distinctions in our software could be made clearly, if we had versioned reference documents. We do on the CIM side. It's problematic for us to use the libCF implementation as a kind of substitute for a specification of the GridSpec convention. The convention needs to be able to evolve, and to support multiple reference implementations. Using libCF could be terrific and valuable for us (and something to talk more about offline) but unfortunately it doesn't address the problem at hand. We could say in our documentation that we support the version of GridSpec approved last May, but without a version number or static doc, and without a path to evolve the convention forward, the solution feels pretty shaky. I think there's an awkward question that needs to be answered about whether your team wants to keep managing and evolving the specification on your wiki - that evolution needs to be done somewhere. Or maybe the expectation is that the whole text will be incorporated into the main CF document and evolved there? There already seem to be some ideas, with respect to UGRID and staggering, about how GridSpec might grow or change. For GridSpec in CF it might make the most sense to resolve the question about where the thing lives (in a perfect world, the approach might be consistent for GridSpec, UGRID, and discrete geometries), and wherever that is, put an initial GridSpec version on it and generate a static doc, and then sit back and think about evolution. Best, - Cecelia On 12/6/2012 9:10 AM, Alexander Pletzer wrote: Hi Cecelia, Great to hear that ESMF is implementing gridspec. We're in the same boat, we had to settle on some specifics before implementing gridspec into libCF. Like Bert, I'm a little worried about a proliferation of gridspec flavours, even with versions attached. Another concern is that our funding for this particular work has stopped, we will not be able to provide continuous support for gridspec ad infinitum, in the short term this means fixing small things. The good news is the specification has been accepted by the CF committee in its present form, although suggestions for further improvement have since come, particularly in view of consolidating the syntax between ugrid and gridspec. This leaves a couple of options for ESMF: use libCF to handle the gridspec aggregation. We made an effort to make libCF compatible with the specification. The library is pure C so should be easy to link against and it is hosted on sourceforge. Or state that ESMF implements the gridspec approved by the committee last May. On our side, we can try to highlight any changes on the wiki. Cheers, --Alex On 12/05/2012 05:16 PM, Cecelia DeLuca wrote: Steve, I agree that solutions may be different in the near term and long term, that seems reasonable - and Alex, I understand about coordinating with Jeff. I still worry about maintaining versions on a wiki ... I think to work in practice it would need to be set up with some attention to navigation. Some more background... ESMF is already building versioned software based on UGRID and (sort of, in a single tile way) GridSpec. We need to cite distinct and static versions of the conventions for both developers and users, since our software may not work, or our documentation may be wrong, if the conventions evolve. Further, since we're committed to supporting ESMF versions for up to three years, we expect, at any given time, to have access to multiple static vers
Re: [CF-metadata] inclusion of GridSpec in CF?
Hi Cecelia, We can certainly add a version to the wiki. I'd like to wait for Jeff to come back from vacation and see what the typical procedure is in this case (if there is one), if that's ok with you. Otherwise I cannot make some version number, put it on the wiki some time next week Cheers, --Alex On 12/06/2012 11:43 AM, Cecelia DeLuca wrote: Hi Alex, Our current GridSpec input file implementation is minimal at best, because we just did a single tile and anything distinctly GridSpeccian comes in with multiple tiles. It's as much intention as implementation at this point. Still, we have had to worry about multiple GridSpec flavors already. ESMF also supports the Common Information Model (CIM) grids package, a slightly different version of GridSpec, as a metadata output format. This is part of our general CIM XML output support. That's a pain, but not unworkable: the distinctions in our software could be made clearly, if we had versioned reference documents. We do on the CIM side. It's problematic for us to use the libCF implementation as a kind of substitute for a specification of the GridSpec convention. The convention needs to be able to evolve, and to support multiple reference implementations. Using libCF could be terrific and valuable for us (and something to talk more about offline) but unfortunately it doesn't address the problem at hand. We could say in our documentation that we support the version of GridSpec approved last May, but without a version number or static doc, and without a path to evolve the convention forward, the solution feels pretty shaky. I think there's an awkward question that needs to be answered about whether your team wants to keep managing and evolving the specification on your wiki - that evolution needs to be done somewhere. Or maybe the expectation is that the whole text will be incorporated into the main CF document and evolved there? There already seem to be some ideas, with respect to UGRID and staggering, about how GridSpec might grow or change. For GridSpec in CF it might make the most sense to resolve the question about where the thing lives (in a perfect world, the approach might be consistent for GridSpec, UGRID, and discrete geometries), and wherever that is, put an initial GridSpec version on it and generate a static doc, and then sit back and think about evolution. Best, - Cecelia On 12/6/2012 9:10 AM, Alexander Pletzer wrote: Hi Cecelia, Great to hear that ESMF is implementing gridspec. We're in the same boat, we had to settle on some specifics before implementing gridspec into libCF. Like Bert, I'm a little worried about a proliferation of gridspec flavours, even with versions attached. Another concern is that our funding for this particular work has stopped, we will not be able to provide continuous support for gridspec ad infinitum, in the short term this means fixing small things. The good news is the specification has been accepted by the CF committee in its present form, although suggestions for further improvement have since come, particularly in view of consolidating the syntax between ugrid and gridspec. This leaves a couple of options for ESMF: use libCF to handle the gridspec aggregation. We made an effort to make libCF compatible with the specification. The library is pure C so should be easy to link against and it is hosted on sourceforge. Or state that ESMF implements the gridspec approved by the committee last May. On our side, we can try to highlight any changes on the wiki. Cheers, --Alex On 12/05/2012 05:16 PM, Cecelia DeLuca wrote: Steve, I agree that solutions may be different in the near term and long term, that seems reasonable - and Alex, I understand about coordinating with Jeff. I still worry about maintaining versions on a wiki ... I think to work in practice it would need to be set up with some attention to navigation. Some more background... ESMF is already building versioned software based on UGRID and (sort of, in a single tile way) GridSpec. We need to cite distinct and static versions of the conventions for both developers and users, since our software may not work, or our documentation may be wrong, if the conventions evolve. Further, since we're committed to supporting ESMF versions for up to three years, we expect, at any given time, to have access to multiple static versions of these conventions. For the last couple ESMF releases, we used the URL of the UGRID wiki with an associated date as the version reference, but that's not the greatest solution. Hence the request for static versioned documents, but I know they are a pain to produce. At a minimum, if GridSpec and UGRID stay on wikis, it would be helpful to have a version table with links on the front wiki page so that someone who ended up there could quickly see and navigate to other versions. Does that make sense? Thanks, - Cecelia On 12/5/2012 2:17 PM
Re: [CF-metadata] CF calendars
All, On 12/6/2012 5:49 PM, Dave Allured - NOAA Affiliate wrote: All, I agree that the current calendar definitions and usage have a lot of problems. Following are some proposals that are geared towards both accuracy and backward compatibility. 1. Please, NO default calendar. Starting with the next CF version, the calendar attribute should be required on all time coordinate variables. This is an appealing option. It works and removes some of the "CF is for data, not models" issues with the current default. 2. Define a new and unique name for the mixed Julian/Gregorian calendar. My favorite would be "julian-gregorian". I could tolerate "mixed julian-gregorian". This is good as well. 3. Recommend that mixed Julian/Gregorian never be used, except (a) in genuine historical data sets that properly need to span the crossover; and (b) to facilitate backward compatibility in existing data sets. Yup, still good. 4. Redefine "gregorian" to mean ONLY the proper Gregorian calendar starting on the crossover date 1582 October 15. Add the formal constraint on this calendar, that any usage with dates earlier than the crossover is prohibited. Note that many existing data sets are automatically compatible with this constraint, with no changes needed. Up to the data folks ... 5. Recommend gregorian, NOT proleptic_gregorian, as the CF preferred calendar for dates associated with the modern era. Don't see how this works for climate modelers who need a continuous calendar, and will continue the divide between models and data. Cecelia 6. Stop using "standard" as a calendar name. Please be explicit. --Dave On Thu, Dec 6, 2012 at 2:37 PM, John Caron wrote: Hi Steve, Cecelia: I agree we should move to a better default calendar, probably proleptic_gregorian. We are stuck with mixed Gregorian for previous CF versions. I think we just need a proposal that says "As of version X, the default calender is proleptic_gregorian. You should explicitly add the calendar attribute for clarity. Udunits is no longer considered to be the reference software for date/time." It would be nice to add advice to the user about calendar tradeoffs and how to do historical date matching for modelers, assuming we have useful things to say. I wonder if allowing ISO formatted strings in place of / in addition to the "time since reference date" form might be useful for historical time matching? Arguably having calendar reference libraries in Python and Java would be sufficient, possibly Python is preferable even to one in C. John On 12/6/2012 1:45 PM, Cecelia DeLuca wrote: Hi John and all, Thanks for this information. A few questions: * So you are not supporting "standard Gregorian calendar" even though thats the CF default? Correct, the climate modelers that we work with don't use it. AFAIK the decision of CESM was to reject the CF default as unreasonable for modeling and use proleptic Gregorian. GFDL might support it, I don't know. We could probably add standard Gregorian as a new calendar if people on the data side needed it. However, we would be happier to join Steve in putting a stake in its heart! * Do modelers need to match "historical dates"? If so, what calendar do they use? I think the calendar used would depend on what was supported by the model and configuration, as well as the form of the date. If you wanted to represent the date of something like a volcanic eruption you could probably make it work with most of the calendars. * Is the time library suitable to be released seperate from the ESMF code, eg as standalone C++? The time library can be used apart from other ESMF interfaces, and some modeling groups do use it that way. I don't think we'd be willing to extract that part and support a separate build. We did that years ago and it was a pain to maintain. (We could try to isolate the documentation though, so users didn't have to wade through the whole reference manual to find the API.) With ESMP(ython), the scope of the interface is much smaller than ESMF proper and I think Ryan could set time functions up nicely. It might make sense to represent it as a separate python package in that approach. Would python work for you, or would you really prefer C? - Cecelia John On 12/5/2012 3:01 PM, Cecelia DeLuca wrote: Hi John and all, ESMF has a mature time management library with calendars that are commonly used in climate/weather/related modeling. It supports the following: 360 day, no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day, and custom (including custom calendars that may change the length of days). ESMF, like CESM, doesn't support the standard Gregorian calendar because it doesn't make sense for modeling groups who may need to cross the Julian/Gregorian weirdness line (we've never gotten a request to support standard Gregorian from modelers). ESMF has calendar, time, time interval, clock, and alarm constructs, can run time forward or backward, a
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear all I think that we should be very cautious about backwards-incompatibility, even when changing CF versions. I would be worried about making a change which means metadata would have a different meaning according to which version of the convention is used. I know in principle it is safe because the file states the version, but I don't think we can depend on the version to be coded correctly, or on software to make correct use of it. Although these are errors, we ought nonetheless to design the convention to be as robust as possible in an error-prone world. Therefore I don't think it is safe to change the the default calendar from gregorian to proleptic_gregorian, because that could change the date you get for a given time coordinate. However, I do think it would be safe to make it illegal to use the default calendar for reference dates earlier than 1582-10-15, or for negative time coordinates (to make sure that dates before 1582-10-15 cannot be specified). In that case, real-world data for which there is no problem would be unaffected, but software would reject time coordinates whose interpretation is problematic. Software could provide an option whereby the user could state what the calendar actually is, when not specified, in order to override the error if the user knows what it truly is. Alternatively, they could easily repair the file with ncatted or some such tool. It should still be possible to use the real-world calendar before 1582-10-15, because CF might be used for real-world historical data earlier than that date, but it would be necessary to state the calendar explicitly. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] new standard name proposal for total ozone in DU
Dear Martin > Philip also raises a good point with respect to alias names: has it been > stated clearly that they must refer to "exactly the same quantity"? I believe > they should, because if we allow "trivial" unit conversions to count as > aliases, then even "wavelength" and "frequency" could be considered of > aliases, which surely no one would want. It does not say explicitly in your terms, but the convention (Appendix B) implies that the alias has the same definition as the quantity of which it is an alias, which I would say means it is exactly the same quantity. As Philip says, these quantities cannot be the same because they have different physical dimensions, even though the values may be numerically equal. I tend to think it should be sufficient to point out the alternative in the definition of each of these quantities (the one in mm and the one in mol m-2). Deprecation of a quantity would be a step further than CF usually takes. CF provides metadata for things people want to describe, rather than prescribing which things they ought to describe. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
