Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Cecelia > is it codified that > recommendations should generate warnings in tools if they are not > followed? (I assume the mixed calendar itself would not be > deprecated, just its use as the default?) I don't think it's codified, but that's what the CF checker does. Requirements and recommendations are clearly separated in the CF conformance document. Breaking a requirement gives an error; not following a recommendation gives a warning. In this case, if we agree to keep a default, I think we would recommend that the calendar should always be specified, so that a warning would result if it wasn't (in which case the default applies). Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Jonathan, I do think that it would be right to deprecate the use of the mixed Julian- Gregorian calendar for all cases except where there is really a need for time coordinates before the change of calendar. There may be no such cases (as I agree in my previous posting, to Steve) but we cannot be sure and I would guess it might occasionally be needed. I've probably said all this before, but to summarise, my first choice would be to change the default to be strict Gregorian from 1582, which excludes the worst problem, but if others argue that there is need to prevent confusion between real-world and model calendars for reference dates more recent than 1582, I would accept my second choice, which is not to allow a default at all. That's my second choice because I think it would cause annoyance for many people who've been used to coding COARDS-like real-world time coordinates without defining their calendar, whose datasets might become illegal, either because their software is upgraded to use the new version of CF in which this change is made, or because their software changes its default behaviour without paying attention to the Conventions attribute, as Chris also mentions. Please consider me on board with the solution you've proposed. You, John, Steve, Cathy and others are in a better position than I am to judge the impacts of such a change on data set users. I'm disappointed that the solution is still a bit of a snub for modelers, but in practice it probably doesn't matter much. I do wonder about deprecation of the current default and the recommendation to specify calendars, and whether those behaviors are well-defined. For example, is a time period defined after which the current default is not supported at all? And is it codified that recommendations should generate warnings in tools if they are not followed? (I assume the mixed calendar itself would not be deprecated, just its use as the default?) Best, Cecelia ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Chris, >> I think a solution shouldn't break current files which followed what had >> been a standard for a long time (however ill-advised the standard was).[...] while I fully support your pledge for backward compatibility, we should also avoid stagnation because of too many old hassles that need to be supported. Isn't this situation exactly why we need different CF versions? The existing files you talk about will carry a CF attribute <= 1.5. If the rules become stricter in CF 1.6 (and "1-1-1" will no longer be allowed, for example), there shouldn't be any issue with backward compatibility, because any software that claims it can read CF-1.5 data will have to allow for "1-1-1" dates and should interpret these correctly. In this case, the backward compatibility issue will only apply for data files that are newly created, and this is exactly the situation where we would like to get away from "1-1-1", isn't it? The other potential conflict could arise if someone decides to "clean" an old file and bring it to the CF-1.6 standard. Well, in this case, cleaning of the date values and attributes would have to be part of the cleanup process. Cheers, Martin Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
I also think we should recommend that the calendar attribute should always be defined (if we continue to allow a default). Recommending this would mean, for instance, that the cf-checker would give a warning if the calendar is not defined, but it would not be an error. Jonathan On Wed, Dec 19, 2012 at 09:23:34AM +, Jonathan Gregory wrote: > Date: Wed, 19 Dec 2012 09:23:34 + > From: Jonathan Gregory > To: cf-metadata@cgd.ucar.edu > Subject: [CF-metadata] CF calendars (was: problem with times in PSD dataset) > User-Agent: Mutt/1.5.21 (2010-09-15) > > Dear Cecelia et al. > > > An "entirely unproblematic" calendar attribute such as days since > > 2012-1-1 could be > > quite problematic if it is March 1, if it is unclear if you are on a > > Gregorian, no leap, > > or 360 day calendar, all in active use in modeling and all yielding > > potentially different > > answers. A no default strategy might lessen the chances for > > mismatches (or at > > least, steer people from the implicit assumption that dates are > > likely to be Gregorian). > > Clearly I should be more precise in what I say! > > When I said "entirely unproblematic" in this case I meant that the mixed > Julian-Gregorian calendar was not a problem for it. If it is known to be in > the real world, it is well-defined, because it is later than the change from > Julian to Gregorian in all countries. I agree of course that Gregorian, 360-, > 365- and 366-day calendars would all give different answers still. Hence that > choice does have to be clear. > > I do think that it would be right to deprecate the use of the mixed Julian- > Gregorian calendar for all cases except where there is really a need for time > coordinates before the change of calendar. There may be no such cases (as I > agree in my previous posting, to Steve) but we cannot be sure and I would > guess it might occasionally be needed. > > I don't think deprecation (which should be done anyway) changes my own view > about the default. I think the default should be the real world *but* I think > it would be fine to redefine the default to exclude the mixed calendar (i.e. > make it strict Gregorian). Allowing strict Gregorian to go back to 1582 is the > most generous choice. We could choose a later date. However, since Greece and > Russia (and maybe elsewhere) changed in the 20th century, we cannot choose a > date that is both late enough to avoid the transition in all countries and > early enough to include the period of instrumental data, which goes back to > the 19th century and earlier, for which there is a very likely need in CF. > Thus 1582 seems a simple and logical choice, and still helps as it would make > reference dates of year 0 and 1 fail. I suspect they are the commonest > choices that are made when the reference date has not been specifically chosen > to suit the data. > > I've probably said all this before, but to summarise, my first choice would > be to change the default to be strict Gregorian from 1582, which excludes the > worst problem, but if others argue that there is need to prevent confusion > between real-world and model calendars for reference dates more recent than > 1582, I would accept my second choice, which is not to allow a default at all. > That's my second choice because I think it would cause annoyance for many > people who've been used to coding COARDS-like real-world time coordinates > without defining their calendar, whose datasets might become illegal, either > because their software is upgraded to use the new version of CF in which this > change is made, or because their software changes its default behaviour > without > paying attention to the Conventions attribute, as Chris also mentions. > > Best wishes > > Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Cecelia et al. > An "entirely unproblematic" calendar attribute such as days since > 2012-1-1 could be > quite problematic if it is March 1, if it is unclear if you are on a > Gregorian, no leap, > or 360 day calendar, all in active use in modeling and all yielding > potentially different > answers. A no default strategy might lessen the chances for > mismatches (or at > least, steer people from the implicit assumption that dates are > likely to be Gregorian). Clearly I should be more precise in what I say! When I said "entirely unproblematic" in this case I meant that the mixed Julian-Gregorian calendar was not a problem for it. If it is known to be in the real world, it is well-defined, because it is later than the change from Julian to Gregorian in all countries. I agree of course that Gregorian, 360-, 365- and 366-day calendars would all give different answers still. Hence that choice does have to be clear. I do think that it would be right to deprecate the use of the mixed Julian- Gregorian calendar for all cases except where there is really a need for time coordinates before the change of calendar. There may be no such cases (as I agree in my previous posting, to Steve) but we cannot be sure and I would guess it might occasionally be needed. I don't think deprecation (which should be done anyway) changes my own view about the default. I think the default should be the real world *but* I think it would be fine to redefine the default to exclude the mixed calendar (i.e. make it strict Gregorian). Allowing strict Gregorian to go back to 1582 is the most generous choice. We could choose a later date. However, since Greece and Russia (and maybe elsewhere) changed in the 20th century, we cannot choose a date that is both late enough to avoid the transition in all countries and early enough to include the period of instrumental data, which goes back to the 19th century and earlier, for which there is a very likely need in CF. Thus 1582 seems a simple and logical choice, and still helps as it would make reference dates of year 0 and 1 fail. I suspect they are the commonest choices that are made when the reference date has not been specifically chosen to suit the data. I've probably said all this before, but to summarise, my first choice would be to change the default to be strict Gregorian from 1582, which excludes the worst problem, but if others argue that there is need to prevent confusion between real-world and model calendars for reference dates more recent than 1582, I would accept my second choice, which is not to allow a default at all. That's my second choice because I think it would cause annoyance for many people who've been used to coding COARDS-like real-world time coordinates without defining their calendar, whose datasets might become illegal, either because their software is upgraded to use the new version of CF in which this change is made, or because their software changes its default behaviour without paying attention to the Conventions attribute, as Chris also mentions. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Jonathan and all, 2 remove the Julian-Gregorian calendar as default, and have no default calendar (grid analogy) 4 replace the Julian-Gregorian calendar as default with a strict Gregorian calendar I think either of these would work. 2 causes more aggravation. It means that data which doesn't state the calendar attribute is illegal and will produce errors, even if it's entirely unproblematic such as "days since 2012-1-1". It would make CF more intolerant of existing practices than it usually has been. At the moment, CF accepts COARDS time coordinates; with this change, COARDS data would not be acceptable. Hence I would still prefer 4. The aim of this would be to make the default illegal in cases where there is a serious chance of unsafe time units, and the obvious criterion seemed to prevent dates before the invention of the Gregorian calendar. In particular that will exclude reference years of 0 and 1, which are often problematic. However, I don't feel strongly about it. If a deprecation process were in place, would that affect your ranking of these options? An "entirely unproblematic" calendar attribute such as days since 2012-1-1 could be quite problematic if it is March 1, if it is unclear if you are on a Gregorian, no leap, or 360 day calendar, all in active use in modeling and all yielding potentially different answers. A no default strategy might lessen the chances for mismatches (or at least, steer people from the implicit assumption that dates are likely to be Gregorian). The other reasons do still stand - the strict Gregorian calendar is unsuitable for many climate model experiments that must move smoothly to dates before the 1500's, and the Gregorian calendar is frequently not used as the default in models in any case (in favor of something easier, like no leap). It does still seem cleaner to have tools request a calendar attribute or generate an obvious error rather than generating errors only on some dates, which could be harder to catch. (However, I do see the benefit of a strict Gregorian calendar default still working for current dates for observational data, and could imagine this outweighs other concerns.) Anyway, I thought the notion of deprecation, and an adjustment period, might make it easier to make a "bigger" change and remove the default - what do you think? Cecelia -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Cathy, We have not talked yet about a deprecation process. It would be possible to deprecate the mixed Gregorian calendar for an extended period (say 3 years), during which time people would be encouraged to use a new strategy, and would be warned that the default would be removed or changing at the end of the period. That would enable more people to weigh in on the change before the plug was pulled. Do you think that would help make a change more manageable? Cecelia On 12/17/2012 4:13 PM, Cathy Smith (NOAA Affiliate) wrote: Cecelia I think a solution shouldn't break current files which followed what had been a standard for a long time (however ill-advised the standard was). I don't have a good sense of what might break if the standard changed in terms of software so I can' speak for all users but I do know many people have downloaded our mean files with 1-1-1 base dates (ignoring the climatologies for now). While we can potentially change what we have either by changing the dates and/or adding a calendar attribute, changing the default calendar may create errors in reading dates for users who already have those files (which are currently CF complaint ). And, they won't have the same ability to change the files and they wouldn't necessarily know they needed to. I think no default at all would be problematic as what would software do? So, I would support the inclusion of a calendar attribute that would be used by software if there but using the old default calendar if not. Also, I would support making a calendar attribute for new files mandatory (with an updated CF version) but I would keep backwards compatibility unless it were reliably shown it was not an issue with users. I'm not convinced that the users needs (as opposed to developers) have been adequately researched. Cathy On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote: Cathy, Of the other options, do you find some (or all) completely unacceptable? Are some better than others? - Cecelia On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote: Cecelia I support 1) mostly for backward compatibility. I would also strongly encourage but not demand that users change their base dates to after 1800 when it makes sense to do so. And, I (again) want to make sure that LTMs and their time values are addressed before any decisions are made as to negative times and using base dates of 1-1-1 and the issue of what year to use for climatologies. LTM dates are a problem when one needs to use a calendar based on real dates. Cathy On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: Hi Steve, Jonathan and all, There are not that many options being discussed. With respect to the default calendar: 1 keep the Julian-Gregorian calendar as default (no change) 2 remove the Julian-Gregorian calendar as default, and have no default calendar (grid analogy) 3 replace the Julian-Gregorian calendar as default with the proleptic Gregorian calendar 4 replace the Julian-Gregorian calendar as default with a strict Gregorian calendar Maybe it makes sense for people to cite (or rank) their preference at this point? There were a couple other proposals, depending on which of above is selected: 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed for 4) 6 remove the Julian-Gregorian calendar (impossible for 1, optional for 2, 3, 4) Again, maybe worth it to see where people are after the round of discussion? Best, Cecelia On 12/10/2012 12:40 PM, Steve Hankin wrote: Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: Dear Jon Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because*_it's the one we really use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numerical climate modeling we use the proleptic Greogorian
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
On Mon, Dec 17, 2012 at 4:57 PM, John Caron wrote: > The current proposal(s) would not change files that are written with > :Conventions="CF-1.x", where x <= 6. Files with x > 6 could still use the > (ill-advised) old way if they want to, by putting an explicit calendar > attribute in. But if theres no explicit calendar attribute, then these new > files will be interpreted in a way that is less likely to give incorrect > dates. > > So, im not sure why you keep saying "shouldn't break current files", since > there is no such proposal on the table. The trick is that I suspect a lot of client software may not check CF version carefully, or at all, or with a version>=something check. I know that I"ve never thought about calendars, but then again, I don't think I've even seen a "since 1-1-1" file either -- that does seem an odd choice! >> But anyway, I wonder if folks currently using such files are actually >> getting the "correct" results, when they do. > those using udunits get the correct result even when they cross the line. i > suspect its correct > because using a different implementation (joda time > library) gets the same results for the > small sample i have tested. Good to know -- though strictly speaking, you need to not only use udunits, but use it correctly -- i.e. with the right calendar. Anyway, I was suggesting that there may be a lot of mis-use of the current default anyway -- I have no idea what folks are using for clients. If there are a lot of clients that don't currently "do the right thing", then maybe we can be less concerned about "breakage". -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Chris: On 12/17/2012 4:50 PM, Chris Barker - NOAA Federal wrote: I think a solution shouldn't break current files which followed what had been a standard for a long time (however ill-advised the standard was). I don't have a good sense of what might break if the standard changed in terms of software so I can' speak for all users but I do know many people have downloaded our mean files with 1-1-1 base dates Hmm -- this seems to contradict Steve's "no known users". there are no known users using mixed gregorian/julian who actually need to use it (cross the transition date line). if there are any such users, please speak up, so we can think about your use case. BUt anyway, I wonder if folks currently using such files are actually getting the "correct" results, when they do. those using udunits get the correct result even when they cross the line. i suspect its correct because using a different implementation (joda time library) gets the same results for the small sample i have tested. For instance, in my naiveté, I never thought about calendars when using netcdf files -- I simple converted "something since some date" to Python datetime objects using python's datetime package, which uses "An idealized naive date, assuming the current Gregorian calendar always was, and always will be, in effect" So I would have gotten the "wrong" results. it sounds like you were using what is called proplectic gregorian. so you would have gotten wrong results on the PSD dataset, which uses "time since 1-1-1". Thats the argument for not using mixed gregorian, that simple things arent simple. I've lost track whether UDunits does the mixed calendar right, but I'm sure not all libraries one might use do. So is this a use case se need to preserve? we have datasets out there who are using "time since 1-1-1" unfortunately. Theres no doubt we need to leave those "working"; and even that they can continue their dangerous ways if they want to explicitly include a calendar attribute in CF-1.7 convention file. The proposal is just to change the default in CF-1.7 to use a more common calendar, and to encourage people not to cross the time warp zone unecessarily. John ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Cathy: I think that you are using "backwards compatible" in a different way. The current proposal(s) would not change files that are written with :Conventions="CF-1.x", where x <= 6. Files with x > 6 could still use the (ill-advised) old way if they want to, by putting an explicit calendar attribute in. But if theres no explicit calendar attribute, then these new files will be interpreted in a way that is less likely to give incorrect dates. So, im not sure why you keep saying "shouldn't break current files", since there is no such proposal on the table. John On 12/17/2012 4:13 PM, Cathy Smith (NOAA Affiliate) wrote: Cecelia I think a solution shouldn't break current files which followed what had been a standard for a long time (however ill-advised the standard was). I don't have a good sense of what might break if the standard changed in terms of software so I can' speak for all users but I do know many people have downloaded our mean files with 1-1-1 base dates (ignoring the climatologies for now). While we can potentially change what we have either by changing the dates and/or adding a calendar attribute, changing the default calendar may create errors in reading dates for users who already have those files (which are currently CF complaint ). And, they won't have the same ability to change the files and they wouldn't necessarily know they needed to. I think no default at all would be problematic as what would software do? So, I would support the inclusion of a calendar attribute that would be used by software if there but using the old default calendar if not. Also, I would support making a calendar attribute for new files mandatory (with an updated CF version) but I would keep backwards compatibility unless it were reliably shown it was not an issue with users. I'm not convinced that the users needs (as opposed to developers) have been adequately researched. Cathy On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote: Cathy, Of the other options, do you find some (or all) completely unacceptable? Are some better than others? - Cecelia On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote: Cecelia I support 1) mostly for backward compatibility. I would also strongly encourage but not demand that users change their base dates to after 1800 when it makes sense to do so. And, I (again) want to make sure that LTMs and their time values are addressed before any decisions are made as to negative times and using base dates of 1-1-1 and the issue of what year to use for climatologies. LTM dates are a problem when one needs to use a calendar based on real dates. Cathy On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: Hi Steve, Jonathan and all, There are not that many options being discussed. With respect to the default calendar: 1 keep the Julian-Gregorian calendar as default (no change) 2 remove the Julian-Gregorian calendar as default, and have no default calendar (grid analogy) 3 replace the Julian-Gregorian calendar as default with the proleptic Gregorian calendar 4 replace the Julian-Gregorian calendar as default with a strict Gregorian calendar Maybe it makes sense for people to cite (or rank) their preference at this point? There were a couple other proposals, depending on which of above is selected: 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed for 4) 6 remove the Julian-Gregorian calendar (impossible for 1, optional for 2, 3, 4) Again, maybe worth it to see where people are after the round of discussion? Best, Cecelia On 12/10/2012 12:40 PM, Steve Hankin wrote: Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: Dear Jon Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because*_it's the one we really use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numeric
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
> I think a solution shouldn't break current files which followed what had > been a standard for a long time (however ill-advised the standard was). I > don't have a good sense of what might break if the standard changed in terms > of software so I can' speak for all users but I do know many people have > downloaded our mean files with 1-1-1 base dates Hmm -- this seems to contradict Steve's "no known users". BUt anyway, I wonder if folks currently using such files are actually getting the "correct" results, when they do. For instance, in my naiveté, I never thought about calendars when using netcdf files -- I simple converted "something since some date" to Python datetime objects using python's datetime package, which uses "An idealized naive date, assuming the current Gregorian calendar always was, and always will be, in effect" So I would have gotten the "wrong" results. I've lost track whether UDunits does the mixed calendar right, but I'm sure not all libraries one might use do. So is this a use case se need to preserve? -Chris (ignoring the climatologies > for now). While we can potentially change what we have either by changing > the dates and/or adding a calendar attribute, changing the default calendar > may create errors in reading dates for users who already have those files > (which are currently CF complaint ). And, they won't have the same ability > to change the files and they wouldn't necessarily know they needed to. I > think no default at all would be problematic as what would software do? So, > I would support the inclusion of a calendar attribute that would be used by > software if there but using the old default calendar if not. Also, I would > support making a calendar attribute for new files mandatory (with an updated > CF version) but I would keep backwards compatibility unless it were reliably > shown it was not an issue with users. > I'm not convinced that the users needs (as opposed to developers) have been > adequately researched. > > Cathy > > > > On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote: > > Cathy, > > Of the other options, do you find some (or all) completely unacceptable? > Are some better than others? > > - Cecelia > > On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote: > > Cecelia > I support 1) mostly for backward compatibility. I would also strongly > encourage but not demand that users change their base dates to after 1800 > when it makes sense to do so. > > And, I (again) want to make sure that LTMs and their time values are > addressed before any decisions are made as to negative times and using base > dates of 1-1-1 and the issue of what year to use for climatologies. LTM > dates are a problem when one needs to use a calendar based on real dates. > > Cathy > > > On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: > > Hi Steve, Jonathan and all, > > There are not that many options being discussed. > > With respect to the default calendar: > > 1 keep the Julian-Gregorian calendar as default (no change) > 2 remove the Julian-Gregorian calendar as default, and have no default > calendar (grid analogy) > 3 replace the Julian-Gregorian calendar as default with the proleptic > Gregorian calendar > 4 replace the Julian-Gregorian calendar as default with a strict Gregorian > calendar > > Maybe it makes sense for people to cite (or rank) their preference at this > point? > > There were a couple other proposals, depending on which of above is > selected: > 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed for 4) > 6 remove the Julian-Gregorian calendar (impossible for 1, optional for 2, 3, > 4) > > Again, maybe worth it to see where people are after the round of discussion? > > Best, > Cecelia > > > > On 12/10/2012 12:40 PM, Steve Hankin wrote: > > Hi Jonathan, > > I'm not sure if my remarks below conflict with your proposed resolution. > But they do dispute the facts you assert, and these waters are so muddy that > agreeing on the facts seems an important first step. > > On 12/10/2012 1:21 AM, Jonathan Gregory wrote: > > Dear Jon > > Just to repeat a remark that Steve Hankin made whose implications have not > been explored in this discussion: different countries adopted the Gregorian > calendar at different times. (Greece didn't adopt it till 1923!) So what > is considered a valid Gregorian date varies from country to country (and > some of those countries don't even exist any more, or at least the > boundaries have changed...) > > 2. The non-proleptic Gregorian calendar is extremely problematic for > historical observations as well as for models (astronomers use the Julian > calendar consistently for this reason). > > Yes, that's right. Nonetheless I don't think we can abolish the real-world > calendar, despite its ambiguities, because it's the one we really use! > > > Are you sure this is true? Evidence seems to suggest that our community has > no use for the mixed Gregorian/Julian calendar at all, exc
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Cecelia I think a solution shouldn't break current files which followed what had been a standard for a long time (however ill-advised the standard was). I don't have a good sense of what might break if the standard changed in terms of software so I can' speak for all users but I do know many people have downloaded our mean files with 1-1-1 base dates (ignoring the climatologies for now). While we can potentially change what we have either by changing the dates and/or adding a calendar attribute, changing the default calendar may create errors in reading dates for users who already have those files (which are currently CF complaint ). And, they won't have the same ability to change the files and they wouldn't necessarily know they needed to. I think no default at all would be problematic as what would software do? So, I would support the inclusion of a calendar attribute that would be used by software if there but using the old default calendar if not. Also, I would support making a calendar attribute for new files mandatory (with an updated CF version) but I would keep backwards compatibility unless it were reliably shown it was not an issue with users. I'm not convinced that the users needs (as opposed to developers) have been adequately researched. Cathy On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote: > Cathy, > > Of the other options, do you find some (or all) completely unacceptable? > Are some better than others? > > - Cecelia > > On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote: >> Cecelia >> I support 1) mostly for backward compatibility. I would also strongly >> encourage but not demand that users change their base dates to after >> 1800 >> when it makes sense to do so. >> >> And, I (again) want to make sure that LTMs and their time values are >> addressed before any decisions are made as to negative times and >> using base dates of 1-1-1 and the issue of what year to use for >> climatologies. LTM dates are a problem when one needs to use a >> calendar based on real dates. >> >> Cathy >> >> >> On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: >>> Hi Steve, Jonathan and all, >>> >>> There are not that many options being discussed. >>> >>> With respect to the default calendar: >>> >>> 1 keep the Julian-Gregorian calendar as default (no change) >>> 2 remove the Julian-Gregorian calendar as default, and have no >>> default calendar (grid analogy) >>> 3 replace the Julian-Gregorian calendar as default with the >>> proleptic Gregorian calendar >>> 4 replace the Julian-Gregorian calendar as default with a strict >>> Gregorian calendar >>> >>> Maybe it makes sense for people to cite (or rank) their preference >>> at this point? >>> >>> There were a couple other proposals, depending on which of above is >>> selected: >>> 5 create a strict Gregorian calendar (optional for 1, 2, 3 and >>> needed for 4) >>> 6 remove the Julian-Gregorian calendar (impossible for 1, optional >>> for 2, 3, 4) >>> >>> Again, maybe worth it to see where people are after the round of >>> discussion? >>> >>> Best, >>> Cecelia >>> >>> >>> >>> On 12/10/2012 12:40 PM, Steve Hankin wrote: Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: > Dear Jon > >> Just to repeat a remark that Steve Hankin made whose implications have >> not been explored in this discussion: different countries adopted the >> Gregorian calendar at different times. (Greece didn't adopt it till >> 1923!) So what is considered a valid Gregorian date varies from country >> to country (and some of those countries don't even exist any more, or at >> least the boundaries have changed...) >> 2. The non-proleptic Gregorian calendar is extremely problematic for >> historical observations as well as for models (astronomers use the >> Julian calendar consistently for this reason). > Yes, that's right. Nonetheless I don't think we can abolish the real-world > calendar, despite its ambiguities, because *_it's the one we really > use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numerical climate modeling we use the proleptic Greogorian calendar. (I'll wager you there is no serious paleo-modeling done with an 11 day discontinuity in its time axis. ) * What do Renaissance historians use when discussing dates that are rendered ambiguous by dif
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Cathy, Of the other options, do you find some (or all) completely unacceptable? Are some better than others? - Cecelia On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote: Cecelia I support 1) mostly for backward compatibility. I would also strongly encourage but not demand that users change their base dates to after 1800 when it makes sense to do so. And, I (again) want to make sure that LTMs and their time values are addressed before any decisions are made as to negative times and using base dates of 1-1-1 and the issue of what year to use for climatologies. LTM dates are a problem when one needs to use a calendar based on real dates. Cathy On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: Hi Steve, Jonathan and all, There are not that many options being discussed. With respect to the default calendar: 1 keep the Julian-Gregorian calendar as default (no change) 2 remove the Julian-Gregorian calendar as default, and have no default calendar (grid analogy) 3 replace the Julian-Gregorian calendar as default with the proleptic Gregorian calendar 4 replace the Julian-Gregorian calendar as default with a strict Gregorian calendar Maybe it makes sense for people to cite (or rank) their preference at this point? There were a couple other proposals, depending on which of above is selected: 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed for 4) 6 remove the Julian-Gregorian calendar (impossible for 1, optional for 2, 3, 4) Again, maybe worth it to see where people are after the round of discussion? Best, Cecelia On 12/10/2012 12:40 PM, Steve Hankin wrote: Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: Dear Jon Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because*_it's the one we really use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numerical climate modeling we use the proleptic Greogorian calendar. (I'll wager you there is no serious paleo-modeling done with an 11 day discontinuity in its time axis. ) * What do Renaissance historians use when discussing dates that are rendered ambiguous by differing timings of the Julian/Gregorian transition in different locations? Do any of us know? Does it effect any use of CF that we are aware of? As you say, we should be clearer about what the real-world calendar means, in cases where_users really want to use it._ Who are these users? Where is the user who intersects with our community and really wants to use the mixed Julian/Gregorian calendar? The only potential user I can think of would be a Renaissance historian looking at paleo climate model output. That hypothetical person would already understand that manual calendar translations were needed to make sense of precise dates at that time of history (and would almost surely shrug off an 11 day timing uncertainty in a paleo climate model outputs in any case). As Cecelia said, lets drive a stake through the heart of this madness ... at least to the maximum degree we can given inescapable backwards compatibility concerns. - Steve ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email:cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- -- NOAA/ESRL PSD and CIRES CDC 303-497-6263 http://www.esrl.noaa.gov/psd/people/cathy.smith/ Emails about data/webpages may g
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Cecelia I support 1) mostly for backward compatibility. I would also strongly encourage but not demand that users change their base dates to after 1800 when it makes sense to do so. And, I (again) want to make sure that LTMs and their time values are addressed before any decisions are made as to negative times and using base dates of 1-1-1 and the issue of what year to use for climatologies. LTM dates are a problem when one needs to use a calendar based on real dates. Cathy On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote: > Hi Steve, Jonathan and all, > > There are not that many options being discussed. > > With respect to the default calendar: > > 1 keep the Julian-Gregorian calendar as default (no change) > 2 remove the Julian-Gregorian calendar as default, and have no default > calendar (grid analogy) > 3 replace the Julian-Gregorian calendar as default with the proleptic > Gregorian calendar > 4 replace the Julian-Gregorian calendar as default with a strict > Gregorian calendar > > Maybe it makes sense for people to cite (or rank) their preference at > this point? > > There were a couple other proposals, depending on which of above is > selected: > 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed > for 4) > 6 remove the Julian-Gregorian calendar (impossible for 1, optional for > 2, 3, 4) > > Again, maybe worth it to see where people are after the round of > discussion? > > Best, > Cecelia > > > > On 12/10/2012 12:40 PM, Steve Hankin wrote: >> Hi Jonathan, >> >> I'm not sure if my remarks below conflict with your proposed >> resolution. But they do dispute the facts you assert, and these >> waters are so muddy that agreeing on the facts seems an important >> first step. >> >> On 12/10/2012 1:21 AM, Jonathan Gregory wrote: >>> Dear Jon >>> Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). >>> Yes, that's right. Nonetheless I don't think we can abolish the real-world >>> calendar, despite its ambiguities, because *_it's the one we really use!_* >> >> Are you sure this is true? Evidence seems to suggest that our >> community has _no use for the mixed Gregorian/Julian calendar at >> all_, except the need to resolve the backwards compatibility mess we >> have created for ourselves. >> >> * In everyday life we use is the modern Gregorian calendar, and are >> not concerned with historical calendar changes. >> * In numerical climate modeling we use the proleptic Greogorian >> calendar. (I'll wager you there is no serious paleo-modeling >> done with an 11 day discontinuity in its time axis. ) >> * What do Renaissance historians use when discussing dates that are >> rendered ambiguous by differing timings of the Julian/Gregorian >> transition in different locations? Do any of us know? Does it >> effect any use of CF that we are aware of? >> >>> As you say, we should be clearer about what the real-world calendar means, >>> in >>> cases where _users really want to use it._ >> >> Who are these users? Where is the user who intersects with our >> community and really wants to use the mixed Julian/Gregorian >> calendar? The only potential user I can think of would be a >> Renaissance historian looking at paleo climate model output. That >> hypothetical person would already understand that manual calendar >> translations were needed to make sense of precise dates at that time >> of history (and would almost surely shrug off an 11 day timing >> uncertainty in a paleo climate model outputs in any case). >> >> As Cecelia said, lets drive a stake through the heart of this madness >> ... at least to the maximum degree we can given inescapable backwards >> compatibility concerns. >> >> - Steve >> >> >> ___ >> CF-metadata mailing list >> CF-metadata@cgd.ucar.edu >> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > -- > === > Cecelia DeLuca > NESII/CIRES/NOAA Earth System Research Laboratory > 325 Broadway, Boulder 80305-337 > Email: cecelia.del...@noaa.gov > Phone: 303-497-3604 > > > ___ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- -- NOAA/ESRL PSD and CIRES CDC 303-497-6263 http://www.esrl.noaa.gov/psd/p
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
On 12/14/2012 9:35 AM, Jonathan Gregory wrote: Dear Cecilia, Steve et al. Steve is right that mostly we use the Gregorian calendar. That is what I meant mostly when I said that the default is the calendar we use. The real world is mixed Julian-Gregorian, and I don't think dealing with this calendar is an issue only for Renaissance historians. I can't give you examples, but I think it is conceivable or likely that at some point people would want to record real-world data in CF earlier than the Renaissance, or have already done so. For instance, what about astronomical data, such as the dates of eclipses. These are real-world events, on precise dates which are translated into the mixed Julian-Gregorian calendar. Hi Jonathan, If scientists somewhere have encoded the dates of these historical events as data(*) using a mixed Gregorian-Julian calendar Lord help 'em. Those poor folks have to face an 11 day discontinuity in their own data, as well as in ours. I'm not meaning to be snarky. I just want to stamp out this pesky calendar issue. It has been tripping us up for too many years. Your points below are definitely the real guts of the discussion, but in this email I am addressing just the one single point. I'm afraid we will never heal ourselves from this virus if we do not eradicate it from our thinking. - Steve (*) attaching a date to an historical narrative is different from using a date as a time coordinate. It's metadata versus data. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi all: Heres what I understand of the conversation: 1. Theres nothing to do about existing files CF-1.6 and before. we are stuck with the udunits mixed calendar. 2. Starting with the next version after acceptance, (1.7 currently), we can do something different. I agree that forcing people to put in a calendar attribute makes simple things not simple. So lets choose a reasonable default, either gregorian_propleptic or gregorian_strict is ok with me. 3. everyone agrees that in the unit "time since date", date is interpreted in the calendar that is specified, or the default if not specified. 4. i thing we should add some advice not to start counting from 0001 when recording modern dates, no matter if you do it "right" or not. John ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Cecilia, Steve et al. Steve is right that mostly we use the Gregorian calendar. That is what I meant mostly when I said that the default is the calendar we use. The real world is mixed Julian-Gregorian, and I don't think dealing with this calendar is an issue only for Renaissance historians. I can't give you examples, but I think it is conceivable or likely that at some point people would want to record real-world data in CF earlier than the Renaissance, or have already done so. For instance, what about astronomical data, such as the dates of eclipses. These are real-world events, on precise dates which are translated into the mixed Julian-Gregorian calendar. It would not be sensible to insist on translating real-world dates into the non-real-world proleptic Gregorian calendar. Hence we have to continue to support the mixed calendar. Abolishing it in CF is not one of Cecilia's four options, which only concern what the default should be. > With respect to the default calendar: > 3 replace the Julian-Gregorian calendar as default with the > proleptic Gregorian calendar I don't think this is acceptable since it changes the meaning of existing data. > 1 keep the Julian-Gregorian calendar as default (no change) Since the current default is a pitfall, changing the default would be preferable. > 2 remove the Julian-Gregorian calendar as default, and have no > default calendar (grid analogy) > 4 replace the Julian-Gregorian calendar as default with a strict > Gregorian calendar I think either of these would work. 2 causes more aggravation. It means that data which doesn't state the calendar attribute is illegal and will produce errors, even if it's entirely unproblematic such as "days since 2012-1-1". It would make CF more intolerant of existing practices than it usually has been. At the moment, CF accepts COARDS time coordinates; with this change, COARDS data would not be acceptable. Hence I would still prefer 4. The aim of this would be to make the default illegal in cases where there is a serious chance of unsafe time units, and the obvious criterion seemed to prevent dates before the invention of the Gregorian calendar. In particular that will exclude reference years of 0 and 1, which are often problematic. However, I don't feel strongly about it. As has been said in other postings, CF has its own (non-COARDS) ways of expressing climatological time. I think that calendar-based units of time cannot be introduced without a new syntax for time units, and some rules about how to interpret the cases when adding months to a reference date gives an impossible date. We could make such changes to the convention. Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Steve, Jonathan and all, There are not that many options being discussed. With respect to the default calendar: 1 keep the Julian-Gregorian calendar as default (no change) 2 remove the Julian-Gregorian calendar as default, and have no default calendar (grid analogy) 3 replace the Julian-Gregorian calendar as default with the proleptic Gregorian calendar 4 replace the Julian-Gregorian calendar as default with a strict Gregorian calendar Maybe it makes sense for people to cite (or rank) their preference at this point? There were a couple other proposals, depending on which of above is selected: 5 create a strict Gregorian calendar (optional for 1, 2, 3 and needed for 4) 6 remove the Julian-Gregorian calendar (impossible for 1, optional for 2, 3, 4) Again, maybe worth it to see where people are after the round of discussion? Best, Cecelia On 12/10/2012 12:40 PM, Steve Hankin wrote: Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: Dear Jon Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because*_it's the one we really use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numerical climate modeling we use the proleptic Greogorian calendar. (I'll wager you there is no serious paleo-modeling done with an 11 day discontinuity in its time axis. ) * What do Renaissance historians use when discussing dates that are rendered ambiguous by differing timings of the Julian/Gregorian transition in different locations? Do any of us know? Does it effect any use of CF that we are aware of? As you say, we should be clearer about what the real-world calendar means, in cases where_users really want to use it._ Who are these users? Where is the user who intersects with our community and really wants to use the mixed Julian/Gregorian calendar? The only potential user I can think of would be a Renaissance historian looking at paleo climate model output. That hypothetical person would already understand that manual calendar translations were needed to make sense of precise dates at that time of history (and would almost surely shrug off an 11 day timing uncertainty in a paleo climate model outputs in any case). As Cecelia said, lets drive a stake through the heart of this madness ... at least to the maximum degree we can given inescapable backwards compatibility concerns. - Steve ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
The datum should always be interpreted according to the calendar system specified in the file. That's how the time coordinate is handled in the output from every model that I've seen (which is at least a dozen), so I'd say do it that way just for the sake of backwards compatibility. But it's not clear to me that any alternative is even possible. What would it mean to say that my time coordinate uses a 360-day calendar, but that the start date is Jan 1, 1900 in the mixed gregorian calendar? Or worse, if the start date was Jan 31? It doesn't really make any sense. Likewise, units of months and years should also be interpreted according to the specified calendar. Using months and years as units is cautioned against in the spec, but if one were to do it, I'd say the expectation of nearly every end user and data provider would be that they would be in the same units as the calendar. Certainly, if you had a file of monthly data with a units attribute of "months" on the time variable, it would be obvious what was meant if the time coordinate went 1-2-3-4-5, and very confusing if it had some kind of non- integer spacing that corresponded to (days in month)*12/365.25... Cheers, --Seth On Sun, 9 Dec 2012 19:55:39 + Jon Blower wrote: > >Also, there are a couple of ambiguities in CF, which have also been brought up >before but not resolved. Time axes are recorded as a sequence of numbers >representing a certain number of intervals since a specified datum. The datum >is part of the units string of the time axis and is recorded in a kind of >pseudo-ISO format (e.g. "1970-1-1 0:0:0"). Should the datum always be >interpreted as a date/time in the mixed Gregorian/Julian calendar (as >specified by UDUNITS) or in the calendar system specified in the CF file >(which would probably be more sensible)? Also, should interval lengths (e.g. >months and years) be interpreted as the UDUNITS lengths (this is specified in >CF but is not always helpful) or should these vary by calendar (which is again >perhaps sensible, and some data providers do this)? ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Jonathan, I'm not sure if my remarks below conflict with your proposed resolution. But they do dispute the facts you assert, and these waters are so muddy that agreeing on the facts seems an important first step. On 12/10/2012 1:21 AM, Jonathan Gregory wrote: Dear Jon Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because*_it's the one we really use!_* Are you sure this is true? Evidence seems to suggest that our community has _no use for the mixed Gregorian/Julian calendar at all_, except the need to resolve the backwards compatibility mess we have created for ourselves. * In everyday life we use is the modern Gregorian calendar, and are not concerned with historical calendar changes. * In numerical climate modeling we use the proleptic Greogorian calendar. (I'll wager you there is no serious paleo-modeling done with an 11 day discontinuity in its time axis. ) * What do Renaissance historians use when discussing dates that are rendered ambiguous by differing timings of the Julian/Gregorian transition in different locations? Do any of us know? Does it effect any use of CF that we are aware of? As you say, we should be clearer about what the real-world calendar means, in cases where_users really want to use it._ Who are these users? Where is the user who intersects with our community and really wants to use the mixed Julian/Gregorian calendar? The only potential user I can think of would be a Renaissance historian looking at paleo climate model output. That hypothetical person would already understand that manual calendar translations were needed to make sense of precise dates at that time of history (and would almost surely shrug off an 11 day timing uncertainty in a paleo climate model outputs in any case). As Cecelia said, lets drive a stake through the heart of this madness ... at least to the maximum degree we can given inescapable backwards compatibility concerns. - Steve ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Jonathan, On 12/9/2012 8:45 AM, Jonathan Gregory wrote: Dear Cecilia Thanks for your posting. Are we in agreement? I think that we are. I think we are close. The difference I see is that you are recommending a change to a strict Gregorian calendar as the default, and in my last mail I had advocated no default. it is a good idea to add another, strict Gregorian (error before 1582). Thanks for seeing it like that. Yes, perhaps we should define what I suggested as a new calendar: strict_gregorian, which is not allowed to have a ref date before 1582, or to use negative time coordinates, or to describe dates before 1582 (which would be impossible given the first two conditions). (Note, I am not personally the eponymous strict Gregory.) A relative advantage of no default is that it's not necessary to either change the definition of the current CF mixed Gregorian calendar or create a new CF strict Gregorian calendar. It sounds like there are issues with making the strict Gregorian calendar the default on the data side, and it does not work any better than the mixed Gregorian on the modeling side. I can still imagine reasons to want to define a strict Gregorian calendar in CF, but with more time, and thought, and perhaps in a less than central role. Maybe the main thing no default could provide is a path to move forward - a chance to develop additional calendars and conventions that work better for particular purposes, and put them into use without needing to worry about creating or conflicting with the default as an implicit recommendation. It's more of a view of the calendar as analogous to a grid type, where the notion of a default is generally not well defined. Do you see a reason why a default is needed? In that case, we could change the default in the next release from gregorian or standard to strict_gregorian. For software which is careful about versions, this would mean that dates in the default calendar before 1582 would give an error. This would be a safe failure, rather than a wrong answer. With respect to tools, it may be that the no default solution is a safer change... If the CF version is not taken into account, and the calendar not specified: - with no default, tools would either work as previous, or would fail on all data points. - with the strict Gregorian default, tools would either work as previous, or would fail on some data points. Failure for some data points seems more prone to create confusion about the behavior of tools before and after the change, and more likely to create user errors when, for example, it is not noticed that some subset of data points has an invalid date. This may be worrying too much - maybe it is reasonable to expect tools to always check the CF version? That just doesn't seem to be the way that life (or software) turns out. Some data sets would be CF compliant only under particular versions of CF. Is this the first time that a change to CF would have such an impact? In the sense of old data becoming invalid in a *later* version, yes I think it is. Of course new features always allow datasets which are invalid in an *earlier* version. That's impressive... and does beg the question of whether a change in default calendar would be worth it. Best, Cecelia Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Jon > Just to repeat a remark that Steve Hankin made whose implications have not > been explored in this discussion: different countries adopted the Gregorian > calendar at different times. (Greece didn't adopt it till 1923!) So what is > considered a valid Gregorian date varies from country to country (and some of > those countries don't even exist any more, or at least the boundaries have > changed...) > 2. The non-proleptic Gregorian calendar is extremely problematic for > historical observations as well as for models (astronomers use the Julian > calendar consistently for this reason). Yes, that's right. Nonetheless I don't think we can abolish the real-world calendar, despite its ambiguities, because it's the one we really use! It is not realistic to expect users of CF for historical obs to convert their dates to the Julian calendar, I suspect, even if we say they should. My proposal for the strict_gregorian calendar is to limit the danger of the *default* calendar by forcing the calendar to be specified in cases where there's a high risk of problems e.g. with a reference date of 1-1-1, so that people are made aware of the danger. We could instead make it mandatory always to specify the calendar, as has already been suggested i.e. not allow a default at all. Would you prefer that? As you say, we should be clearer about what the real-world calendar means, in cases where users really want to use it. Adopting the earliest Julian-Gregorian switch-over date is consistent with udunits and has some logic. It means that dates would have to be converted for historical obs from countries which did not convert at first. People working with historical data are aware of different calendars being in use, so that seems more likely to be done than converting to an other-wordly Julian calendar. Alternatively we could leave it undefined, as it is now, but I don't think that's satisfactory. There must be instrumental data affected by late switch-over e.g. in Greece and Russia. We ought to say which calendar CF expects to be used for this. udunits doesn't say what calendar the ref date is in because udunits supports only the real-world calendar. I agree CF should be clear about this and the obvious rule is that the ref date is in the calendar you are using. As you say, that would be sensible. Any other choice would be rather bizarre. The use of units of time which are calendar-based (months and years) is deprecated in CF. That's because those units are not helpful, as you say, unless you have calendar-aware software to interpret them, and some additional rules on how that interpretation should be done e.g. what does 31 Jan 2012 + 1 month mean? I think this is a different issue from calendars, since it applies to any calendar which has variable-length months. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Jonathon We have the issue that we store many long terms means (subdaily,daily, monthly...). These long terms means have no "years" per se though the rest of the date is an actual 'date'. We have been storing them as year 1 generally speaking (usually base 1-1-1) as we want to avoid using a date within the date range of the dataset as we feel it confuses people. So, in your proposal below, we couldn't use year 1 as the date because we couldn't use 1-1-1., We also couldn't use 1800-1-1- and year 1 (negative dates). So, we would have to use some other year that didn't look like a real year but was >1800. We could use something like would work but would lead to large values of the time variable and would be hard to read. We could switch the base date for the LTM's (instead of 1800-1-1 use -1-1) but I'm not sure that is a good idea to switch base dates within a dataset. For us, the best thing for LTM's is to use 1-1-1 and year 1 as that is fairly easy to read and won't tend to confuse people. CF doesn't have a standard and looking at the doc page, the examples all show values that use a year from the range (something that really does confuse some users in our experience). We could also switch the default calendar and maybe that is sufficient but in my experience only people who care about calendars would notice any calendar attribute (e.g. not scientific users) so potentially that could prove an issue. If we did switch, I could see that possibly being a problem with applications as well, unless coded well. I'm really not sure the best solution to this problem but I do think any calender solution needs to take into account how to store climatological dates. Cathy On 12/9/12 8:45 AM, Jonathan Gregory wrote: > Dear Cecilia > > Thanks for your posting. Are we in agreement? I think that we are. > >> However, the current default is also problematic - reasons I see are: >> 1) it does send a message to modelers who find the default unusable, >> 2) because different calendar defaults are used on the modeling and >> data side, current tools are limited to particular calendars, >> affecting users, >> and 3) the mixed Julian-Gregorian calendar is an ugly beast to peg as the >> standard forevermore. > I agree with these drawbacks. If CF only dealt with the real world, obviously > the real-world calendar (mixed Julian-Gregorian) would be the default and only > calendar it had to deal with. However, at its outset CF was for models, many > of > which do not use this calendar, which was nonetheless chosen as the default > because of udunits. I agree with you and others that providing such an > inconvenient default is unsatisfactory in retrospect. But here we are! > >> it is a good idea to add another, strict Gregorian (error before 1582). > Thanks for seeing it like that. Yes, perhaps we should define what I suggested > as a new calendar: strict_gregorian, which is not allowed to have a ref date > before 1582, or to use negative time coordinates, or to describe dates before > 1582 (which would be impossible given the first two conditions). (Note, I am > not personally the eponymous strict Gregory.) > > In that case, we could change the default in the next release from gregorian > or standard to strict_gregorian. For software which is careful about versions, > this would mean that dates in the default calendar before 1582 would give an > error. This would be a safe failure, rather than a wrong answer. > >> Some data sets would be CF compliant only under particular versions of CF. >> Is this the first time that a change to CF would have such an impact? > In the sense of old data becoming invalid in a *later* version, yes I think > it is. Of course new features always allow datasets which are invalid in an > *earlier* version. > > In the failing cases, I think we agree that tools might offer the facility >> to provide the calendar if none was found. >> However, I imagine most tools would continue to assume the current default. > Yes, I expect so. They would still be unsafe, if datasets have been wrongly > coded with the default, but it would be no worse than now. > >> This does not seem so bad to me. The removal of the default solution should >> not produce the nastier kinds of errors that you would get if you >> changed the default. > That's what I think, except that formally the above suggestion is a change to > to the default - one which installs a fail-safe mechanism. > > Best wishes > > Jonathan > ___ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- -- NOAA/ESRL PSD and CIRES CDC 303-497-6263 http://www.esrl.noaa.gov/psd/people/cathy.smith/ Emails about data/webpages may get quicker responses from emailing esrl.psd.d...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Jonathan, all, Just to repeat a remark that Steve Hankin made whose implications have not been explored in this discussion: different countries adopted the Gregorian calendar at different times. (Greece didn't adopt it till 1923!) So what is considered a valid Gregorian date varies from country to country (and some of those countries don't even exist any more, or at least the boundaries have changed...) This has two main implications: 1. I don't agree with the proposal for a "strict Gregorian" calendar - the proposed switchover of 1582 is largely meaningless. 2. The non-proleptic Gregorian calendar is extremely problematic for historical observations as well as for models (astronomers use the Julian calendar consistently for this reason). Also, there are a couple of ambiguities in CF, which have also been brought up before but not resolved. Time axes are recorded as a sequence of numbers representing a certain number of intervals since a specified datum. The datum is part of the units string of the time axis and is recorded in a kind of pseudo-ISO format (e.g. "1970-1-1 0:0:0"). Should the datum always be interpreted as a date/time in the mixed Gregorian/Julian calendar (as specified by UDUNITS) or in the calendar system specified in the CF file (which would probably be more sensible)? Also, should interval lengths (e.g. months and years) be interpreted as the UDUNITS lengths (this is specified in CF but is not always helpful) or should these vary by calendar (which is again perhaps sensible, and some data providers do this)? Hope this helps, Jon -Original Message- From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory Sent: 09 December 2012 15:46 To: cf-metadata@cgd.ucar.edu Subject: [CF-metadata] CF calendars (was: problem with times in PSD dataset) Dear Cecilia Thanks for your posting. Are we in agreement? I think that we are. > However, the current default is also problematic - reasons I see are: > 1) it does send a message to modelers who find the default unusable, > 2) because different calendar defaults are used on the modeling and > data side, current tools are limited to particular calendars, > affecting users, and 3) the mixed Julian-Gregorian calendar is an ugly > beast to peg as the standard forevermore. I agree with these drawbacks. If CF only dealt with the real world, obviously the real-world calendar (mixed Julian-Gregorian) would be the default and only calendar it had to deal with. However, at its outset CF was for models, many of which do not use this calendar, which was nonetheless chosen as the default because of udunits. I agree with you and others that providing such an inconvenient default is unsatisfactory in retrospect. But here we are! > it is a good idea to add another, strict Gregorian (error before 1582). Thanks for seeing it like that. Yes, perhaps we should define what I suggested as a new calendar: strict_gregorian, which is not allowed to have a ref date before 1582, or to use negative time coordinates, or to describe dates before 1582 (which would be impossible given the first two conditions). (Note, I am not personally the eponymous strict Gregory.) In that case, we could change the default in the next release from gregorian or standard to strict_gregorian. For software which is careful about versions, this would mean that dates in the default calendar before 1582 would give an error. This would be a safe failure, rather than a wrong answer. > Some data sets would be CF compliant only under particular versions of CF. > Is this the first time that a change to CF would have such an impact? In the sense of old data becoming invalid in a *later* version, yes I think it is. Of course new features always allow datasets which are invalid in an *earlier* version. In the failing cases, I think we agree that tools might offer the facility > to provide the calendar if none was found. > However, I imagine most tools would continue to assume the current default. Yes, I expect so. They would still be unsafe, if datasets have been wrongly coded with the default, but it would be no worse than now. > This does not seem so bad to me. The removal of the default solution > should not produce the nastier kinds of errors that you would get if > you changed the default. That's what I think, except that formally the above suggestion is a change to to the default - one which installs a fail-safe mechanism. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear Cecilia Thanks for your posting. Are we in agreement? I think that we are. > However, the current default is also problematic - reasons I see are: > 1) it does send a message to modelers who find the default unusable, > 2) because different calendar defaults are used on the modeling and > data side, current tools are limited to particular calendars, > affecting users, > and 3) the mixed Julian-Gregorian calendar is an ugly beast to peg as the > standard forevermore. I agree with these drawbacks. If CF only dealt with the real world, obviously the real-world calendar (mixed Julian-Gregorian) would be the default and only calendar it had to deal with. However, at its outset CF was for models, many of which do not use this calendar, which was nonetheless chosen as the default because of udunits. I agree with you and others that providing such an inconvenient default is unsatisfactory in retrospect. But here we are! > it is a good idea to add another, strict Gregorian (error before 1582). Thanks for seeing it like that. Yes, perhaps we should define what I suggested as a new calendar: strict_gregorian, which is not allowed to have a ref date before 1582, or to use negative time coordinates, or to describe dates before 1582 (which would be impossible given the first two conditions). (Note, I am not personally the eponymous strict Gregory.) In that case, we could change the default in the next release from gregorian or standard to strict_gregorian. For software which is careful about versions, this would mean that dates in the default calendar before 1582 would give an error. This would be a safe failure, rather than a wrong answer. > Some data sets would be CF compliant only under particular versions of CF. > Is this the first time that a change to CF would have such an impact? In the sense of old data becoming invalid in a *later* version, yes I think it is. Of course new features always allow datasets which are invalid in an *earlier* version. In the failing cases, I think we agree that tools might offer the facility > to provide the calendar if none was found. > However, I imagine most tools would continue to assume the current default. Yes, I expect so. They would still be unsafe, if datasets have been wrongly coded with the default, but it would be no worse than now. > This does not seem so bad to me. The removal of the default solution should > not produce the nastier kinds of errors that you would get if you > changed the default. That's what I think, except that formally the above suggestion is a change to to the default - one which installs a fail-safe mechanism. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
Jonathan and all There are aspects of this decision that relate to data sets, models, model/data set alignment, tools, and the scope of the CF user base. I agree that changing the default is problematic. Tools (and people) would have a hard time with prior to change/post change datasets that used the default, in which the only calendar indication was the version of CF used. However, the current default is also problematic - reasons I see are: 1) it does send a message to modelers who find the default unusable, 2) because different calendar defaults are used on the modeling and data side, current tools are limited to particular calendars, affecting users, and 3) the mixed Julian-Gregorian calendar is an ugly beast to peg as the standard forevermore. It's probably possible to overcome 2) by updating tools to include additional calendars. The calendar definitions in CF are pretty clear and it does not seem necessary to change existing ones, but maybe it is a good idea to add another, strict Gregorian (error before 1582). If the calendar default was removed in a future version of CF, this would create some issues, but it would also have benefits that may outweigh them: Some data sets would be CF compliant only under particular versions of CF. Is this the first time that a change to CF would have such an impact? Some tools might be modified to expect all data sets to specify calendars. These would produce errors if no calendar was specified, or in some cases, as Jonathan suggested, the user could be asked to provide the calendar if none was found. However, I imagine most tools would continue to assume the current default. The way to talk about these tools could be that they would work on CF-compliant data sets that spanned versions of CF pre/post removal of the default calendar I would expect that most software based on CF would not have to change. This does not seem so bad to me. The removal of the default solution should not produce the nastier kinds of errors that you would get if you changed the default. People will weigh costs and benefits differently - and I may be missing some major consideration above - but for me, it seems like it's worth it to remove the weird default calendar and move on. Best, - Cecelia On 12/7/2012 6:53 AM, Jonathan Gregory wrote: Dear all I think that we should be very cautious about backwards-incompatibility, even when changing CF versions. I would be worried about making a change which means metadata would have a different meaning according to which version of the convention is used. I know in principle it is safe because the file states the version, but I don't think we can depend on the version to be coded correctly, or on software to make correct use of it. Although these are errors, we ought nonetheless to design the convention to be as robust as possible in an error-prone world. Therefore I don't think it is safe to change the the default calendar from gregorian to proleptic_gregorian, because that could change the date you get for a given time coordinate. However, I do think it would be safe to make it illegal to use the default calendar for reference dates earlier than 1582-10-15, or for negative time coordinates (to make sure that dates before 1582-10-15 cannot be specified). In that case, real-world data for which there is no problem would be unaffected, but software would reject time coordinates whose interpretation is problematic. Software could provide an option whereby the user could state what the calendar actually is, when not specified, in order to override the error if the user knows what it truly is. Alternatively, they could easily repair the file with ncatted or some such tool. It should still be possible to use the real-world calendar before 1582-10-15, because CF might be used for real-world historical data earlier than that date, but it would be necessary to state the calendar explicitly. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- === Cecelia DeLuca NESII/CIRES/NOAA Earth System Research Laboratory 325 Broadway, Boulder 80305-337 Email: cecelia.del...@noaa.gov Phone: 303-497-3604 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Dear all I think that we should be very cautious about backwards-incompatibility, even when changing CF versions. I would be worried about making a change which means metadata would have a different meaning according to which version of the convention is used. I know in principle it is safe because the file states the version, but I don't think we can depend on the version to be coded correctly, or on software to make correct use of it. Although these are errors, we ought nonetheless to design the convention to be as robust as possible in an error-prone world. Therefore I don't think it is safe to change the the default calendar from gregorian to proleptic_gregorian, because that could change the date you get for a given time coordinate. However, I do think it would be safe to make it illegal to use the default calendar for reference dates earlier than 1582-10-15, or for negative time coordinates (to make sure that dates before 1582-10-15 cannot be specified). In that case, real-world data for which there is no problem would be unaffected, but software would reject time coordinates whose interpretation is problematic. Software could provide an option whereby the user could state what the calendar actually is, when not specified, in order to override the error if the user knows what it truly is. Alternatively, they could easily repair the file with ncatted or some such tool. It should still be possible to use the real-world calendar before 1582-10-15, because CF might be used for real-world historical data earlier than that date, but it would be necessary to state the calendar explicitly. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)
All, I agree that the current calendar definitions and usage have a lot of problems. Following are some proposals that are geared towards both accuracy and backward compatibility. 1. Please, NO default calendar. Starting with the next CF version, the calendar attribute should be required on all time coordinate variables. 2. Define a new and unique name for the mixed Julian/Gregorian calendar. My favorite would be "julian-gregorian". I could tolerate "mixed julian-gregorian". 3. Recommend that mixed Julian/Gregorian never be used, except (a) in genuine historical data sets that properly need to span the crossover; and (b) to facilitate backward compatibility in existing data sets. 4. Redefine "gregorian" to mean ONLY the proper Gregorian calendar starting on the crossover date 1582 October 15. Add the formal constraint on this calendar, that any usage with dates earlier than the crossover is prohibited. Note that many existing data sets are automatically compatible with this constraint, with no changes needed. 5. Recommend gregorian, NOT proleptic_gregorian, as the CF preferred calendar for dates associated with the modern era. 6. Stop using "standard" as a calendar name. Please be explicit. --Dave On Thu, Dec 6, 2012 at 2:37 PM, John Caron wrote: > Hi Steve, Cecelia: > > I agree we should move to a better default calendar, probably > proleptic_gregorian. We are stuck with mixed Gregorian for previous CF > versions. > > I think we just need a proposal that says "As of version X, the default > calender is proleptic_gregorian. You should explicitly add the calendar > attribute for clarity. Udunits is no longer considered to be the reference > software for date/time." > > It would be nice to add advice to the user about calendar tradeoffs and how > to do historical date matching for modelers, assuming we have useful things > to say. > > I wonder if allowing ISO formatted strings in place of / in addition to the > "time since reference date" form might be useful for historical time > matching? > > Arguably having calendar reference libraries in Python and Java would be > sufficient, possibly Python is preferable even to one in C. > > John > > On 12/6/2012 1:45 PM, Cecelia DeLuca wrote: >> >> >> Hi John and all, >>> >>> >>> Thanks for this information. A few questions: >>> >>> * So you are not supporting "standard Gregorian calendar" even though >>> thats the CF default? >> >> Correct, the climate modelers that we work with don't use it. AFAIK the >> decision of >> CESM was to reject the CF default as unreasonable for modeling and use >> proleptic >> Gregorian. GFDL might support it, I don't know. >> >> We could probably add standard Gregorian as a new calendar if people on >> the >> data side needed it. However, we would be happier to join Steve in >> putting >> a stake in its heart! >>> >>> >>> * Do modelers need to match "historical dates"? If so, what calendar >>> do they use? >> >> I think the calendar used would depend on what was supported by the >> model and >> configuration, as well as the form of the date. If you wanted to >> represent the date >> of something like a volcanic eruption you could probably make it work >> with most >> of the calendars. >> >>> >>> * Is the time library suitable to be released seperate from the ESMF >>> code, eg as standalone C++? >> >> The time library can be used apart from other ESMF interfaces, and some >> modeling groups do use it that way. I don't think we'd be willing to >> extract that >> part and support a separate build. We did that years ago and it was a >> pain >> to maintain. (We could try to isolate the documentation though, so users >> didn't have to wade through the whole reference manual to find the API.) >> >> With ESMP(ython), the scope of the interface is much smaller than >> ESMF proper and I think Ryan could set time functions up nicely. It might >> make sense to represent it as a separate python package in that approach. >> Would python work for you, or would you really prefer C? >> >> - Cecelia >> >>> >>> John >>> >>> On 12/5/2012 3:01 PM, Cecelia DeLuca wrote: Hi John and all, ESMF has a mature time management library with calendars that are commonly used in climate/weather/related modeling. It supports the following: 360 day, no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day, and custom (including custom calendars that may change the length of days). ESMF, like CESM, doesn't support the standard Gregorian calendar because it doesn't make sense for modeling groups who may need to cross the Julian/Gregorian weirdness line (we've never gotten a request to support standard Gregorian from modelers). ESMF has calendar, time, time interval, clock, and alarm constructs, can run time forward or backward, and prints times and time intervals in ISO8601 format. CESM/CAM, NASA, NOAA, Navy and other codes use
[CF-metadata] CF calendars (was: problem with times in PSD dataset)
Hi Steve, Cecelia:
I agree we should move to a better default calendar, probably
proleptic_gregorian. We are stuck with mixed Gregorian for previous CF
versions.
I think we just need a proposal that says "As of version X, the default
calender is proleptic_gregorian. You should explicitly add the calendar
attribute for clarity. Udunits is no longer considered to be the
reference software for date/time."
It would be nice to add advice to the user about calendar tradeoffs and
how to do historical date matching for modelers, assuming we have useful
things to say.
I wonder if allowing ISO formatted strings in place of / in addition to
the "time since reference date" form might be useful for historical time
matching?
Arguably having calendar reference libraries in Python and Java would be
sufficient, possibly Python is preferable even to one in C.
John
On 12/6/2012 1:45 PM, Cecelia DeLuca wrote:
Hi John and all,
Thanks for this information. A few questions:
* So you are not supporting "standard Gregorian calendar" even though
thats the CF default?
Correct, the climate modelers that we work with don't use it. AFAIK the
decision of
CESM was to reject the CF default as unreasonable for modeling and use
proleptic
Gregorian. GFDL might support it, I don't know.
We could probably add standard Gregorian as a new calendar if people on the
data side needed it. However, we would be happier to join Steve in putting
a stake in its heart!
* Do modelers need to match "historical dates"? If so, what calendar
do they use?
I think the calendar used would depend on what was supported by the
model and
configuration, as well as the form of the date. If you wanted to
represent the date
of something like a volcanic eruption you could probably make it work
with most
of the calendars.
* Is the time library suitable to be released seperate from the ESMF
code, eg as standalone C++?
The time library can be used apart from other ESMF interfaces, and some
modeling groups do use it that way. I don't think we'd be willing to
extract that
part and support a separate build. We did that years ago and it was a
pain
to maintain. (We could try to isolate the documentation though, so users
didn't have to wade through the whole reference manual to find the API.)
With ESMP(ython), the scope of the interface is much smaller than
ESMF proper and I think Ryan could set time functions up nicely. It might
make sense to represent it as a separate python package in that approach.
Would python work for you, or would you really prefer C?
- Cecelia
John
On 12/5/2012 3:01 PM, Cecelia DeLuca wrote:
Hi John and all,
ESMF has a mature time management library with calendars that are
commonly
used in climate/weather/related modeling. It supports the following: 360
day,
no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day,
and custom
(including custom calendars that may change the length of days). ESMF,
like CESM,
doesn't support the standard Gregorian calendar because it doesn't make
sense for modeling groups who may need to cross the Julian/Gregorian
weirdness line (we've never gotten a request to support standard
Gregorian from
modelers). ESMF has calendar, time, time interval, clock, and alarm
constructs,
can run time forward or backward, and prints times and time intervals in
ISO8601
format. CESM/CAM, NASA, NOAA, Navy and other codes use it.
The most complete interface to the time lib is Fortran, and there are
some basic
time functions exposed in C (the lib is actually written in C++).
However, we could
wrap the time functions in Python and include them in ESMP, which is
currently a
set of ESMF regridding functions wrapped in Python. We could probably do
that
in the late/winter spring timeframe, with some guidance on what
functions were
desired and a pass through our prioritization board.
Best,
-- Cecelia
On 12/5/2012 12:25 PM, John Caron wrote:
Hi all:
Its probably the right thing to do to make gregorian ("Mixed
Gregorian/Julian calendar") the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.
And I would strongly suggest that data writers stop using "time since
1-1-1". Ive never seen a dataset where "time since 1-1-1" using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.
If you really need "historical accuracy", then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. "time since
reference date" is not good for very large ranges of time.
Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard
