Re: [CF-metadata] udunits time unit question
On 3/31/11 10:18 AM, Seth McGinnis wrote: For those who have never encountered a non-standard calendar in the wild, here's one place they show up: Thanks for the clarification. I draw the following conclusions: 1) converting between Calendars is ripe for error/mis-interpretation, and probably shouldn't be attempted. 2) months and year really should be considered a calendar concept, NOT a unit of time, like seconds and hours, and thus should only be used in the context of calendar-specific calculation and specification. I don't know about days and weeks. The point of storing the data using the 360-day calendar is that it faithfully reflects the structure of the model output. I totally agree -- and further processing should respect that calendar, too. I'm wondering what essential methods a calendar library needs to have to support, eg 360 day calendars? The only one that comes to mind is to calculate the difference between two dates in, say, seconds? What else? I think using a data type that represents a date-time and another one for a time-delta, and system to work with them is really helpful. a time-delta is a span of time, which could be expresses in seconds, hour, etc. However, when you pick a unit (like seconds) you are specifying a resolution and range, so it's kind of nice to have a universal one -- for instance, Python's timedelta stores days, seconds and microseconds internall to get a good range and microsecond resolution -- of course microsecond resolution isn't adequate for all use, so that may not be ideal. Perhaps udinits's time functionality is already good for this. A date-time represents a particular point in time (rather than a span), and must be tied to a given calendar. So: The difference between two date-times is a time-delta. You can add a time-delta to a date-time to get another date-time. etc. With date-times, and good calender support, you could express things like: x calendar months from a given date-time. next thursday from a given date-time and all sorts of nifty things like that, but they are distinctly different operations from: date_time + 45 hours. The mxDateTime package: http://www.egenix.com/products/python/mxBase/mxDateTime/doc/ provides a RelativeDateTime object to handle that kind of calendar concepts/calculsations. That's a pretty nice approach. HTH, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
Dear All, This debate between Chris and Benno hinges upon how one determines the sampling interval for a time series. Benno depends upon semantics encoded in the units of measure field - something I have done in the past although I am now convinced that it is not best practice. Chris advocates obtaining it by analysis of the time channel. Although I see Benno's usage as not best practice, it is established practice and therefore I for one feel that it should continue to be supported. Cheers, Roy. -Original Message- From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Christopher Barker Sent: 30 March 2011 23:27 To: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] udunits time unit question On 3/29/11 12:24 PM, Benno Blumenthal wrote: Well I thought I was helping, but maybe not. you certainly are helping -- we all need to know what folks' use cases are to determine a good solution. -- no one who uses months since date-time is using the udunits interpretation -- all of us who use it are using the calendar 360_day interpretation, which is part of the standard, and beyond udunits. uhm, how can you be sure what all of us are doing? Not to pick on Chris, You aren't picking on me, you're picking on my understanding of the situation -- which is fair game. but I think it is clear over the course of this conversation that many of the participants do not understand the calendar attribute, which is causing a great deal of pain (or at least conversation). yes, i think you are right. I just went to read the CF standard about this again, and it helps, but I'm still confused about how Calendars like 360_day can be used in practice, and I don't see why anyone needs: 5 days since 1960-01-01 calendar 365_days rather than: days since 1960-01-01 calendar 365_days and multiply the time variable by 5. So I think the use of units like 5 days, and use of various calendars are orthogonal. on to my question about calenders. Form the docs: 360_day All years are 360 days divided into 30 day months. OK, so I take it that a day is 24 hours, so a year is 24 * 360 hours long. I can see the utility in this, but then i guess 1960-01-31 is illegal? And if you do years since or even months since, the meaning is going to get very offset after a few year from the usual calendar. I'm also not sure what 1960-01-01 means in a 360_day calendar -- what is year 0? Or do you use the Gregorian calendar for the point in time part? Maybe that doesn't matter, as long as we aren't concerned about absolute dates. This does satisfy Steve's point about having a meaningful axis for doing math -- every month and year is the same length. However, at the end of the day, I don't see the use -- you really can't talk about dates in this calendar anyway, so why not pick an arbitrary point in time and use hours? When if comes down to it, I can certainly see why one might want to do calculations and plotting, etc, in those calendars, but I don't see why your netcdf file has to store the data that way. And there is a lot of data out there with year being the right sense of things -- the fuzziness is real -- tree ring data, coral data, ice cores, all count years more-or-less, yup. many of us are modeling the earth, which means that the earth-bound descriptions of time are what we need to concisely describe what we are doing. right -- and I think that the calendar_year concept makes sense for this -- but you'd better use something like the Gregorian calendar, or your years won't line up with the seasons for long! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- This message (and any attachments) is for the recipient only NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
On 3/31/11 12:42 AM, Lowry, Roy K. wrote: Dear All, This debate between Chris and Benno hinges upon how one determines the sampling interval for a time series. Benno depends upon semantics encoded in the units of measure field - something I have done in the past although I am now convinced that it is not best practice. Chris advocates obtaining it by analysis of the time channel. Yes, I think that does sum it up well. If nothing else, there is a clear need for a little more detail (or a reference) for the calendar definition in the CF docs: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.1/ch04s04.html Perhaps things like 360_day calendar are standard enough in the climate modeling community that they don't nee definition, but these brief descriptions are terrible unclear/imprecise for folks like me. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
For those who have never encountered a non-standard calendar in the wild, here's one place they show up: 360-day and 365-day calendars are used in climate modeling. It requires a lot of extra work to write code that can handle leap years by making the year length variable, and the payoff for doing so is minimal. So in most (maybe all) cases, nobody's ever done it. Even the irregularity of variable-length months is too much work for some model developers. Yes, 1960-01-31 is an illegal date in a 360-day calendar. The year 1960 would generally mean the year when prescribed model conditions correspond to the real-world conditions of 1960. You wouldn't get a cumulative offset in the dates using years since because there's no one-to-one mapping between model days and historical days -- if you want to compare the model output with data that uses a different calendar, the first thing you do is aggregate it to a monthly or longer timescale where there is a one-to-one correspondence. (And usually it doesn't make much sense to compare things day-by-day anyway.) The point of storing the data using the 360-day calendar is that it faithfully reflects the structure of the model output. You could massage the data to fit a Gregorian calendar, but it would be extra work, it would introduce gaps in the time coordinate, and the result might not accurately reflect the descriptions of what went into the model. For example, if some land-cover boundary condition was prescribed on a monthly basis, you'd infer the wrong value for day 60 if the calendar was Gregorian (March 1st) instead of 360-day (Feb. 30th). So it's just better all ways round to define a non-standard calendar that matches how the model works. Does that help clarify what's going on with these strange beasts? Cheers, --Seth 360_day All years are 360 days divided into 30 day months. OK, so I take it that a day is 24 hours, so a year is 24 * 360 hours long. I can see the utility in this, but then i guess 1960-01-31 is illegal? And if you do years since or even months since, the meaning is going to get very offset after a few year from the usual calendar. I'm also not sure what 1960-01-01 means in a 360_day calendar -- what is year 0? Or do you use the Gregorian calendar for the point in time part? Maybe that doesn't matter, as long as we aren't concerned about absolute dates. This does satisfy Steve's point about having a meaningful axis for doing math -- every month and year is the same length. However, at the end of the day, I don't see the use -- you really can't talk about dates in this calendar anyway, so why not pick an arbitrary point in time and use hours? When if comes down to it, I can certainly see why one might want to do calculations and plotting, etc, in those calendars, but I don't see why your netcdf file has to store the data that way. - Seth McGinnis NARCCAP Data Manager Associate Scientist IMAGe / NCAR -- ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
On Tue, Mar 29, 2011 at 5:16 PM, Jon Blower j.d.blo...@reading.ac.ukwrote: I tend to agree with Chris’s points by the way – I am not sure that Benno’s formulations make things clearer. If Chris has misunderstood the use of the calendar attribute then so have I and so have many others – IMHO this points to a lack of clarity in the standard rather than a failure of the community to “get it”. Just my viewpoint. I did not say that there was a failure of the community to get it. What I meant was that the calendar attribute controls the interpretation of time, and many of the e-mails ignored it, making the conversation hard to understand. I'll not participate in this conversation again, Benno *From:* cf-metadata-boun...@cgd.ucar.edu [mailto: cf-metadata-boun...@cgd.ucar.edu] *On Behalf Of *Benno Blumenthal *Sent:* 29 March 2011 20:25 *To:* Christopher Barker *Cc:* cf-metadata@cgd.ucar.edu *Subject:* Re: [CF-metadata] udunits time unit question Well I thought I was helping, but maybe not. On Tue, Mar 29, 2011 at 12:35 PM, Christopher Barker chris.bar...@noaa.gov wrote: There are data sets that use months since date-time in existence. However, anyone using those data sets with the udunits lib is interpreting that data in a way that may not be what the data creator intended -- this is simply broken, It can not be enshrined as a standard. and I agree. I would take it a bit farther -- no one who uses months since date-time is using the udunits interpretation -- all of use who use it are using the calendar 360_day interpretation, which is part of the standard, and beyond udunits. Not to pick on Chris, but I think it is clear over the course of this conversation that many of the participants do not understand the calendar attribute, which is causing a great deal of pain (or at least conversation). My examples indirectly show how I handled the problem before the calendar attribute existed -- I defined two fundamental time units within udunits (seconds *and* years), and thus did not let udunits convert between them (outside code, which understands calendar, does convert between them). This is not a major code change, but a configuration change (udunits.dat in my day). Anyway, calendar is now part of the standard, which makes things clearer (or so we would like to think). Anyway, we have a basic problem: while since seems to let us convert between duration (which has physical units), and an instant in time, there is a precision problem which calendar partly addresses/papers over: sometime we just talk about years (or much much longer), sometimes we ignore the variation in month length (calendar 360_days), sometimes we ignore the leap years (calendar 365_days or calendar 366_days), and we almost always ignore leap_seconds (at least I do, anyway). The standard has to come to grips with this aspect of the conversion, which is always there, physical units being what they are. And there is a lot of data out there with year being the right sense of things -- the fuzziness is real -- tree ring data, coral data, ice cores, all count years more-or-less, though one wonders if a year is catastrophic enough whether it messes up the count. Not to mention the data where year seems a bit small. Yes, we gave up on the passing of the seasons for defining time a while ago, but many of us are modelling the earth, which means that the earth-bound descriptions of time are what we need to concisely describe what we are doing. Benno On 3/26/11 6:07 PM, Benno Blumenthal wrote: 1/365 years since 1960-01-01 calendar 365_days (equivalent to days in 365_day calendar) Is it? what varies here, the length of the year or the length of the day? if the length of the year is well defined (what is it here?), the 1/365 years is well defined, and it very well may not be 24 hours. If if is, then why the heck do you not use days since? 1/360 years since 1960-01-01 calendar 360_days (equivalent to days in 360_day calendar) 1/12 years since 1960-01-01 calendar 360_days (the much maligned months) but which month? one with a defined number of hours, all the same, or one that varies depending on where in the calendar year the data is? As I read these, I'm quite confused -- I can certainly see why one might what to collect or store data at such frequencies, but that's not what the CF standard is about. It is about clearly defining the data when stored, transmitted and shared -- ALL of the above examples are ripe for confusion, and could just as easily be expressed as hours since or days since. months since 1960-01-01 calendar 360_days what is a calendar 360_day? (just as a reminder -- it is important to us, no matter how many people write to say months are meaningless) I don't think any of us think months are meaningless -- simple that they are not a good choice for well-defined time durations. It seems to me
Re: [CF-metadata] udunits time unit question
Benno, the calendar attribute controls the interpretation of time All I'm trying to say is that the standard needs more clarity about how the calendar attribute controls the interpretation of time. It's not clear to me, and apparently it's not clear to others. I don't think it's fair to say that the previous emails have ignored this issue, although we may not share the same understanding that you have. But now I understand your previous emails better and why you have come up with the formulations you have. Sorry if we had crossed wires. Jon From: bennoblument...@gmail.com [mailto:bennoblument...@gmail.com] On Behalf Of Benno Blumenthal Sent: 30 March 2011 08:46 To: Jon Blower Cc: Christopher Barker; cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] udunits time unit question On Tue, Mar 29, 2011 at 5:16 PM, Jon Blower j.d.blo...@reading.ac.ukmailto:j.d.blo...@reading.ac.uk wrote: I tend to agree with Chris's points by the way - I am not sure that Benno's formulations make things clearer. If Chris has misunderstood the use of the calendar attribute then so have I and so have many others - IMHO this points to a lack of clarity in the standard rather than a failure of the community to get it. Just my viewpoint. I did not say that there was a failure of the community to get it. What I meant was that the calendar attribute controls the interpretation of time, and many of the e-mails ignored it, making the conversation hard to understand. I'll not participate in this conversation again, Benno From: cf-metadata-boun...@cgd.ucar.edumailto:cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edumailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Benno Blumenthal Sent: 29 March 2011 20:25 To: Christopher Barker Cc: cf-metadata@cgd.ucar.edumailto:cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] udunits time unit question Well I thought I was helping, but maybe not. On Tue, Mar 29, 2011 at 12:35 PM, Christopher Barker chris.bar...@noaa.govmailto:chris.bar...@noaa.gov wrote: There are data sets that use months since date-time in existence. However, anyone using those data sets with the udunits lib is interpreting that data in a way that may not be what the data creator intended -- this is simply broken, It can not be enshrined as a standard. and I agree. I would take it a bit farther -- no one who uses months since date-time is using the udunits interpretation -- all of use who use it are using the calendar 360_day interpretation, which is part of the standard, and beyond udunits. Not to pick on Chris, but I think it is clear over the course of this conversation that many of the participants do not understand the calendar attribute, which is causing a great deal of pain (or at least conversation). My examples indirectly show how I handled the problem before the calendar attribute existed -- I defined two fundamental time units within udunits (seconds *and* years), and thus did not let udunits convert between them (outside code, which understands calendar, does convert between them). This is not a major code change, but a configuration change (udunits.dat in my day). Anyway, calendar is now part of the standard, which makes things clearer (or so we would like to think). Anyway, we have a basic problem: while since seems to let us convert between duration (which has physical units), and an instant in time, there is a precision problem which calendar partly addresses/papers over: sometime we just talk about years (or much much longer), sometimes we ignore the variation in month length (calendar 360_days), sometimes we ignore the leap years (calendar 365_days or calendar 366_days), and we almost always ignore leap_seconds (at least I do, anyway). The standard has to come to grips with this aspect of the conversion, which is always there, physical units being what they are. And there is a lot of data out there with year being the right sense of things -- the fuzziness is real -- tree ring data, coral data, ice cores, all count years more-or-less, though one wonders if a year is catastrophic enough whether it messes up the count. Not to mention the data where year seems a bit small. Yes, we gave up on the passing of the seasons for defining time a while ago, but many of us are modelling the earth, which means that the earth-bound descriptions of time are what we need to concisely describe what we are doing. Benno On 3/26/11 6:07 PM, Benno Blumenthal wrote: 1/365 years since 1960-01-01 calendar 365_days (equivalent to days in 365_day calendar) Is it? what varies here, the length of the year or the length of the day? if the length of the year is well defined (what is it here?), the 1/365 years is well defined, and it very well may not be 24 hours. If if is, then why the heck do you not use days since? 1/360 years since 1960-01-01 calendar 360_days (equivalent to days in 360_day calendar) 1/12 years since 1960-01
Re: [CF-metadata] udunits time unit question
On 3/29/11 12:24 PM, Benno Blumenthal wrote: Well I thought I was helping, but maybe not. you certainly are helping -- we all need to know what folks' use cases are to determine a good solution. -- no one who uses months since date-time is using the udunits interpretation -- all of us who use it are using the calendar 360_day interpretation, which is part of the standard, and beyond udunits. uhm, how can you be sure what all of us are doing? Not to pick on Chris, You aren't picking on me, you're picking on my understanding of the situation -- which is fair game. but I think it is clear over the course of this conversation that many of the participants do not understand the calendar attribute, which is causing a great deal of pain (or at least conversation). yes, i think you are right. I just went to read the CF standard about this again, and it helps, but I'm still confused about how Calendars like 360_day can be used in practice, and I don't see why anyone needs: 5 days since 1960-01-01 calendar 365_days rather than: days since 1960-01-01 calendar 365_days and multiply the time variable by 5. So I think the use of units like 5 days, and use of various calendars are orthogonal. on to my question about calenders. Form the docs: 360_day All years are 360 days divided into 30 day months. OK, so I take it that a day is 24 hours, so a year is 24 * 360 hours long. I can see the utility in this, but then i guess 1960-01-31 is illegal? And if you do years since or even months since, the meaning is going to get very offset after a few year from the usual calendar. I'm also not sure what 1960-01-01 means in a 360_day calendar -- what is year 0? Or do you use the Gregorian calendar for the point in time part? Maybe that doesn't matter, as long as we aren't concerned about absolute dates. This does satisfy Steve's point about having a meaningful axis for doing math -- every month and year is the same length. However, at the end of the day, I don't see the use -- you really can't talk about dates in this calendar anyway, so why not pick an arbitrary point in time and use hours? When if comes down to it, I can certainly see why one might want to do calculations and plotting, etc, in those calendars, but I don't see why your netcdf file has to store the data that way. And there is a lot of data out there with year being the right sense of things -- the fuzziness is real -- tree ring data, coral data, ice cores, all count years more-or-less, yup. many of us are modeling the earth, which means that the earth-bound descriptions of time are what we need to concisely describe what we are doing. right -- and I think that the calendar_year concept makes sense for this -- but you'd better use something like the Gregorian calendar, or your years won't line up with the seasons for long! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
On 3/26/11 6:07 PM, Benno Blumenthal wrote: 1/365 years since 1960-01-01 calendar 365_days (equivalent to days in 365_day calendar) Is it? what varies here, the length of the year or the length of the day? if the length of the year is well defined (what is it here?), the 1/365 years is well defined, and it very well may not be 24 hours. If if is, then why the heck do you not use days since? 1/360 years since 1960-01-01 calendar 360_days (equivalent to days in 360_day calendar) 1/12 years since 1960-01-01 calendar 360_days (the much maligned months) but which month? one with a defined number of hours, all the same, or one that varies depending on where in the calendar year the data is? As I read these, I'm quite confused -- I can certainly see why one might what to collect or store data at such frequencies, but that's not what the CF standard is about. It is about clearly defining the data when stored, transmitted and shared -- ALL of the above examples are ripe for confusion, and could just as easily be expressed as hours since or days since. months since 1960-01-01 calendar 360_days what is a calendar 360_day? (just as a reminder -- it is important to us, no matter how many people write to say months are meaningless) I don't think any of us think months are meaningless -- simple that they are not a good choice for well-defined time durations. It seems to me that there are two cases: 1) The data is defined on some kind of regular interval (or irregular, but on clearly defined points in the time continuum) - in which case use an appropriate unit: seconds, hours, days. (days is open to debate, I suppose) or 2) the data correspond to calendar months, i.e. monthly averaged data, etc -- a way to specify calendar unit makes sense: calendar months since 1989-01 but using all these peculiar combination of units, so that the time variable can be: (1,2,3,4), rather than say, (0, 5, 10, 15) makes no sense to me. And is it used in any other context? If you have an instrument that is gathering data at 1/2hz, would you do: 2 seconds since date-time then have your time variable be: (0,1,2,2)? I think not, you'd do: seconds since date-time and have your time variable be: (1,2,4,6) However, then you lose the semantics of a 3 month interval. As Benno (sorry for spelling your name wrong last time) showed, the semantics of the qualifier for x since have real meaning in climate datasets. I am looking at how hard that would be to support, but it does add perhaps unneeded complexity. But it adds semantics of the time periods in question. The question is is the added information worth the added complexity?. Also, one of the goal of the CF standard is to make it easier to have client software that can easily work with data from a variety of data sets -- this kind of thing makes it harder, not easier. If you want to give the user some information about the data collection interval, put it in optional meta-data. There is a lot of talk about udunits as the reference library, and while I do see the value of a reference library, I also think that we need to remember that: 1) we can define a standard without a specific reference lib actually existing. 2) not every one is going to use udunits -- which means that if we add all this complexity to the standard, we need to not only add a bunch of code to handle it in udunits, but also everyone else that uses other libraries for units has to deal with it -- please don't make me write that code! I have no idea how anyone else handles time in their client code, but what I do is: - first convert the time access to date-time objects - second -- convert to whatever I need to do with it. I do this so that my client code can be all the same, I don't have to deal with multiple units, reference dates, etc in most of my code. On 3/28/11 9:31 AM, Steve Hankin wrote: 1. the *use of months as a unit of measure*, with the intention that this refer to calendar months of varying length ... not a unit of measure by the normal meaning of that word It's not clear to me that that is the intention, actually -- I htink it means that in some uses, and means 1/12 of a year in others (even though year isn't clearly defined, either!) In fact, since udunits is the official reference lib,and it does define a month has being a specific length, I think current practice is the later use. The question before us is, does the fact that there are some existing CF files that utilize these encodings, require that those encodings be formalized into CF? It is hard to say no in such cases, because doing so creates inconvenience for our colleagues and their users. Sure, but a standrad is defeined as everything in current use, there isin't really much point in having it at all. I think my point above is relevant here: There are data sets that use months since date-time in existence. However, anyone using
Re: [CF-metadata] udunits time unit question
Hi Benno, all, no one who uses months since date-time is using the udunits interpretation I'm not sure how you can say this with certainty. CF deprecates, but does not ban, the use of months since in the udunits sense. Therefore, by a literal reading of CF, I think an interpretation of udunits months since would be allowable. the calendar 360_day interpretation, which is part of the standard, and beyond udunits. The 360_day calendar is part of CF, but nowhere in the standard can I find something that says that if you use the 360_day calendar the definition of the month somehow changes from the udunits definition, even if that is the intention. Is the definition of the day constant between calendars? If so, constant at what? I tend to agree with Chris's points by the way - I am not sure that Benno's formulations make things clearer. If Chris has misunderstood the use of the calendar attribute then so have I and so have many others - IMHO this points to a lack of clarity in the standard rather than a failure of the community to get it. Just my viewpoint. Jon From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Benno Blumenthal Sent: 29 March 2011 20:25 To: Christopher Barker Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] udunits time unit question Well I thought I was helping, but maybe not. On Tue, Mar 29, 2011 at 12:35 PM, Christopher Barker chris.bar...@noaa.govmailto:chris.bar...@noaa.gov wrote: There are data sets that use months since date-time in existence. However, anyone using those data sets with the udunits lib is interpreting that data in a way that may not be what the data creator intended -- this is simply broken, It can not be enshrined as a standard. and I agree. I would take it a bit farther -- no one who uses months since date-time is using the udunits interpretation -- all of use who use it are using the calendar 360_day interpretation, which is part of the standard, and beyond udunits. Not to pick on Chris, but I think it is clear over the course of this conversation that many of the participants do not understand the calendar attribute, which is causing a great deal of pain (or at least conversation). My examples indirectly show how I handled the problem before the calendar attribute existed -- I defined two fundamental time units within udunits (seconds *and* years), and thus did not let udunits convert between them (outside code, which understands calendar, does convert between them). This is not a major code change, but a configuration change (udunits.dat in my day). Anyway, calendar is now part of the standard, which makes things clearer (or so we would like to think). Anyway, we have a basic problem: while since seems to let us convert between duration (which has physical units), and an instant in time, there is a precision problem which calendar partly addresses/papers over: sometime we just talk about years (or much much longer), sometimes we ignore the variation in month length (calendar 360_days), sometimes we ignore the leap years (calendar 365_days or calendar 366_days), and we almost always ignore leap_seconds (at least I do, anyway). The standard has to come to grips with this aspect of the conversion, which is always there, physical units being what they are. And there is a lot of data out there with year being the right sense of things -- the fuzziness is real -- tree ring data, coral data, ice cores, all count years more-or-less, though one wonders if a year is catastrophic enough whether it messes up the count. Not to mention the data where year seems a bit small. Yes, we gave up on the passing of the seasons for defining time a while ago, but many of us are modelling the earth, which means that the earth-bound descriptions of time are what we need to concisely describe what we are doing. Benno On 3/26/11 6:07 PM, Benno Blumenthal wrote: 1/365 years since 1960-01-01 calendar 365_days (equivalent to days in 365_day calendar) Is it? what varies here, the length of the year or the length of the day? if the length of the year is well defined (what is it here?), the 1/365 years is well defined, and it very well may not be 24 hours. If if is, then why the heck do you not use days since? 1/360 years since 1960-01-01 calendar 360_days (equivalent to days in 360_day calendar) 1/12 years since 1960-01-01 calendar 360_days (the much maligned months) but which month? one with a defined number of hours, all the same, or one that varies depending on where in the calendar year the data is? As I read these, I'm quite confused -- I can certainly see why one might what to collect or store data at such frequencies, but that's not what the CF standard is about. It is about clearly defining the data when stored, transmitted and shared -- ALL of the above examples are ripe for confusion, and could just as easily be expressed as hours since or days since
Re: [CF-metadata] udunits time unit question
Hi all, I've stayed out of this for a long while, but this is bringing back fond memories of the early days of ESMF:-). Ah, that ESMF time manager... it was a thing of beauty. I think the issue is that time instants and time intervals are different quantities. Time instants can have a reference time instant (hence, the since), a calendar, and calendar-dependent units, such as month. Time intervals can only have SI units (seconds, minutes, hours). Days and years introduce difficulties once you take on the requirement of accommodating other planets in your software. Months of course are a problem. (The expectation that you can add and subtract the same interval from an instant and get back to the original is violated with months). Don't all our difficulties arise out of trying to describe intervals and instants using the single thing called time? Steve Hankin writes: On 3/26/2011 9:11 AM, Karl Taylor wrote: Dear all, I think 3 days since 1970-01-01 is a sensible statement, with 3 being the number and days since 1970-01-01 being the units. Would anybody normally interpret 5 3 days since 1970-01-01 as meaning 15 days since 1970-01-01? If not, then I don't think we should allow a unit of 3 days since 1970-01-01. This would be just as confusing as allowing a unit of 3 meters and interpreting 5 3 meters as 15 meters. Since we don't allow 3 meters as a unit, why should we allow 3 days since 1970-01-01 as a unit? Hi All, Karl has challenged us to think about a broader (and difficult) question. CF has not been as thorough as it needed to be in standardizing time units. It merely appealed to udunits, despite known limitations. This has resulted over time in various encodings that were not properly considered in the formulation of CF * the use of *multiple calendars* -- after discussions this resulted in enhancements to CF document ... and thereby implicit conflicts with udunits, since it does not support the calendars and 1. the *use of months as a unit of measure*, with the intention that this refer to calendar months of varying length ... not a unit of measure by the normal meaning of that word 2. *pre-pending a number to some units* in order to create custom units -- e.g. 3 days since 1970-01-01 The question before us is, does the fact that there are some existing CF files that utilize these encodings, require that those encodings be formalized into CF? It is hard to say no in such cases, because doing so creates inconvenience for our colleagues and their users. Next time around we could find ourselves on the wrong side of the no answer and be inconvenienced ourselves. On the other hand, we need to recognize this impulse as a problem, itself. It is one of the design by committee impulses that leads standards to grow bloated and inconsistent over time. Quote Henning http://queue.acm.org/detail.cfm?id=1142044, 'To create quality software [or standards], the ability to say no is usually far more important than the ability to say yes.' Karl's example illustrates the community problem. An encoding of times with units of 3 days since 1970-01-01 may be convenient and readable to the project that performed a 3-day measurement cycle and created the file, but what about the generic application reading the file? How should software label the time axis of a plot? Should it have tic marks spaced at 3 day intervals documented as intervals of 3 days since 1970-01-01? How peculiar! If the software takes the more conventional approach of labeling the time in days or in calendar formatting, seeing the plot marks at 3 day intervals will convey the contents of the data very effectively. In which case, what has CF gained by allowing units of 3 days since 1970-01-01? And what has it lost? We should look at each of these questions from the point of view of the complexity of the software *reading* the file. From our colleague Bryan Lawrence, In general my rule of thumb is organise the data for the consumers, not the providers! - Steve == Best regards, Karl On 3/26/11 6:46 AM, Don Murray wrote: John- On 3/25/11 4:54 PM, John Caron wrote: On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) I agree with Beno that it should be legal. GrADS gives their units in terms of N (minutes, hours, days, months, years) from a reference time. When I wrote the GrADS IOSP, I originally was using this syntax, because then
Re: [CF-metadata] udunits time unit question
John- On 3/26/11 12:51 PM, John Caron wrote: It seems like the essential thing you need is calendar field manipulation, to get around the variable month problem. yes. If you didnt have the 3 in 3 months since 2011-04-01, you could just multiply by 3 in your coordinate values. However, then you lose the semantics of a 3 month interval. As Benno (sorry for spelling your name wrong last time) showed, the semantics of the qualifier for x since have real meaning in climate datasets. I am looking at how hard that would be to support, but it does add perhaps unneeded complexity. But it adds semantics of the time periods in question. Don On 3/26/2011 7:46 AM, Don Murray wrote: John- On 3/25/11 4:54 PM, John Caron wrote: On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) I agree with Beno that it should be legal. GrADS gives their units in terms of N (minutes, hours, days, months, years) from a reference time. When I wrote the GrADS IOSP, I originally was using this syntax, because then your time coordinate values are 0,1,2,. However, 3mo intervals came up with the problems that you have shown here, so I converted everything to hours since the base date. But, if we had a library that would compute 3mo since 2011-04-01 as 2011-07-01, I would revert to that syntax because it is closer to the original GrADS definition. Don ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- Don Murray NOAA/ESRL/PSD and CIRES 303-497-3596 http://www.esrl.noaa.gov/psd/people/don.murray/ ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
John- On 3/25/11 4:54 PM, John Caron wrote: On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) I agree with Beno that it should be legal. GrADS gives their units in terms of N (minutes, hours, days, months, years) from a reference time. When I wrote the GrADS IOSP, I originally was using this syntax, because then your time coordinate values are 0,1,2,. However, 3mo intervals came up with the problems that you have shown here, so I converted everything to hours since the base date. But, if we had a library that would compute 3mo since 2011-04-01 as 2011-07-01, I would revert to that syntax because it is closer to the original GrADS definition. Don -- Don Murray NOAA/ESRL/PSD and CIRES 303-497-3596 http://www.esrl.noaa.gov/psd/people/don.murray/ ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
Dear all, I think 3 days since 1970-01-01 is a sensible statement, with 3 being the number and days since 1970-01-01 being the units. Would anybody normally interpret 5 3 days since 1970-01-01 as meaning 15 days since 1970-01-01? If not, then I don't think we should allow a unit of 3 days since 1970-01-01. This would be just as confusing as allowing a unit of 3 meters and interpreting 5 3 meters as 15 meters. Since we don't allow 3 meters as a unit, why should we allow 3 days since 1970-01-01 as a unit? Best regards, Karl On 3/26/11 6:46 AM, Don Murray wrote: John- On 3/25/11 4:54 PM, John Caron wrote: On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) I agree with Beno that it should be legal. GrADS gives their units in terms of N (minutes, hours, days, months, years) from a reference time. When I wrote the GrADS IOSP, I originally was using this syntax, because then your time coordinate values are 0,1,2,. However, 3mo intervals came up with the problems that you have shown here, so I converted everything to hours since the base date. But, if we had a library that would compute 3mo since 2011-04-01 as 2011-07-01, I would revert to that syntax because it is closer to the original GrADS definition. Don ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
Hi Don: It seems like the essential thing you need is calendar field manipulation, to get around the variable month problem. If you didnt have the 3 in 3 months since 2011-04-01, you could just multiply by 3 in your coordinate values. I am looking at how hard that would be to support, but it does add perhaps unneeded complexity. John On 3/26/2011 7:46 AM, Don Murray wrote: John- On 3/25/11 4:54 PM, John Caron wrote: On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) I agree with Beno that it should be legal. GrADS gives their units in terms of N (minutes, hours, days, months, years) from a reference time. When I wrote the GrADS IOSP, I originally was using this syntax, because then your time coordinate values are 0,1,2,. However, 3mo intervals came up with the problems that you have shown here, so I converted everything to hours since the base date. But, if we had a library that would compute 3mo since 2011-04-01 as 2011-07-01, I would revert to that syntax because it is closer to the original GrADS definition. Don ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] udunits time unit question
On 3/22/2011 6:53 AM, John Caron wrote: Consider: int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = days since 1970-01-01; vs int time(sample=1001); :long_name = Measurement time; :standard_name = time; :units = 3 days since 1970-01-01; values = 1, 2, 3, ... are these equivalent or does the second one mean every 3 days ? Is the second one illegal ? Im am going to assume that the second form is illegal, that is, you may not have a number in front of the unit in a time coordinate unit (CF section 4.4) ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
