Re: [CF-metadata] CDM calendar date handling

[Don Murray]

John and Jonathan-

On 8/24/11 9:23 AM, Jonathan Gregory wrote:


So now we are taking about the "N calendar UNITS" part, when that part means
a large number of years.  I think this format:

"50 calendar Myears since 1980-01-01"

is the best one. The word "calendar" is useful to make clear it is not udunits
years. Also, it's good to have a reference explicitly. For example, "before
present" sometimes means before 1950, and it's nice to have that spelled out.


I think this form is the best representation as well.

Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jonathan Gregory]

Dear John

> since im going to propose some grammar that we will be stuck with
> for the next 50m years ...
:-)

We've talking about the syntax which describes the reference date now, is
that right? That is, the complete syntax is
N [calendar] UNITS since DATE
and we have so far said that it's not a good a idea for N to be -50 million
+- small increments, so instead we want to put the large offset into DATE.
Usually DATE is a string as described in your section "W3C profile of ISO8601".

You suggest
> "01-01-01 12:00 epoch 50m BCE"
> "01-01-01 12:00 reference 50m BCE"
but for this purpose I think we could assume we mean 0Z on 1 Jan. The aim is
to have an offset of a large number of years. Offsets within the year can be
dealt with as usual by the N [calendar] UNITS. So I suggest that DATE should
be allowed to be "N BCE" or "Nk BCE" or "NM BCE" (big M rather than little m
for mega) where N is an integer interpreted as calendar years e.g.
"n calendar years since 65500k BCE" is for counting time in months since the
end of the Cretaceous. :-) To be more helpful you could accept floating-point
numbers if k or M was specified, so we could have "65.5M BCE" in this case.

> >The other kind of example of palaeoclimate is the
> >one where large periods of time are spanned by the time axis

So now we are taking about the "N calendar UNITS" part, when that part means
a large number of years.  I think this format:
> "50 calendar Myears since 1980-01-01"
is the best one. The word "calendar" is useful to make clear it is not udunits
years. Also, it's good to have a reference explicitly. For example, "before
present" sometimes means before 1950, and it's nice to have that spelled out.

> >it would be convenient to have time-bounds for a cell of e.g. "1 calendar 
> >month
> >since reference" and "2 calendar months since reference", with the time-coord
> >of "1.5 calendar months since reference".
> 
> it seems like a more informative coordinate value would be, eg "Feb
> 2001" when the bounds are [1,2] calendar months since 2001-01-01 ?

That is true, that is what it means in this case. (The old GDT convention had
a syntax for specifying a subset of the components of time in this way.) But
it feels to me like we'd be doing something rather more complicated now if we
start to allow months to be named absolutely. Also, even though it is
arbitrary what you choose, sometimes it is used as an exact number, for
example if you differentiate in time. You could translate 1.5 months since
2001-01-01 into an exact datetime. It means half-way between 2001-02-01 and
2001-03-01.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jim Biard]

I've got a question and a thought to stir the pot with.

Is there any problem with having negative values for the dates, or are 
we talking about having only positive and increasing values?


It seems that providing a way to specify the base unit of time would be 
helpful from a conceptual standpoint, even if it is a pain (and 
understandably avoided when possible) to handle in code.  So if you want 
your unit of time to be 1 kyear or Myear or Gyear, you could work in 
that basis (with a corresponding loss of precision).


On 8/24/2011 9:16 AM, John Caron wrote:

On 8/24/2011 6:23 AM, Jonathan Gregory wrote:

Dear John


It seems to me it would be better to somehow denote the "epoch"
seperately, because its kind of silly keeping track of # millisecs
between two dates separated by 50 million years. plus its hard.

what about:

"01-01-01 12:00 epoch 50m BCE"

where the "epoch 50m BCE" is probably just carried along in the
string representation of the date.
I agree that that's a sensible way to do it for dates which are 
separated
by relatively short periods in an epoch which is a long time ago. 
However
it could be useful if the epoch wasn't just a string. If it was a 
longinteger
number of calendar years (relative to the normal year 1) you would be 
able
to interconvert dates expressed relative to different epochs. That 
is, you
could tell that 10 calendar Myears since 01-01-01 epoch 20M BCE means 
the
same as 0 calendar years since 01-01-01 epoch 10M BCE. This 
conversion would
be accurate provided the time unit in calendar years is stored as an 
integer.
Accuracy is lost if it is turned into floating-point milliseconds, I 
suppose.


I agree we should be able to calculate intervals between epochs. 
calendar years would also be my first choice as the unit of this.


since im going to propose some grammar that we will be stuck with for 
the next 50m years, id like your (or anyone's) thoughts on the wording.


currently i have thought of

"01-01-01 12:00 epoch 50m BCE"
"01-01-01 12:00 reference 50m BCE"


but im open to anything



The other kind of example of palaeoclimate is the
one where large periods of time are spanned by the time axis e.g. if you
had a timeseries whose time axis spanned all the geological periods 
since
600 Myear ago. (That would probably not be from a GCM! - but it could 
be from
geological data or other Earth system models.) That sort of axis 
would like

to use units of calendar Myear and they need to be stored accurately.


some possibilities that occur to me (not yet sure whats feasible in 
parser):


"50 calendar Myears since 1980-01-01"

"50 Myears since 1980-01-01" (Myears == million calendar years)

"50 Myears" (assume reference is 0 CE)



About the multiples of calendar units: You have said they should be 
integers.
However, perhaps fractions could be rather useful. For example, we 
have the
recurrent question of where the time coordinate should be positioned 
in the
interval if the data actually applies to the interval rather than the 
instant.
It's a bit arbitrary and the usual advice is to put it in the middle. 
For this
it would be convenient to have time-bounds for a cell of e.g. "1 
calendar month
since reference" and "2 calendar months since reference", with the 
time-coord

of "1.5 calendar months since reference".


it seems like a more informative coordinate value would be, eg "Feb 
2001" when the bounds are [1,2] calendar months since 2001-01-01 ?


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


--
Jim Biard

Government Contractor, STG Inc.
Remote Sensing and Applications Division (RSAD)
National Climatic Data Center
151 Patton Ave.
Asheville, NC 28801-5001

jim.bi...@noaa.gov
828-271-4900

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/24/2011 6:23 AM, Jonathan Gregory wrote:

Dear John


It seems to me it would be better to somehow denote the "epoch"
seperately, because its kind of silly keeping track of # millisecs
between two dates separated by 50 million years. plus its hard.

what about:

"01-01-01 12:00 epoch 50m BCE"

where the "epoch 50m BCE" is probably just carried along in the
string representation of the date.

I agree that that's a sensible way to do it for dates which are separated
by relatively short periods in an epoch which is a long time ago. However
it could be useful if the epoch wasn't just a string. If it was a longinteger
number of calendar years (relative to the normal year 1) you would be able
to interconvert dates expressed relative to different epochs. That is, you
could tell that 10 calendar Myears since 01-01-01 epoch 20M BCE means the
same as 0 calendar years since 01-01-01 epoch 10M BCE. This conversion would
be accurate provided the time unit in calendar years is stored as an integer.
Accuracy is lost if it is turned into floating-point milliseconds, I suppose.


I agree we should be able to calculate intervals between epochs. 
calendar years would also be my first choice as the unit of this.


since im going to propose some grammar that we will be stuck with for 
the next 50m years, id like your (or anyone's) thoughts on the wording.


currently i have thought of

"01-01-01 12:00 epoch 50m BCE"
"01-01-01 12:00 reference 50m BCE"


but im open to anything



The other kind of example of palaeoclimate is the
one where large periods of time are spanned by the time axis e.g. if you
had a timeseries whose time axis spanned all the geological periods since
600 Myear ago. (That would probably not be from a GCM! - but it could be from
geological data or other Earth system models.) That sort of axis would like
to use units of calendar Myear and they need to be stored accurately.


some possibilities that occur to me (not yet sure whats feasible in parser):

"50 calendar Myears since 1980-01-01"

"50 Myears since 1980-01-01" (Myears == million calendar years)

"50 Myears" (assume reference is 0 CE)



About the multiples of calendar units: You have said they should be integers.
However, perhaps fractions could be rather useful. For example, we have the
recurrent question of where the time coordinate should be positioned in the
interval if the data actually applies to the interval rather than the instant.
It's a bit arbitrary and the usual advice is to put it in the middle. For this
it would be convenient to have time-bounds for a cell of e.g. "1 calendar month
since reference" and "2 calendar months since reference", with the time-coord
of "1.5 calendar months since reference".


it seems like a more informative coordinate value would be, eg "Feb 
2001" when the bounds are [1,2] calendar months since 2001-01-01 ?


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] CDM calendar date handling

[Jonathan Gregory]

Dear John

> It seems to me it would be better to somehow denote the "epoch"
> seperately, because its kind of silly keeping track of # millisecs
> between two dates separated by 50 million years. plus its hard.
> 
> what about:
> 
> "01-01-01 12:00 epoch 50m BCE"
> 
> where the "epoch 50m BCE" is probably just carried along in the
> string representation of the date.

I agree that that's a sensible way to do it for dates which are separated
by relatively short periods in an epoch which is a long time ago. However
it could be useful if the epoch wasn't just a string. If it was a longinteger
number of calendar years (relative to the normal year 1) you would be able
to interconvert dates expressed relative to different epochs. That is, you
could tell that 10 calendar Myears since 01-01-01 epoch 20M BCE means the 
same as 0 calendar years since 01-01-01 epoch 10M BCE. This conversion would
be accurate provided the time unit in calendar years is stored as an integer.
Accuracy is lost if it is turned into floating-point milliseconds, I suppose.

The other kind of example of palaeoclimate is the
one where large periods of time are spanned by the time axis e.g. if you
had a timeseries whose time axis spanned all the geological periods since
600 Myear ago. (That would probably not be from a GCM! - but it could be from
geological data or other Earth system models.) That sort of axis would like
to use units of calendar Myear and they need to be stored accurately.

About the multiples of calendar units: You have said they should be integers.
However, perhaps fractions could be rather useful. For example, we have the
recurrent question of where the time coordinate should be positioned in the
interval if the data actually applies to the interval rather than the instant.
It's a bit arbitrary and the usual advice is to put it in the middle. For this
it would be convenient to have time-bounds for a cell of e.g. "1 calendar month
since reference" and "2 calendar months since reference", with the time-coord
of "1.5 calendar months since reference".

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/23/2011 6:13 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 22, 2011, at 6:36 PM, John Caron wrote:


On 8/22/2011 6:37 AM, Jonathan Gregory wrote:

Dear Chris


Perhaps there could be an attribute we could set that says whether we have 
accounted for leap seconds?  With the absence of such an attribute to be 
presumed as meaning leap seconds have been ignored.

Perhaps the real-world calendars with and without leap seconds should be
regarded as two different calendars, since they have different encodings
(meaning decoding/encoding as YMD HMS<->   time-interval since reference-time).
The "true" real-world calendar is the one with leap seconds.

CF has a calendar
proleptic_gregorian

 A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is 
divisible by 400.

What if we clarified this calendar as not having leap seconds? Then it could
be used for real-world applications for recent dates meaning that it was just
like the real world except that it doesn't have leap seconds.

Model calendars, which are already idealised wrt length of year, don't have
leap seconds anyway, I am sure.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

I agree that a separate calendar is needed if we want to have leap
seconds. I think the common form is UTC (or TAI?). Chris, what does the
satellite community use?

Both UTC and TAI, actually.


Then i would propose adding those two calendars, if they are really 
used. OTOH, i dont have an implementation of them, although there are 
plans to add to java 8.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Hi Don:

Both java date and joda-time use a long to represent millisecs since ref 
date.


so min/max year is:

max = 292,278,994
min = -292,275,055

heres what im getting with your example in CDM 4.3 using joda-time:

5000 years since 1970-01-01 00:00:00Z == 50001928-10-07T01:30:00.064Z

5000 calendar years since 1970-01-01 00:00:00Z == 
50001970-01-01T00:00:00.000Z


calendar periods are integers, so you are limited to  +/- 2 Gyears.

It seems to me it would be better to somehow denote the "epoch" 
seperately, because its kind of silly keeping track of # millisecs 
between two dates separated by 50 million years. plus its hard.


what about:

"01-01-01 12:00 epoch 50m BCE"

where the "epoch 50m BCE" is probably just carried along in the string 
representation of the date.


(Im not sure why Myears is even accepted in Stu's example, since its a 
time unit, not a date unit).


john

On 8/22/2011 9:16 AM, Don Murray wrote:

John-

An example of how this could be handled (provided by Stuart Wier of 
UNAVCO) is available here:


http://geon.unavco.org/unavco/geodynamics/Lithgow-Bertelloni_Richards_Mesozoic_Cenozoic_Plate_Velocities.cdl 



described on the page:

http://geon.unavco.org/unavco/IDV_datasource_plates.html#c

Here, the time coordinate is listed as:

float time(time) ;
time:units = "Myear" ;
time:standard_name = "time" ;

with values of:

time = -170.0, -96.0, -94.0, -84.0, -74.0, -64.0, -56.0, -48.0, -43.0, 
-25.0, -10.0 ;


The problem is that udunits ends up computing times for -64 Myear as:

63998634-12-14 00:00:00 BCE

so you lose precision on the year.

Don


On 8/19/11 10:45 AM, John Caron wrote:



Regarding paleoclimate, a point I forgot is that some modellers may
wish to
have years which are very large negative numbers (many more than four
digits)
if they set up the model with the "true" date for the run. Although 
for

geological timescales you might say that this isn't necessary and you
might
as well choose an arbitrary year, there is a good reason for it in
Pleistocene
when you might be using the dates to relate to orbital forcing or
atmospheric
composition.


so the idea is that you are simulating some year, so you really need
time down to the hour or second. but the climate is from 5 million
years ago, so you need the year field to be able to handle that?


Im just thinking that fitting this into the ISO date format
"500-01-01 12:00" seems awkward, esp as it indicates unwarranted
precision.

seems something like "01-01 12:00 reference 50m BCE" would be better.
What do paleo modellers actually use, eg in the figures that they 
publish?

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

Yes, TAI is an epochal time, number of seconds since a reference time. In 
EOSDIS, folks often use TAI93, number of seconds since 01 January 1993 00:00:00.

We have tools that translate between TAI93 and UTC, which rely on an external 
file (leapsec.dat) to compute the time conversions. 

On Aug 23, 2011, at 8:47 AM, Jim Biard wrote:

> Hi.
> 
> According to the almighty Wikipedia ;), UTC is "a time standard based on 
> International Atomic Time (TAI) with leap seconds added at irregular 
> intervals to synchronize with the Earth's rotation."  So TAI doesn't attempt 
> to stay synchronized with the Earth's rotation.
> 
> Another quote from the Wikipedia article on UTC states
> UTC is a discontinuous timescale, so it is not possible to compute the exact 
> time interval elapsed between two UTC timestamps without consulting a table 
> that describes how many leap seconds occurred during that interval. 
> Therefore, many scientific applications that require precise measurement of 
> long (multi-year) intervals use TAI instead.
> I'm not advocating for anything, just contributing some factoids.
> 
> Grace and peace,
> 
> Jim Biard
> 
> On 8/23/2011 8:13 AM, Lynnes, Christopher S. (GSFC-6102) wrote:
>> On Aug 22, 2011, at 6:36 PM, John Caron wrote:
>> 
>> 
>>> On 8/22/2011 6:37 AM, Jonathan Gregory wrote:
>>> 
 Dear Chris
 
 
> Perhaps there could be an attribute we could set that says whether we 
> have accounted for leap seconds?  With the absence of such an attribute 
> to be presumed as meaning leap seconds have been ignored.
> 
 Perhaps the real-world calendars with and without leap seconds should be
 regarded as two different calendars, since they have different encodings
 (meaning decoding/encoding as YMD HMS<->  time-interval since 
 reference-time).
 The "true" real-world calendar is the one with leap seconds.
 
 CF has a calendar
 proleptic_gregorian
 
 A Gregorian calendar extended to dates before 1582-10-15. That is, a 
 year is a leap year if either (i) it is divisible by 4 but not by 100 or 
 (ii) it is divisible by 400.
 
 What if we clarified this calendar as not having leap seconds? Then it 
 could
 be used for real-world applications for recent dates meaning that it was 
 just
 like the real world except that it doesn't have leap seconds.
 
 Model calendars, which are already idealised wrt length of year, don't have
 leap seconds anyway, I am sure.
 
 Best wishes
 
 Jonathan
 ___
 CF-metadata mailing list
 
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>> I agree that a separate calendar is needed if we want to have leap 
>>> seconds. I think the common form is UTC (or TAI?). Chris, what does the 
>>> satellite community use?
>>> 
>> Both UTC and TAI, actually.
>> 
>> 
>>> ___
>>> CF-metadata mailing list
>>> 
>>> CF-metadata@cgd.ucar.edu
>>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>> Christopher Lynnes 
>> Goddard Earth Sciences Data and Information Center, NASA/GSFC
>> 301-614-5185
>> 
>> ___
>> CF-metadata mailing list
>> 
>> CF-metadata@cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> 
> -- 
> Jim Biard
> 
> Government Contractor, STG Inc.
> Remote Sensing and Applications Division (RSAD)
> National Climatic Data Center
> 151 Patton Ave.
> Asheville, NC 28801-5001
> 
> 
> jim.bi...@noaa.gov
> 
> 828-271-4900
> 
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone: 301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jim Biard]

Hi.

According to the almighty Wikipedia ;), UTC is "a time standard based on 
International Atomic Time 
 (TAI) with leap 
seconds  added at irregular 
intervals to synchronize with the Earth's rotation 
."  So TAI doesn't 
attempt to stay synchronized with the Earth's rotation.


Another quote from the Wikipedia article on UTC states

   UTC is a discontinuous timescale, so it is not possible to compute
   the exact time interval  elapsed
   between two UTC timestamps without consulting a table that describes
   how many leap seconds occurred during that interval. Therefore, many
   scientific applications that require precise measurement of long
   (multi-year) intervals use TAI
    instead.

I'm not advocating for anything, just contributing some factoids.

Grace and peace,

Jim Biard

On 8/23/2011 8:13 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 22, 2011, at 6:36 PM, John Caron wrote:


On 8/22/2011 6:37 AM, Jonathan Gregory wrote:

Dear Chris


Perhaps there could be an attribute we could set that says whether we have 
accounted for leap seconds?  With the absence of such an attribute to be 
presumed as meaning leap seconds have been ignored.

Perhaps the real-world calendars with and without leap seconds should be
regarded as two different calendars, since they have different encodings
(meaning decoding/encoding as YMD HMS<->   time-interval since reference-time).
The "true" real-world calendar is the one with leap seconds.

CF has a calendar
proleptic_gregorian

 A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is 
divisible by 400.

What if we clarified this calendar as not having leap seconds? Then it could
be used for real-world applications for recent dates meaning that it was just
like the real world except that it doesn't have leap seconds.

Model calendars, which are already idealised wrt length of year, don't have
leap seconds anyway, I am sure.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

I agree that a separate calendar is needed if we want to have leap
seconds. I think the common form is UTC (or TAI?). Chris, what does the
satellite community use?

Both UTC and TAI, actually.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Christopher Lynnes
Goddard Earth Sciences Data and Information Center, NASA/GSFC
301-614-5185

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


--
Jim Biard

Government Contractor, STG Inc.
Remote Sensing and Applications Division (RSAD)
National Climatic Data Center
151 Patton Ave.
Asheville, NC 28801-5001

jim.bi...@noaa.gov
828-271-4900

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

On Aug 22, 2011, at 6:36 PM, John Caron wrote:

> On 8/22/2011 6:37 AM, Jonathan Gregory wrote:
>> Dear Chris
>> 
>>> Perhaps there could be an attribute we could set that says whether we have 
>>> accounted for leap seconds?  With the absence of such an attribute to be 
>>> presumed as meaning leap seconds have been ignored.
>> Perhaps the real-world calendars with and without leap seconds should be
>> regarded as two different calendars, since they have different encodings
>> (meaning decoding/encoding as YMD HMS<->  time-interval since 
>> reference-time).
>> The "true" real-world calendar is the one with leap seconds.
>> 
>> CF has a calendar
>> proleptic_gregorian
>> 
>> A Gregorian calendar extended to dates before 1582-10-15. That is, a 
>> year is a leap year if either (i) it is divisible by 4 but not by 100 or 
>> (ii) it is divisible by 400.
>> 
>> What if we clarified this calendar as not having leap seconds? Then it could
>> be used for real-world applications for recent dates meaning that it was just
>> like the real world except that it doesn't have leap seconds.
>> 
>> Model calendars, which are already idealised wrt length of year, don't have
>> leap seconds anyway, I am sure.
>> 
>> Best wishes
>> 
>> Jonathan
>> ___
>> CF-metadata mailing list
>> CF-metadata@cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> 
> I agree that a separate calendar is needed if we want to have leap 
> seconds. I think the common form is UTC (or TAI?). Chris, what does the 
> satellite community use?

Both UTC and TAI, actually.

> 
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Christopher Lynnes 
Goddard Earth Sciences Data and Information Center, NASA/GSFC
301-614-5185

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/22/2011 6:37 AM, Jonathan Gregory wrote:

Dear Chris


Perhaps there could be an attribute we could set that says whether we have 
accounted for leap seconds?  With the absence of such an attribute to be 
presumed as meaning leap seconds have been ignored.

Perhaps the real-world calendars with and without leap seconds should be
regarded as two different calendars, since they have different encodings
(meaning decoding/encoding as YMD HMS<->  time-interval since reference-time).
The "true" real-world calendar is the one with leap seconds.

CF has a calendar
proleptic_gregorian

 A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is 
divisible by 400.

What if we clarified this calendar as not having leap seconds? Then it could
be used for real-world applications for recent dates meaning that it was just
like the real world except that it doesn't have leap seconds.

Model calendars, which are already idealised wrt length of year, don't have
leap seconds anyway, I am sure.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


I agree that a separate calendar is needed if we want to have leap 
seconds. I think the common form is UTC (or TAI?). Chris, what does the 
satellite community use?


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

OK, that looks like it might have an equivalent effect.

On Aug 22, 2011, at 8:37 AM, Jonathan Gregory wrote:

> Dear Chris
> 
>> Perhaps there could be an attribute we could set that says whether we have 
>> accounted for leap seconds?  With the absence of such an attribute to be 
>> presumed as meaning leap seconds have been ignored.
> 
> Perhaps the real-world calendars with and without leap seconds should be
> regarded as two different calendars, since they have different encodings
> (meaning decoding/encoding as YMD HMS <-> time-interval since reference-time).
> The "true" real-world calendar is the one with leap seconds.
> 
> CF has a calendar
> proleptic_gregorian
> 
>A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
> is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it 
> is divisible by 400.
> 
> What if we clarified this calendar as not having leap seconds? Then it could
> be used for real-world applications for recent dates meaning that it was just
> like the real world except that it doesn't have leap seconds.
> 
> Model calendars, which are already idealised wrt length of year, don't have
> leap seconds anyway, I am sure.
> 
> Best wishes
> 
> Jonathan
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone: 301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Don Murray]

John-

An example of how this could be handled (provided by Stuart Wier of 
UNAVCO) is available here:


http://geon.unavco.org/unavco/geodynamics/Lithgow-Bertelloni_Richards_Mesozoic_Cenozoic_Plate_Velocities.cdl

described on the page:

http://geon.unavco.org/unavco/IDV_datasource_plates.html#c

Here, the time coordinate is listed as:

float time(time) ;
time:units = "Myear" ;
time:standard_name = "time" ;

with values of:

time = -170.0, -96.0, -94.0, -84.0, -74.0, -64.0, -56.0, -48.0, -43.0, 
-25.0, -10.0 ;


The problem is that udunits ends up computing times for -64 Myear as:

63998634-12-14 00:00:00 BCE

so you lose precision on the year.

Don


On 8/19/11 10:45 AM, John Caron wrote:



Regarding paleoclimate, a point I forgot is that some modellers may
wish to
have years which are very large negative numbers (many more than four
digits)
if they set up the model with the "true" date for the run. Although for
geological timescales you might say that this isn't necessary and you
might
as well choose an arbitrary year, there is a good reason for it in
Pleistocene
when you might be using the dates to relate to orbital forcing or
atmospheric
composition.


so the idea is that you are simulating some year, so you really need
time down to the hour or second. but the climate is from 5 million
years ago, so you need the year field to be able to handle that?


Im just thinking that fitting this into the ISO date format
"500-01-01 12:00" seems awkward, esp as it indicates unwarranted
precision.

seems something like "01-01 12:00 reference 50m BCE" would be better.
What do paleo modellers actually use, eg in the figures that they publish?
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jonathan Gregory]

Dear Chris

> Perhaps there could be an attribute we could set that says whether we have 
> accounted for leap seconds?  With the absence of such an attribute to be 
> presumed as meaning leap seconds have been ignored.

Perhaps the real-world calendars with and without leap seconds should be
regarded as two different calendars, since they have different encodings
(meaning decoding/encoding as YMD HMS <-> time-interval since reference-time).
The "true" real-world calendar is the one with leap seconds.

CF has a calendar
proleptic_gregorian

A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is 
divisible by 400.

What if we clarified this calendar as not having leap seconds? Then it could
be used for real-world applications for recent dates meaning that it was just
like the real world except that it doesn't have leap seconds.

Model calendars, which are already idealised wrt length of year, don't have
leap seconds anyway, I am sure.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Don Murray]

John-

On 8/19/11 10:32 AM, John Caron wrote:

On 8/19/2011 8:02 AM, Jonathan Gregory wrote:



Im not sure I can actually implement prefixes for calendar fields,
since in general that would violate the need for integers.
millisecond makes sense, but millihour doesnt.

Yes, I see. Could we include ms then, and also a, ka and Ma for a year,
1000 years and a million years, used in paleoclimate?


i will add ms.

im not familiar with a, ka, and Ma. what are those?


a is an abbreviation for year (annus/anno) and recognized by udunits. 
(see http://en.wikipedia.org/wiki/Year)


This is similar to what I was asking for with geophysical data. 
Sometimes you'll see Mya (millions of years ago) and a common reference 
date is 1950.  I'm not suggesting that you support Mya, but
"Ma before reference_date" would be useful.  Perhaps Jonathan knows how 
this is currently specified for paleoclimate data.


Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

On Aug 19, 2011, at 4:12 PM, Karl Taylor wrote:

> Hi all,
> 
>  My understanding is that leap-seconds are inserted so that the calendar 
> remains (almost) perfectly synchronized with the seasons.  If,  under the CF 
> convention, leap seconds are ignored and time is recorded as, say, "days 
> since 2007", and two times were recorded, 0.5 and 365.5, those times would 
> correspond to 12 noon of Jan 1, 2007 and  12 noon of Jan 1, 2008.  If, 
> however, the CF calendar were required to be truly realistic, the leap second 
> which in fact was inserted at the end of 2007 would mean the second time 
> should be 11:59:59 of Jan 1, 2008.  
> 
> I am pretty sure that the vast majority of datasets stored under the CF 
> convention will not take into account leap seconds and will record times 
> assuming they are not there.  I think it is unrealistic to assume users will 
> adhere to a truly realistic calendar when recording their times.  In the 
> above example, if the measurements were made at noon of each day, then you 
> would have to record times of 0.5 and 365.5+1./(24*60*60) = 365.500011574 
> (rather than 0.5 and 365.5). 
> 
> I therefore think the CF convention should specify that the times recorded in 
> files should be interpreted according to the specified calendar, but assuming 
> that leap-seconds have been ignored.

I can't disagree with your reasoning as existing datasets in CF are likely 
dominated by model outputs.  I should mention that it may cause some 
consternation among folks who are just now struggling to make more satellite 
data available in CF. The implication to them will be that they need to 
uncorrect (oops, not a word according to spellcheck) the leap seconds in their 
data in order to be truly CF-compliant. (This won't be an issue with Level 3 
data, but we were just starting to make progress with Level 1 and Level 2 
data...)

Perhaps there could be an attribute we could set that says whether we have 
accounted for leap seconds?  With the absence of such an attribute to be 
presumed as meaning leap seconds have been ignored.

> 
> cheers,
> Karl
> 
> 
> On 8/19/11 9:28 AM, John Caron wrote:
>> On 8/19/2011 7:48 AM, Lynnes, Christopher S. (GSFC-6102) wrote:
>> 
>>> On Aug 19, 2011, at 7:54 AM, Jon Blower wrote:
>>> 
>>> 
 Hi  Chris,
 
 Does the calendar system usually define whether leap-seconds are taken 
 into account or not?
 
>>> Generally speaking, I don't believe so, if only because calendar systems 
>>> usually go down to the day granularity. Seconds-based doordinate time 
>>> standards, such as UTC and TAI do, because they have to.  Julian date (NOT 
>>> the day of year many associate with the name) is an exception, with its 
>>> fractional dates.
>>> 
>>> 
 In other words, given knowledge of which calendar system is in use, could 
 a library make the correct calculation?  Or is other information needed 
 too?
 
>> In my limited understanding, a calendar system might track leap seconds 
>> or not. If it does, then all calendar fields (except seconds and 
>> millisecs) would be variable length. But for any given calendar, the 
>> interval (in seconds) between two valid dates would be well defined. Of 
>> course, these intervals will differ between calendars.
>> 
>> Therefore, one can talk about a "correct" implementation of a calendar 
>> system, and one can talk about the difference between two calendar 
>> systems, one of which is the correct one to use in some context. 
>> Otherwise, im not sure there really is a "correct calculation", or at 
>> least that might be a confusing way to state the issue.
>> ___
>> CF-metadata mailing list
>> 
>> CF-metadata@cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone: 301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Karl Taylor]

Hi all,

 My understanding is that leap-seconds are inserted so that the 
calendar remains (almost) perfectly synchronized with the seasons. If,  
under the CF convention, leap seconds are ignored and time is recorded 
as, say, "days since 2007", and two times were recorded, 0.5 and 365.5, 
those times would correspond to 12 noon of Jan 1, 2007 and  12 noon of 
Jan 1, 2008.  If, however, the CF calendar were required to be truly 
realistic, the leap second which in fact was inserted at the end of 2007 
would mean the second time should be 11:59:59 of Jan 1, 2008.


I am pretty sure that the vast majority of datasets stored under the CF 
convention will not take into account leap seconds and will record times 
assuming they are not there.  I think it is unrealistic to assume users 
will adhere to a truly realistic calendar when recording their times.  
In the above example, if the measurements were made at noon of each day, 
then you would have to record times of 0.5 and 365.5+1./(24*60*60) = 
365.500011574 (rather than 0.5 and 365.5).


I therefore think the CF convention should specify that the times 
recorded in files should be interpreted according to the specified 
calendar, but assuming that leap-seconds have been ignored.


cheers,
Karl


On 8/19/11 9:28 AM, John Caron wrote:

On 8/19/2011 7:48 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 19, 2011, at 7:54 AM, Jon Blower wrote:


Hi  Chris,

Does the calendar system usually define whether leap-seconds are taken into 
account or not?

Generally speaking, I don't believe so, if only because calendar systems 
usually go down to the day granularity. Seconds-based doordinate time 
standards, such as UTC and TAI do, because they have to.  Julian date (NOT the 
day of year many associate with the name) is an exception, with its fractional 
dates.


In other words, given knowledge of which calendar system is in use, could a 
library make the correct calculation?  Or is other information needed too?

In my limited understanding, a calendar system might track leap seconds
or not. If it does, then all calendar fields (except seconds and
millisecs) would be variable length. But for any given calendar, the
interval (in seconds) between two valid dates would be well defined. Of
course, these intervals will differ between calendars.

Therefore, one can talk about a "correct" implementation of a calendar
system, and one can talk about the difference between two calendar
systems, one of which is the correct one to use in some context.
Otherwise, im not sure there really is a "correct calculation", or at
least that might be a confusing way to state the issue.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Regarding paleoclimate, a point I forgot is that some modellers may 
wish to
have years which are very large negative numbers (many more than four 
digits)

if they set up the model with the "true" date for the run. Although for
geological timescales you might say that this isn't necessary and you 
might
as well choose an arbitrary year, there is a good reason for it in 
Pleistocene
when you might be using the dates to relate to orbital forcing or 
atmospheric

composition.


so the idea is that you are simulating some year, so you really need 
time down to the hour or second. but the climate is from 5 million 
years ago, so you need the year field to be able to handle that?


Im just thinking that fitting this into the ISO date format 
"500-01-01 12:00" seems awkward, esp as it indicates unwarranted 
precision.


seems something like "01-01 12:00 reference 50m BCE" would be better. 
What do paleo modellers actually use, eg in the figures that they publish?

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/19/2011 8:02 AM, Jonathan Gregory wrote:

Dear John


Im not sure I can actually implement prefixes for calendar fields,
since in general that would violate the need for integers.
millisecond makes sense, but millihour doesnt.

Yes, I see. Could we include ms then, and also a, ka and Ma for a year,
1000 years and a million years, used in paleoclimate?


i will add ms.

im not familiar with a, ka, and Ma. what are those?



Regarding paleoclimate, a point I forgot is that some modellers may wish to
have years which are very large negative numbers (many more than four digits)
if they set up the model with the "true" date for the run. Although for
geological timescales you might say that this isn't necessary and you might
as well choose an arbitrary year, there is a good reason for it in Pleistocene
when you might be using the dates to relate to orbital forcing or atmospheric
composition.


so the idea is that you are simulating some year, so you really need 
time down to the hour or second. but the climate is from 5 million years 
ago, so you need the year field to be able to handle that?




Thanks for your other replies as well.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/19/2011 7:48 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 19, 2011, at 7:54 AM, Jon Blower wrote:


Hi  Chris,

Does the calendar system usually define whether leap-seconds are taken into 
account or not?

Generally speaking, I don't believe so, if only because calendar systems 
usually go down to the day granularity. Seconds-based doordinate time 
standards, such as UTC and TAI do, because they have to.  Julian date (NOT the 
day of year many associate with the name) is an exception, with its fractional 
dates.


In other words, given knowledge of which calendar system is in use, could a 
library make the correct calculation?  Or is other information needed too?




In my limited understanding, a calendar system might track leap seconds 
or not. If it does, then all calendar fields (except seconds and 
millisecs) would be variable length. But for any given calendar, the 
interval (in seconds) between two valid dates would be well defined. Of 
course, these intervals will differ between calendars.


Therefore, one can talk about a "correct" implementation of a calendar 
system, and one can talk about the difference between two calendar 
systems, one of which is the correct one to use in some context. 
Otherwise, im not sure there really is a "correct calculation", or at 
least that might be a confusing way to state the issue.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jonathan Gregory]

Dear John

> Im not sure I can actually implement prefixes for calendar fields,
> since in general that would violate the need for integers.
> millisecond makes sense, but millihour doesnt.

Yes, I see. Could we include ms then, and also a, ka and Ma for a year,
1000 years and a million years, used in paleoclimate?

Regarding paleoclimate, a point I forgot is that some modellers may wish to
have years which are very large negative numbers (many more than four digits)
if they set up the model with the "true" date for the run. Although for
geological timescales you might say that this isn't necessary and you might
as well choose an arbitrary year, there is a good reason for it in Pleistocene
when you might be using the dates to relate to orbital forcing or atmospheric
composition.

Thanks for your other replies as well.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

On Aug 19, 2011, at 7:54 AM, Jon Blower wrote:

> Hi  Chris,
> 
> Does the calendar system usually define whether leap-seconds are taken into 
> account or not?  

Generally speaking, I don't believe so, if only because calendar systems 
usually go down to the day granularity. Seconds-based doordinate time 
standards, such as UTC and TAI do, because they have to.  Julian date (NOT the 
day of year many associate with the name) is an exception, with its fractional 
dates. 
 
> In other words, given knowledge of which calendar system is in use, could a 
> library make the correct calculation?  Or is other information needed too?

Actually, it may be more a matter of what the user/application wants as output.

Alas, I'm afraid I am not enough of an expert on coordinate time standards, as 
they are exceedingly complex, almost byzantine. See such items on Wikipedia as:
http://en.wikipedia.org/wiki/Terrestrial_Time, 
http://en.wikipedia.org/wiki/Universal_Time and 
http://en.wikipedia.org/wiki/International_Atomic_Time, 
http://en.wikipedia.org/wiki/Julian_Date, ...


> 
> Presumably this would only affect real-world calendars, and perhaps only UTC?
> 
> Cheers, Jon
> 
> -Original Message-
> From: cf-metadata-boun...@cgd.ucar.edu 
> [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Lynnes, Christopher S. 
> (GSFC-6102)
> Sent: 19 August 2011 12:46
> To: John Caron
> Cc: cf-metadata@cgd.ucar.edu
> Subject: Re: [CF-metadata] CDM calendar date handling
> 
> On Aug 18, 2011, at 6:23 PM, "John Caron"  wrote:
> 
>>> In order to do calculations
>>> with times, which are often necessary, we need to be able to convert 
>>> them into a form which has a fixed-length unit since a reference time 
>>> (like udunits), even though that isn't the most convenient way to express 
>>> them.
>> I think that given two calendar dates in the same calendar, there will 
>> be a well-defined # seconds between them.
> 
> Yes...and no. There is the question of leap seconds. In the satellite data 
> world, most people account for them, but not everyone. Some (most?) off the 
> shelf packages do not. 
> --
> Chris Lynnes
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone: 301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jon Blower]

Hi  Chris,

Does the calendar system usually define whether leap-seconds are taken into 
account or not?  In other words, given knowledge of which calendar system is in 
use, could a library make the correct calculation?  Or is other information 
needed too?

Presumably this would only affect real-world calendars, and perhaps only UTC?

Cheers, Jon

-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu 
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Lynnes, Christopher S. 
(GSFC-6102)
Sent: 19 August 2011 12:46
To: John Caron
Cc: cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] CDM calendar date handling

On Aug 18, 2011, at 6:23 PM, "John Caron"  wrote:

>> In order to do calculations
>> with times, which are often necessary, we need to be able to convert 
>> them into a form which has a fixed-length unit since a reference time 
>> (like udunits), even though that isn't the most convenient way to express 
>> them.
> I think that given two calendar dates in the same calendar, there will 
> be a well-defined # seconds between them.

Yes...and no. There is the question of leap seconds. In the satellite data 
world, most people account for them, but not everyone. Some (most?) off the 
shelf packages do not. 
--
Chris Lynnes
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Lynnes, Christopher S. (GSFC-6102)]

On Aug 18, 2011, at 6:23 PM, "John Caron"  wrote:

>> In order to do calculations
>> with times, which are often necessary, we need to be able to convert them 
>> into
>> a form which has a fixed-length unit since a reference time (like udunits),
>> even though that isn't the most convenient way to express them.
> I think that given two calendar dates in the same calendar, there will 
> be a well-defined # seconds between them. 

Yes...and no. There is the question of leap seconds. In the satellite data 
world, most people account for them, but not everyone. Some (most?) off the 
shelf packages do not. 
--
Chris Lynnes
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/18/2011 4:23 PM, John Caron wrote:
1) There are some dates that are invalid in some calendars eg 29 Feb 
2009, and also apparently 1582-10-05 to 1582-10-14, as well as year 0 
in the gregorian calendar. So if you try to use those in the 
calendar_field, you will get an error. 


sorry that should say reference_date, not calendar_field
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Hi Jonathan:

On 8/18/2011 9:41 AM, Jonathan Gregory wrote:

Dear John


   http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html

If there's interest, I can propose as a CF convention. Otherwise it
can remain a CDM extension.

Thank you for doing this. I think it's an attractive idea to have calendar
handling in CF time coordinates. One issue to be dealt with would be the
need to be able to process these strings using any programming language
which might be used to process CF-netCDF data. Could the software be ported
to other languages in which the netCDF lib exists?


As Jon said, it should be straightforward to develop an extension of 
udunits (C library) to support this, once the specification is worked 
out. However, at the moment, we are not sure of who/how/if this will be 
done. A lot would depend on finding an open-source calendar libary in C 
that does the right thing. We may be able to leverage the ESMF library.



In order to do calculations
with times, which are often necessary, we need to be able to convert them into
a form which has a fixed-length unit since a reference time (like udunits),
even though that isn't the most convenient way to express them.
I think that given two calendar dates in the same calendar, there will 
be a well-defined # seconds between them. I think thats the essence of 
whats needed.


What you have proposed for the syntax looks very sensible to me. I have some
minor points:

* A bit related to Don's: it would be good to allow SI prefixes, like udunits
does. For instance, ms is the usual SI symbol for millisecond.
Im not sure I can actually implement prefixes for calendar fields, since 
in general that would violate the need for integers. millisecond makes 
sense, but millihour doesnt.


* What happens if n months or n years since a certain date is not a valid
date in the calendar? You probably have a rule to resolve that. Most safely,
it should be an error to encode such a time, I suppose (e.g. 1 year since
29 Feb 2008). Do you insist that the ref data is valid in the calendar?

Very good question.

1) There are some dates that are invalid in some calendars eg 29 Feb 
2009, and also apparently 1582-10-05 to 1582-10-14, as well as year 0 in 
the gregorian calendar. So if you try to use those in the 
calendar_field, you will get an error.


2) Then theres the question of "1 year since 29 Feb 2008". This is what 
the library im using does, which seems right to me:


 1 calendar years since 2008-02-29 00:00:00Z == 2009-02-28T00:00:00.000Z
 2 calendar years since 2008-02-29 00:00:00Z == 2010-02-28T00:00:00.000Z
 3 calendar years since 2008-02-29 00:00:00Z == 2011-02-28T00:00:00.000Z
 4 calendar years since 2008-02-29 00:00:00Z == 2012-02-29T00:00:00.000Z
 5 calendar years since 2008-02-29 00:00:00Z == 2013-02-28T00:00:00.000Z
 6 calendar years since 2008-02-29 00:00:00Z == 2014-02-28T00:00:00.000Z
 7 calendar years since 2008-02-29 00:00:00Z == 2015-02-28T00:00:00.000Z
 8 calendar years since 2008-02-29 00:00:00Z == 2016-02-29T00:00:00.000Z
 9 calendar years since 2008-02-29 00:00:00Z == 2017-02-28T00:00:00.000Z
10 calendar years since 2008-02-29 00:00:00Z == 2018-02-28T00:00:00.000Z
11 calendar years since 2008-02-29 00:00:00Z == 2019-02-28T00:00:00.000Z
12 calendar years since 2008-02-29 00:00:00Z == 2020-02-29T00:00:00.000Z
13 calendar years since 2008-02-29 00:00:00Z == 2021-02-28T00:00:00.000Z
14 calendar years since 2008-02-29 00:00:00Z == 2022-02-28T00:00:00.000Z

also:

 1 calendar months since 1930-01-31 00:00:00Z == 1930-02-28T00:00:00.000Z
 2 calendar months since 1930-01-31 00:00:00Z == 1930-03-31T00:00:00.000Z
 3 calendar months since 1930-01-31 00:00:00Z == 1930-04-30T00:00:00.000Z
 4 calendar months since 1930-01-31 00:00:00Z == 1930-05-31T00:00:00.000Z
 5 calendar months since 1930-01-31 00:00:00Z == 1930-06-30T00:00:00.000Z
 6 calendar months since 1930-01-31 00:00:00Z == 1930-07-31T00:00:00.000Z
 7 calendar months since 1930-01-31 00:00:00Z == 1930-08-31T00:00:00.000Z
 8 calendar months since 1930-01-31 00:00:00Z == 1930-09-30T00:00:00.000Z
 9 calendar months since 1930-01-31 00:00:00Z == 1930-10-31T00:00:00.000Z
10 calendar months since 1930-01-31 00:00:00Z == 1930-11-30T00:00:00.000Z
11 calendar months since 1930-01-31 00:00:00Z == 1930-12-31T00:00:00.000Z
12 calendar months since 1930-01-31 00:00:00Z == 1931-01-31T00:00:00.000Z




* The udunits ref time implies missing components e.g. 1 hour since 2011-1-1
means 1 hour since 00:00 on 2001-1-1. What do missing components of ISO strings
imply in your implementation? I think it has got to imply something, because
these datetimes are still complete time specifications, aren't they.

good point, i will add a sentence saying missing fields must be set to 0.


* For further udunits compatibility, it would be convenient to be able to omit
the time zone, which would imply Z. All global model data is in UTC, I should
think, and I suppose it must be the obvious choice for all global datasets.

ok,

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/18/2011 6:54 AM, Don Murray wrote:

Hi John-

On 8/17/11 5:50 PM, John Caron wrote:

In March I promised to do something with the "udunits handling of fuzzy
time units" email conversation. I now have a preliminary implementation
of extended date coordinates. docs are here:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html

If there's interest, I can propose as a CF convention. Otherwise it can
remain a CDM extension.




Hi Don:

Thanks for tackling this. I'm just starting a project with 
non-standard calendars and this will help tremendously.


A couple of comment/questions:

- Why is the calendar field limited to integer types?  It seems like 
one could use NcML to change the unit of an existing double time 
coordinate variable to be a calendar unit so it can be handled 
correctly, but then you'd have a scope mismatch.


A calendar field is inherently an integer: what does 1.5 months mean? If 
you want to use fractions, you really want to use udunits, which are a 
fixed amount of time, and then fractions make sense.


Note that support for non-standard calendars should work for both udunit 
and calendar units.


- Some geophysical datasets use units of millions of years before a 
date, e.g. Myr before 1950-01-01.   It would be good to support the 
"before" syntax and also support "yr" as an abbreviation for year.


Ive added "mon" and "yr".

I have to say that "Myr before 1950-01-01" doesnt make much sense to me. 
There must be something better to do.



- perhaps you could support "mon" and "mo" for month as well.
- "grammer" should be "grammar" ;-)


thakns!

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jon Blower]

Just a very quick comment:

> Could the software be ported to other languages in which the netCDF lib 
> exists?

The good news is that the "non-standard" calendars (360_day etc) are easier to 
program in libraries than the standard Gregorian since the years (and sometimes 
months) are often fixed-length.  Most languages will provide code that handles 
Gregorian dates and I don't think it's too hard to implement the others.

> In order to do calculations with times, which are often necessary, we need to 
> be able to convert them into
> a form which has a fixed-length unit since a reference time (like udunits), 
> even though that isn't the most
> convenient way to express them.

Yes - I think the "engine" of this is a routine that subtracts two times in a 
given calendar system and expresses the result as a certain number of, say, 
seconds.  Again, not too hard in the "non-standard" calendars and usually 
implemented already for the Gregorian system in standard libs.

Jon

-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu 
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory
Sent: 18 August 2011 16:42
To: cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] CDM calendar date handling

Dear John

>   http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html
> 
> If there's interest, I can propose as a CF convention. Otherwise it 
> can remain a CDM extension.

Thank you for doing this. I think it's an attractive idea to have calendar 
handling in CF time coordinates. One issue to be dealt with would be the need 
to be able to process these strings using any programming language which might 
be used to process CF-netCDF data. Could the software be ported to other 
languages in which the netCDF lib exists? In order to do calculations with 
times, which are often necessary, we need to be able to convert them into a 
form which has a fixed-length unit since a reference time (like udunits), even 
though that isn't the most convenient way to express them.

What you have proposed for the syntax looks very sensible to me. I have some 
minor points:

* A bit related to Don's: it would be good to allow SI prefixes, like udunits 
does. For instance, ms is the usual SI symbol for millisecond.

* What happens if n months or n years since a certain date is not a valid date 
in the calendar? You probably have a rule to resolve that. Most safely, it 
should be an error to encode such a time, I suppose (e.g. 1 year since
29 Feb 2008). Do you insist that the ref data is valid in the calendar?

* The udunits ref time implies missing components e.g. 1 hour since 2011-1-1 
means 1 hour since 00:00 on 2001-1-1. What do missing components of ISO strings 
imply in your implementation? I think it has got to imply something, because 
these datetimes are still complete time specifications, aren't they.

* For further udunits compatibility, it would be convenient to be able to omit 
the time zone, which would imply Z. All global model data is in UTC, I should 
think, and I suppose it must be the obvious choice for all global datasets.

I think that we should use standard_names of time for datetimes only, and other 
words, such as period, for intervals of time in standard_names. Then we can 
give them different sorts of units.

Best wishes and thanks

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Jonathan Gregory]

Dear John

>   http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html
> 
> If there's interest, I can propose as a CF convention. Otherwise it
> can remain a CDM extension.

Thank you for doing this. I think it's an attractive idea to have calendar
handling in CF time coordinates. One issue to be dealt with would be the
need to be able to process these strings using any programming language
which might be used to process CF-netCDF data. Could the software be ported
to other languages in which the netCDF lib exists? In order to do calculations
with times, which are often necessary, we need to be able to convert them into
a form which has a fixed-length unit since a reference time (like udunits),
even though that isn't the most convenient way to express them.

What you have proposed for the syntax looks very sensible to me. I have some
minor points:

* A bit related to Don's: it would be good to allow SI prefixes, like udunits
does. For instance, ms is the usual SI symbol for millisecond.

* What happens if n months or n years since a certain date is not a valid
date in the calendar? You probably have a rule to resolve that. Most safely,
it should be an error to encode such a time, I suppose (e.g. 1 year since
29 Feb 2008). Do you insist that the ref data is valid in the calendar?

* The udunits ref time implies missing components e.g. 1 hour since 2011-1-1
means 1 hour since 00:00 on 2001-1-1. What do missing components of ISO strings
imply in your implementation? I think it has got to imply something, because
these datetimes are still complete time specifications, aren't they.

* For further udunits compatibility, it would be convenient to be able to omit
the time zone, which would imply Z. All global model data is in UTC, I should
think, and I suppose it must be the obvious choice for all global datasets.

I think that we should use standard_names of time for datetimes only, and other
words, such as period, for intervals of time in standard_names. Then we can
give them different sorts of units.

Best wishes and thanks

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Upendra Dadi]

 Thanks John! calendar grammer could be quite useful when representing 
climatology bounds. I find it difficult to use udunits for representing 
monthly and seasonal climatology bounds. It would definitely be useful 
to have it in CF.


Upendra


On 8/17/2011 7:50 PM, John Caron wrote:
In March I promised to do something with the "udunits handling of 
fuzzy time units" email conversation. I now have a preliminary 
implementation of extended date coordinates. docs are here:


  http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html 



If there's interest, I can propose as a CF convention. Otherwise it 
can remain a CDM extension.


John


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[Don Murray]

Hi John-

On 8/17/11 5:50 PM, John Caron wrote:

In March I promised to do something with the "udunits handling of fuzzy
time units" email conversation. I now have a preliminary implementation
of extended date coordinates. docs are here:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html

If there's interest, I can propose as a CF convention. Otherwise it can
remain a CDM extension.


Thanks for tackling this. I'm just starting a project with non-standard 
calendars and this will help tremendously.


A couple of comment/questions:

- Why is the calendar field limited to integer types?  It seems like one 
could use NcML to change the unit of an existing double time coordinate 
variable to be a calendar unit so it can be handled correctly, but then 
you'd have a scope mismatch.
- Some geophysical datasets use units of millions of years before a 
date, e.g. Myr before 1950-01-01.   It would be good to support the 
"before" syntax and also support "yr" as an abbreviation for year.

- perhaps you could support "mon" and "mo" for month as well.
- "grammer" should be "grammar" ;-)
- I don't think the "udunit date grammer" is any more difficult to 
understand than the ISO8601 grammar.  It's basically the same.


Thanks again for tackling this.

Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] CDM calendar date handling

[John Caron]

In March I promised to do something with the "udunits handling of fuzzy 
time units" email conversation. I now have a preliminary implementation 
of extended date coordinates. docs are here:


  http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html 



If there's interest, I can propose as a CF convention. Otherwise it can 
remain a CDM extension.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata