Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

Here are the blogs:

http://www.unidata.ucar.edu/blogs/developer/en/entry/dimensions_scales
http://www.unidata.ucar.edu/blogs/developer/en/entry/dimension_scale2
http://www.unidata.ucar.edu/blogs/developer/en/entry/dimension_scales_part_3
http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf4_shared_dimensions
http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf4_use_of_dimension_scales

On Fri, Apr 22, 2016 at 7:57 AM, Pedro Vicente <
pedro.vice...@space-research.org> wrote:

> John
>
> >>>i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> >>>case you want details.
>
> yes, I am interested, I had the impression by looking at the code some
> years ago that netCDF writes some unique name attributes somewhere
>
> --
> Pedro Vicente
> pedro.vice...@space-research.org
> https://twitter.com/_pedro__vicente
> http://www.space-research.org/
>
>
>
>
> - Original Message -
> *From:* John Caron <jcaron1...@gmail.com>
> *To:* Pedro Vicente <pedro.vice...@space-research.org>
> *Cc:* cf-metadata@cgd.ucar.edu ; Discussion forum for the NeXus data
> format <ne...@nexusformat.org> ; netcdfgr...@unidata.ucar.edu ; Dennis
> Heimbigner <d...@ucar.edu> ; NetCDF-Java community
> <netcdf-j...@unidata.ucar.edu>
> *Sent:* Thursday, April 21, 2016 11:11 PM
> *Subject:* Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF
> versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>
> 1) I completely agree with the idea of adding system metadata that
> indicates the library version(s) that wrote the file.
>
> 2) the way shared dimensions are implemented by netcdf4 is a unique
> signature that would likely identify (100 - epsilon) % of real data files
> in the wild. One could add such detection to the netcdf4 and/or hdf5
> libraries, and/or write a utility program to detect.
>
> there are 2 variants:
>
> 2.1) one could write a netcdf4 file without shared dimensions, though im
> pretty sure no one does. but you could argue then that its fine to just
> treat it as an hdf5 file and read through hdf5 library
>
> 2.2) one could write a netcdf4 file with hdf5 library, if you knew what
> you are doing. i have heard of this happening. but then you could argue
> that its really a netcdf4 file and you should use netcdf library to read.
>
> i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> case you want details.
>
> On Thu, Apr 21, 2016 at 4:18 PM, Pedro Vicente <
> pedro.vice...@space-research.org> wrote:
>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>>
>>
>>
>> ok, thank you
>>
>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>>
>>
>>
>> Excellent idea , Dennis
>>
>> --
>> Pedro Vicente
>> pedro.vice...@space-research.org
>> https://twitter.com/_pedro__vicente
>> http://www.space-research.org/
>>
>>
>> - Original Message - From: <d...@ucar.edu>
>> To: "Pedro Vicente" <pedro.vice...@space-research.org>; <
>> cf-metadata@cgd.ucar.edu>; "Discussion forum for the NeXus data format" <
>> ne...@nexusformat.org>; <netcdfgr...@unidata.ucar.edu>
>> Sent: Thursday, April 21, 2016 5:02 PM
>> Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 --
>> PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>>
>>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>> =Dennis Heimbigner
>>>   Unidata
>>>
>>>
>>> On 4/21/2016 2:17 PM, Pedro Vicente wrote:
>>>
>>>> Dennis
>>>>
>>>> I am in the process of adding a global attribute in the root group
>>>>>>>>
>>>>>>> that captures both the netcdf library version and the hdf5 library
>>>>> version
>>>>> whenever a netcdf file is created. The current  form is
>>>>>

Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

Hi Pedro:

there are 4 bl;ogs, starting with this one:


On Fri, Apr 22, 2016 at 7:57 AM, Pedro Vicente <
pedro.vice...@space-research.org> wrote:

> John
>
> >>>i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> >>>case you want details.
>
> yes, I am interested, I had the impression by looking at the code some
> years ago that netCDF writes some unique name attributes somewhere
>
> --
> Pedro Vicente
> pedro.vice...@space-research.org
> https://twitter.com/_pedro__vicente
> http://www.space-research.org/
>
>
>
>
> - Original Message -
> *From:* John Caron <jcaron1...@gmail.com>
> *To:* Pedro Vicente <pedro.vice...@space-research.org>
> *Cc:* cf-metadata@cgd.ucar.edu ; Discussion forum for the NeXus data
> format <ne...@nexusformat.org> ; netcdfgr...@unidata.ucar.edu ; Dennis
> Heimbigner <d...@ucar.edu> ; NetCDF-Java community
> <netcdf-j...@unidata.ucar.edu>
> *Sent:* Thursday, April 21, 2016 11:11 PM
> *Subject:* Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF
> versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>
> 1) I completely agree with the idea of adding system metadata that
> indicates the library version(s) that wrote the file.
>
> 2) the way shared dimensions are implemented by netcdf4 is a unique
> signature that would likely identify (100 - epsilon) % of real data files
> in the wild. One could add such detection to the netcdf4 and/or hdf5
> libraries, and/or write a utility program to detect.
>
> there are 2 variants:
>
> 2.1) one could write a netcdf4 file without shared dimensions, though im
> pretty sure no one does. but you could argue then that its fine to just
> treat it as an hdf5 file and read through hdf5 library
>
> 2.2) one could write a netcdf4 file with hdf5 library, if you knew what
> you are doing. i have heard of this happening. but then you could argue
> that its really a netcdf4 file and you should use netcdf library to read.
>
> i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> case you want details.
>
> On Thu, Apr 21, 2016 at 4:18 PM, Pedro Vicente <
> pedro.vice...@space-research.org> wrote:
>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>>
>>
>>
>> ok, thank you
>>
>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>>
>>
>>
>> Excellent idea , Dennis
>>
>> --
>> Pedro Vicente
>> pedro.vice...@space-research.org
>> https://twitter.com/_pedro__vicente
>> http://www.space-research.org/
>>
>>
>> - Original Message - From: <d...@ucar.edu>
>> To: "Pedro Vicente" <pedro.vice...@space-research.org>; <
>> cf-metadata@cgd.ucar.edu>; "Discussion forum for the NeXus data format" <
>> ne...@nexusformat.org>; <netcdfgr...@unidata.ucar.edu>
>> Sent: Thursday, April 21, 2016 5:02 PM
>> Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 --
>> PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>>
>>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>> =Dennis Heimbigner
>>>   Unidata
>>>
>>>
>>> On 4/21/2016 2:17 PM, Pedro Vicente wrote:
>>>
>>>> Dennis
>>>>
>>>> I am in the process of adding a global attribute in the root group
>>>>>>>>
>>>>>>> that captures both the netcdf library version and the hdf5 library
>>>>> version
>>>>> whenever a netcdf file is created. The current  form is
>>>>> _NCProperties="version=...|netcdflibversion=...|hdflibversion=..."
>>>>>
>>>>
>>>>
>>>> ok, good to know, thank you
>>>>
>>>>
>>>> > 1. I am open to suggestions about changing the format or adding
>>>>>>> info > to it.
&g

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

Hi David:

At the risk of giving more useful answers to the wrong question, i will say
that we could do something other than require ancillary or coordinate
variables to only have dimensions that the parent variable has. There just
must be a simple and explicit rule for mapping between parent and child
dimension.

We went down that path with DSG, in order to efficiently store ragged
arrays. Another possible use case is to represent contiguous bounds, where
lower_bounds(i+1) == upper_bounds(i), then one only needs n+1 values
instead of 2*n.

Anyway, i think the rule of only allowing dimensions that are in the parent
is a good one, but for compelling use cases, we could also allow "simple
and explicit" alternatives.

regards,
John

On Tue, Dec 22, 2015 at 4:07 AM, David Hassell 
wrote:

> Hello all,
>
> Many thanks for the replies. It seems there is only support for the
> case that ancillary variable dimensions must be a subset of their
> parent variable dimensions (treating scalar coordinate variables like
> size 1 coordinate variables for the sake of a snappier sentence - oh,
> that's blown it).
>
> This reminded of an old thread about this that Mark started
> (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056162.html). In
> this thread Jim convincingly argues for an ancillary variable which
> can have size-greater-than-one dimensions which the parent variable
> does not span
> (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056215.html):
>
>   "As an example, let's say that I have a variable a[X,Y,T], which
>contains the total energy deposited per day into the atmosphere by
>charged particles (from the solar wind & Van Allen belts) on a
>lon/lat grid as a function of time.  (Something I used to work on
>many years ago.)  Let's then say that I have another variable,
>b[X,Y,Z], which represents the model atmosphere density profile on
>the same lon/lat grid as a function of altitude.  This density
>profile was used in the calculation of the energy deposition values
>stored in variable a.  Variable b is clearly valid as an ancillary
>variable for a, isn't it?"
>
> Mark raised a defect ticket (#98) about this, but withdrew it in light
> of the lack of supprt from the mailing list discussion.
>
> However, variable b in Jim's example seems to me to be storing
> "sub-grid scale information", so if we are to allow that we perhaps we
> ought to allow coordinate variables which sub-grid values and which do
> not span their any of their variable dimensions. For example, we might
> want to store the longitude coordinates which contributed to a zonal
> mean. The size 1 dimension case I originally raised is merely a
> special case of this, I feel.
>
> But I do not think that there is any appetite for this at this time,
> and certainly not from me. John - your comment here was actually very
> useful!
>
> So, I now think that ancillary variable dimensions should be a subset
> of their asssociated data variable dimensions (which includes the
> possibility of ancillary variables have no dimensions). Perhaps, in
> the new year, ticket #98 could be reopened as an enhancement ...
>
> All the best,
>
> David
>
>
>  Original message from Karl Taylor (01PM 21 Dec 15)
>
> > Date: Mon, 21 Dec 2015 13:59:33 -0800
> > From: Karl Taylor 
> > To: cf-metadata@cgd.ucar.edu
> > Subject: Re: [CF-metadata] [CF Metadata] Question about ancillary
> >  variables/formula terms variables
> > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
> >  Gecko/20100101 Thunderbird/38.4.0
> >
> > Hi David,
> >
> > I can't think of anyone needing these constructs, although there are
> > cases when both the "parent" and ancillary or formula_terms variable
> > might *both* have a scalar dimension.
> >
> > Karl
> >
> > On 12/21/15 7:41 AM, David Hassell wrote:
> > >Hello,
> > >
> > >I was wondering if anyone has ever had use for an ancillary variable
> > >(or data variable pointed to from a formula_terms attribute), having a
> > >size 1 dimension which the "parent" field does *not* have. The
> > >convention do not say anything about this.
> > >
> > >For example:
> > >
> > >   dimensions:
> > > t = 1;
> > > x = 96;
> > > y = 73;
> > >   variables:
> > > float q(x, y) ;
> > >   q:standard_name = "specific_humidity" ;
> > >   q:units = "g/g" ;
> > >   q:ancillary_variables = "q_error_limit" ;
> > > float q_error_limit(t, x, y)
> > >   q_error_limit:standard_name = "specific_humidity standard_error"
> ;
> > >   q_error_limit:units = "g/g" ;
> > >Similary for a scalar coordinate variable, which is logically
> > >equivalent to the size 1 dimension case (this time shown on a formula
> > >terms variable):
> > >   dimensions:
> > > x = 96;
> > > y = 73;
> > > z = 19;
> > >   variables:
> > > float q(z, x, y) ;
> > >   q:standard_name = "specific_humidity" ;
> > >   q:units = "g/g" ;
> > 

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

oops, sorry misread the question

On Mon, Dec 21, 2015 at 1:06 PM, Jim Biard <jbi...@cicsnc.org> wrote:

> John,
>
> But ancillary variables aren't coordinates, are they?
>
> Jim
>
>
> On 12/21/15 1:08 PM, John Caron wrote:
>
> i think a crucial property of coordinates is that their dimensions are a
> subset of the variable's dimensions. otherwise one cant assign the correct
> coordinate.
>
> the length of a char valued coordinate is an exception, really char(n) is
> logically the same as string.
>
> while a dim=1 coordinate could be used, it would violate that simple
> principle for no obvious gain. scalar here is preffered i think.
>
> we added some language to CF recently to clarify this. not sure where it
> is.
>
> john
>
>
>
> On Mon, Dec 21, 2015 at 8:49 AM, Jim Biard <jbi...@cicsnc.org> wrote:
>
>> David,
>>
>> I've not come across this particular case, but I've had cases where the
>> ancillary data that I wish to store has less dimensions than the
>> corresponding parent field. The Conventions seem to require that the
>> dimensions all match. I'm of the opinion that it should not be so
>> restrictive.
>>
>> In your particular instance, what is the significance of your time
>> variable?
>>
>> Grace and peace,
>>
>> Jim
>>
>>
>> On 12/21/15 10:41 AM, David Hassell wrote:
>>
>> Hello,
>>
>> I was wondering if anyone has ever had use for an ancillary variable
>> (or data variable pointed to from a formula_terms attribute), having a
>> size 1 dimension which the "parent" field does *not* have. The
>> convention do not say anything about this.
>>
>> For example:
>>
>>   dimensions:
>> t = 1;
>> x = 96;
>> y = 73;
>>
>>   variables:
>> float q(x, y) ;
>>   q:standard_name = "specific_humidity" ;
>>   q:units = "g/g" ;
>>   q:ancillary_variables = "q_error_limit" ;
>>
>> float q_error_limit(t, x, y)
>>   q_error_limit:standard_name = "specific_humidity standard_error" ;
>>   q_error_limit:units = "g/g" ;
>>
>>
>> Similary for a scalar coordinate variable, which is logically
>> equivalent to the size 1 dimension case (this time shown on a formula
>> terms variable):
>>
>>   dimensions:
>> x = 96;
>> y = 73;
>> z = 19;
>>
>>   variables:
>> float q(z, x, y) ;
>>   q:standard_name = "specific_humidity" ;
>>   q:units = "g/g" ;
>>
>> float z(z):
>>   z:standard_name = "atmosphere_hybrid_height_coordinate";
>>   z.formula_terms = "a: var1 b: var2 orog: var3"
>>
>> float var3(x, y):
>>   time:coordinates = "time";
>>
>> float time:
>>   time:standard_name = "time";
>>   time:units = "days since 2000-12-1";
>>
>>
>> Thanks for your help,
>>
>> All the best,
>>
>> David
>>
>>
>> --
>> David Hassell
>> National Centre for Atmospheric Science (NCAS)
>> Department of Meteorology, University of Reading,
>> Earley Gate, PO Box 243,
>> Reading RG6 6BB, U.K.
>>
>> Tel   : +44 118 3785613
>> E-mail: d.c.hass...@reading.ac.uk
>> ___
>> CF-metadata mailing 
>> listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>>
>> --
>> [image: CICS-NC] <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA National Centers for Environmental Information
>> <http://ncdc.noaa.gov/>
>> *formerly NOAA’s National Climatic Data Center*
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbi...@cicsnc.org
>> o: +1 828 271 4900
>>
>> *Connect with us on Facebook for climate
>> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
>> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow us on
>> Twitter at @NOAANCEIclimate <https://twitter.com/NOAANCEIclimate> and
>> @NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. *
>>
>> ___
>> CF-metadata mailing list
>> CF-metadata@

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

i think a crucial property of coordinates is that their dimensions are a
subset of the variable's dimensions. otherwise one cant assign the correct
coordinate.

the length of a char valued coordinate is an exception, really char(n) is
logically the same as string.

while a dim=1 coordinate could be used, it would violate that simple
principle for no obvious gain. scalar here is preffered i think.

we added some language to CF recently to clarify this. not sure where it is.

john



On Mon, Dec 21, 2015 at 8:49 AM, Jim Biard  wrote:

> David,
>
> I've not come across this particular case, but I've had cases where the
> ancillary data that I wish to store has less dimensions than the
> corresponding parent field. The Conventions seem to require that the
> dimensions all match. I'm of the opinion that it should not be so
> restrictive.
>
> In your particular instance, what is the significance of your time
> variable?
>
> Grace and peace,
>
> Jim
>
>
> On 12/21/15 10:41 AM, David Hassell wrote:
>
> Hello,
>
> I was wondering if anyone has ever had use for an ancillary variable
> (or data variable pointed to from a formula_terms attribute), having a
> size 1 dimension which the "parent" field does *not* have. The
> convention do not say anything about this.
>
> For example:
>
>   dimensions:
> t = 1;
> x = 96;
> y = 73;
>
>   variables:
> float q(x, y) ;
>   q:standard_name = "specific_humidity" ;
>   q:units = "g/g" ;
>   q:ancillary_variables = "q_error_limit" ;
>
> float q_error_limit(t, x, y)
>   q_error_limit:standard_name = "specific_humidity standard_error" ;
>   q_error_limit:units = "g/g" ;
>
>
> Similary for a scalar coordinate variable, which is logically
> equivalent to the size 1 dimension case (this time shown on a formula
> terms variable):
>
>   dimensions:
> x = 96;
> y = 73;
> z = 19;
>
>   variables:
> float q(z, x, y) ;
>   q:standard_name = "specific_humidity" ;
>   q:units = "g/g" ;
>
> float z(z):
>   z:standard_name = "atmosphere_hybrid_height_coordinate";
>   z.formula_terms = "a: var1 b: var2 orog: var3"
>
> float var3(x, y):
>   time:coordinates = "time";
>
> float time:
>   time:standard_name = "time";
>   time:units = "days since 2000-12-1";
>
>
> Thanks for your help,
>
> All the best,
>
> David
>
>
> --
> David Hassell
> National Centre for Atmospheric Science (NCAS)
> Department of Meteorology, University of Reading,
> Earley Gate, PO Box 243,
> Reading RG6 6BB, U.K.
>
> Tel   : +44 118 3785613
> E-mail: d.c.hass...@reading.ac.uk
> ___
> CF-metadata mailing 
> listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
> --
> [image: CICS-NC]  Visit us on
> Facebook  *Jim Biard*
> *Research Scholar*
> Cooperative Institute for Climate and Satellites NC 
> North Carolina State University 
> NOAA National Centers for Environmental Information
> 
> *formerly NOAA’s National Climatic Data Center*
> 151 Patton Ave, Asheville, NC 28801
> e: jbi...@cicsnc.org
> o: +1 828 271 4900
>
> *Connect with us on Facebook for climate
>  and ocean and geophysics
>  information, and follow us on
> Twitter at @NOAANCEIclimate  and
> @NOAANCEIocngeo . *
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Fwd: Geos grid mapping

[John Caron]

i think its just that CF 1.7 document has been stuck for going on 3 years
now.

-- Forwarded message --
From: Ethan Davis eda...@ucar.edu
Date: Wed, May 6, 2015 at 3:29 PM
Subject: Re: [CF-metadata] Geos grid mapping
To: Randy Horne rho...@excaliburlabs.com
Cc: John Caron ca...@ucar.edu


Hi Randy,

Sounds like you might have missed the last netCDF-Java support
message/email John sent, back in Sept 2014, on this topic. It said

We are releaseing 4.5.3 today that has this CF geostationary projection in
it. Please let us know if you see any problems.


Also, there is a related JIRA issue:

https://bugtracking.unidata.ucar.edu/browse/TDS-538


Sorry for any confusion.
Hope that helps,

Ethan

On Wed, May 6, 2015 at 2:12 PM, Randy Horne rho...@excaliburlabs.com
wrote:

 Martin et al:


 I think we ended up with the projection name “geostationary”.

 Also, I had some contact with the Unidata netcdf Java support team (RE:
 taking item netCDF #ZCD-209044]), and have not heard anything for over a
 year.

 I would have thought it would find its way into CF 1.7.


 very respectfully,

 randy




  On May 6, 2015, at 3:41 PM, Martin Raspaud martin.rasp...@smhi.se
 wrote:
 
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1
 
  Hi all,
 
  A while ago (3 years ago), we submitted ticket #72 to add the geos
  grid mapping to the conventions. However, it doesn't seem to go
  anywhere. Is there anyone here that knows why, and could advise me how
  I could help to make this happen ?
 
  Best regards,
  Martin
  -BEGIN PGP SIGNATURE-
  Version: GnuPG v2.0.22 (GNU/Linux)
 
  iQEcBAEBAgAGBQJVSm5qAAoJEBdvyODiyJI4cxAH/33l7QKOgIWWIXYJEOqSvCRA
  faivR8tNX21jPbJOLxg4dBhRRbg+lARClJjmapdLGn9a9FJU2OB/sTdSuZYyeDFh
  ySVmPFscPzK4xFVa0WDMphytNPu6xkTmhciEmnSyP1ep+cgerrAwcOWFB+TTLpMa
  mme5P+gBoMAWtwf/Lrwu0Ro3Q9aB98+frc1b3I6h/hmw3BCxdz1nqrlAAE2H9Xyb
  2+ddEMDRp3a/1VC2Emon7mwOB3yHspQdwbbmIfUTzjrj8695K3wS0yjAYd00pXvy
  qJ9JCHcvtWzeNbv2ZHH7HwcYYLQWJqBmPzBjSxWuTBDs1uFDdNjMMcVHA8OSrfc=
  =mz78
  -END PGP SIGNATURE-
  martin_raspaud.vcf___
  CF-metadata mailing list
  CF-metadata@cgd.ucar.edu
  http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


 

 Randy C. Horne (rho...@excaliburlabs.com)
 Principal Engineer, Excalibur Laboratories Inc.
 voice  fax: (321) 952.5100
 url: http://www.excaliburlabs.com





 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] geostationary vs vertical_perspective in CF 1.7.

[John Caron]

great, thanks for the clarification.

the reference in CF vertical perspective projections has this proj4
reference:

  http://www.remotesensing.org/geotiff/proj_list/geos.html

is it wrong/misleading?

is geostationary a special case of vertical perspective?

On Tue, Nov 18, 2014 at 6:28 PM, Randy Horne rho...@excaliburlabs.com
wrote:

 John:

 geostationary projection and vertical perspective projections are not the
 same.

 an example of a vertical projection image is one taken by a camera in
 space, which is not necessarily at the equator, and the data for the entire
 image is observed using a single exposure.

 the geostationary projection is that associated with scanning imagers from
 geostationary orbit where each image pixel covers a fixed number of
 radians/degrees in east/west and north/south relative to the camera’s
 aperture.

 very respectfully,

 randy



 On Nov 18, 2014, at 3:19 PM, John Caron ca...@ucar.edu wrote:

 CF TRAC ticket 72 defines geostationary projection, accepted for 1.7.

 1) I _think_ that Vertical Perspective is an earlier version of this,
 and it should be deprecated in favor of geostationary. Does anyone have a
 different opinion?

 2) The draft 1.7 does not yet include it, so its been hard to refer users
 to it.


 http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/cf-conventions.html#appendix-grid-mappings
 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



 

 Randy C. Horne (rho...@excaliburlabs.com)
 Principal Engineer, Excalibur Laboratories Inc.
 voice  fax: (321) 952.5100
 url: http://www.excaliburlabs.com





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Correct mail group address?

[John Caron]

I have both

cf-metadata@cgd.ucar.edu


and


cf-metad...@lists.llnl.gov


it appears the many (most?) still use cf-metadata@cgd.ucar.edu, i
guess its being forwarded to cf-metad...@lists.llnl.gov
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Correct mail group address?

[John Caron]

I see, so we should continue to use cf-metadata@cgd.ucar.edu. for normal
conversation?

BTW, the TRAC system is either down or slow enough to time out.

On Tue, Nov 18, 2014 at 12:57 PM, Jeffrey F. Painter paint...@llnl.gov
wrote:

  Use cf-metadata@cgd.ucar.edu.
 The one at llnl.gov is part of the system for connecting the Trac
 issue-tracker to the main mailing list at ucar.edu.
 - Jeff


 On 11/18/14, 11:52 AM, John Caron wrote:

 I have both

  cf-metadata@cgd.ucar.edu

  and

  cf-metad...@lists.llnl.gov

  it appears the many (most?) still use cf-metadata@cgd.ucar.edu, i guess its 
 being forwarded to cf-metad...@lists.llnl.gov



 ___
 CF-metadata mailing 
 listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] geostationary vs vertical_perspective in CF 1.7.

[John Caron]

CF TRAC ticket 72 defines geostationary projection, accepted for 1.7.

1) I _think_ that Vertical Perspective is an earlier version of this, and
it should be deprecated in favor of geostationary. Does anyone have a
different opinion?

2) The draft 1.7 does not yet include it, so its been hard to refer users
to it.

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/cf-conventions.html#appendix-grid-mappings
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] string valued coordinates

[John Caron]

My preference is that one explicitly puts in the units. For dimensionless,
1 or  is ok for udunits. If the units attribute isnt there, I assume
that the user forgot to specify it, so the units are unknown.

Im not sure what CF actually says, but it would be good to clarify.

John

On Thu, Oct 30, 2014 at 2:37 AM, Hedley, Mark mark.hed...@metoffice.gov.uk
wrote:

  Hello CF

  From: CF-metadata [cf-metadata-boun...@cgd.ucar.edu] on behalf of
 Jonathan Gregory [j.m.greg...@reading.ac.uk]

  Yes, there are some standard names which imply string values, as Karl
 says. If the standard_name table says 1, that means the quantity is
 dimensionless, so it's also fine to omit the units, as Jim says.

 I would like to raise question about this statement.  Omitting the units
 and stating that the units are '1' are two very different things;
 dimensionless != no_unit
 is an important statement which should be clear to data consumers and
 producers.

 If the standard name table defines a canonical unit for a standard_name of
 '1' then I expect this quantity to be dimensionless, with a unit of '1' or
 some multiple there of.
 If the standard name states that the canonical unit for a standard_name is
 '' then I expect that quantity to have no unit stated.
 Any deviation from this behaviour is a break with the conventions.  I have
 code which explicitly checks this for data sets.

 Are people aware of examples of the pattern of use described by Jonathan,
 such as a categorical quantities identified by a standard_name with a
 canonical unit of '1'?

 thank you
 mark

 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] string valued coordinates

[John Caron]

I understand that netCDF coordinate variables have to be strictly
monotonic, and no-one wants to define what this means for the general case
of strings; that is fine.

in CDM, monontonicity is required to make the 1D coordinate maps
invertible. For string valued coordinates, the equivilent requirement is
uniqueness. Im not sure if CF data model states this or not.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[John Caron]

Hi all:

Yes, 1.10 would follow 1.9, if needed, on that branch.

In software, we are used to major version increments meaning API may have
changed. Since any CF file using features of the extended model will not
be fully visable (or even break) software that only understands the classic
model, it seems right to increment the major version number of the CF
Convention.

John



On Wed, Sep 24, 2014 at 2:54 AM, stephen.pas...@stfc.ac.uk wrote:

 Tim,

 The usual convention is that version numbers are not decimal but
 .-separated integers.  Therefore, following 1.9 would be 1.10 and there
 is no necessity to merge into 2.0.

 On the subject of git and github.  I would thoroughly recommend embracing
 github as a collaborative tool.  The github ticket system is excellent and
 the freedom to experiment that comes with git branches and forks is truly
 revolutionary.  For me, the point about git is that all parties are in
 control -- no-one needs authorisation to try something new and the
 official branch does not loose control by allowing experimentation.  It
 is a technology which enables a true meritocracy.

 Having said that, I understand that it is a big culture shift which people
 don't get until they have had to use it for a while.

 My final point -- bear in mind that those of us who's job it is to archive
 data have no choice but to remain backwards compatible :-).

 Stephen.

 ---
 Stephen Pascoe  +44 (0)1235 445980
 Centre of Environmental Data Archival
 STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK


 -Original Message-
 From: Timothy Patterson [mailto:timothy.patter...@eumetsat.int]
 Sent: 24 September 2014 09:16
 To: 'Charlie Zender'; CF Metadata Mail List
 Subject: Re: [CF-metadata] CF-2.0 Convention discussion

 I'm wondering whether CF Convention 2.0 is the right naming convention to
 be using to refer to this new branch.

 Firstly, as the classic CF Convention has moved from 1.0 to 1.7 in 0.1
 increments, is the original branch then forced after CF 1.9 to either merge
 with CF 2.0 or to adopt a new naming convention (1.9, 1.91, 1.92...1.991,
 1.992...)?

 Secondly, as a newcomer to the CF Convention, if I find documents called
 CF Convention 1.7 and CF Convention 2.0, I'm probably going to assume that
 CF 2.0 is the document I should be using.

 As an alternative, the new branch could be given a unique name e.g.  CF
 Conventions Enhanced 1.0 (to imply the use of netCDF-4 enhanced features).
 Then we could refer to CF Enhanced 1.0 and CF 1.7 or CF Classic 1.7 when we
 need to differentiate between the two branches.

 Regards,

 Tim

 -

 Dr. Timothy Patterson
 Instrument Data Simulation
 Product Format Specification

 EUMETSAT
 Eumetsat-Allee 1
 64295 Darmstadt
 Germany



 -Original Message-
 From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
 Charlie Zender
 Sent: Friday, September 19, 2014 6:20 PM
 To: CF Metadata Mail List
 Subject: [CF-metadata] CF-2.0 Convention discussion

 I like the idea of a Google Doc to consolidate the issue list.
 Whether it will prove unwieldy is hard to guess.

 A 3-month period for listing use-cases seems reasonable.
 It seems more CF-like to be use-case driven than to presume the entire
 netCDF4 data model will be adopted.
 This has the advantage of parsimony--only implement what is necessary.
 From a technical point-of-view, however, it is advantageous to know already
 what netCDF4 can and cannot easily do. In my view the intersection of
 important use-cases and netCDF4 capabilities is the sweet spot for CF-2.0.

 Once the smoke clears I may summarize use-cases for handling
 ensembles/replicants and for handling large integer numbers. These could
 raise the issues of having groups and 64-bit ints in CF-2.0.
 But first let's settle the framework for discussions.

 Best,
 Charlie
 --
 Charlie Zender, Earth System Sci.  Computer Sci.
 University of California, Irvine 949-891-2429 )'(
 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
 --
 Scanned by iCritical.
 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Why surface_altitude instead of platform_altitude?

[John Caron]

As I recall, the original proposal was for station_altitude. We decided to
change station to platform. At the same time it was thought that the
existing standard name of surface altitude would be synonymous. I at
least was thinking of ground stations. So I think we make a mistake there
and platform_altitude would be the right correction.

An altitude of course needs a datum, and I think we have not been clear
enough on that. I think we should review our use or non-use of vertical
datum. A quick look seems to imply that WGS 84 is assumed (?)

On Thu, Sep 18, 2014 at 1:42 PM, Signell, Richard rsign...@usgs.gov wrote:

 John,
 So then the surface needs to be defined relative to some known datum, no?

 Maybe we need platform_altitude_above_datum  and a specification of
 the vertical datum (EPSG:5701 (MSL), EPSG:5703 (NAVD88), etc)

 On Thu, Sep 18, 2014 at 1:47 PM, John Graybeal
 john.grayb...@marinexplore.com wrote:
  I assume surface_altitude is an important variable for providing the
 vertical location of measurements relative to a surface (as opposed to
 relative to a geoid -- notwithstanding the definition issue).
 
  John
 
  On Sep 18, 2014, at 08:30, Signell, Richard rsign...@usgs.gov wrote:
 
  Maybe a simpler approach would be to just adopt platform_altitude as
  an alias for surface_altitude and suggest deprecating the use of
  surface_altitude?
 
  On Thu, Sep 18, 2014 at 11:15 AM, John Graybeal
  john.grayb...@marinexplore.com wrote:
  Interesting that there is so little discussion of this language in the
 mail list, only in John Caron's 2011.09.16 mail on standard names for
 stations (which refers to words already in draft 1.6, I think) -- which
 came at the tail end of a long thread on platform names/IDs.
 
  From those words, I infer that the original drafter thought
 surface_altitude was just as good for describing platform location, as it
 was for describing observation location. I suspect the assumption was that
 any corresponding observations were at the same location as the platform.
 
  Since this is not always true, I'm with you that there should be a
 term platform altitude, and it should be the one used in this sentence.
 
  I hereby propose the standard name platform_surface_altitude (m),
 Standard names for platform describe the motion and orientation of the
 vehicle from which observations are made e.g. aeroplane, ship or satellite.
  The surface called surface means the lower boundary of the
 atmosphere. Altitude is the (geometric) height above the horizontal
 reference surface.
 
  Note I've changed the standard wording of the _altitude definition,
 which generally says .. above the geoid, which is the reference
 geopotential surface. The geoid is similar to mean sea level. This seems
 clearly in conflict with the definition of surface_altitude and this new
 term, and I think it should be changed in surface_altitude's definition too.
 
  I suppose if people agree with you and me, we need to do a Trac ticket
 for the corresponding change to the standard.
 
  John
 
 
 
  On Sep 18, 2014, at 06:40, Signell, Richard rsign...@usgs.gov wrote:
 
  In the CF-1.6 and CF-1.7 draft doc, in section H.2, we have:
 
  It is recommended that there should be station variables with
  standard_name attributes  platform_name ,  surface_altitude  and “
  platform_id ” when applicable.
 
  Why is this surface_altitude instead of platform_altitude?
 
  In the ocean, we have lots of upward-looking Acoustic Doppler Current
  Profilers (ADCP), where the instrument with transducer and other
  sensors is located some distance below the ocean surface.   While
  velocity and other properties are measured in vertical bins above the
  instrument (timeSeriesProfile), other properties like pressure and
  temperature are measured at the instrument.
 
  Since the instrument is not at the surface, it seems misleading to use
  the standard_name surface_altitude instead of platform_altitude,
  particularly when we already have platform_name and platform_id.
 
  In this example CF_1.6 ADCP dataset:
 
 
 http://geoport.whoi.edu/thredds/dodsC/usgs/data2/rsignell/data/adcp/7201adc-a_cf16.nc.html
 
  the variable platform_altitude has a value of -10.4522 m:
 
 http://geoport.whoi.edu/thredds/dodsC/usgs/data2/rsignell/data/adcp/7201adc-a_cf16.nc.ascii?platform_altitude
 
  but we are forced to use a standard_name of surface_altitude.
 
  Why not platform_altitude?
 
  Thanks,
  Rich
 
  --
  Dr. Richard P. Signell   (508) 457-2229
  USGS, 384 Woods Hole Rd.
  Woods Hole, MA 02543-1598
  ___
  CF-metadata mailing list
  CF-metadata@cgd.ucar.edu
  http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
 
 
 
 
  --
  Dr. Richard P. Signell   (508) 457-2229
  USGS, 384 Woods Hole Rd.
  Woods Hole, MA 02543-1598
 



 --
 Dr. Richard P. Signell   (508) 457-2229
 USGS, 384 Woods Hole Rd.
 Woods Hole, MA 02543-1598
 ___
 CF-metadata mailing 

Re: [CF-metadata] CF Conventions and netCDF-4 enhanced model

[John Caron]

Hi Timothy:

From my POV, groups are a way to, um, group things (variables, coord
systems, etc) together. They should be used for the same reason we use
directories in a file system, to prevent a big mess of things in one
namespace. On the surface, your layout above seems reasonable for those
reasons.

The main restriction from the CDM data model is that variables can only use
dimensions from their own group or a parent group.  the result is to keep
coordinate processing simple and close to the classic model. it does affect
the way you lay out your groups, which may be different from how you think
about your data. HDF-EOS tends to stick coordinates in a different group,
for example. But I havent seen a use case where it makes something
impossible that really is needed. You just need to push common dimensions
up into a common parent.

John

On Thu, Sep 11, 2014 at 4:27 AM, Timothy Patterson 
timothy.patter...@eumetsat.int wrote:


 I spent an instructive evening reading through the previous discussions
 (thanks for the links, Corey) and the arguments for and against using
 hierarchal structures. I also re-read the CF conventions documents again
 (1.6 and 1.7 draft) and it seems the standard currently ignores groups
 rather than explicitly forbidding their use. It seems to me that a netCDF
 dataset with groups could still conform to the CF conventions as they are
 currently written, even with all the other restrictions that the standard
 imposes. I'd be interested in seeing and possibly helping with CF
 conventions for supporting the enhanced model.

 After reading the previous discussions, I thought it might be interesting
 to the list to explain our use of groups in netCDF products as it is
 somewhat different from the other cases that were discussed.

 Our netCDF datasets have to cope with a number of different needs from
 various parties - archive, end-users, higher-level processing,
 reprocessing, monitoring, etc. To keep things simple, we wanted a single
 format per instrument/processing level that is flexible enough to contain
 all the data or a subset of the data depending upon the consumer needs. To
 do this, we created a hierarchal data structure that encapsulates data in
 related, but independent groups. These groups can be present in or missing
 from the dataset as required by the needs of the consumer. So a level 2
 processing function might receive a product containing 20 instrument
 channels at 2 different resolutions, whereas the dissemination function
 might receive a product with just 5 of these channels at the lowest
 resolution. Both of these products are described by a single format
 specification.

 This model of including or omitting independent groups also supports other
 needs, for example being able to add data that is produced at irregular
 intervals but needs to be in the product when it is available. Also, by
 tagging groups with a specific attribute, we should also be able to have a
 single, generic method for end-users to be able to subset data retrieved
 from the archive without requiring specific knowledge of each netCDF
 product. They should be able to select only the tagged groups (which might
 correspond to instrument channels for instance) that they want in their
 retrieved datasets.

 This gives us a single, easily understood format definition that
 encompasses a wide variety of possible variations.

 Any feedback on the idiocy or genius of this (ab)use of the netCDF format
 is welcome.

 Thanks,

 Tim

 -

 Dr. Timothy Patterson
 Instrument Data Simulation
 Product Format Specification

 EUMETSAT
 Eumetsat-Allee 1
 64295 Darmstadt
 Germany
 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF Conventions and netCDF-4 enhanced model

[John Caron]

Hi Karl and all:

NetCDF-4 compression and chunking are transparent to the user, and are
compatible with the classic data model.

I think we should be gathering experiences with  the enhanced data model,
and start a CF-2.X convention draft document that uses the enhanced model.
It would also be a good time to remove deprecated features and in general
not require backwards compatibility. Perhaps if there are 5-6 people we
could start a working group to discuss.

John


On Wed, Sep 10, 2014 at 10:35 AM, Corey Bettenhausen 
corey.bettenhau...@ssaihq.com wrote:

 Tim,
 There was a discussion of this last year. See the archives:
 http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/author.html

 Particularly, the thread Towards recognizing and exploiting hierarchical
 groups:
 http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056827.html

 Cheers,
 -Corey

 On Sep 10, 2014, at 9:53 AM, Timothy Patterson wrote:

  Is it correct to say that, although they don't explicitly state it, the
 CF conventions (1.6 and the draft 1.7) restrict compliant netCDF products
 to be either netCDF-3 or netCDF-4 in classic format? There are no
 conventions for the enhanced features such as groups and user-defined types
 like enumerated variables, and Section 2.2, as an example, bars the use of
 unsigned integer variables or string variables (which are even stated not
 to exist, again implying classic-model only).
 
  There are some features of the enhanced model we want to use for our
 future datasets (such as groups) and some features which would make life
 easier but could be worked around if it led to CF compliance (enumerated
 types, unsigned integers, string types, etc.). Are there any plans to
 introduce conventions for the use of these enhanced features at some point
 in the future or would non-classic model datasets always be seen as
 non-compliant?
 
  Thanks for your insights on this issue!
 
  Regards,
 
  Tim Patterson
 
 
 
  -
 
  Dr. Timothy Patterson
  Instrument Data Simulation
  Product Format Specification
 
  EUMETSAT, Eumetsatallee 1, D-64295 Darmstadt, Germany
  ___
  CF-metadata mailing list
  CF-metadata@cgd.ucar.edu
  http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

 --
 Corey Bettenhausen
 Science Systems and Applications, Inc
 NASA Goddard Space Flight Center
 301 614 5383
 corey.bettenhau...@ssaihq.com

 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Date/time and leap seconds

[John Caron]

Hi Jim:

Thanks for explaining some of the details of TAI and UTC time standards.
Obviously Im not much of an expert on them. What I am 99 44/100 % sure of
is that udunits uses 60 seconds per minute no matter what, as does
joda-time, which the netcdf-java library uses. So thats what I mean by udunits
does not support leap seconds. Im unclear whether that means udunits and
joda-time are not really UTC.

You are right that its all about the conversion from time/date strings to
counts of seconds and back . I have advocated that ISO 8601 datetime
strings be an optional way to specify time coordinates in CF, with the
argument that they are unambiguous and easy enough for even rocket
scientists to understand ;) That hasnt been accepted in CF, although
netcdf-java will allow them, and one can always use them
as auxiliary coordinates, in addition to the required udunits secs since
epoch coordinate.

I think, like Jonathan, an additional calendar option in CF could be the
way forward here. Java 8 has a new date/time package (based on joda-time)
that will allow for correct UTC leap-seconds. So I think I can get them
into netcdf-java (eventually) if that was needed.

What do you think?

Regards,
John




On Thu, Jul 17, 2014 at 7:14 AM, Jim Biard jbi...@cicsnc.org wrote:

  John,

 I'm afraid that your response has confused me quite a lot. If you don't
 mind, please clarify what you mean by udunits does not support leap
 seconds. TAI does not use leap seconds. It is a running count of seconds
 since Jan 1, 1958 00:00:00 UTC. UTC time messes around with leap seconds in
 order to account for the fact that a day is not exactly 864000 seconds
 long, and that the rotation rate varies over time. The leap seconds are
 used to keep time/date strings synchronized to the diurnal cycle, much
 the same way that leap days are used to keep date strings synchronized to
 the Vernal Equinox.

 I think this problem may have more to do with the conversion from
 time/date strings to counts of seconds and back than anything about the
 counts of seconds themselves.

 Grace and peace,

 Jim

 A true count of seconds since an epoch matches the approach of TAI, not
 UTC.

 On 7/16/14, 5:58 PM, John Caron wrote:

 Hi Maik:

  Unfortunately, CF references the udunits package which is no longer
 being developed, at least for date/time coordinates.

  udunits does not support leap seconds.

  your best bet is to add an auxiliary time coordinate which uses leap
 seconds, eg TAI. your specialized software can use that

  also add a time coordinate which uses the standard non-leap seconds from
 udunits. visualization software will use this to display calendar dates, i
 think if you are careful this will only be wrong on a leap second.

  You could propose adding a leap second-aware calendar to CF.


 On Wed, Jul 16, 2014 at 2:48 PM, Maik Riechert maik.riech...@arcor.de
 wrote:

 Hi Jim

  I see where the potential for mismatches when using time values with
  unexpected encoding means that we should provide a way to declare what
  kind of time-since-epoch values are used in a particular time variable.

  And that's exactly the problem. The CDF website summarizes it quite
 nicely: Science data should have well-defined times to enable more
 accurate cross-comparison between missions and as documentation for
 future archive use. netcdf-CF is basically CDF before introduction of
 the CDF_TIME_TT2000 type.

 In the end, people use libraries to read datasets, and I'm pretty sure
 that the netcdf4 library in Python is used a lot, and this one assumes a
 certain time encoding, without actually having it specified within the
 CF conventions. And I find this all a bit worrying, but many people
 probably don't care as their time resolution is higher than 1 second.
 Still, I think this issue should be addressed, better sooner than later.
 Don't you think?

 Cheers
 Maik

 
  Grace and peace,
 
  Jim
 
  On 7/16/14, 4:06 AM, Maik Riechert wrote:
  Hi,
 
  after reading
  http://cdf.gsfc.nasa.gov/html/leapseconds_requirements.html I'm a bit
  confused on how CF handles times for the netcdf world. It seems as if
  the 'seconds since' convention is basically POSIX time in regards to
  how leap seconds are handled. Quoting wiki  Due to its handling of
  leap seconds, it is neither a linear representation of time nor a true
  representation of UTC. This means that any dataset with = 1 second
  resolution which covers the moment a leap second happens will have a
  problem encoding the time using the existing units convention.
 
  I first posted this issue on the netcdf4-python tracker:
  https://github.com/Unidata/netcdf4-python/issues/280
 
  Is this an issue which was discussed before and is there any solution?
 
  Cheers
  Maik

 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



 --
   [image: CICS-NC] http://www.cicsnc.org/ Visit us on
  Facebook

Re: [CF-metadata] Date/time and leap seconds

[John Caron]

Hi Maik:

Unfortunately, CF references the udunits package which is no longer being
developed, at least for date/time coordinates.

udunits does not support leap seconds.

your best bet is to add an auxiliary time coordinate which uses leap
seconds, eg TAI. your specialized software can use that

also add a time coordinate which uses the standard non-leap seconds from
udunits. visualization software will use this to display calendar dates, i
think if you are careful this will only be wrong on a leap second.

You could propose adding a leap second-aware calendar to CF.


On Wed, Jul 16, 2014 at 2:48 PM, Maik Riechert maik.riech...@arcor.de
wrote:

 Hi Jim

  I see where the potential for mismatches when using time values with
  unexpected encoding means that we should provide a way to declare what
  kind of time-since-epoch values are used in a particular time variable.

 And that's exactly the problem. The CDF website summarizes it quite
 nicely: Science data should have well-defined times to enable more
 accurate cross-comparison between missions and as documentation for
 future archive use. netcdf-CF is basically CDF before introduction of
 the CDF_TIME_TT2000 type.

 In the end, people use libraries to read datasets, and I'm pretty sure
 that the netcdf4 library in Python is used a lot, and this one assumes a
 certain time encoding, without actually having it specified within the
 CF conventions. And I find this all a bit worrying, but many people
 probably don't care as their time resolution is higher than 1 second.
 Still, I think this issue should be addressed, better sooner than later.
 Don't you think?

 Cheers
 Maik

 
  Grace and peace,
 
  Jim
 
  On 7/16/14, 4:06 AM, Maik Riechert wrote:
  Hi,
 
  after reading
  http://cdf.gsfc.nasa.gov/html/leapseconds_requirements.html I'm a bit
  confused on how CF handles times for the netcdf world. It seems as if
  the 'seconds since' convention is basically POSIX time in regards to
  how leap seconds are handled. Quoting wiki  Due to its handling of
  leap seconds, it is neither a linear representation of time nor a true
  representation of UTC. This means that any dataset with = 1 second
  resolution which covers the moment a leap second happens will have a
  problem encoding the time using the existing units convention.
 
  I first posted this issue on the netcdf4-python tracker:
  https://github.com/Unidata/netcdf4-python/issues/280
 
  Is this an issue which was discussed before and is there any solution?
 
  Cheers
  Maik

 ___
 CF-metadata mailing list
 CF-metadata@cgd.ucar.edu
 http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Are ensembles a compelling use case for group-aware metadata? (CZ)

[john caron]

On 9/19/2013 3:58 PM, Schultz, Martin wrote:

Now, here is another use case, which we haven't implemented yet - partly 
because we didn't see how it can be done in a CF consistent way:
While there has been a definition of a standard file layout for data from multiple 
stations (a contribution from Ben Domenico and Stefano Nativi if I am not mistaken), this 
concept cannot be applied to multiple aircraft flight data. The station data can be 
packaged together with help of a non-geophysical station coordinate, because 
all stations share the same time axis. With aircraft flights, the time axes often don't 
overlap, and forcing all data onto the superset of time would be a tremendous waste of 
space. Groups would seem as  the natural solution to this problem! Why not flat files? 
Because you might wish to retrieve all the aircraft data which were sampled in a given 
region during a specific period (a natural use case for a catalogue query it seems) in 
one entity, and not in N entities, where you cannot even predict N.

Hi Martin:

if i understand this example, you could use any of the DSG trajectory 
forms to have seperate time coordinates for each aircraft track, eg for 
multidimensional representation (4.1) :


 double time(trajectory, obs) ;

this one assumes you have (about) the same number of observation for 
each flight. The ragged array representation (4.3 and 4.4) would allow 
you to efficiently store any number of obs for each flight, and each obs 
would have its own time coordinate.


The DSG was indeed intended to enable spatial/temporal queries over the 
dataset. A natural organization might be to time partition the data 
(all flights for a day in one file). One either searches through the 
files for spatial subsets or employs an external  spatial index. The TDS 
will do spatial/temporal subsetting on time partitioned DSG files, 
though we dont yet have a spatial index.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Are ensembles a compelling use case for group-aware metadata? (CZ)

[john caron]

Hi all:

A few thoughts from my (possibly limited) POV.

1. Best strategy for storing very large collections of data in flat 
files like netcdf?


- store coherent chunks of the dataset in each file. Seems like a good 
file size these days is 100M - 1 Gbyte.
- choose coherence by deciding on the reader's most likely access 
pattern, assuming write once - read many.
- build an external index of the entire dataset to make access as fast 
as possible. the index can be rebuilt as needed without modifying the 
data files.


2. Whats the best use for groups in a netcdf file?

I think its best as a mimic to heirarchical file systems.  when there 
are more than a (few?) hundred files in a directory, the listing become 
unwieldy and hard to navigate. also, a file directory is a namespace 
that allows the same filename to be used in different directories. i 
think groups should be used in the same way for the same reasons. from 
this POV, an array of groups i think is not a good idea.


Zip files are interesting in that they have internal directories, and 
tools like 7zip and Windows file explorer transparently let you browse 
the OS file heirarchy and the internal zip heirarchy.


In THREDDS, we have found a good use for groups as follows. Given a 
collection of GRIB files, if there are multiple horizontal grids, we 
create a seperate group for each horizontal grid, and put all the 
variables that use that horizontal grid into that group. This creates a 
dataset with multiple groups. Each group is actually self-contained, so 
the TDS, when serving the dataset out,  will break the dataset back into 
multiple datasets, one for each group. This simplifies things for 
software that is not group aware.


A netcdf file is itself a container that creates a namespace for the 
variables within it. So you dont usually need groups for a file that has 
less than N variables in it (N = 50? 200??). The case of multiple 
coordinate systems is a reasonable exception, because often you have 
exactly the same variables in each group, so you need a seperate namespace.


One could apply this argument to ensemble data, and i think its 
reasonable. But as others have pointd out, you likely have much more 
data than you want to place into a single file. So then the file becomes 
the namespace and internal groups are less compelling.


Suppose we are talking about the logical structure of a multi-file 
collection? One could decide that internal groups are a good way to keep 
things clear, even though you perhaps never have more than one group in 
a file. Then, when you aggregate the files into a logical dataset, the 
user sees a structure that makes it clear what things are. One would 
also want to be able to specify such a structure in an external 
configuration that describes the collection. No doubt this issue will 
come back when CF works on conventions for multifile collections.


3. What does the TDS currently do with groups?

The TDS data model (aka CDM) handles groups just fine. You can aggregate 
variables in groups across files, as long as they are identical. We are 
waiting for DAP4 to be able to serve these though OPeNDAP, but the other 
services (WMS, WCS, NCSS, cdmremote)  work now with groups. OTOH, we 
dont see them much used yet, except in HDF-EOS.


4. Things are hard to predict, especially in the future.

I think software will evolve to handle the extended data model. I think 
data providers should start to use it, and CF could start examining best 
practices for extended data model Conventions (Perhaps CF-2 ?). That 
is to say, providers cant wait for CF conventions, they need to try new 
stuff that CF eventually incorporates (or not). Jim Baird's satellite 
data seems to be a good example of this.




On 9/29/2013 9:46 PM, Charlie Zender wrote:

Hi Steve,

Thank you for your continued engagement and responses.
Looks like CF hasn't the appetite for group hierarchies anytime soon.
I'll report the lessons learned in this fruitful discussion to our
NASA WG next week. My concluding (for now) remarks are interleaved.

Best,
Charlie

Le 25/09/2013 12:33, Steve Hankin a écrit :

On 9/24/2013 9:45 PM, Charlie Zender wrote:

It is not my place to determine whether there is a consensus, or how
close we are, but it's clear to me there is no consensus yet. Bryan
Lawrence, Steve Hankin, Jonathan Gregory, Karl Taylor, and Philip
Cameron-Smith are not on board. I hope they will speak-up and say if
they concur that maintaining the status quo (flat files) is best
(period), or whether they do wish to extend CF to hierarchies
(starting now), or the additional information they would need to decide.

Hi Charlie et. al.,

Since you have asked   I have heard two points that seemed to
bolster Bryan's pov that the multi-model use case is  great but not
compelling.  (See a more positive spin at the end.)

  1. file size.   Model outputs today are typically too large for even a
 single variable from a single model to be packaged in a 

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi Sean:

What feature of CF are you using that you need to preserve the version?

John

If you torture data long enough, it will confess.
  -- Economist Ronald Coase



On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote:

Hi,
Here at the British Oceanographic Data Centre we use THREDDS to deliver
and subset our numerical model data that we hold in CF netcdf format.
I’ve just been made aware that during the delivery and subsetting
process, THREDDS seems to be converting the CF files we hold from
whatever CF convention they have been supplied in, to CF 1.0.
Is this something that the rest of the community are aware of, and if
so, do people consider it to be of major importance that the files
outputted by thredds are in a different convention to the source files
thredds interrogates?
Cheers
Sean Gaffney
BODC

_   _
This message (and any attachments) is for the recipient only. NERC is
subject to the Freedom of Information Act 2000 and the contents of this
email and any reply you make may be disclosed by NERC unless it is
exempt from release under the Act. Any material supplied to NERC may be
stored in an electronic records management system.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi John and all:

For sure its a good idea to preserve the CF Conventions attribute if it 
exists. I will add that to our issue tracker.


Remember that THREDDS is a general purpose subsetter, not specific to 
CF. So we deal with lots of source datasets (eg GRIB) that arent CF. The 
point is that we convert whatever we have to CF.


Up to this point, as CF versions have evolved, there hasnt been any 
difference in the way that we write out these files. So im interested if 
Ive missed something and there is some actual change in the file 
structure that we should be tracking from Sean's POV. Other than the 
actual version number.


John


On 8/28/2013 11:11 AM, John Graybeal wrote:

It isn't just about feature presentation, right?  On the assumption that the 
latest version is always most current (so everyone will use the latest version 
if they can), I use the version as a proxy for how 
current/up-to-date/sophisticated the data provider is.

It also gives me a clue about which standard name table version they might use, 
though that version info is often available elsewhere.

If true, this may help explain why so few data providers seemed to be using 
1.6, anyway

John

On Aug 28, 2013, at 08:56, John Caron ca...@unidata.ucar.edu wrote:


Hi Sean:

What feature of CF are you using that you need to preserve the version?

John

If you torture data long enough, it will confess.
  -- Economist Ronald Coase



On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote:

Hi,
Here at the British Oceanographic Data Centre we use THREDDS to deliver
and subset our numerical model data that we hold in CF netcdf format.
I’ve just been made aware that during the delivery and subsetting
process, THREDDS seems to be converting the CF files we hold from
whatever CF convention they have been supplied in, to CF 1.0.
Is this something that the rest of the community are aware of, and if
so, do people consider it to be of major importance that the files
outputted by thredds are in a different convention to the source files
thredds interrogates?
Cheers
Sean Gaffney
BODC

_   _
This message (and any attachments) is for the recipient only. NERC is
subject to the Freedom of Information Act 2000 and the contents of this
email and any reply you make may be disclosed by NERC unless it is
exempt from release under the Act. Any material supplied to NERC may be
stored in an electronic records management system.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



John Graybeal
Senior Data Manager, Metadata and Semantics

T +1 (408) 675-5545
F +1 (408) 616-1626
skype: graybealski

Marinexplore
920 Stewart Drive
Sunnyvale, CA



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi Seth:

On 8/28/2013 12:59 PM, Seth McGinnis wrote:

Hi Sean,

Personally, I would regard that as suspect behavior.

I'm of the opinion that it's best practice for a data service to
affect the data it operates on in very targeted and transparent
way, and that it should pass everything else through untouched.
If it doesn't do that, it's harder to trust the service, because
the more ways that it modifies the data, the more opportunities
there are for it to get something wrong, and the more likely it
is that the error will go unnoticed because it's hiding amongst a
bunch of other changes.


We take this philosophy also.



Case in point, I have recently discovered that when the THREDDS
server subsets CF data that uses a coordinate system other that
lat-lon, it mangles a bunch of the ancillary data in the file.  I
suspect this happens, ironically enough, precisely because it's
trying to make the data CF-1.0 conformant instead of just leaving
it as-is...


Let us know what the specifics are and we can try to make it better.



So yes, I think it's significant that the data coming out of
THREDDS is in a different convention than the source files, and
that it's cause for concern.


A famous story from NCAR data portal early days is that some users added 
the Conventions = CF-1.0 attribute to their data because they had to 
make their data CF compliant. But thats all they did. Hey, it passed the 
compliance checker! ;)


Regards,
John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series lat/lon coordinates

[John Caron]

Hi Mark and Jonathan:

I will say straight up that I dont understand the implications of this 
particular discussion. But in case its helpful, I have found it 
important to separate the data model and the encoding of the model in 
the netCDF file. Probably you are both already doing that, but I wasnt sure.


John

On 6/7/2013 3:23 AM, Hedley, Mark wrote:

Hello Jonathan


As a result of this discussion, it seems me that for a DSG (which is indicated 
by the presence of featureType), scalar coordinate variables have to be 
interpreted as auxiliary coordinate variables of an omitted size-one instance 
dimension. That is what is implied by section 9.2. It's different from the 
interpretation that is implied by section 5.7,  which should exclude DSGs (and 
predates DSGs).


I don't think this approach holds together safely.  9.1 describes data 
variables and space-time (auxiliary) coordinate variables as varying with index 
dimensions.

The different feature types have different expectations for the relationship 
between dimensions, data and space-time coordinates.

9.2 describes the special case where the instance dimension is omitted and the 
space-time coordinates are scalar coordinate variables.

Interpreting sampling geometry scalar coordinates as a set of special cases, 
all different from the special cases of interpreting scalar coordinates in the 
absence of a featureType, is placing extensive complexity in the conventions 
and their interpretations.  I think this is asking too much of data producers 
and data consumers.

In many cases I think data producers have interpreted and will interpret the 
conventions (paraphrasing extensively) as 'these are just scalars'.

We also have to be mindful of the large selection of data sets with some 
sampling geometry like properties but which do not use the featureType 
attribute, e.g. from CF 1.5 or earlier.


I see no problem with having different interpretations for different purposes.


I see particular problems with this statement.  Treating the core types of the 
CF conventions differently depending on the context, such as by featureType and 
variable type introduces extensive complexity in data creation and 
interpretation, which I think CF should be very careful of. I do not think this 
is necessary.

Continuing and expanding the special casing for scalar coordinate variables in 
various contexts will cause significant issues for data consumers.

I am confident that scalar coordinate variable interpretation can be clearly 
defined in a manner consistent with the use of scalar coordinates in previous 
versions of the conventions.  I think that delivering a coherent definition of 
simple scalar coordinates for CF 1.7 will continue to support current usage and 
provide helpful clarification for future usage.

Examples 5.11, H4, H5 and H9 all use scalar coordinate variables in a way which 
I feel can be neatly supported by a simple definition of the scalar coordinate 
variable, distinct from vector coordinates.

Some more examples will help to clarify this for users of the conventions.

mark
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series lat/lon coordinates

[John Caron]

Hi Jonathan:

On 6/4/2013 4:17 AM, Jonathan Gregory wrote:

Dear John Caron and John Maurer

I agree with John C that the problem arises when the coordinate variables
are not size one. John M's example


   float lon(lon) ;
   float lat(lat) ;
   float alt(alt) ;
   float temp(time, alt, lat, lon) ;
   temp:standard_name = ?air_temperature? ;
   temp:units = Celsius ;
   temp:coordinates = time lat lon alt ;
   temp:_FillValue = -999.9;


with alt=lat=lon=1 is legal in CF. In fact the coordinates attribute is not
needed, because these are all (Unidata) coordinate variables (1D, with name
of the variable equal to the name of the dimension). Ignoring the coordinates
attribute, this example is fine in COARDS as well. In the data model, lon lat
alt time are all dimension coordinate constructs with separate domain axes.

But when there are *two* timeseries, you would not have alt=lat=lon=2. That
would mean three independent size-2 dimensions. This would also be legal in
CF and COARDS, but it means timeseries at a grid of 8 points, not two points.
To deal with this situation, we introduce an index dimension of size 2, and
make alt lat lon auxiliary coordinate variables of this single dimension. In
the data model, there is then only one domain axis for alt lat lon.

Back to the case of one timeseries: Example H4 shows scalar coordinate
variables for alt lat lon. That is, these size-1 dimensions have been omitted.
In this case, the coordinates attribute is needed; that's how scalar coordinate
variables are attached to the data variable. In the data model (in my opinion)
this is logically equivalent including the size-1 dimensions.


Lets see, its currently not legal as a DSG file, according to the spec. 
The CDM will barf on it, though I could put in a workaround.


Should it be legal? That is, should people be allowed to put in 
extraneous dimensions that only make sense for some values of the 
dimension length (eg 1 but not 1) ? I think it would be a rather ugly 
wart, and you would gain no new functionality.


I also think it would confuse dimensions with coordinates (which has 
already happened because the distinction isnt obvious). I think we 
should try to be clear about the use of dimensions because it makes the 
data model more powerful. So I would prefer not.




Maybe this question raises an issue for chapter 9 and Example H4. The example
is following Section 9.2:

If there is only a single feature to be stored in a data variable, there is no
need for an instance dimension and it is permitted to omit it. The data will
then be one-dimensional, which is a special (degenerate) case of the
multidimensional array representation.  The instance variables will be scalar
coordinate variables; the data variable and other auxiliary coordinate
variables will have only an element dimension and not have an instance
dimension, e.g. data(o) and t(o) for a single timeSeries.

In the multidimensional array representation, featureType doesn't have to be
coded, because this representation has always existed in CF. We could say that
*if* you encode featureType, there *must* be an instance dimension (of size 1
if appropriate) and that alt lat lon must be auxiliary coordinate variables
with this dimension. That would be a restriction we don't have in CF 1.6, so
it would be a change to CF. What do you think, John C?


I think this is a seperate question yes?

From my POV, its just a practical question to make things clear between 
data producer and consumer. I think we allowed the scalar instance 
coordinates because it was a natural way for producers to think when 
there was only one feature in the file, ie why do you want me to add 
this extra dimension? As long as the featureType attribute is 
present, as well as the coordinates attribute I think the meaning is 
unambiguous. Requiring a dimension=1 is maybe a bit simpler, but i would 
still have to deal with the scalar case for versions before we change, 
so its not really better for me.


John Caron
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series lat/lon coordinates

[John Caron]

Hi Jonathan:

If we use the time series featureType as example

(from 
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8307552)

AFAIU, the orthogonal multidimensional representation would be:

  float humidity(station,time)

not 

  float humidity(lat, lon, time)


Regards,
John


On 6/4/2013 11:33 AM, Jonathan Gregory wrote:
 Dear John C
 
 I think the two questions are linked actually. There is nothing wrong with
 the size-1 dimensions in general, I would say. You could see it as a single
 point extracted from a larger gridded field (time,atl,lat,lon) in which all
 the dimensions were originally greater than one. It's legal in the orthogonal
 multidimensional representation, which is the one representation which has
 always existed in CF. When we added ch9, we brought that representation into
 the same chapter.
 
 If the CDM doesn't like the orthogonal multidimensional representation for
 DSG, we could disallow featureType if that representation is used. Then the
 CDM wouldn't attempt to process it as a DSG. It would be a pity to lose the
 extra metadata that the featureType provides, though.
 
 Best wishes
 
 Jonathan
 
 
 - Forwarded message from John Caron ca...@unidata.ucar.edu -
 
 Date: Tue, 4 Jun 2013 10:54:45 -0600
 From: John Caron ca...@unidata.ucar.edu
 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509
  Thunderbird/17.0.6
 To: cf-metadata@cgd.ucar.edu
 Subject: Re: [CF-metadata] CF-1.6 DSG clarification: time series  lat/lon
  coordinates

 Hi Jonathan:

 On 6/4/2013 4:17 AM, Jonathan Gregory wrote:
 Dear John Caron and John Maurer

 I agree with John C that the problem arises when the coordinate variables
 are not size one. John M's example

float lon(lon) ;
float lat(lat) ;
float alt(alt) ;
float temp(time, alt, lat, lon) ;
temp:standard_name = ?air_temperature? ;
temp:units = Celsius ;
temp:coordinates = time lat lon alt ;
temp:_FillValue = -999.9;

 with alt=lat=lon=1 is legal in CF. In fact the coordinates attribute is not
 needed, because these are all (Unidata) coordinate variables (1D, with name
 of the variable equal to the name of the dimension). Ignoring the 
 coordinates
 attribute, this example is fine in COARDS as well. In the data model, lon 
 lat
 alt time are all dimension coordinate constructs with separate domain axes.

 But when there are *two* timeseries, you would not have alt=lat=lon=2. That
 would mean three independent size-2 dimensions. This would also be legal in
 CF and COARDS, but it means timeseries at a grid of 8 points, not two 
 points.
 To deal with this situation, we introduce an index dimension of size 2, and
 make alt lat lon auxiliary coordinate variables of this single dimension. In
 the data model, there is then only one domain axis for alt lat lon.

 Back to the case of one timeseries: Example H4 shows scalar coordinate
 variables for alt lat lon. That is, these size-1 dimensions have been 
 omitted.
 In this case, the coordinates attribute is needed; that's how scalar 
 coordinate
 variables are attached to the data variable. In the data model (in my 
 opinion)
 this is logically equivalent including the size-1 dimensions.

 Lets see, its currently not legal as a DSG file, according to the
 spec. The CDM will barf on it, though I could put in a workaround.

 Should it be legal? That is, should people be allowed to put in
 extraneous dimensions that only make sense for some values of the
 dimension length (eg 1 but not 1) ? I think it would be a rather
 ugly wart, and you would gain no new functionality.

 I also think it would confuse dimensions with coordinates (which has
 already happened because the distinction isnt obvious). I think we
 should try to be clear about the use of dimensions because it makes
 the data model more powerful. So I would prefer not.


 Maybe this question raises an issue for chapter 9 and Example H4. The 
 example
 is following Section 9.2:

 If there is only a single feature to be stored in a data variable, there 
 is no
 need for an instance dimension and it is permitted to omit it. The data will
 then be one-dimensional, which is a special (degenerate) case of the
 multidimensional array representation.  The instance variables will be 
 scalar
 coordinate variables; the data variable and other auxiliary coordinate
 variables will have only an element dimension and not have an instance
 dimension, e.g. data(o) and t(o) for a single timeSeries.

 In the multidimensional array representation, featureType doesn't have to be
 coded, because this representation has always existed in CF. We could say 
 that
 *if* you encode featureType, there *must* be an instance dimension (of size 
 1
 if appropriate) and that alt lat lon must be auxiliary coordinate variables
 with this dimension. That would be a restriction we don't have in CF 1.6, so
 it would be a change to CF. What do you

Re: [CF-metadata] CF-1.6 DSG clarification: time series lat/lon coordinates

[John Caron]

Hi John:

1) The coordinates attribute is not new to DSG, it has been around since 
the beginning of CF. This defines the auxiliary coordinate variables, 
that is, coordinate variables that do not follow lat(lat) template. I 
think it would be good if Grads et al could be upgraded to use them.


2) DSG data is 1 dimensional. The addition of extra dimensions of length 
1 seems harmless when you know you can ignore them, but is wrong when 
you want dimensions to represent independent coordinates of the 
sampling. To see that, ask yourself what the data variables would look 
like if the dimension had length 2.


  float humidity(sample) -  float humidity(sample, alt, lat, lon);

  float humidity(sample, 1, 1, 1); // seems ok
  float humidity(sample, 2, 2, 2); // huh??

it looks like a grid, and its not. there are not 2 x 2 x 2 independent 
coordinates. point data is not degenerate grid data. it has a different 
connectedness between the indices, and its this connectedness that makes 
multidimensional arrays much faster than, say, an RDBM, so throw that 
away at your peril.


Im interested in hearing what Jonathan et al's data model would say 
about this. I can tell you that the CDM/TDS feature collection layer 
would break if you added the extraneous dimensions.


John


On 5/29/2013 12:35 PM, John Maurer wrote:

Hi all,
We ran into a glitch after converting one of our buoys to the new CF-1.6
discrete sampling geometries (DSG) format, and I'm looking for advice.
This dataset uses the single time series format, like the one provided
in the template of the CF document in Example H.4:
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8320208.
The problem we discovered is that the dataset is no longer readable in
software clients like GrADS and Dapper/DChart (likely others) because it
can't find the latitude and longitude coordinates anymore.

While the dataset previously included the lat/lon dimensions of the buoy
in NetCDF coordinate variables, CF-1.6 DSG no longer defines lat/lon as
coordinate variables. Instead, they are scalar variables which are
referenced by other variables (temp, salinity, etc.) via a new
variable-level coordinates attribute: e.g.

temp:coordinates = time lat lon alt ;

In order to allow the DSG datasets to continue working in existing
software clients that do not know yet to look for this new coordinates
variable attribute, does it break CF-1.6 DSG compliance to *also* change
the lat/lon scalars to coordinate variables like they used to operate?
Building on Example H.4, the modified dataset would look like this,
which is a melding of the old and new ways:

dimensions:
   time = 100233 ;
   name_strlen = 23 ;
   alt = 1 ;
   lat = 1 ;
   lon = 1 ;
variables:
   float lon(lon) ;
   lon:standard_name = longitude;
   lon:long_name = station longitude;
   lon:units = degrees_east;
   float lat(lat) ;
   lat:standard_name = latitude;
   lat:long_name = station latitude ;
   lat:units = degrees_north ;
   float alt(alt) ;
   alt:long_name = vertical distance above the surface ;
   alt:standard_name = height ;
   alt:units = m;
   alt:positive = up;
   alt:axis = Z;
   char station_name(name_strlen) ;
   station_name:long_name = station name ;
   station_name:cf_role = timeseries_id;

   double time(time) ;
   time:standard_name = time;
   time:long_name = time of measurement ;
   time:units = days since 1970-01-01 00:00:00 ;
   time:missing_value = -999.9;
   float humidity(time, alt, lat, lon) ;
   humidity:standard_name = “specific_humidity” ;
   humidity:coordinates = time lat lon alt ;
   humidity:_FillValue = -999.9;
   float temp(time, alt, lat, lon) ;
   temp:standard_name = “air_temperature” ;
   temp:units = Celsius ;
   temp:coordinates = time lat lon alt ;
   temp:_FillValue = -999.9;

attributes:
   :featureType = timeSeries;



Thanks for any insights on this issue!
Cheers,
John Maurer
Pacific Islands Ocean Observing System (PacIOOS)
University of Hawaii at Manoa




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG Trajectory files: clarification and advice

[John Caron]

Hi Derrick:

Some first reactions:

1. the aggregation or collection will have two trajectories, the first 
is along a time axis time and the second is along a time axis time_avg.


since they are both trajectories, then i think you dont have a problem 
with more than one feature type.


however, theres no convention which says how the two trajectories are 
related. they might be though of as being along the same trajectory, 
with different sampling points. this would be useful for CF coordinate 
systems to capture in a general way.


2. The THREDDS server is implementing with feature collections. You 
have an interesting case here of creating essentially 2 feature 
collections, each collection in multiple files, but the 2 collections 
share the same files.


Maybe I can get some sample files and see how hard it would be to extend 
the TDS.


3. is it legal to have a variable with a valid geometry AND a variable 
that is the result of a cell_method applied to that variable in the same 
file?


i think its legal, but the question is, does any standard software able 
to understand it? Generally, from my POV, a main use of conventions is 
to allow software to understand these relationships.


John

On 5/21/2013 11:11 AM, Derrick Snowden - NOAA Federal wrote:

All,

We are attempting to create a valid CF 1.6 discrete sampling geometry
trajectory file to exchange data from underwater profiling gliders.
There is a schematic of the major sampling features for gliders at the
following web site
(https://github.com/IOOSProfilingGliders/Real-Time-File-Format/wiki/Glider-Types-and-Sampling#sampling-patterns).
There are a few variations of the files we are working on in the
following github repo
(https://github.com/IOOSProfilingGliders/Real-Time-File-Format/tree/master/examples/proposed_templates/trajectory_uv),
andglider_trajectory_uv_template_v.0.0.cdl
https://github.com/IOOSProfilingGliders/Real-Time-File-Format/blob/master/examples/proposed_templates/trajectory_uv/glider_trajectory_uv_template_v.0.0.cdl
is file in question.

We are grappling with the constraint in CF that each file must contain
one and only one feature type.  (Section 9.1 para 1:/the features
contained within a collection must always be of the same type; and all
the collections in a CF file must be of the same feature type/. )  As
it is written, this constraint doesn't say anything about the shape or
dimensionality of the features, only that they are the same type.  Below
I'll try to describe our use case which differs from the examples in
chapter 9 in two ways.
1. The files only contain data from a single glider so the trajectory
variable is always a scalar (i.e. the variable called trajectory with
trajectory:cf_role=trajectory_id is usually just set to one in these tests).
2. The file only contains part of a complete deployment and we need to
aggregate them together into a complete deployment (one deployment may
contain hundreds or thousands of segments) but, each of these segments
still has trajectory=1.

As stated, our files will contain one segment as defined on the web
site above.  A segment is defined as the data collected between two
surface gps fixes of the glider.  When the glider submerges it is
collecting data (temp/sal etc) as a function of time.  But, underwater
there are no gps fixes so the lat/lon is estimated with a dead reckoning
algorithm.  So far we think we've encoded that correctly.  Our problem
is with data that is estimated as an average between gps fixes.

For example, when the glider surfaces at the end of a segment there are
two position fixes available (1: the gps position which is considered
true, and 2: the dead reckoning position which is estimated) The
difference between these two positions can be used to estimate the
depth/time averaged currents that pushed the glider off course during
it's trajectory underwater.  We'd like to encode these depth averaged
currents in the same file as the measured T/S etc but this data has a
different time axis and therefore a different lat/lon coordinate.  In
practice the depth averaged currents are assigned to the midpoint (time
and space) of the subsurface trajectory.  Our goal is for aggregations
of the segment files to represent an entire glider deployment and if
we're successful the aggregation or collection will have two
trajectories, the first is along a time axis time and the second is
along a time axis time_avg.  They are both trajectories but they have
different time axes.  I'm not sure if THREDDS will allow this type of
aggregation so some changes may be necessary.

Does this violate the intent of the convention?  If so, can anyone
suggest a better encoding?  We're trying to keep these two data sets in
the same file but creating two files is possible if not desirable.

A similar related question could be, is it legal to have a variable with
a valid geometry AND a variable that is the result of a cell_method
applied to that variable in the same file?  e.g. a time 

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

On 3/21/2013 9:41 AM, Steve Hankin wrote:


On 3/21/2013 8:25 AM, John Caron wrote:

Probably my proposal comes down to 2 parts, which are separable:

1. Find a suitable replacement for udunits as a reference library for 
CF calendar dates. Unfortunately, udunits used a slightly 
non-standard syntax, which we need to still be legal for backwards 
compatibility. So its unlikely a standard library would work off the 
shelf. Someone needs to commit to making a library that supports CF 
as it evolves. Other libraries can test against it.


2. *Allow* 


Hi John,

The crux of this debate has been whether to allow a new (for CF), 
redundant encoding of time coordinates.  I *think* you are suggesting 
in your bullet #2 to allow the new syntax only in ancillary variables 
(metadata), but not as a representation of coordinates.  Can you clarify?


- Steve

a suitable string syntax for date/times, probably expressed as a 
profile of ISO8601. Its a modest but worthwhile improvement IMO. The 
syntax that is already used in udunits would be ok with me. However 
this syntax has to be defined (it never was, unless you are yacc). In 
my understanding, udunits is more or less a superset of ISO. that is, 
its a bit more lenient in the syntax. However it adds a few 
restrictions, like defaulting to UTC and filling non specified fields 
with 0. So its not strictly a superset. The advantage of using CF 
process to define the syntax (clarify the profile of ISO8601) is 
that we are less likely to miss some subtlety in syntax and meaning.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




Hi Steve:

Im proposing that time coordinates may be expressed as strings, in 
addition to the current way of using time offsets from a base date 
expressed as a string in the unit attribute. The two date string 
syntaxes (standalone and in the unit attribute)  c/should be the same.


Presumably we want to do either this or Aleksandar's proposal ?

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi all:

Another thing to consider is the relationship to the current udunit/CF 
standard for specifying dates in the unit string


  period SINCE date

The udunits documentation 
http://www.unidata.ucar.edu/software/udunits/udunits-2/udunits2lib.html#Grammar 
not being very clear, I wrote up my understanding of current practice here:


http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html#ISO_String

Note the paragraph towards the end starting with with the following 
differences, to allow backwards compatibility with udunits:


as this documents the differences AFAICT that current practice has with 
ISO 8601 standard.


It appears that this grammer is slightly more lenient than the one that 
Aleksandar proposes.


John



On 3/19/2013 6:21 PM, Dave Allured - NOAA Affiliate wrote:

Aleksandar,

I support this standard name proposal.  I like the restrictions on
syntax and use to a sensible subset for scientific applications.

Unlike most standard name proposals, this is more than a simple
definition of a physical meaning.  You are making a set of technical
use specifications, with significant departures from the reference
standard ISO 8601.  Will there be a reference archive for the
particular specifications?  Will this reside exclusively in the CF
standard name table, or in a new CF chapter or appendix, or something?
  I think there needs to be something better than that CF list
discussion of January to March 2013.

Please be exact in the specification of the reference calendar.  Try
to eliminate all possible measurement ambiguities.  Leap seconds are a
problem.  There are no leap seconds in the standard calendars
currently defined by CF, and I was not able to locate exactly where
ISO8601 stands on this issue.  I would really hate to see a Gregorian
calendar or proleptic_gregorian implementation with interval
calculations that differ from the CF defined calendars.

I suggest a requirement that the calendar attribute be included on all
variables using datetime_iso8601.  If leap seconds are excluded, then
the correct attribute value should be proleptic_gregorian.

You have four-digit years.  Are you explicitly restricting the spec
to only years  through ?

Thank you for working on this!

--Dave

On Fri, Jan 11, 2013 at 10:00 AM, Aleksandar Jelenak - NOAA Affiliate
aleksandar.jele...@noaa.gov wrote:

Dear All:

Here's the modified proposal for the datetime_iso8601 standard name:

standard_name: datetime_iso8601

Units: N/A

String representing date-time information according to the ISO
8601:2004(E) standard. Variables with this standard name cannot serve
as coordinate variables. Date-time information is in the Gregorian
calendar. For dates preceding the Gregorian calendar the date-time
information is in the proleptic Gregorian calendar. Possible date-time
string forms are:

datetime = date T time timezone ;

date =  - MM - DD |  - DDD ;

time = hh | hh : mm | hh : mm : ss | hh : mm : ss . S
   | hh : mm : ss , S ;

timezone =  | Z | + hh | + hh : mm | - hh | - hh : mm

Where:

*  is a four-digit year (-).

* MM is a two-digit month of the year (01-12).

* DD is a two-digit day of the month (01-31).

* DDD is a three-digit ordinal day of the year (001-366).

* hh is a two-digit hour (00-23).

* mm is a two-digit minute (00-59)

* ss is a two-digit second (00-59).

* S is one or more digits representing a decimal fraction of the
  second.

* The value of any designator when not specified is zero.

* If timezone is ommitted the default value is Z.


-Aleksandar

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

On 3/20/2013 9:41 AM, David Hassell wrote:

Hello,

My beer/coffee/perocet levels are too low to want to comment broadly on
this, so I'll just make one comment ...


Really the main advantage is that data writers are less likely to
make a mistake specifying

1952-08-15T00:00:00Z

than

2434567 days since -4713-01-01T00:00:00Z.

I'm not sure why the later would be any more prone to mistakes than
the former. Surely the conversion of

   time-as-stored-in-whatever-arcane-format-files-my-model-spits-out

 to

   '2434567 days since -4713-01-01T00:00:00Z'

would be handled by a software library, just as would the conversion
of

   time-as-stored-in-whatever-arcane-format-files-my-model-spits-out

 to

   '1952-08-15T00:00:00Z'

and its fair to assume, I think, that such libraries will do the right
thing.


I guess the point is that its not always fair to assume that, and the 
user wont know when it fails, esp for


'2434567 days since -4713-01-01T00:00:00Z'

unless she also computes

'1952-08-15T00:00:00Z'

which presumably she could do as a double check.


Which library would you use? Does it work for non-standard calendars?

How about if everyone who'g got nothing better to do try it out in the 
library of their choice and report the results?


John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi All:

Ok, its friday afternoon so ill bite on this, and wax philosophical even 
before the beer.


An insidious mistake is to think that problems should be fixed in 
software libraries. Actually the deep mistake is to mistake the 
reference software with the file encoding. Why bother fixing the 
encoding when a few lines in _your_ software can fix the problem 
transparently? Ive seen this occur in all three of the great western 
religious systems of our day: netCDF, HDF and OPeNDAP libraries.


Better is to do the encoding of information as cleanly as possible. 
Post-apocalyptic software engineers who have lost all knowledge of what 
netCDF and CF mean and are painstakingly uncovering climate archives 
with their whisk brooms will thank us.


35246 hours since 1970-01-01 isnt just unreadable; it uses a calendar 
system that may be non-trivial. Calendars are hard; Java has got it 
wrong already twice, and is now trying for a 3rd time (with jsr 310 in 
Java 8, based on experience with joda-time).


1974-01-08T14:00:00Z ( == 35246 hours since 1970-01-01 in the 
standard calendar) is a better representation of that date. because at 
least you know what the user thought the damn date was.


The good argument for 35246 hours since 1970-01-01 representation, is 
that given two of them, at least you know what the user thought the damn 
time interval is between them.


Anyway, I think both are good, and should be allowed. Finish your beer 
and ill order another round.


John

PS: NEMO files use time units of days since -4713-01-01T00:00:00Z. 
Dont know why, and no disrespect intended, but I cant say I like it.



On 3/15/2013 4:30 AM, Hattersley, Richard wrote:

Hi all,
I think Steve has got to the heart of the issue with his comment, IMHO
it is the client libraries that hold the answer to this question.
However you choose to peer into your netCDF files you are seeing them
through the lens of a client library. And it's worth noting that
date/times aren't special in this regard, this is just as true for
floating point numbers.
With the Iris library(*), we are working towards improving the
readability and usability of date/times because we recognise that simply
displaying 35246 hours since 1970-01-01 is essentially meaningless to
a human. From what I've seen, I think we can all agree on that! So we
want our users to be able to see date/times in a human-readable form,
but that certainly doesn't mean we will represent them that way internally.
The only real benefit I can see for the string representation is that
_some_ date/time values can be made more human-readable when viewed
through client libraries which don't support the
machine-readable-to-human-readable conversion. But if the library can't
do machine-to-human then it probably can't do human-to-machine. In which
case there's very little you can actually _do_ with the date/time values
(e.g. determine ordering or compute intervals). If the library is that
limited then adding the string representation to CF isn't really fixing
the right problem. If you'll excuse the analogy, it's like taking the
engine out of a car because the brakes don't work.
In short, I am against the addition of a string representation for
date/times.
*) Iris - a candidate reference implementation of CF implemented in
Python (http://scitools.org.uk/).
Richard Hattersley Iris Benevolent Dictator
*Met Office*  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Tel: +44 (0)1392 885702
Email: richard.hatters...@metoffice.gov.uk
mailto:richard.hatters...@metoffice.gov.uk  Web: www.metoffice.gov.uk
http://www.metoffice.gov.uk/


*From:* CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] *On Behalf
Of *Steve Hankin
*Sent:* 24 February 2013 19:07
*To:* John Caron
*Cc:* cf-metadata@cgd.ucar.edu
*Subject:* Re: [CF-metadata] New standard name: datetime_iso8601


On 2/23/2013 1:41 PM, John Caron wrote:

Hi Chris, and all:

On 1/11/2013 2:37 PM, Chris Barker - NOAA Federal wrote:

On Fri, Jan 11, 2013 at 9:00 AM, Aleksandar Jelenak - NOAA Affiliate
aleksandar.jele...@noaa.gov wrote:


Here's the modified proposal for the datetime_iso8601 standard name:

...

String representing date-time information according to the ISO
8601:2004(E) standard.


I think we should NOT adopt a string option for datetime variables.

To quote Jonathan Gregory:


In CF we have always applied the
principle that we only add to CF when there is a need to do so, i.e.
there is
a use-case for something which cannot already be represented in CF


We already have a way to encode datetimes in CF-netcdf.


Yes, but time since date is not as good as date as an encoding.
The time since date is a result of cramming calendar handling into a
units package.

I would advocate both should be allowed.


Hi John,

The bell is ringing, round three on the ISO dates issue.

The arguments *for *supporting ISO dates are:

 1. they are the clear standard for date/time

Re: [CF-metadata] brand new user

[John Caron]

On 3/14/2013 6:39 AM, Izidor Pelajic wrote:

Hi all,
I'm new at netCDF, new at CF, and moderate MATLAB user. I'm new at
netCDF in a sense of days or week.
So my problem (beyond the fact that I do not have general knowledge)
is next. I was provided with a .nc file. In it there are 3 information
of interest: latitude matrix 160 by 22, longitude matrix 160 by 22 and
SST matrix also 160 by 22. File is created from NOAA satellite data
record (from hdf file, but not by me). I need to convert this custom
made nc file to netCDF CF file.
So far, I managed to open nc file in matlab, edit variables, make some
nice charts and save it as it is. But I do not know how to save in CF
convention. My problem is longitude and latitude that are of same
dimension as variable SST. How do I make sst( lon, lat ) if all lon
and lat elements are of different value? (it was written down in line
as satellite passed over an area)

Any idea, or example?
Izidor
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


Hi Izidor:

i presume that you need to  compute the 2D lat and lon coordinates and 
add them to the file.

If you think of renaming your dimensions like this, perhaps it is easier:

dimension
  scan = 160;
  xscan = 22

variables:
  float sst(scan,xscan);
sst:coordinate = lat lon;

  float lat(scan,xscan);
lat:units= degrees_north;

  float lon(scan,xscan);
lat:units= degrees_east;


the

  sst:coordinate = lat lon;

means add an attribute to the sst variable with name coordinate and 
value lat lon. This is how CF associates the lat and lon coordinates 
to the data variable. The units attribute on lat and lon is how CF 
knows what they are (i.e. not by the variable name, which can be anything).


this is just the minimum, of course you might want to add standard_name 
attribute, etc.


Good luck!

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Heiko:

So

grid_north_pole_longitude = 
normalizeLongitude180(longitudeOfSouthernPoleInDegrees)

grid_north_pole_latitude = -1 * longitudeOfSouthernPoleInDegrees

?

where does one find documentation on proj4's ob_tran routine?

thanks,
John



On 3/6/2013 8:03 AM, Heiko Klein wrote:

Hi John,

all our models use rotated_latitute_longitude. And we have a 
consistent way of translating it between grib, proj and netcdf-java, 
i.e. publicly available at:

http://thredds.met.no/thredds/catalog/metno/proff4km/default/catalog.html


grid_mapping_name: rotated_latitude_longitude
grid_north_pole_longitude: 156.0
grid_north_pole_latitude: 23.5
earth_radius: 6367470.0

to proj:
proj4: +proj=ob_tran +o_proj=longlat +lon_0=-24 +o_lat_p=23.5 
+a=6367470.0 +no_defs


from grib:
lonRot = longitudeOfSouthernPoleInDegrees
latRot = latitudeOfSouthernPoleInDegrees

+proj=ob_tran +o_proj=longlat +lon_0=  
normalizeLongitude180(lonRot)   +o_lat_p=  (-1 * latRot);




This is the set of parameters which have been useful. I'm not sure if 
they are named in a useful way. proj4's ob_tran even allows for a 
o_lon_p parameter, but this doesn't seem to be used by grib or CF.


Best regards,

Heiko



On 2013-03-06 15:18, John Caron wrote:

The Rotated Pole projection here:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#appendix-grid-mappings 




Rotated pole

grid_mapping_name = rotated_latitude_longitude


Map parameters:

  *

grid_north_pole_latitude

  *

grid_north_pole_longitude

  *

north_pole_grid_longitude - This parameter is option (default 
is 0).


Map coordinates:

The rotated latitude and longitude coordinates are identified by
the standard_name attribute values
grid_latitude and grid_longitude respectively.



is problematic. It has no reference to Proj.4, and is not in Snyder.

Its likely the same as the GRIB-2 GDS Template 3.1:

Grid Definition Template 3.1: Rotated Latitude/longitude (or
equidistant cylindrical, or Plate Carrée)

Octet No. Contents

   15-72 Same as Grid Definition Template 3.0 (see Note 1)

   73-76 Latitude of the southern pole of projection

77-80 Longitude of the southern pole of
projection

   81-84 Angle of rotation of projection

   85-nnList of number of points along
each meridian or parallel (These octets are only present for
quasi-regular grids as described in Note 3)

Notes:


(2) Three parameters define a general latitude/longitude
coordinate system, formed by a general rotation of the sphere. One
choice for these parameters is:

(a) The geographic latitude in degrees of the southern pole of
the coordinate system, θp for example.

(b) The geographic longitude in degrees of the southern pole of
the coordinate system, λp for example.

(c) The angle of rotation in degrees about the new polar axis
(measured clockwise when looking from the southern to the northern pole)
of the coordinate system, assuming the new axis to have been obtained by
first rotating the sphere through λp degrees about the geographic polar
axis, and then rotating through (90 + θp) degrees so that the southern
pole moved along the (previously rotated) Greenwich meridian.


Except note that GRIB-2 uses southern pole of projection. Not sure how
that affects the algorithm.

The CF parameter north_pole_grid_longitude likely should be Angle of
rotation.



Anyone using this or have some insight on it ??




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Burkhardt:

Of course we cant change the existing conventions as there are files 
that already use it. But we need to document it so that we know when we 
are doing it right.


Im cleaning up my code and noticed that i have two implementations, one 
for grib and one for CF. So Id like to merge the two.


John

On 3/6/2013 12:32 PM, Burkhardt Rockel wrote:

John,

the rotated pole definition was requested by me in 2002 (see the thread 
projections/default cell_methods). In this thread the present description was 
chosen.
Please note: the rotated pole grid mapping is NOT a PROJECTION!  it is a 
TRANSFORMATION (therefore the GRIB description is misleading). This is 
probably the reason why you did not find it in Proj4.
The a, b, c in the GRIB definition description is fine. Whether  the rotated 
South Pole (grib) or North Pole (CF-Conventions) is used in the definition does 
not matter, the values can be easily converted to each other.

The rotated pole grid mapping as defined in the CF-Conventions document is 
already used widely as netCDF output standard for regional climate models in 
several international projects. It is also the standard in the IPCC-AR5 CORDEX 
model output for regional climate models. Therefore I am not in favor of 
changing the definition. However, a more precise description in the 
CF-Conventions would certainly be helpful for those not familiar with the 
rotated pole transformation.

Regards
Burkhardt

---
Dr. Burkhardt Rockel
Helmholtz-Zentrum Geesthacht
Institute of Coastal Research / Group Regional Atmospheric Modeling
Max-Planck-Strasse 1
D-21502 Geesthacht
Germany
Phone: +49 4152 87 1803
Fax: +49 4152 87 4 1803
Email: Burkhardt.Rockel (at) hzg.de
www: http://rockel.staff.coast.hzg.de
coordinates: 53.40579 N, 10.428647 E
---


Am 06.03.2013 um 17:09 schrieb John Caron ca...@unidata.ucar.edu:


Hi Heiko:

So

grid_north_pole_longitude = 
normalizeLongitude180(longitudeOfSouthernPoleInDegrees)
grid_north_pole_latitude = -1 * longitudeOfSouthernPoleInDegrees

?

where does one find documentation on proj4's ob_tran routine?

thanks,
John



On 3/6/2013 8:03 AM, Heiko Klein wrote:

Hi John,

all our models use rotated_latitute_longitude. And we have a consistent way of 
translating it between grib, proj and netcdf-java, i.e. publicly available at:
http://thredds.met.no/thredds/catalog/metno/proff4km/default/catalog.html


grid_mapping_name: rotated_latitude_longitude
grid_north_pole_longitude: 156.0
grid_north_pole_latitude: 23.5
earth_radius: 6367470.0

to proj:
proj4: +proj=ob_tran +o_proj=longlat +lon_0=-24 +o_lat_p=23.5 +a=6367470.0 
+no_defs

from grib:
lonRot = longitudeOfSouthernPoleInDegrees
latRot = latitudeOfSouthernPoleInDegrees

+proj=ob_tran +o_proj=longlat +lon_0=  normalizeLongitude180(lonRot)   
+o_lat_p=  (-1 * latRot);



This is the set of parameters which have been useful. I'm not sure if they are 
named in a useful way. proj4's ob_tran even allows for a o_lon_p parameter, but 
this doesn't seem to be used by grib or CF.

Best regards,

Heiko



On 2013-03-06 15:18, John Caron wrote:

The Rotated Pole projection here:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#appendix-grid-mappings


Rotated pole

grid_mapping_name = rotated_latitude_longitude


Map parameters:

  *

grid_north_pole_latitude

  *

grid_north_pole_longitude

  *

north_pole_grid_longitude - This parameter is option (default is 0).

Map coordinates:

The rotated latitude and longitude coordinates are identified by
the standard_name attribute values
grid_latitude and grid_longitude respectively.



is problematic. It has no reference to Proj.4, and is not in Snyder.

Its likely the same as the GRIB-2 GDS Template 3.1:

Grid Definition Template 3.1: Rotated Latitude/longitude (or
equidistant cylindrical, or Plate Carrée)

Octet No. Contents

   15-72 Same as Grid Definition Template 3.0 (see Note 1)

   73-76 Latitude of the southern pole of projection

77-80 Longitude of the southern pole of
projection

   81-84 Angle of rotation of projection

   85-nnList of number of points along
each meridian or parallel (These octets are only present for
quasi-regular grids as described in Note 3)

Notes:


(2) Three parameters define a general latitude/longitude
coordinate system, formed by a general rotation of the sphere. One
choice for these parameters is:

(a) The geographic latitude in degrees of the southern pole of
the coordinate system, θp for example.

(b) The geographic longitude in degrees of the southern pole of
the coordinate system, λp for example.

(c) The angle of rotation in degrees about the new polar axis
(measured clockwise when looking from the southern to the northern pole)
of the coordinate system, assuming the new axis to have been obtained by
first rotating the sphere

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Seth:

Thanks for the description, its really helpful.

Im just wondering how we document this in the CF convention, so that 
implementers have something to check against. Is there a reference 
implementation we can use ?


John

On 3/6/2013 1:17 PM, Seth McGinnis wrote:

So there are an infinite number of ways to decompose a 3-D orientation into
three orthogonal rotations.  Specifying the projection in terms of the final
location of the north pole and an optional rotation around that point means
that CF is actually agnostic about exactly how you get there.

That said, given that it's being specified in that way, the decomposition that
maps most straightforwardly to those parameters is a Z-Y-X rotation sequence
using an extrinsic (space-fixed) coordinate system:

First rotate by angle phi around the north pole in space-fixed coordinates.
Then rotate by angle theta around 0N 90E in space-fixed coordinates.
Finally, rotate by angle psi around 0N 0E in space-fixed coordinates.

The location of the grid's north pole in the original lat-lon coordinate system
is now at longitude = phi, colatitude = minus theta.

(I'm not sure how psi relates to north_pole_grid_longitude mathematically; it
defaults to zero and I've never seen it in use, so I think it's included mostly
for completeness. I agree that we may well want to change it to something more
obvious like an azimuth angle.)

Switching to the GRIB2 convention where you use the south pole instead of the
north should be just a matter of adding 180 degrees to phi and negating theta.

Cheers,

--Seth


On Wed, 6 Mar 2013 18:29:47 +
  David Hassell d.c.hass...@reading.ac.uk wrote:

Hello,

It is a bit of mess! As I understand it, the full rotation described
is a sequence of rotations about three different axes:

   Z  = 90S - 90N
   Y' = 90W' - 90E'
   X = 0E - 180E

where it is understood that the definitions of the N-S and W-E axes
change after each rotation (hence the primes and double
primes). Therefore the order in which they are done matters.

I suspect that the usual and assumed order is Z, Y', X?

From the GRIB-2 stuff John posted the north_pole_grid_longitude gives
the rotation about the X axis.

Something like John's Angle of rotation seems right to me.

The apparent lack of consistency between the parameter names irks
me. Perhaps one solution could be to:

i) Add some text to the conventions state the order (Z, Y', X, say)
   and direction of rotations.

ii) add three new, consistent, self-describing parameters (e.g.)

angle_of_rotation_z
angle_of_rotation_y
angle_of_rotation_x

iii) allow aliases for backwards compatibility

grid_north_pole_longitude = angle_of_rotation_z
grid_north_pole_latitude  = angle_of_rotation_y
north_pole_grid_longitude = angle_of_rotation_x

Or would that just obfuscate things even more?

All the best,

David

P.S. If you have a copy available, there are some nice descriptions in
 Coordinate Systems and Map Projections by D. H. Maling

--
David Hassell
National Centre for Atmospheric Science (NCAS)
Department of Meteorology, University of Reading,
Earley Gate, PO Box 243,
Reading RG6 6BB, U.K.

Tel   : 0118 3785613
Fax   : 0118 3788316
E-mail: d.c.hass...@reading.ac.uk
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] calendar-based units of time

[John Caron]

Hi Jonathan:

On 12/17/2012 10:51 AM, Jonathan Gregory wrote:

Dear John


I have prototyped similar functionality in the CDM, documented here:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html

Thanks for this. I recall the earlier discussion and I think what you have
implemented would be a good extension to CF, namely

* Optional prefix calendar to the time-units means the corresponding
component of time is manipulated. You show only years and months. Do you
also accept the unproblematic units of days, hours, minutes and seconds
with calendar prefixed? Do you allow the SI prefixes to these units?


yes, the same periods are allowed that are in the udunits grammer:

period = millisec | msec | second | sec | s | minute | min | hour | hr | day | 
week | month | mon |
year | yr


we do not allow SI prefixes. we do allow plural forms, ie with an 
appended s.





* With calendar, only integer numbers of units are allowed. Are negative
numbers allowed?


yes



* Invalid dates caused by adding months and years are decremented in whole
days until a valid date is reached.

yes

regards,
John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Cathy:

I think that you are using backwards compatible in a different way.

The current proposal(s) would not change files that are written with 
:Conventions=CF-1.x, where x = 6. Files with x  6 could still use 
the (ill-advised) old way if they want to, by putting an explicit 
calendar attribute in. But if theres no explicit calendar attribute, 
then these new files will be interpreted in a way that is less likely to 
give incorrect dates.


So, im not sure why you keep saying shouldn't break current files, 
since there is no such proposal on the table.


John

On 12/17/2012 4:13 PM, Cathy Smith (NOAA Affiliate) wrote:

Cecelia

I think a solution shouldn't break current files which followed what had
been a standard for a long time (however ill-advised the standard was).
I don't have a good sense of what might break if the standard changed in
terms of software so I can' speak for all users but I do know many
people have downloaded our mean files with 1-1-1 base dates (ignoring
the climatologies for now). While we can potentially change what we have
either by changing the dates and/or adding a calendar attribute,
changing the default calendar may create errors in reading dates for
users who already have those files (which are currently CF complaint ).
And,  they won't have the same ability to change the files and they
wouldn't necessarily know they needed to.  I think no default at all
would be problematic as what would software do?  So, I would support the
inclusion of a calendar attribute that would be used by software if
there but using the old default calendar if not. Also, I would support
making a calendar attribute for new files mandatory (with an updated CF
version) but I would keep backwards compatibility unless it were
reliably shown it was not an issue with users.
I'm not convinced that the users needs (as opposed to developers) have
been adequately researched.

Cathy



On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote:

Cathy,

Of the other options, do you find some (or all) completely unacceptable?
Are some better than others?

- Cecelia

On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote:

Cecelia
I support 1) mostly for backward compatibility. I would also strongly
encourage but not demand that users change their base dates to after
1800
when it makes sense to do so.

And, I (again) want to make sure that LTMs and their time values are
addressed before any decisions are made as to negative times and
using base dates of 1-1-1 and the issue of what year to use for
climatologies. LTM dates are a problem when one needs to use a
calendar based on real dates.

Cathy


On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote:

Hi Steve, Jonathan and all,

There are not that many options being discussed.

With respect to the default calendar:

1 keep the Julian-Gregorian calendar as default (no change)
2 remove the Julian-Gregorian calendar as default, and have no
default calendar (grid analogy)
3 replace the Julian-Gregorian calendar as default with the
proleptic Gregorian calendar
4 replace the Julian-Gregorian calendar as default with a strict
Gregorian calendar

Maybe it makes sense for people to cite (or rank)  their preference
at this point?

There were a couple other proposals, depending on which of above is
selected:
5 create a strict Gregorian calendar (optional for 1, 2, 3 and
needed for 4)
6 remove the Julian-Gregorian calendar (impossible for 1, optional
for 2, 3, 4)

Again, maybe worth it to see where people are after the round of
discussion?

Best,
Cecelia



On 12/10/2012 12:40 PM, Steve Hankin wrote:

Hi Jonathan,

I'm not sure if my remarks below conflict with your proposed
resolution.  But they do dispute the facts you assert, and these
waters are so muddy that agreeing on the facts seems an important
first step.

On 12/10/2012 1:21 AM, Jonathan Gregory wrote:

Dear Jon


Just to repeat a remark that Steve Hankin made whose implications have not been 
explored in this discussion: different countries adopted the Gregorian calendar 
at different times.  (Greece didn't adopt it till 1923!)  So what is considered 
a valid Gregorian date varies from country to country (and some of those 
countries don't even exist any more, or at least the boundaries have changed...)
2. The non-proleptic Gregorian calendar is extremely problematic for historical 
observations as well as for models (astronomers use the Julian calendar 
consistently for this reason).

Yes, that's right. Nonetheless I don't think we can abolish the real-world
calendar, despite its ambiguities, because*_it's the one we really use!_*


Are you sure this is true?  Evidence seems to suggest that our
community has _no use for the mixed Gregorian/Julian calendar at
all_, except the need to resolve the backwards compatibility mess
we have created for ourselves.

  * In everyday life we use is the modern Gregorian calendar, and
are not concerned with historical calendar changes.
  * In numerical 

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Chris:

On 12/17/2012 4:50 PM, Chris Barker - NOAA Federal wrote:

I think a solution shouldn't break current files which followed what had
been a standard for a long time (however ill-advised the standard was). I
don't have a good sense of what might break if the standard changed in terms
of software so I can' speak for all users but I do know many people have
downloaded our mean files with 1-1-1 base dates


Hmm -- this seems to contradict Steve's no known users.


there are no known users using mixed gregorian/julian who actually need 
to use it (cross the transition date line). if there are any such users, 
please speak up, so we can think about your use case.




BUt anyway, I wonder if folks currently using such files are actually
getting the correct results, when they do.


those using udunits get the correct result even when they cross the 
line. i suspect its correct because using a different implementation 
(joda time library) gets the same results for the small sample i have 
tested.




For instance, in my naiveté, I never thought about calendars when
using netcdf files -- I simple converted something since some date
to Python datetime objects using python's datetime package, which uses
An idealized naive date, assuming the current Gregorian calendar
always was, and always will be, in effect

So I would have gotten the wrong results.


it sounds like you were using what is called proplectic gregorian. so 
you would have gotten wrong results on the PSD dataset, which uses time 
since 1-1-1. Thats the argument for not using mixed gregorian, that 
simple things arent simple.




I've lost track whether UDunits does the mixed calendar right, but I'm
sure not all libraries one might use do.

So is this a use case se need to preserve?


we have datasets out there who are using time since 1-1-1 
unfortunately. Theres no doubt we need to leave those working; and 
even that they can continue their dangerous ways if they want to 
explicitly include a calendar attribute in CF-1.7 convention file. The 
proposal is just to change the default in CF-1.7 to use a more common 
calendar, and to encourage people not to cross the time warp zone 
unecessarily.


John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi all:

Heres what I understand of the conversation:

1. Theres nothing to do about existing files CF-1.6 and before. we are 
stuck with the udunits mixed calendar.


2. Starting with the next version after acceptance, (1.7 currently), we 
can do something different. I agree that forcing people to put in a 
calendar attribute makes simple things not simple. So lets choose a 
reasonable default, either gregorian_propleptic or gregorian_strict is 
ok with me.


3. everyone agrees that in the unit time since date, date is 
interpreted in the calendar that is specified, or the default if not 
specified.


4. i thing we should add some advice not to start counting from 0001 
when recording modern dates, no matter if you do it right or not.



John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] [netcdf-java] problem with times in PSD dataset

[John Caron]

Hi Cecelia:

Thanks for this information. A few questions:

* So you are not supporting standard Gregorian calendar even though 
thats the CF default?


* Do modelers need to match historical dates? If so, what calendar do 
they use?


* Is the time library suitable to be released seperate from the ESMF 
code, eg as standalone C++?


John

On 12/5/2012 3:01 PM, Cecelia DeLuca wrote:

Hi John and all,

ESMF has a mature time management library with calendars that are commonly
used in climate/weather/related modeling. It supports the following: 360
day,
no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day,
and custom
(including custom calendars that may change the length of days). ESMF,
like CESM,
doesn't support the standard Gregorian calendar because it doesn't make
sense for modeling groups who may need to cross the Julian/Gregorian
weirdness line (we've never gotten a request to support standard
Gregorian from
modelers).  ESMF has calendar, time, time interval, clock, and alarm
constructs,
can run time forward or backward, and prints times and time intervals in
ISO8601
format. CESM/CAM, NASA, NOAA, Navy and other codes use it.

The most complete interface to the time lib is Fortran, and there are
some basic
time functions exposed in C (the lib is actually written in C++).
However, we could
wrap the time functions in Python and include them in ESMP, which is
currently a
set of ESMF regridding functions wrapped in Python. We could probably do
that
in the late/winter spring timeframe, with some guidance on what
functions were
desired and a pass through our prioritization board.

Best,
-- Cecelia


On 12/5/2012 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian (Mixed
Gregorian/Julian calendar) the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using time since
1-1-1. Ive never seen a dataset where time since 1-1-1 using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need historical accuracy, then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. time since
reference date is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard libraries that
could be used. Anyone have candidate libraries in C or Python for
robust calendering

In short, we should rely on clear encoding standards (eg ISO8601) with
reference software, rather than implementations like udunits that
eventually go away.

John

PS: I think ill cross post to cf, just to throw some gasoline on the
fire ;), and maybe some broader viewpoints.

On 12/5/2012 10:24 AM, Don Murray (NOAA Affiliate) wrote:

Hi Gerry-

On 12/5/12 9:42 AM, Gerry Creager - NOAA Affiliate wrote:

There are other datasets with reference to 1-1-1. I've seen them most
recently in some ocean models.


And the ESRL/PSD NCEP reanalysis datasets use it.

Don


On Wed, Dec 5, 2012 at 10:16 AM, Don Murray (NOAA Affiliate)
don.mur...@noaa.gov mailto:don.mur...@noaa.gov wrote:

John-

I meant to send this to support-netcdf-java, but perhaps others on
the list might have the same problem.


On 12/4/12 4:51 PM, John Caron wrote:

On 12/4/2012 4:09 PM, Don Murray (NOAA Affiliate) wrote:

Hi-

I was just trying to access the NOAA/ESRL/PSD Outgoing
Longwave
Radiation (OLR) data using netCDF-Java 4.3 ToolsUI and
noticed that the
times are wrong.  If you open:


dods://www.esrl.noaa.gov/psd/__thredds/dodsC/Datasets/__uninterp_OLR/olr.day.mean.nc



http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/uninterp_OLR/olr.day.mean.nc





in the ToolsUI grid viewer, the last time in the file is
shown as
2012-12-04 00Z.  However, the last time in the file is
actually
2012-12-02 00Z.  Here is the time variable in that file:

 double time(time=3989);
   :units = hours since 1-1-1 00:00:0.0;
   :long_name = Time;
   :actual_range = 1.7540448E7, 1.763616E7; // double
   :delta_t = -00-01 00:00:00;
   :avg_period = -00-01 00:00:00;
   :standard_name = time;
   :axis = T;

netCDF-Java 4.2 and ncdump -t -v time (C version) show the
correct
date/times.


hours from 1-1-1 is rather problematic, since you are crossing
the
julian

[CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Steve, Cecelia:

I agree we should move to a better default calendar, probably 
proleptic_gregorian. We are stuck with mixed Gregorian for previous CF 
versions.


I think we just need a proposal that says As of version X, the default 
calender is proleptic_gregorian. You should explicitly add the calendar 
attribute for clarity. Udunits is no longer considered to be the 
reference software for date/time.


It would be nice to add advice to the user about calendar tradeoffs and 
how to do historical date matching for modelers, assuming we have useful 
things to say.


I wonder if allowing ISO formatted strings in place of / in addition to 
the time since reference date form might be useful for historical time 
matching?


Arguably having calendar reference libraries in Python and Java would be 
sufficient, possibly Python is preferable even to one in C.


John

On 12/6/2012 1:45 PM, Cecelia DeLuca wrote:


Hi John and all,


Thanks for this information. A few questions:

* So you are not supporting standard Gregorian calendar even though
thats the CF default?

Correct, the climate modelers that we work with don't use it.  AFAIK the
decision of
CESM was to reject the CF default as unreasonable for modeling and use
proleptic
Gregorian.  GFDL might support it, I don't know.

We could probably add standard Gregorian as a new calendar if people on the
data side needed it.  However, we would be happier to join Steve in putting
a stake in its heart!


* Do modelers need to match historical dates? If so, what calendar
do they use?

I think the calendar used would depend on what was supported by the
model and
configuration, as well as the form of the date.  If you wanted to
represent the date
of something like a volcanic eruption you could probably make it work
with most
of the calendars.



* Is the time library suitable to be released seperate from the ESMF
code, eg as standalone C++?

The time library can be used apart from other ESMF interfaces, and some
modeling groups do use it that way.  I don't think we'd be willing to
extract that
part and support a separate build.   We did that years ago and it was a
pain
to maintain.  (We could try to isolate the documentation though, so users
didn't have to wade through the whole reference manual to find the API.)

With ESMP(ython), the scope of the interface is much smaller than
ESMF proper and I think Ryan could set time functions up nicely.  It might
make sense to represent it as a separate python package in that approach.
Would python work for you, or would you really prefer C?

- Cecelia



John

On 12/5/2012 3:01 PM, Cecelia DeLuca wrote:

Hi John and all,

ESMF has a mature time management library with calendars that are
commonly
used in climate/weather/related modeling. It supports the following: 360
day,
no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day,
and custom
(including custom calendars that may change the length of days). ESMF,
like CESM,
doesn't support the standard Gregorian calendar because it doesn't make
sense for modeling groups who may need to cross the Julian/Gregorian
weirdness line (we've never gotten a request to support standard
Gregorian from
modelers).  ESMF has calendar, time, time interval, clock, and alarm
constructs,
can run time forward or backward, and prints times and time intervals in
ISO8601
format. CESM/CAM, NASA, NOAA, Navy and other codes use it.

The most complete interface to the time lib is Fortran, and there are
some basic
time functions exposed in C (the lib is actually written in C++).
However, we could
wrap the time functions in Python and include them in ESMP, which is
currently a
set of ESMF regridding functions wrapped in Python. We could probably do
that
in the late/winter spring timeframe, with some guidance on what
functions were
desired and a pass through our prioritization board.

Best,
-- Cecelia


On 12/5/2012 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian (Mixed
Gregorian/Julian calendar) the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using time since
1-1-1. Ive never seen a dataset where time since 1-1-1 using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need historical accuracy, then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. time since
reference date is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard libraries

Re: [CF-metadata] [netcdf-java] problem with times in PSD dataset

[John Caron]

Hi Cathy:

There no question that CF currently defaults to mixed gregorian 
calendar. The discussion is whether thats the best choice (probably 
not), and to advise users not to cross the discontinuity (eg store 
modern dates starting from 1-1-1).


Im curious as to how you generate the dates that you store? That is, how 
do you know that they are correct?


John

On 12/6/2012 4:34 PM, Cathy Smith (NOAA Affiliate) wrote:

John
There is some meteorological data that is available pre-Gregorian
calendar (paleo data, some temperature datasets) and of course there are
other scientific fields where data is pre-1500 (e.g. astronomy,
archeology) . Given that netCDF files with data dates spanning ~1550 probably 
already exist and the large number of preexisting files that
use the 1-1-1 base (we have over 2000), it doesn't seem reasonable to
request that files be changed to accommodate what is essentially a bug;
the date values we store are correct. We started using the 1-1-1 base in the mid
1990's (almost 20 years ago) as part of the COARDS (now CF) agreed upon 
standard.
It is reasonable for us to consider future changes but I don't think it 
reasonable for us to have to change our files because the Java interface is not 
backwards compatible.

Cathy Smith
NOAA/ESRL PSD

On 12/5/12 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian (Mixed
Gregorian/Julian calendar) the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using time since
1-1-1. Ive never seen a dataset where time since 1-1-1 using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need historical accuracy, then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. time since
reference date is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard libraries that
could be used. Anyone have candidate libraries in C or Python for
robust calendering

In short, we should rely on clear encoding standards (eg ISO8601) with
reference software, rather than implementations like udunits that
eventually go away.

John

PS: I think ill cross post to cf, just to throw some gasoline on the
fire ;), and maybe some broader viewpoints.

On 12/5/2012 10:24 AM, Don Murray (NOAA Affiliate) wrote:

Hi Gerry-

On 12/5/12 9:42 AM, Gerry Creager - NOAA Affiliate wrote:

There are other datasets with reference to 1-1-1. I've seen them most
recently in some ocean models.


And the ESRL/PSD NCEP reanalysis datasets use it.

Don


On Wed, Dec 5, 2012 at 10:16 AM, Don Murray (NOAA Affiliate)
don.mur...@noaa.gov mailto:don.mur...@noaa.gov wrote:

 John-

 I meant to send this to support-netcdf-java, but perhaps  others on
 the list might have the same problem.


 On 12/4/12 4:51 PM, John Caron wrote:

 On 12/4/2012 4:09 PM, Don Murray (NOAA Affiliate) wrote:

 Hi-

 I was just trying to access the NOAA/ESRL/PSD  Outgoing
Longwave
 Radiation (OLR) data using netCDF-Java 4.3 ToolsUI and
 noticed that the
 times are wrong.  If you open:


dods://www.esrl.noaa.gov/psd/__thredds/dodsC/Datasets/__uninterp_OLR/olr.day.mean.nc



http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/uninterp_OLR/olr.day.mean.nc





 in the ToolsUI grid viewer, the last time in the file is
 shown as
 2012-12-04 00Z.  However, the last time in the file is
actually
 2012-12-02 00Z.  Here is the time variable in that file:

  double time(time=3989);
:units = hours since 1-1-1 00:00:0.0;
:long_name = Time;
:actual_range = 1.7540448E7, 1.763616E7; // double
:delta_t = -00-01 00:00:00;
:avg_period = -00-01 00:00:00;
:standard_name = time;
:axis = T;

 netCDF-Java 4.2 and ncdump -t -v time (C version) show the
 correct
 date/times.


 hours from 1-1-1 is rather problematic, since you are crossing
the
 julian/gregorian weirdness line (i think thats the technical
term ;)

 Im guessing the trouble lies here:

 Default calendar: for udunits, and therefore for CF, the
default
 calendar is gregorian (Mixed Gregorian/Julian calendar). For
 CDM, the
 default calendar is proleptic_gregorian (ISO8601 standard

Re: [CF-metadata] Choice of fill value for unpacked data

[John Caron]

On 10/10/2012 6:36 AM, Jim Biard wrote:

John,

I agree that there is no ambiguity in the original dataset.  I was just
pointing out a scenario in which the difficulty mentioned by Phil in his
original post would manifest itself.  The methodology referenced by Russ
won't handle it.  If you (for reasons beyond your control) have a packed
fill value that is inside your valid range, then no guidance is provided
by CF to automated client software for how to represent the unpacked
fill value, leaving the user with a requirement to dig back through the
packed values, find the elements marked as fill, and manually mark the
unpacked values.


Its true CF is silent about this, so automated client software has to 
make their own Convention. The netcdf-JAVA package supplies a method


  boolean isMissing(double value);

for this purpose. Alternately, it will just set missing real values to 
NaNs, which works well in Java. I think NaNs are the right choice for 
missing values myself.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Choice of fill value for unpacked data

[John Caron]

Hi Jim:

_FillValue/missing_value refers to the packed value, so theres no 
ambiguity in the original dataset. It is best to make sure its outside 
the range of real values, but even if not, one just has to search for 
that exact bit pattern.


If someone rewrites the data, its their responsibility to choose a 
_FillValue/missing_value that is unambiguous.


If your use case is important, you could add _UnpackedFillValue, so that 
your software did the right thing. Dunno if it would be generally useful.


John

On 9/27/2012 7:00 AM, Jim Biard wrote:

Hi.

Assuming you have the luxury of specifying your _FillValue and/or
missing_value, I agree that this isn't a big deal.  However, I am
working with data where the project has defined fill/missing values that
are wholly within the range of possible values (NPP satellite data).
  The approach defined below fails in such cases.

Grace and peace,

Jim

Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave, Asheville, NC 28801-5001

jim.bi...@noaa.gov mailto:jim.bi...@noaa.gov
828-271-4900

On Sep 25, 2012, at 11:08 PM, Russ Rew r...@unidata.ucar.edu
mailto:r...@unidata.ucar.edu wrote:


Hi Phil,


The final para of section 2.5.1 of the CF conventions document describes
the use of the _FillValue (or missing_value) attribute in the case of
data packed using the scale-and-offset method.  What is not clear - at
least to me - is what the preferred application behaviour should be in
the case where the data is unpacked and then written out to a new netCDF
file. In particular, what fill value should be used for the unpacked
data variable?

I presume that one wouldn't normally want to use the original fill value
since that value (typically an 8- or 16-bit integer) is quite likely to
fall within the normal range of the unpacked data (typically a 32- or
64-bit float).

In the absence of explicitly setting a fill value attribute on the
unpacked data variable I assume that the netCDF default fill value will
be used for the data type in question. Which may not always be desirable
(certainly not for 32-bit floats, where the default fill value can give
rise to subtle precision-related problems).

With this in mind, I was wondering if there is any merit in defining a
new attribute called, say, _UnpackedFillValue (or
unpacked_missing_value)? If client software detected this attribute then
the associated value (same data type as the scale_factor and add_offset
attributes) would be used as the fill value for the unpacked data
variable.

Alternatively, the names _FillValueUnpacked (missing_value_unpacked)
might be preferable since they would then appear together pair-wise in
CDL-type listings, e.g.

short pkd_var(z, y, x) :
  ...
  pkd_var:_FillValue =3D -32768 ;
  pkd_var:_FillValueUnpacked =3D -1.0e30 ;
  pkd_var:add_offset =3D 42.0 ;
  pkd_var:scale_factor =3D 1234.0 ;
  ...


Any merit/mileage in this idea?


A more detailed recommendation for treating special values such as
_FillValue or missing_value for packed data is described, with
associated formulas, in the Packed Data Values section of a
best-practices document that we wrote a few years ago:

http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Packed%20Data%20Values

It provides a recommendation for ensuring the unpacked special value is
not in the range of other unpacked data values.  If that recommendation
is followed, I think there is no need for an additional
_FillValueUnpacked (or missing_value_unpacked) attribute.

If you agree that is an acceptable approach, perhaps we should add it to
CF ...

--Russ
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu mailto:CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Roms: ocean_s_coordinate_g1 and g2?

[John Caron]

Ive updated the docs at

http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html 



Thanks for the report.

On 7/17/2012 11:52 PM, Heiko Klein wrote:

Rich,

it must have been the netcdf-java package where I have seen these 
standard-names. The description on the netcdf-java pages are outdated, 
still using formula_terms a and b. I just checked the code, and that 
is up to date working with the formula_term -array C(k).


It'll be good to have an official version. Thanks for picking this up 
again.


Heiko


On 2012-07-17 17:13, Rich Signell wrote:

Heiko,

We worked with John Caron several years ago to get these into the CDM
in the NetCDF-Java library, so if you have g1 or g2 coordinates they
will work with codes that use NetCDF-java (like the Matlab NCTOOLBOX
and Unidata's IDV),  and I remember we drafted up some documentation
to  submit to CF, but looks like we dropped the ball.I'll pick it
up again.

-Rich

On Mon, Jul 16, 2012 at 10:24 AM, Heiko Klein heiko.kl...@met.no 
wrote:

Hi,

I'm currently working with ROMS ocean model files. The model uses a
generalized ocean_s coordinate as defined by
https://www.myroms.org/wiki/index.php/Vertical_S-coordinate .

The same page says that their s_coordinate is covered by standard_names
ocean_s_coordinate_g1
ocean_s_coordinate_g2
and I believe I have seen at least the g1 case in a draft of a CF 
convention
some month ago. Both cases are also implemented in the netcdf-java 
package

with above standard_names:
http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html 



But I cannot find any of those names any longer, neither in CF-1.6 
nor in

the standard_names tables.

Does somebody remember what happened to these vertical coordinates?
According to the ROMS Wiki, the standard_names have been added in March
2009.



Best regards,

Heiko
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata








___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] You will get email from the CF Trac system.

[John Caron]

yes, i typically get 3 copies. it would be nice not to.


On 6/25/2012 4:29 PM, Seth McGinnis wrote:

I do want to see Trac messages, but since this update happened I'm getting two
or three copies of each one.  Is anyone else having this problem?

What do I need to do to get only one copy?

Thanks,

--Seth

On Thu, 21 Jun 2012 16:44:40 -0700
  Jeffrey F. Painter paint...@llnl.gov wrote:

CF metadata issues get discussed both in this mailing list and in the CF
Metadata Trac system at https://cf-pcmdi.llnl.gov/trac/.  Some time in the
past, everyone on the cf-metadata list also got copies of the discussions in
the Trac system.  But the Trac system's mailing list hadn't been updated for
some years.  So today I updated it.  Everyone on this mailing list will get
messages from the CF Metadata Trac system.  There are rather few such
messages, so you may not notice the difference.

If you really don't want to see the Trac messages, but you want to stay on the
regular cf-metadata mailing list, you can opt out of only the Trac messages.
There is a special mailing list which acts as an intermediary between the
Trac system and cf-metad...@ugd.ucar.edu.  You can just unsubscribe to the
special mailing list as follows:  Send mail to majord...@lists.llnl.gov, from
the address at which you receive the mail.   The body of your message should
be the line unsubscribe cf-metadata.  This will unsubscribe you only from
the special intermediary list, not the regular CF Metadata mailing list at
ucar.edu.

Thanks to Jonathan Gregory for suggesting that I address this issue.

- Jeff Painter

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Accept geostationary projection

[John Caron]

If no one has any objections, i move that we accept this ticket:

Adding the geostationary projection.

https://cf-pcmdi.llnl.gov/trac/ticket/72

the changes needed to the spec are at the bottom of the ticket, we didnt 
know how to change the top part.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

On 4/30/2012 8:40 PM, andrew walsh wrote:

Hi John and CF-Metadata list,

Based on your earlier advice I decided using the Scalar way to represent
the coordinate lat, long and time rather than Vector way i.e lat(lat0, 
lon(lon), time(time)

mainly for reason of simplicity.


the correct other choice is lat(profile), lon(profile), time(profile). 
not sure if you caught that.


the scalar choice is fine for 1 profile per file. the other is used to 
store multiple profiles in one file.




I have run the Scalar sample through the BADC netCDF checker (*Note)
at http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl and I now get 
this error

on each of the lat, lon, time variables:

ERROR (4): Axis attribute is not allowed for auxillary coordinate 
variables.
I don't get this error with the Vector approach. It looks like the 
checker thinks
my scalar lat, lon, time are 'auxilliary coordinate variables' and 
axis attributes

are not allowed on these. Is the checker interpreting things correctly?


i think we recently clarified that axis is acceptable on auxiliary 
coordinates. OTOH, its unecessary, as long as you follow the other rules 
for identifying coordinates (chapter 4).


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Reverse-time trajectory

[John Caron]

On 5/1/2012 9:02 AM, Jonathan Gregory wrote:

Dear Richard


The CF definitions of discrete sampling geometries define the trajectory feature type as a series of 
data points along a path through space with monotonically increasing times. This is a stricter stance than the 
usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the 
increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time.

Good point. I would think that monotonically would be enough, not necessarily
increasing. I can't remember a reason for increasing; I assume it's just
because sect 9 was conceived for observations in the first place. However, John
Caron may well have a comment. I don't think anything prevents your storing
the data in the orthogonal multidimensional representation, which existed
before sect 9 did and doesn't require increasing coordinates.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


i agree, increasing is not needed, and could be changed with a defect 
ticket.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

On 4/29/2012 5:33 PM, andrew walsh wrote:

Hi John,

My responses inline below.

Andrew


- Original Message - From: John Caron ca...@unidata.ucar.edu
To: cf-metadata@cgd.ucar.edu
Sent: Saturday, April 28, 2012 2:39 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi Andrew:

You can use a dimension=1 instead of a scalar.


OK, you mean if one used the  vector method.


yes





But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)


So, do you mean in the vector method have this? (simpler I guess to
have common profile dimension = 1)


yes, you cant use incorrect dimensions, even when they are all = 1.



dimensions:
profile=1
pressure = 57

variables:
lat(profile)
lon(profile)
time(profile)
pressure(pressure)
temperature(pressure)

Instead of this?

Dimensions:
time=1
lat=1
lon=1
pressure=57

Variables:
time(time)
lat(lat)
lon(lon)
pressure(pressure)
temperature(pressure)

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

yes, good point.

if it is a multidimension representation, you would also need the 
profile dimension. if its a ragged array, you would need either the 
rowSize or parentIndex variable.


On 4/30/2012 7:22 AM, Jim Biard wrote:
Wouldn't you also need the profile dimension on the temperature and 
pressure variables in order to connect them all as part of the same 
profile collection?


Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave, Asheville, NC 28801-5001

jim.bi...@noaa.gov mailto:jim.bi...@noaa.gov
828-271-4900

On Apr 30, 2012, at 8:46 AM, John Caron wrote:


On 4/29/2012 5:33 PM, andrew walsh wrote:

Hi John,

My responses inline below.

Andrew


- Original Message - From: John Caron 
ca...@unidata.ucar.edu mailto:ca...@unidata.ucar.edu

To: cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu
Sent: Saturday, April 28, 2012 2:39 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi Andrew:

You can use a dimension=1 instead of a scalar.


OK, you mean if one used the  vector method.


yes





But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)


So, do you mean in the vector method have this? (simpler I guess to
have common profile dimension = 1)


yes, you cant use incorrect dimensions, even when they are all = 1.



dimensions:
profile=1
pressure = 57

variables:
lat(profile)
lon(profile)
time(profile)
pressure(pressure)
temperature(pressure)

Instead of this?

Dimensions:
time=1
lat=1
lon=1
pressure=57

Variables:
time(time)
lat(lat)
lon(lon)
pressure(pressure)
temperature(pressure)

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu mailto:CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi Andrew:

You can use a dimension=1 instead of a scalar.

But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)

John

On 4/27/2012 12:03 AM, andrew walsh wrote:

Hi John,

Thanks for your advice.
For a single vertical profile and single valued time, lat, long.
it seems there are two acceptable ways i.e 1 element vector or single 
valued scalar.


Quoting from 'section 2.4 Dimensions' of CF standard:

1 element vector:

Dimensions may be of any size, including unity. When a single value 
of some coordinate applies to all the values in a variable, the 
recommended means of attaching this information to the variable is by 
use of a dimension of size unity with a one-element coordinate variable.


and in the next sentence a scalar:

It is also acceptable to use a scalar coordinate variable which 
eliminates the need for an associated size one dimension in the data 
variable.


1) ***Vector co-ordinate variables***

Dimensions:
time=1
pressure=56
lat=1
lon=1

Variables:
time(time)
lat(lat)
lon(lon)
temperature(pressure)

2) **Scalar co-ordinate varaibles

Dimensions:

pressure =56

Variables:
time
lat
lon
temperature(pressure)

The scalar approach you suggest as in section H.3.3. Single profile 
of the CF v1.6 standard

is simpler than the vector approach so we will take your advice.

Regards,

Andrew


- Original Message - From: John Caron ca...@unidata.ucar.edu
To: andrew walsh awa...@metoc.gov.au
Cc: cf-metadata@cgd.ucar.edu; Luke Callcut l...@metoc.gov.au; 
g...@metoc.gov.au

Sent: Thursday, April 26, 2012 10:48 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi andrew:

The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a 
single profile. Assuming that, you would use the following template:


http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 



note that lat, lon and time coordinates all have to be scalars.

John

On 4/18/2012 8:24 PM, andrew walsh wrote:

Hi John,

We have now constructed a real netCDF file for you to check. QC flag
variables have been added in. The QC flagging method is based on the
proposed IODE scheme in (1) below.

Attached are the sample .nc file, text (ncdump) of same and a document
specifying the CDL.

Thanks and Regards,

Andrew Walsh


Ref.

(1) Konovalov et. al (March 2012), Proposal to adopt a quality flag 
scheme standard

for oceanographic and marine meteorological data, Version 1.2.

- Original Message - From: John Caron 
ca...@unidata.ucar.edu

To: andrew walsh awa...@metoc.gov.au
Sent: Thursday, April 05, 2012 10:44 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



ok

On 4/4/2012 6:42 PM, andrew walsh wrote:

Hi John,

Thank for your offer to check a sample netCDF CTD data file. At 
moment we don't have

some real .nc file but when we do I can send you a file for checking.

Andrew

- Original Message - From: John Caron
To: cf-metadata@cgd.ucar.edu
Sent: Wednesday, April 04, 2012 5:55 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions 
v1.6



Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I 
can check if it works in the CDM with the new 1.6 conventions, and 
maybe give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
joinNew  and joinExisting NCML aggregation. I call it 
syntactic aggregation because it doesnt know what its 
aggregating, and the homogeneity requirements are strict. 2) 
Feature Type collections (aka semantic aggregation) are the 
more recent development. These understand the coordinate 
information of the data, and so can handle variations in the 
representation, eg ragged arrays, scalar coordinates, etc. In this 
case, the CDM understands a dataset as a collection of features 
objects, which can be stored in a collection of files.  The 
interfaces to these collections is still under development. Most 
current and future work in the CDM is in this category.


John

snip 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata








___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi andrew:

The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a 
single profile. Assuming that, you would use the following template:


http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696

note that lat, lon and time coordinates all have to be scalars.

John

On 4/18/2012 8:24 PM, andrew walsh wrote:

Hi John,

We have now constructed a real netCDF file for you to check. QC flag
variables have been added in. The QC flagging method is based on the
proposed IODE scheme in (1) below.

Attached are the sample .nc file, text (ncdump) of same and a document
specifying the CDL.

Thanks and Regards,

Andrew Walsh


Ref.

(1) Konovalov et. al (March 2012), Proposal to adopt a quality flag 
scheme standard

for oceanographic and marine meteorological data, Version 1.2.

- Original Message - From: John Caron ca...@unidata.ucar.edu
To: andrew walsh awa...@metoc.gov.au
Sent: Thursday, April 05, 2012 10:44 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



ok

On 4/4/2012 6:42 PM, andrew walsh wrote:

Hi John,

Thank for your offer to check a sample netCDF CTD data file. At 
moment we don't have

some real .nc file but when we do I can send you a file for checking.

Andrew

- Original Message - From: John Caron
To: cf-metadata@cgd.ucar.edu
Sent: Wednesday, April 04, 2012 5:55 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6


Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I can 
check if it works in the CDM with the new 1.6 conventions, and maybe 
give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
joinNew  and joinExisting NCML aggregation. I call it syntactic 
aggregation because it doesnt know what its aggregating, and the 
homogeneity requirements are strict. 2) Feature Type collections 
(aka semantic aggregation) are the more recent development. These 
understand the coordinate information of the data, and so can handle 
variations in the representation, eg ragged arrays, scalar 
coordinates, etc. In this case, the CDM understands a dataset as a 
collection of features objects, which can be stored in a 
collection of files.  The interfaces to these collections is still 
under development. Most current and future work in the CDM is in 
this category.


John

snip 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] identification of vector components

[John Caron]

On 4/24/2012 4:16 AM, Hedley, Mark wrote:

There are particular issue here with format interoperability and conversion 
with respect to phenomenon.  In GRIB2, for example, there are codes which 
identify vector components:
   Wind direction (from which blowing) degree true
   Wind speed ms
   u-component of wind ms
   v-component of wind
but no code for eastward_...

The current situation poses significant challenges for GRIB - CF 
interoperability.

On 4/24/2012 10:42 AM, Hedley, Mark wrote:

  Can similar tests be used with Grib conversion? I assume that Grib must 
provide sufficient metadata to indicate whether the grid is lon-lat.

Yes and your assumption is correct, there is sufficient information to deliver 
interoperability, but implementing this is adding complexity which I feel is 
not necessary and may cause future issues maintaining the special cases.  One 
of the reasons I raise GRIB is that it treats vector component as one 
phenomenon, as I have described, not two as is done in CF currently.


A few comments on my favorite topic, GRIB:

AFAIU, like CF, GRIB does not associate, say, the u and v components of 
wind together, or identify them as vectors. You just have to know what 
the naming conventions are to make the association.


More generally, it seems that the crux of the issue is that one wants to 
know if the vector components of, say, wind, are reletive to the grid or 
to lat/lon.  GRIB does encode this info in the GDS, the equivalent of 
the CF projection. Again you just have to know if that information is 
actually pertinent to the GRIB record you are looking at. If you know 
this, one could use it in the correct standard CF name.


In terms of mapping between GRIB and CF standard_names, these is 
unfortunately no such up-to-date mapping, AFAIK. It would be very useful 
and quite challenging to do so. One reason its hard is that a GRIB 
variable must be discovered through hueristics that involve the 
parameter name, the level type, and a whole bunch of other things that 
you just have to know. Some details for the foolhardy are at a URL you 
just have to know, but since I know it, here it is:


http://www.unidata.ucar.edu/blogs/developer/en/entry/dataset_schemas_are_lost_in

John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] code that does semantic checking of CF headers

[John Caron]

On 4/20/2012 4:54 AM, Gaffney, Sean P. wrote:

Hi everyone, Thanks for all your feedback.

It's made things clearer for me now. Just to summarise then, a suite of 
attributes that would provide a precise numeric description of the data within 
a variable could then be (ignoring things such as scaling factors and offsets)

Actual_min, actual_max or actual_range: which describe the actual variable 
values held in the file, their minimum and maximum (e.g. for 
sea_water_salinity, I could have data with actual_min of 25.732, actual_max of 
34.994)
Valid_min, valid_max and valid_range: these detail the feasible range of data, 
so for salinity they could range from 0 to 41.5 (if following the BODC 
parameter vocabulary)
_FillValue which then presets the value that should be used for absent data - 
in the case of salinity, if keeping to the BODC parameter vocabulary, a value 
of -1.
My own idiom is to use _FillValue only for values that are never 
written, which was its original intent. If I need missing values then I 
add an explicit attribute missing_value; if there are multiple missing 
values, the attribute can be a vector.


So, I never use _FillValue explicitly, nor valid_min or valid_max, and 
always use missing_value if needed.


I always code to write every value out, never relying on unwritten 
values == missing values. That means I can set fill to off, which can 
save a lot of time when writing (up to 2x)


As I say, thats just my own personal idiom for writing netcdf.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] identification of vector components

[John Caron]

Heres my two cents:

The meaning of the x_coordinate and y_coordinate is actually well 
defined. But it does not mean x=east and y=north. It means the input to 
the projection function proj(x,y) - (lat,lon), which are defined in 
appendix F, with pointers to reference software. AFAIU, these 
coordinates are only meaningful in the context of this projection. Of 
course, for some projections it does mean east and north, but not always.


im not sure how that impacts this discussion on standard names, but i 
want to be clear that x and y are well defined. Its just that their 
properties vary depending on the projection.


John

On 4/18/2012 7:49 AM, Hedley, Mark wrote:

Hello Bryan


Sorry, silence doesn't mean consent.

I didn't think it did, but prodding that notion can encourage people to pitch 
in.

My reasoning is that I do not think it is the responsibility of the standard name to 
define what is meant by x.  The initial parts of the definition in the table: 
'x indicates a vector component along the grid x-axis' ... ', positive with 
increasing x'; say everything there is to say.

It is the responsibility of the coordinate and grid_mapping variables to define 
what 'x' means.

Rather than this we have the case now where significant metadata inspection on 
coordinate system and coordinate is required to determine the correct 
standard_name from two mutually exclusive choices when writing CF NetCDF.  This 
feels to me to be an unnecessary complication which delivers little benefit 
from a data and metadata definition perspective.


If you are importing something where x is used as the coordinate, and it is 
longitude, then why not put that in other metadata?

I would say that I have defined this explicitly, using the approach I propose.  
I define that the data variable is x-wind and I define that x is longitude, 
therefore I can infer that the x-wind data variable is eastward wind, with 
respect to the defined grid_mapping.  Forcing me to put it in the standard_name 
adds complexity to software which writes data and increases the opportunity for 
data to be written incorrectly.

For example, does the cf_checker cross reference the 'x' coordinate and any 
standard names to ensure that datasets defined with respect to a true longitude 
coordinate variable do not use standard names with the 'x' modifier?


The you say x, I say x, and we both mean different things, is what we need to 
avoid

This cannot be avoided, in almost all cases x means different things in 
different datasets. It can even mean different things in the same file.


in particular we must not change definitions of existing quantitities.

I don't think that it is safe to make that strong a statement on definition 
changes over time.  I can understand the desire to avoid invalidating datasets 
by narrowing definitions after they are defined; but I don't think that a 
constrained broadening of the definition of a modifier should be refused on 
principle.  Such changes sometimes need to take place to keep the standard as 
applicable to its community as possible.


That's not to say 'eastward' isn't a useful standard name: there is a good case 
for model intercomparison, as there is no guarantee that my 'x' is anything 
like your 'x' for a given dataset: we can agree to publish data as 'eastward' 
to allow quick and easy intercomparison.

even this becomes slightly problematic at small scales, as eastward is with 
respect to a coordinate reference system, so my east may be subtly different 
from yours.

many thanks
mark

-Original Message-
From: Bryan Lawrence [mailto:bryan.lawre...@ncas.ac.uk]
Sent: Wed 18/04/2012 11:34
To: cf-metadata@cgd.ucar.edu
Cc: Hedley, Mark
Subject: Re: [CF-metadata] identification of vector components

Hi Mark

Sorry, silence doesn't mean consent.

I think it is exactly the place of standard names to be completely proscriptive 
about what terms mean.

The you say x, I say x, and we both mean different things, is what we need to 
avoid, and in particular we must not change definitions of existing 
quantitities.

Admittedly, your change wouldn't strictly change anything retrospectively, 
since it's an inclusive change, but it's probably a dangerous thing to do. (My 
sense of deja vu tells me we've been here before, and I may even have been on 
the other side of the argument :-).

If you are importing something where x is used as the coordinate, and it is 
longitude, then why not put that in other metadata? The point of the CF 
standard is that it just that 

Bryan


There have not been any responses to this post in the last 10 days.

I know that this is a dangerous philosophy, but can I suggest that, in this 
case, silence equals consent?

If it is, I would like to see these amendments in the standard_name 
publications as soon as possible.  Would this cause concern?

many thanks
mark

-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu on behalf of Hedley, Mark
Sent: Thu 05/04/2012 17:35

Re: [CF-metadata] code that does semantic checking of CF headers

[John Caron]

On 4/19/2012 9:13 AM, Gaffney, Sean P. wrote:

Hi all,

My name is Sean Gaffney, from the British Oceanographic Data Centre, and I'm 
working on a project dealing with numerical model data that are in CF compliant 
NetCDF, so I thought I'd sign up to the community.

The project I am working on aims to develop a web-based delivery system for 
oceanographic numerical model data and has a module which allows visualisation 
of the data. We've been using test CF data to fine-tune some of the technical 
aspects of this visualisation.

I have a particular issue at the moment which I hope someone out there might be 
able to assist me with.

My problem started when I found that the test CF data were passing the BADC CF 
compliance checker, but not visualising properly. A check with the people who 
developed the visualisation module led to the discovery that, while the CF 
metadata were formatted correctly, the actual values within the metadata were 
incorrect e.g. the valid_min and valid_max attributes for both the latitude and 
longitude and dimensional variables had values which did not reflect the actual 
range of data in the file. The visualisation was setting itself up based on the 
values stored in the attributes and was therefore not displaying any data.


you should not use valid_min, valid_max, use actual_min, actual_max to 
document the actual range of data. Unnecessarily using valid_min, 
valid_max means that conforming software has to check each data value.


this hasnt made it into the docs yet, but you can see

https://cf-pcmdi.llnl.gov/trac/ticket/31



Has anyone in the CF community come across this sort of issue before and if so, 
what solutions would you recommend? My initial thoughts were that I'd have to 
develop some sort of code which interrogates the data file and compares the 
entries in the CF metadata header against the actual data values in the file, 
but I'd be interested to see what people think. Please bear in mind that I 
won't actually be generating model runs myself, but will be receiving data from 
people that have done so and need to know that I'm being given valid data and 
metadata.


In my experience, one can have a valid CF file, but the metadata is 
incorrect or insufficient to georeference data, since theres no 
requirement in CF that data actually be georeferenced.


The CDM validator gives feedback on what data it can georeference 
/visualize. Since we try to keep up with CF, its a good double check for 
this problem, a complement to the CF validator(s).


http://motherlode.ucar.edu:8080/cdmvalidator/validate.html

However, it would not have caught your problem. For that, you would want 
to visualize it and see that the picture was not what you would expect. 
You could automate that process in an integration testing framework.




Sorry for making my first message to the CF community so long.

Looking forward to your responses

Yours

Sean Gaffney
BODC



Sean Gaffney
Data Scientist
British Oceanographic Data Centre
Joseph Proudman Building
6 Brownlow Street
Liverpool
L3 5DA
UK
+44 (0)151 795 4950




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Using add_offset scale_factor with coordinate variables

[John Caron]

On 4/11/2012 3:59 PM, Jonathan Gregory wrote:

Dear Randy

While I understand why this would be convenient for you, I'm inclined to think
that your convenience would be outweighed by the inconvenience to the users of
the data in this case. I don't expect there is any existing CF-aware analysis
software which would expect to unpack coordinate variables, so the likely
consequence is that the coordinates would be presented to the user as packed
values, which isn't what you intend.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Hi all:

Well, the netcdf-Java library will unpack coordinate variables, so any 
applications built with it will also.


If one already knows how to unpack data variables, its not any harder to 
unpack coordinate variables.


I guess the question is: what are the CF-aware analysis software that we 
know of?


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I can 
check if it works in the CDM with the new 1.6 conventions, and maybe 
give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
joinNew  and joinExisting NCML aggregation. I call it syntactic 
aggregation because it doesnt know what its aggregating, and the 
homogeneity requirements are strict.  2) Feature Type collections (aka 
semantic aggregation) are the more recent development. These 
understand the coordinate information of the data, and so can handle 
variations in the representation, eg ragged arrays, scalar coordinates, 
etc. In this case, the CDM understands a dataset as a collection of 
features objects, which can be stored in a collection of files.  The 
interfaces to these collections is still under development. Most current 
and future work in the CDM is in this category.


John

On 4/3/2012 12:48 PM, Upendra Dadi wrote:

Hi,
   I think Jim is talking about NCML (virtual) aggregation. THREDDS 
can aggregate even when the variable is a scalar using joinNew. But 
I think THREDDS can aggregate only over regular arrays. That is, all 
the dimensions other than over which a variable is aggregated should 
be of same size across all the files. This is possible, for example, 
only if all the profiles have same depth( or pressure) levels. Which 
in general is not true. NCML aggregation I guess is of limited use 
when dealing with jagged arrays. But I agree with Jim in that ability 
to aggregate should be an important consideration when coming up with 
a representation. Physical aggregation is still possible. But I prefer 
virtual aggregation, since letting data to be stored in individual 
files in often operationally convenient, but users benefit from 
aggregated views. I wonder if NCML could be extended to be able to 
aggregate jagged arrays into incomplete multidimensional array 
representations.  I heard that ERDDAP has similar capability though I 
don't think it has anything like NCML where users can create views 
over remote data, not sure though.


Upendra



On 4/3/2012 1:27 PM, Jim Biard wrote:

Hi.

A colleague of mine has told me that if you use size-1 array for a 
variable, then data servers such as THREDDS can aggregate the 
variable over multiple files and deliver a single file in which the 
variables will be arrays of size  1.  He said that if a variable is 
a scalar, THREDDS won't do this.  (I don't mess with THREDDS, so I am 
just parroting what he said.)


If this is correct, then you might want to consider this point when 
deciding which way to represent coordinates.


Grace and peace,

Jim

On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi upendra.d...@noaa.gov 
mailto:upendra.d...@noaa.gov wrote:


Andrew,

However I have another small concern about all of this. Given
there is more than
1 way to treat dimensions does the netCDF plotting and
visualising sofware out
there understand both approaches. That is can we expect
something like
ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure
and work OK with BOTH approaches to coding the netCDF?

  Software like ncBrowse do not understand CF conventions. They
can read netCDF files, but wouldn't know how to interpret the
attributes like coordinates. So if you want to plot the profile
location on a map, for example, it wouldn't be able to do it by
itself. It would need to be told how to interpret the
coordinates.  A CF based software wouldn't have to be supplied
with the additional semantics since it can read from the file by
itself. But if you want to plot salinity vs pressure,  software
like ncBrowse can already do it since lot of these software make
it easy to make plots based on a shared dimension.  And here
salinity and pressure share the same dimension.  So I guess, the
correct answer to your question is - it depends on what kind of
plot or task you want to do and if the software can understand CF
conventions.

Upendra




Thanks and Regards All,
Andrew Walsh

- Original Message -
*From:* Lowry, Roy K. mailto:r...@bodc.ac.uk
*To:* Upendra Dadi mailto:upendra.d...@noaa.gov ; andrew
walsh mailto:awa...@metoc.gov.au
*Cc:* Luke Callcut mailto:l...@metoc.gov.au ;
cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu ;
sdn2-t...@seadatanet.org mailto:sdn2-t...@seadatanet.org
*Sent:* Tuesday, April 03, 2012 9:31 AM
*Subject:* RE: [CF-metadata] Ocean CTD data following CF
Conventions v1.6

Hi Upendra,
I like the idea of a station dimension.  It goes a long way
to resolving the issue raised in my response to Jim which
was based on the tunnel vision of having 

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/28/2012 6:26 AM, Jonathan Gregory wrote:

Dear all

Brian has a good point that sect 5.3 allows that there might be missing data
in aux coord variables. Appendix A - which is equally ancient in the story of
CF :-) - is not consistent with this, because it didn't allow _FillValue or
missing_value atts for coordinates until sect 9 was introduced in version 1.6.
But never mind the history, the point is to clarify what we need now.

In Randy's, Rich's and Bert's examples, if I understand correctly, there are
non-existent points, at which both data and aux coord vars are missing values.
That is also what sect 9 requires. I don't see any problem with this.

Nan's example is different, because it has missing values in aux coord vars
at points where there is non-missing data. If we all agree that this is OK
too, then fine. Speaking for myself, I could agree to it, but I'm less happy,
because clearly the aux coord var is not doing its job of locating the data.

I think you said, Nan, that you might fill them in at some later stage. At
that stage, they should certainly be aux coord vars. Before they are filled
in, of course I am not saying they should be excluded from the file, but I
am asking if they should be regarded as data, rather than coordinates. A
pressure value which was destined to be an aux coord var, but is actually a
data variable measured by a sensor and has missing values in it, could be
named by the ancillary_variables attribute. It is really per-point metadata,
which is what ancillary_variables are
(http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#ancillary-data)
but you could argue it should not be regarded as an aux coord var if it can't
provide information for every point where there is data.

I'm just asking. I don't have a very strong opinion about this, but I'd like
to know if others have the same concern that I do.

My existing ticket (85) connected with this subject is a defect ticket. It
is only able to correct mistakes. It can't make a substantive change to the
convention, just clarify it. If we can decide easily how to clarify it, that
is fine. I'll amend the ticket if we have a clear decision. Otherwise, we
should use a different ticket.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


I think we have a number of valid use cases for missing data in aux 
coordinates, and i would vote to allow that.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/28/2012 10:49 AM, John Caron wrote:
I think we have a number of valid use cases for missing data in aux 
coordinates, and i would vote to allow that. 


sorry i didnt really answer jonathan's concern. I would suggest this 
wording:


Auxiliary coordinates do not have to be monotonic or have unique values, 
and may contain missing values. Data providers may choose to put valid 
data where the auxiliary coordinates are missing, but applications are 
free to assume that data is missing where the auxiliary coordinates are 
missing.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/26/2012 2:24 AM, Jonathan Gregory wrote:

Dear Ros

Regarding this requirement:


9.6 Where any auxiliary coordinate variable contains a missing value, all
other coordinate, auxiliary coordinate and data values corresponding to that
element should also contain missing values.

Appendix A notes that missing data is allowed in aux coord vars only in the
case of discrete sampling geometries. This means the checker could regard it as
an error also if it finds any missing data in aux coord vars, unless a valid
featureType att is present, arising from 9.6, in addition to the above.

However, I think the convention should be clear that this is not allowed except
for discrete sampling geometries, so I'll open a defect ticket for that.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


I think this may be more complicated.

Consider a geosynch satellite with lat/lon aux coordinates. the nature 
of the image is that there are many points around the outside of the 
image that are not located on the earth. i dont see any good choices 
other than to put missing values there for lat/lon..


To add insult to injury, it seems possible that there are valid data 
values at these locations. Not sure about that however. Can anyone with 
geosynch data expertise comment?


It seems at minimum this is another case where missing values in aux 
coordinates are needed.



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

Hi all:

From a CDM developer perspective, an auxiliary coordinate is just as 
good as a regular coordinate variable. The extra requirements on 
coordinate variables are helpful in knowing when to optimize, eg 
monotonicity allows one to efficiently find the index given the 
coordinate value.


John

On 3/26/2012 10:05 AM, Jim Biard wrote:
I am working with satellite data, and I, for example, have timestamps 
that arrive in the data stream along with sensor measurements.  I can 
have independent missing values in both my time variable and my 
measurement variables.  I want to preserve all the incoming data, and 
the restriction on true coordinate variables having no missing 
values prevents me from designating my time variable as a coordinate 
variable.  If I want to represent the relationship between the time 
variable and the measurement variables, the only recourse I have is to 
designate the time variable as an auxiliary coordinate variable in my 
measurement variables.


In the observational world, you cannot always be assured of meeting 
all the restrictions imposed on true coordinate variables.  In fact, 
other restrictions imposed on coordinate variables might not be met 
(monotonicity, for example), even though the contents of the variable 
do function properly as coordinate information for another variable. 
 The only mechanism that I am aware of within CF to document the 
relationship is the auxiliary coordinate attribute.


On Mon, Mar 26, 2012 at 11:20 AM, Nan Galbraith ngalbra...@whoi.edu 
mailto:ngalbra...@whoi.edu wrote:


Hi Jonathan -

For underway CTD profiles (gliders and floats, too, I'd think) if
the pressure
sensor fails intermittently, you can approximate Z by
interpolating over
time, assuming there are good values along the track.  In final
data, we might
do that, but we might like to present raw data in CF files, too.
So, yes, this data
would be useful, with fill values here and there in the pressure
record.

Are we getting into a grey area that might be outside the scope of
the CF
standard -  judgements made about the usefulness of data?   Having
all your
coordinates seems like an excellent  NetCDF best practice and
could certainly
be a rule for many tools, but it could preempt the use of the CF
standard for
some kinds of observational data.

Cheers -
Nan


On 3/26/12 10:48 AM, Jonathan Gregory wrote:

Dear Nan and John

It's a good thing we're having this discussion! In my
understanding, we have
always prohibiting missing data in aux coord vars, and in
section 9 we
explicitly allowed for the first time. Evidently we should be
clear, one way
or the other (which was one of the intentions of the defect
ticket I opened).

The restriction on coord vars not having missing data is, I
think, hard to
avoid because they are required to be distinct and monotonic.

In Nan's case:

For something like towed CTD data, you might have a period
of time where
data from the pressure sensor is missing. If neither the
coordinate or aux
coordinate can contain null values, does this mean the
only options are
interpolating Z or removing that section of data?

When the sensor is missing, does that mean the data can't be
geolocated? As
you know, geolocation is one thing CF tries hard to do. Is the
data useful if
you don't know where it comes from? Perhaps you know in some
other way?

In John's case

Consider a geosynch satellite with lat/lon aux
coordinates. the
nature of the image is that there are many points around
the outside
of the image that are not located on the earth. i dont see
any good
choices other than to put missing values there for lat/lon..

Bert has coincidentally mentioned a similar case (not on the
list). I tend to
agree that if index space contains locations that simply do
not exist, you
could put missing data in aux coord vars, but I think this
needs a change to
the CF convention.

To add insult to injury, it seems possible that there are
valid data
values at these locations. Not sure about that however.
Can anyone
with geosynch data expertise comment?

As in Nan's case, I am curious about how the data can be
useful if it doesn't
have a location.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu mailto:CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



-- 

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/23/2012 1:59 PM, Jim Biard wrote:

Hi.

Jonathan's reply contained the section:

9.6 Where any auxiliary coordinate variable contains a missing
value, all
other coordinate, auxiliary coordinate and data values
corresponding to that
element should also contain missing values.


I thought I understood that missing values were forbidden for true 
coordinate variables.  Has this changed?
No, coordinate variable cant have missing values but auxiliary 
coordinates can.


This requirement seems wrong to me anyway.  If the values in the data 
and auxiliary coordinate variables come from an external data source, 
it is entirely possible that you could have a measurement missing from 
one without it being missing from the other.  Why force a missing 
value into the data when you might, in fact, have a valid value?



An auxiliary coordinate is important to have for valid data, not some 
optional information. If it is optional, dont make it an auxiliary 
coordinate.


Allowing missing values in auxiliary coordinates is very useful, eg see 
the discrete sampling proposal, allowing you to use a rectangular array 
to store ragged arrays.



John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Example 5.10. British National Grid has wrong parameter names

[John Caron]

Example 5.10. British National Grid

  int crs ;
crs:grid_mapping_name = transverse_mercator;
crs:semi_major_axis = 6377563.396 ;
crs:semi_minor_axis = 6356256.910 ;
crs:inverse_flattening = 299.3249646 ;
crs:latitude_of_projection_origin = 49.0 ;
crs:longitude_of_projection_origin = -2.0 ;
crs:false_easting = 40.0 ;
crs:false_northing = -10.0 ;
crs:scale_factor_at_projection_origin = 0.9996012717 ;


but Appendix F has:

Transverse Mercator
grid_mapping_name = transverse_mercator

Map parameters:
  scale_factor_at_central_meridian
  longitude_of_central_meridian
  latitude_of_projection_origin
  false_easting
  false_northing

I think we can presume that

  scale_factor_at_central_meridian == scale_factor_at_projection_origin
  longitude_of_central_meridian == longitude_of_projection_origin

central_meridian is probably clearer; so maybe just update the example ??

or we can add this amendment to ticket #75= fix documentation and 
definitions of 3 grid mapping definitions

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] FW: GOES-R metadata prototype

[John Caron]

hi michael

1) There are a few typos (see attached) that made it invalid NcML. also 
.ncml suffix is preferred over .xml
2) for cdm_data_type, i might not call this a grid, maybe image is 
better. the satellite group should probably come up with a feature type 
name for geostationary data.
3) dont use valid_min / valid_max unless you really have data stored 
outside that range that you want marked as missing.

use actual_min/actual_max if you want to document actual ranges.

if you can give me a real file with data and coordinates in it, i can 
test the projection parameters.


john


On 3/7/2012 3:30 PM, Carlomusto, Michael wrote:

_
*From:* Carlomusto, Michael
*Sent:* Wednesday, March 07, 2012 5:20 PM
*To:* 'CF-metadata@cgd.ucar.edu'
*Subject:* GOES-R metadata prototype
The attached NcML file has been vetted by a small group of people from 
the CF metadata list, especially with regard to the representation of 
the proposed GOES projection.

I would like to submit it for comments to the larger group.
I encourage a critical evaluation.
Thanks.
Mike Carlomusto
*/This information is only intended for the use of the individual or 
entity named in this email. If you are not the intended recipient, you 
are hereby notified that any disclosure, copying, distribution, or use 
of the contents of this information is strictly prohibited. If you 
have received this email in error, please notify me and delete it 
immediately./*

Michael Carlomusto
mcarl...@harris.com
Harris Corp.
Government Communications Systems Division (GCSD)
Melbourne, FL
(321) 309-7905


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


?xml version='1.0' encoding='UTF-8'?
netcdf xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2  http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd;
xmlns=http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2;

  !-- Title : Volcanic Ash Product --
  !-- Identifier / Metadata Reference --

  !-- global attributes --
  attribute name=id value=ef175830-cc1c-11e0-bfd6-0800200c9a66 type=string/
  attribute name=dataset_name value=OR_ABI-L2-VAAF-M3_G16_s2011177060530_e2011177062030_c2011177080607.nc
 type=string/
  attribute name=naming_authority value=gov.noaa.goes-r, SE-21_GS_8021130_Metadata_Model_Rev-A_Final.docx
 type=string/
  attribute name=iso_series_metadata_id value=a70be540-c38b-11e0-962b-0800200c9a66/
  attribute name=Conventions value=CF-1.6 type=string/
  attribute name=Metadata_Conventions value=Unidata Dataset Discovery v1.0, CF-1.6 type=string/
  attribute name=standard_name type=string value=volcanic_ash_detection_and_height/
  attribute name=standard_name_vocabulary value=CF Standard Name Table (v18, 22 July 2011) type=string/
  attribute name=title value=Advanced Baseline Imager (ABI) Level 2+ Volcanic Ash: Detection and Height
 type=string/
  attribute name=summary
 value=The volcanic ash product provides an estimation of cloud height and mass loading (mass per unit area) of volcanic ash clouds observed by the Advanced Baseline Imager (ABI)...
 type=string/
  attribute name=keywords value=Spectral/EngineeringSensor CharacteristicsImager type=string/
  attribute name=keywords_vocabulary
 value=NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 6.0 type=string/
  attribute name=platform value=GOES-16 type=string/
  attribute name=sensor value=GOES 16 Imager type=string/
  attribute name=license value=Unclassified data.  Access is restricted to approved users only. type=string/
  attribute name=processing_level value=National Aeronautics and Space Administration (NASA) L2 type=string/
  attribute name=institution value=DOC/NOAA/NESDIS U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services type=string/
  attribute name=date_created value=2016-09-09T14:45:00Z type=string/
  attribute name=processing_site value=NSOF type=string/
  attribute name=satellite_position_flag type=string value=G16/
  attribute name=spatial_resolution type=string value=2km at nadir/
  attribute name=scene_id value=Full Disk type=string/
  attribute name=cdm_data_type value=Grid type=string/
  attribute name=project value=GOES type=string/
  attribute name=time_coverage_start value=2016-09-09T14:30:00Z type=string/
  attribute name=time_coverage_end value=2016-09-09T14:35:00Z type=string/
  attribute name=timeline_id value=ABI Mode 3 type=string/
  attribute name=platform_ID value=G16 type=string/
  attribute name=instrument_ID value=ABI-1 type=string/
  attribute name=percent_uncorrectable_L0_errors type=float value=3.45/
  attribute name=nominal_satellite_subpoint_lat type=float value=0.0/
  

Re: [CF-metadata] machine-accessible CF compliance checker?

[John Caron]

Hi all:

If the checker can read from opendap, then putting the files on an 
opendap server might be helpful, in

that only the metadata is read instead of the entire file.

There are a few difference among servers (Hyrax, TDS, GDS, etc) but im 
pretty sure this would work for
netcdf files well enough for the checker's needs. could be worth finding 
out.


john

On 2/29/2012 7:55 AM, Bentley, Philip wrote:

Hi Stephen,

I may well be wrong, but I suspect that one of the reasons - perhaps the
main reason - why folks would like access to a stand-alone cfchecker
utility is the large size of their netcdf files. (It's certainly the
reason we use a local version.)

As you know only too well (from CMIP5), most of the netcdf files we're
dealing with these days run into the high 100s of MB or low GBs. It's
not really practical to upload such files in order to perform a CF
compliance check. At least not on more than a few files.

I wonder, therefore, if a possible solution is to develop an alternative
checker utility that operates not on the netcdf file itself, but rather
on the header info output by the ncdump -h command. Generating and
uploading those would be trivial by comparison.

Off the top of my head (and having given it almost zero thought!) the
only test that I can think of which examines the actual data values is
the check for coord vars being monotonic. But I suspect there are others
I'm unaware of.

If nothing else, such a utility might have some value as a 'first pass'
checker.

Any mileage in this? Apologies if it's already done the rounds!

Regards,
Phil



-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
stephen.pas...@stfc.ac.uk
Sent: 29 February 2012 11:39
To: christopher.s.lyn...@nasa.gov; cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] machine-accessible CF compliance checker?

Hi Christopher,

Are you looking for a way to run the cfchecker locally or
submit checking requests to one of the websites in a script?

I think both would be possible with a little help.  The
cfchecker is now available on the Python Package Index at
http://pypi.python.org/pypi/cfchecker.  It needs a little
help to work out of the box -- you need to configure UDUNITS
and the location of several CF XML files.  It's on my
long-term TODO list to make this configuration easier.

Also the CF-checker at BADC and NCAS-CMS has a very simple
HTTP interface that could be scripted.

Cheers,
Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK


-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
Lynnes, Christopher S. (GSFC-6102)
Sent: 28 February 2012 16:03
To: cf-metadata@cgd.ucar.edu
Subject: [CF-metadata] machine-accessible CF compliance checker?

I know about the web sites where you use a form to upload
files to check CF compliance, but is there a service
somewhere that can be accessed by a script?
--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone:
301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
Scanned by iCritical.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] defects in example 7.11, 7.12

[John Caron]

Possible defects:

1) Example 7.11. Extreme statistics and spell-lengths

 climatology_bounds=2007-12-1 6:00, 2000-8-2 6:00;

should be

 climatology_bounds=2007-12-1 6:00, 2008-3-1 6:00;

perhaps clearer as

 climatology_bounds=2007-12-1, 2008-3-1;

2) flip order of 7.11 and 7.12, since 7.12 refers to 7.10.

3) 7.12 Temperature for each hour of the typical climatological day

 temperature:cell_methods=time: mean within days ,
  time: mean over days time: mean over years;

may be confused with array of strings, perhaps should be:

 temperature:cell_methods=time: mean within days time: mean over days 
time: mean over years;



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] How to represent 6-hourly diurnal monthly means in CF ?

[John Caron]

Thanks, Jonathan.

On 2/21/2012 11:18 AM, Jonathan Gregory wrote:

Dear John


However, im not sure what this sentence in 7.4 means: When
considering intervals within days, if the earlier time of day is
equal to the later time of day, then the method is applied to a full
24 hour day.

It doesn't matter in the instantaneous case, but it does matter for statistics
that apply to the whole cell, like means and accumulations, such as


 precipitation:cell_methods=time: sum within days time: mean over days;

If the sums were for 24-hour intervals, the bounds might be


   climatology_bounds=2008-08-01T00:00Z, 2008-09-01T00:00Z,

Here, the time of day is the same (00:00Z) at both ends. That means it
applies to full days. That's what the bit quoted is talking about. However,
this doesn't arise in your six-hourly case.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] How to represent 6-hourly diurnal monthly means in CF ?

[John Caron]

Im working with NOAA's CFSR dataset, reading it into the CDM, which 
essentially means converting GRIB into CF/netCDF.


ref:
http://nomads.ncdc.noaa.gov/data.php#cfs-reanal-data
http://dss.ucar.edu/datasets/ds093.2/

from NCAR's DSS page:

 CFSR monthly atmospheric, oceanic and land surface output products 
are available as 6-hourly diurnal monthly means (00Z, 06Z, 12Z, and 18Z)


If I understand this correctly, the monthly mean data is as follows:

  for each month:
  for each day of the month:
compute separate averages of the data at the 0,6,12, and 18Z 
time coordinate.


So one ends up with 4 time coordinates per month, and the data 
represents the average for that month at that time. The time coordinate 
is not a real calendar date, for example for the month 2008-08, the time 
coordinate is not really


   2008-08-01T00:00:00.000Z
   2008-08-01T06:00:00.000Z
   2008-08-01T12:00:00.000Z
   2008-08-01T18:00:00.000Z

but more like:

   2008-08-*T00:00:00.000Z
   2008-08-*T06:00:00.000Z
   2008-08-*T12:00:00.000Z
   2008-08-*T18:00:00.000Z

where the * indicates an average over the days in that month.

From

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#climatological-statistics

It appears that CF would represent this as:

dimensions:
  time=4;
  nv=2;

 variables:
  float temperature(time,lat,lon);
temperature:long_name=surface air temperature;
temperature:cell_methods=time: point within days time: mean over days;
temperature:units=K;

 double time(time);
time:climatology=climatology_bounds;
time:units=days since 2008-01-01;

 double climatology_bounds(time,nv);

data: 
  time=2008-08-01T00:00Z,2008-08-01T06:00Z,2008-08-01T12:00Z,2008-08-01T18:00Z ;


  climatology_bounds=2008-08-01T00:00Z, 2008-08-31T00:00Z,
 2008-08-01T06:00Z, 2008-08-31T06:00Z,
 2008-08-01T12:00Z, 2008-08-31T12:00Z,
 2008-08-01T18:00Z, 2008-08-31T18:00Z ;

Using point within days because  we have an instantaneous quantity.

However, im not sure what this sentence in 7.4 means: When considering 
intervals within days, if the earlier time of day is equal to the later 
time of day, then the method is applied to a full 24 hour day.



Another case is when the data are on time intervals, for example Total 
Precipitation Accumulation. In this case, the original data are on time 
intervals which i represent as (start,end) dates:


2008-08-01T00:00:00.000Z, 2008-08-01T00:00:06.000Z
2008-08-01T06:00:00.000Z, 2008-08-01T00:00:12.000Z
2008-08-01T12:00:00.000Z, 2008-08-01T00:00:18.000Z
2008-08-01T18:00:00.000Z, 2008-08-01T00:00:24.000Z
2008-08-02T00:00:00.000Z, 2008-08-02T00:00:06.000Z
2008-08-02T06:00:00.000Z, 2008-08-02T00:00:12.000Z
2008-08-02T12:00:00.000Z, 2008-08-02T00:00:18.000Z
2008-08-02T18:00:00.000Z, 2008-08-02T00:00:24.000Z
2008-08-03T00:00:00.000Z, 2008-08-03T00:00:06.000Z
...

which are are averaged over days, giving 4 averages for each month, 
which i denote as:


2008-08-*T00:00:00.000Z, 2008-08-*T00:00:06.000Z
2008-08-*T06:00:00.000Z, 2008-08-*T00:00:12.000Z
2008-08-*T12:00:00.000Z, 2008-08-*T00:00:18.000Z
2008-08-*T18:00:00.000Z, 2008-08-*T00:00:24.000Z

So perhaps this is:

dimensions:
  time=4;
  nv=2;

 variables:
  float precipitation(time,lat,lon);
precipitation:long_name=surface air temperature;
precipitation:cell_methods=time: sum within days time: mean over days;
precipitation:units=K;

 double time(time);
time:climatology=climatology_bounds;
time:units=days since 2008-01-01;

 double climatology_bounds(time,nv);

data: 
  time=2008-08-01T03:00Z,2008-08-01T09:00Z,2008-08-01T15:00Z,2008-08-01T21:00Z ;


  climatology_bounds=2008-08-01T00:00Z, 2008-08-31T06:00Z,
 2008-08-01T06:00Z, 2008-08-31T12:00Z,
 2008-08-01T12:00Z, 2008-08-31T18:00Z,
 2008-08-01T18:00Z, 2008-08-31T24:00Z ;

??




thanks for any insights,
 John



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Convention attribute

[john caron]

On 12/22/2011 2:11 AM, Jonathan Gregory wrote:

Dear all

The existing Unidata recommendation is OK and we could incorporate it into
CF but it would help to be more precise, for instance: If the Conventions att
includes no commas, it is interpreted as a blank-separated list of conventions;
if it contains at least one comma, it is interpreted as a comma-separated list.
Blank-separated lists are more CF-like - many CF attributes use that syntax -
but obviously we can't insist that other conventions don't have blanks in their
names, and it would be simpler therefore to use a comma-separated list for
this attribute, despite the Unidata recommendation.

I see no problem with allowing multiple conventions except the important
proviso that if the file follows more that one convention it is the
responsibility of the data-writer to ensure there is no inconsistency between
the metadata following these conventions. That is, they must serve
complementary purposes. It would be impossible to check this automatically so
we have to depend on the data-writer doing it correctly.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

I think this would solve the problem:

If the Conventions att
includes no commas, it is interpreted as a blank-separated list of conventions;
if it contains at least one comma, it is interpreted as a comma-separated list.

I could also point out that reading software has a list of conventions it 
recognizes, so in practice one takes the result of this parsing and compares to 
a known list. also, the netcdf-4 data model allows attribute values to be a 1-d 
array of Strings.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] inconsistencies in translating projection parameters OGC WKT - CF-1.5

[John Caron]

Hi Etienne:

Generally, CF has followed proj4 conventions as much as we could, and 
John Snyder's reference book is often underneath:


http://pubs.usgs.gov/pp/1395/report.pdf

OGC specs tended to come from experts using the EPSG, and generally we 
are less familiar with that.


some specific comments below, based on my own understandings.

On 12/13/2011 2:21 PM, Etienne Tourigny wrote:

Hi all,

I am opening a seperate thread at this deals merely with the problems
encountered in translating projection parameters to/from OGC WKT and
CF-1.5.

I would appreciate any guidance in resolving these issues.

In summary:

1) LCC (Lambert conformal): should include a scale_factor parameter in
the 1 standard parallel case
2) CEA (Lambert Cylindrical Equal Area): how to convert the
'scale_factor_at_projection_origin' alternative to standard parallel
to OGC WKT?
3) PS (Polar stereographic):This projection's parameters in CF-1 are
significantly different from OGC/PROJ.4 ,  CF-1 standard parameters
'Either standard_parallel or scale_factor_at_projection_origin' is
incompatible with OGC WKT, which requires both

More detailed information (reproduced here) can be found at
http://trac.osgeo.org/gdal/wiki/NetCDF_ProjectionTestingStatus

1) LCC 1SP (Lambert conformal)

The OGC WKT 'Lambert_Conformal_Conic_2SP' maps clearly to LCC in CF-1
with 2 std parallels.

The CF-1 spec claims that a 1SP version can be used, but it doesn't
include a 'scale_factor' as does the OGC WKT
'Lambert_Conformal_Conic_1SP'.

For import: Given discussion and example on #3324, and also support
from NARCCAP project: it seems the CF-1 LCC 1SP projection is a
'degenerate' LCC-2SP with both SP's at the same location. Thus seems
we can read with both 'std_parallel' (1 value) and
'latitude_of_projection_origin' mapping to latitude_of_origin (WKT)
and setting scale_factor=1? This is the code's current behaviour
(though perhaps could do more error-checking).
yes, we handle both 1  and 2 standard parallels in the same projection. 
the 1 parallel can be thought of as a degenerate case of the 2 parellels.


AFAIU, the scale factor is determined by the standard parallels, so is 
reduntant to specify, see Snyder, equation 15-4.


its possible that the more general case is to use another constant to 
multiply the one given in 15-4, but i have never seen that.


Im not sure what #3324  refers to.



For export: given the LCC-1SP can specifically include a scale_factor
to control the projection (see
http://www.remotesensing.org/geotiff/proj_list/lambert_conic_conformal_1sp.html),
it seems this can't be directly exported to NetCDF without throwing
away the scale factor.
Perhaps the reasonable thing for now is to export the scale_factor to
NetCDF, so it can at least be read back in. Otherwise, we'd have to
try to translate the LCC-1SP to a LCC-2SP on the fly.


2) CEA (Lambert Cylindrical Equal Area):

The CF-1 conventions claim this can be encoded with a
'scale_factor_at_projection_origin' alternative to standard parallel -
how would this conversion be done from/to OGC WKT (which requires
central_meridian and standard_parallel_1)?


It does appear at first glance that there should be a seperate scale 
factor. I am less familiar since I havent coded this one.



3) PS (Polar stereographic)

This projection's parameters in CF-1 are significantly different from
http://www.remotesensing.org/geotiff/proj_list/polar_stereographic.html.
In particular the CF-1 equivalent for 'latitude of natural origin'
seems to be 'standard_parallel'.


http://www.remotesensing.org/geotiff/proj_list/polar_stereographic.htmli 
agree, i dont know where these CF names came from.




CF-1 defines either standard_parallel or
scale_factor_at_projection_origin, which is incompatible with OGC WKT
(and OGR) that requires both. Note: we have verified the current
approach we take to mapping is the same used by at least one other
NetCDF CF-1 user in the UCAR NARCCAP project.

Current implementation (as of Dec. 13) uses the following rules:
- WKT scale_factor maps to CF scale_factor_at_projection_origin
- WKT central_meridian maps to CF straight_vertical_longitude_from_pole
- WKT latitude_of_origin maps to WKT standard_parallel
- WKT to CF: if WKT latitude_of_origin is positive then CF
latitude_of_projection_origin is +90, else it is -90 (North vs. South
Pole)
- CF to WKT: unsupported if standard_parallel is not defined, as
OGRSpatialReference::SetPS() requires both CF
scale_factor_at_projection_origin
Im guessing these rules may have to be specific to a projection, rather 
than assuming you can do a global mapping of the attribute names.


I would like to incorporate your findings into the CDM library when I 
can. Do you have any official documentation describing this mapping?




Thanks
Etienne
___


The question for CF is whether we should start to follow the WKT spec 
closer, and add aliases where needed.  I would prefer this since most of 
us arent 

Re: [CF-metadata] udunits corresponding to Forel-Ule, milliequivalent

[john caron]

hi:

I think I would also advocate adding another standard attribute, 
something like units_label which would be a label for the units in a 
plot, not necessarily udunit compliant.


eg:
  var : units = ;
  var : units_label = milligrams per gram of dry sediment;

john

On 12/9/2011 2:09 AM, Lowry, Roy K. wrote:

Hi Upendra,

The Forel-Ule is another example where parameter semantics have been off-loaded 
in the units of measure, such as 'milligrams per gram of dry sediment'.  I have 
been working to eliminate this by moving the semantics into the parameter 
description to leave a UDUNITS-compatible UoM.

In BODC we have an equivalent to a Standard Name of 'Colour of the water body 
by visual estimation and conversion to a number on the Forel-Ule scale' which 
has units of dimensionless. This would require a new Standard Name were this 
approach to be used in CF.

I'm guessing you want milliequivalents for total alkalinity data.  Previously 
in CF the standard name 'sea_water_alkalinity_expressed_as_mole_equivalent' has 
been used with the canonical unit 'Moles per cubic metre', which is a pragmatic 
solution to your problem that works in the oceanographic domain because 
alkalinity is the only thing we encounter in Equivalents and luckily for us the 
chemistry is such that 1 Mole = 1 Equivalent.

Cheers, Roy.

-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu 
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Steve Emmerson
Sent: 08 December 2011 23:58
To: Upendra Dadi
Cc: cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] udunits corresponding to Forel-Ule, milliequivalent

Upendra,

 From the perspective of the UDUNITS unit-packages, the unit Forel-Ule
is the unit Forel multiplied by the unit Ule -- neither, of which,
is known by the UDUNITS or UDUNITS2 packages. I suggest you contact the
author.

Regards,
Steve Emmerson

On 12/08/2011 09:05 AM, Upendra Dadi wrote:

Hi,
   I have some data which has Forel-Ule for the units used. Is there a
udunits corresponding to this? Also, what should I use for
milliequivalent ? Thanks for the help.
Upendra
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Proposal for better handling vector quantities in CF

[John Caron]

On 12/9/2011 11:37 AM, Jonathan Gregory wrote:

Dear John

I prefer the idea that Thomas has put forward of an umbrella, rather than
containing the vector/tensor components in one data variable, because

* I really don't like the idea of units being mixed within a data variable.
I think things with different units must certainly be different quantities
and could not be regarded as the same field. You can get away with it if they
are all m s-1, for instance, but not so easily if the vertical one is orders
of magnitude different from the horizontal, and not at all if the vector is
expressed in polar coordinates.


I think the common case is that the vector components have the same 
unit. One could restrict to that case.




* I think it would be very inconvenient, and would break a lot of existing
software, if the coordinates were not what they appeared to be, because an
offset had to be added. Also, in general, the component fields of a staggered
grid do not have the same dimensions, as well as differing in the coordinates.

Im not sure what an offset had to be added means.

I think the common case of staggered grids could be handled with a 
convention defining the staggering, rather than seperate dimensions. I 
pull out the one Rich Signell and I cam up with a long time ago, for its 
own sake.




* It avoids the need to define a convention for labelling vector/tensor
components.
I think this convention would be about as complex as the one you will 
need for Thomas' proposal.




* It is completely backwards-compatible as regards the component fields, which
are exactly as before; we're just adding some separate information linking
them. This seems neat to me.


I agree thats a strong reason for Thomas' method.

OTOH, if we start thinking in terms of the extended model, a Structure 
(compound type in HDF5 parlance) might be useful. What do you think 
about starting to think about possible uses of extended data model?


Thanks for your thoughts, as always, interesting.

John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Proposal for better handling vector quantities in CF

[John Caron]

On 11/24/2011 2:53 PM, Thomas Lavergne wrote:

Dear all,

This email is a proposal to strenghthen the storage and exploitation of 
vector/tensor data in CF. Thanks to Jonathan for commenting an early version of 
this note.

As far as I can tell, vectors are not handled as such by CF, only their components (via 
the standard names defining them, e.g. sea_ice_x_velocity, northward_sea_ice_velocity, 
eastern_wind, etc...). Life and some applications (e.g. plotting) would be easier if it 
was possible to group all components of a vector field into a single vector 
object.

Here is my use case: I have an ice drift product, thus two datasets to define my vectors: 
sea_ice_x_displacement, and sea_ice_y_displacement. Note that it could be any combination of x/y, north/east, 
module/direction. It is moreover not limited to ice drift, but rather applies to any 2D (3D as well) 
variables such as vectors. As far as I know, the current CF does not provide me a way to group 
these two components an re-unite them into a vector. Two consequences: 1) I cannot define a third variable 
(say status_flag) that would apply to the vector object (thus to both its components). And 2) computer 
programmes (that for example want to draw vectors instead of colour contours) have to guess that 
my CF file contains a vector. The software has to skim through my variables, check that any two pairs of 
standard names define a vector, and propose a vector plot option to the user. This might work in 
simple files, but will fail if my CF files contains 2 sets of vectors, say on

e

  from model, the other from satellite: X_model, Y_model, X_sat, Y_sat. Will a 
software be smart enough to avoid proposing a (X_model,Y_sat) vector plots when 
all the 4 share the same standard_names: sea_ice_(x|y)_displacements?

Here, an approach could be that the X dataset defines its corresponding Y dataset as an 
auxiliary variable (and the Y dataset would do the same with X). This would 
probably work, but does not solve my concern number 1 to share a 3rd variable with both X 
and Y.

The solution I propose for discussion is to allow an umbrella dummy dataset (like the proj/mapping ones: no dimension, no 
data, just attributes). This umbrella variable would have a valid standard name sea_ice_displacement_vector (definition of 
vector). We would then define a new standard attribute pattern: components =space separated list of components, 
e.g. dX dY dir. The string values in  the list are the name of the datasets containing the components of the vector. Note 
that even for a 2D vector, I could choose to have both x/y and speed/dir in the same CF file, hence the need to allow more than just 2 
components, even for a 2D vector. We must have at least 2.

So in my case:

The two X and Y datasets and the direction:

float dX(time, yc, xc) ;
  dX:long_name = component of the displacement along the x axis of the grid ;
  dX:standard_name = sea_ice_x_displacement ;
  dX:units = km ;
  dX:_FillValue = -1.e+10f ;
  dX:coordinates = lat lon ;
  dX:grid_mapping = Polar_Stereographic_Grid ;

float dY(time, yc, xc) ;
  dY:long_name = component of the displacement along the y axis of the grid ;
  dY:standard_name = sea_ice_y_displacement ;
  dY:units = km ;
  dY:_FillValue = -1.e+10f ;
  dX:coordinates = lat lon ;
  dX:grid_mapping = Polar_Stereographic_Grid ;

float dir(time, yc, xc) ;
  dY:long_name = direction of the displacement ;
  dY:standard_name = direction_of_sea_ice_displacement ;
  dY:units = degrees ;
  dY:_FillValue = -1.e+10f ;
  dX:coordinates = lat lon ;
  dX:grid_mapping = Polar_Stereographic_Grid ;


The new dummy umbrella:

int ice_drift_vector;
  drift_vector:standard_name = sea_ice_displacement ;
  drift_vector:long_name = sea ice drift vector ;
  drift_vector:components = dX dY dir ;

A status flag for the vector:

byte status_flag(time, yc, xc) ;
  status_flag:standard_name = sea_ice_displacement status_flag ;
  status_flag:long_name = rejection and quality level flag ;
  status_flag:valid_min = 0b ;
  status_flag:valid_max = 30b ;
  status_flag:grid_mapping = Polar_Stereographic_Grid ;
  status_flag:coordinates = lat lon ;
  status_flag:flag_values = 0b, 1b,..., 22b, 30b ;
  status_flag:flag_meanings = missing_input_data over_land ... interpolated 
nominal_quality ;

When browsing through the file, a software would immediately see that there are 
vectors available (e.g. for display) and which datasets hold the components. It 
still have to read the component datasets to know how to use them (by reading 
the standard_name). We could even imagine that tools are able to automatically 
compute (for display or comparison)the vector length although only the x/y 
components are available.

Some new standard names would be needed, of course. As well as the components 
attribute. A revisit/cleanup of many definitions of existing standard names of component 
variables could also be envisaged.

This proposal needs to be more thouroughly discussed. Those CF 

Re: [CF-metadata] [thredds] Compression by Gathering working with Thredds?

[John Caron]

Hi Matthias:

The CDM/TDS does not support Compression by Gathering, nor does any 
other package i know of. There was some internal talk in CF of 
deprecating this. I am cc'ing the CF list in case anyone there wants to 
comment.


Have you looked at using netcdf-4 with compression?

John

On 12/7/2011 1:07 AM, Matthias Müller wrote:

Dear Thredds experts,

in order to save us some disk space and maybe increase the performance 
of a Thredds/ncWMS server we are planning to apply Compression by 
Gathering (CF Conventions 1.6, section 8.2) to our landuse and 
land-bound climate datasets. Before adapting our data massaging 
routines I would like to make sure that the results will be compatible 
with the Thredds server.


Can you please comfirm that Thredds supports netCDF files that utilize 
Compression by Gathering?


Thank you,
Matthias





___
thredds mailing list
thre...@unidata.ucar.edu
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] FW: netcdf for particle trajectories

[John Caron]

Hi Ute:

On 11/25/2011 6:01 AM, Ute Brönner wrote:

Hi folks,

I kind of lost track of our latest discussions and had the feeling that this 
was partly outside the mailing group; so I will try to sum up what we were 
discussing.
My latest try was to produce NetCDF for particle trajectory trying to write out 
the concentration grid which resulted in a 11GB netFCDF3 file :-(

So we have different motivations for discussion particle trajectory and netcdf4.

First question:
Does anybody know if and if yes, when writing netCDF4 will be incorporated into 
the NetCDF Java library? Or will we use Python with the help of Jython etc. 
(http://www.slideshare.net/onyame/mixing-python-and-java) to write netCDF4?


Im intending to incorporate the netcdf-4 C library into the netcdf-java 
library using JNI. Im hoping to have something working in the next few 
months, but we'll see. This will be an optional component, and will 
obviously make portability an issue. If you want to use Python, probably 
the one to use is Jeff Whittaker's at 
http://code.google.com/p/netcdf4-python/, which is also an interface to 
the netcdf-4 C library.



Second question:
Is there a de facto standard / proposal for writing Particle Trajectory Data which 
could be CF:featureType:whatever we agree on? The suggestion below is not 
suitable because:
1) we don't track a particle the whole time, it may disappear and show up again 
later, but if I have 1000 particles in time step 1 and 1000 in time step 2 we 
cannot be sure these 1000 are the same as before.
2) I cannot know the number of time steps in advance.



I think its time to start using netcdf-4 for large collections of point 
data which need to be compressed. Instead of first making a standard, we 
need to try out the possibilities and see how it performs. I think you 
want to use Structures, as well as multiple unlimited dimensions. With 
netcdf, we dont need the ragged array mecahnism - thats only needed to 
overcome the limitations of the classic model.


Has anyone started down this path? If so, can you post example netcdf-4 
files?



I would like sth. like
dimensions:
particle = UNLIMITED; //because it may change each time step
time = UNLIMITED; // because I don't know

then every variable is like
latitude (particle, time)
longitude (particle, time)

and I might have
int number_particles_per_timestep(time);
  :units = 1;
  :long_name = number particles per current timestep;
  :CF:ragged_row_count = particle;

That some of you need to know which spill a particle came from, may be solved 
with a 3rd dimension spill
dimensions:
spill = 3; // or how many one has
particle = UNLIMITED; //because it may change each time step
time = UNLIMITED; // because I don't know

particle (spill, time)

then every variable is like
latitude (particle)
longitude (particle)

how would one write this? With coordinates or as hierarchical data structure?
At least we need the ability to use several unlimited dimensions and the 
ragged-array feature.

Third question:
How can we compress big netCDF3 files? Or is it smarter to go for netCDF4 
directly with hierarchical data. As in my example above I would need to write 
out a 11 GB file and then deflate it like described here 
http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2010/msg00095.html
  or with Rich's script; but is that really necessary?


Hoping to get up the discussion again and that we agree on a standard quite 
soon!
Have a nice weekend!

Best,
Ute

 Original Message 
Subject: [CF-metadata] Particle Track Feature Type (was: Re: point observation 
data in CF 1.4)
Date: Fri, 19 Nov 2010 04:15:35 +0100
From: John Caronca...@unidata.ucar.edu
To: cf-metadata@cgd.ucar.educf-metadata@cgd.ucar.edu

Im thinking that we need a new feature type for this. Im calling it 
particleTrack but theres probably a better name.

My reasoning is that the nested table representation of trajectories is:

Table {
traj_id;
Table {
   time;
   lat, lon, z;
   data;
}
}

but this case has the inner and outer table inverted:

Table {
time;
Table {
   particle_id;
   lat, lon, z;
   data;
   data2;
}
}

So, following that line of thought, the possibilities in CDL are:

1) If avg number of particles ~ max number of particles at any time step, then 
one could use multdimensional arrays:

dimensions:
maxParticles = 1000 ;
time =  ; // may be UNLIMITED

variables:

double time(time) ;

int particle_id(time, maxParticles) ;
float lon(time, maxParticles) ;
float lat(time, maxParticles) ;
float z(time, maxParticles) ;
float data(time, maxParticles) ;

attributes:
:featureType = particleTrack;

note maxParticles is the max number of particles at any one time step, not 
total particle tracks. The particle trajectories have to be found by examining 
the values of particle_id(time, maxParticles).

2) The CDL of the ragged case would 

[CF-metadata] Fwd: Re: Proposed addition to CF principles: outside conventions

[John Caron]

 Original Message 
Subject: 	Re: [CF-metadata] Proposed addition to CF principles: outside 
conventions

Date:   Mon, 21 Nov 2011 14:39:13 -0700
From:   Seth McGinnis mcgin...@ucar.edu
To: John Caron ca...@unidata.ucar.edu



Hi John,

I'm generally in favor of this proposal.  The value of standards lies in their
widespread adoption, so if a popular solution already exists, we should adopt
it and bolster both standards rather than creating yet another competing
option.

It also make sense to delegate the design of a convention to those communities
that needed it more urgently and therefore have focused more on it than the CF
community has.

Cheers,

--Seth McGinnis


On Mon, 21 Nov 2011 07:56:02 -0700
 John Caronca...@unidata.ucar.edu  wrote:

The following is being proposed to add to CF principles, and comments from all
would be welcome:

CF may incorporate an outside convention into it when the following
conditions hold:

 1. The semantics of the convention are important to the CF community.
 2. The convention is already in wide use by other communities, and the
adoption by CF significantly helps other communities adopt CF.
 3. The convention is not in conflict with existing CF standards.
 4. In the case that the new convention overlaps existing CF
conventions, software should be developed to detect inconsistencies
and provide feedback to the data producer.



The current context is the proposal Specification of Coordinate Reference
System properties in Well-Known Text format :

https://cf-pcmdi.llnl.gov/trac/ticket/69
https://cf-pcmdi.llnl.gov/trac/ticket/69#comment:31

however, the intention is for this principle to guide future CF proposals
also.



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] standards for probabilities

[John Caron]

Hi Vegard:

I see some of these kinds of things from NCEP, encoded in GRIB, and Im 
still trying to understand what they are. So, some questions from a 
non-modeler:


On 11/15/2011 2:10 AM, Vegard Bønes wrote:

Hi!

I am trying to create a document containing various probability values for 
weather forecasts. But I do have some problems finding out how to express what 
I want to say using the cf metadata standard.

I want to express such things as 25th percentile precipitation amount (based 
on ensemble data), and probability that air temperature will be within 2.5 degrees of the 
forecast. How should I do this?


these are 2 different things, i guess?

1)  25th percentile precipitation amount (based on ensemble data)

* so here the data values are precip amounts? calculated from the 
cumulative distribution function (cdf) from an ensemble?
* do you typically have other percentile amounts in the same file, eg 50 
and 75?
* presumably this is some distillation of the cdf, used when the 
individual ensemble values are not in the file?
* is there any special handling that a generic tool could do, or is it a 
matter a just making this data available to some specialized application 
that you write?



2) probability that air temperature will be within 2.5 degrees of the 
forecast


* so here the data values are probabilities between 0 and 1 ?
* do you typically have other probabilities in the same file, eg within 
1 degree, or 5 degrees?

* is there any special handling that a generic tool could do with such info?

john
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] standard names for stations

[John Caron]

Heres a few comments on this discussion from my POV:

1) to summarize whats already in CF1.6:

section A9.2:

It is strongly recommended that there should be a station variable 
(which may be of any type) with the attribute cf_role=”timeseries_id”, 
whose values uniquely identify the stations.
It is recommended that there should be station variables with 
standard_name attributes station_description, surface_altitude and 
“station_wmo_id” when applicable.


Since surface_altitude already exists, the other two are called out at 
the end:


New standard names to be added to the standard name table
- station_description : variable of character type containing a 
description of a time

series station
- station_wmo_id : variable of character or integer type, containing the 
WMO

identifier of an observing station

(i dont see this last part on the web site at

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6

so here is the final version in pdf for reference:

http://www.unidata.ucar.edu/staff/caron/public/CFch9-may10.pdf

note that this is not a draft, but been accepted for 1.6. However, we 
can always amend and extend it for 1.7.)



2) the NetCDF Attribute Convention for Dataset Discovery is at

http://www.unidata.ucar.edu/software/netcdf-java/formats/DataDiscoveryAttConvention.html

but doesnt have anything about stations. it does have a naming 
authority which was intended to create globally unique dataset ids



3) the attribute cf_role=”timeseries_id” has the same effect as a 
standard name. our intention was to start to separate structural 
meatdata vs naming physical quantities via standard names. so 
cf_role=”timeseries_id” indicates a unique identifier for the station.



4) There is an important wrinkle introduced in 1.6 wrt the global vs 
variable attributes. The info for a particular station is associated by 
way of the station dimension, and all variables with just that 
dimension are station variables. The set of variables for a station 
are also associated by various mechanism involving dimensions. So:


1. any metadata intended to describe the station should be a station 
variable or an attribute on a station variable.
2. if the data, for example, came from multiple instruments, you might 
want to annotate the variables with that info, understanding that the 
variable is already associated with a specific station and must be 
consistent.


5) Generally i like the idea of richer metadata for stations and 
platforms etc, and a naming authority is a really good idea. In service 
of Getting Things Done, i would recommend that we agree on something 
that works for human readable metadata, and then start to experiment 
with machine readable versions, eg JSON.


whether the naming authority is part of the name or not is a bit of 
style, but ill say that i like it.


6) So what would be helpful would be to start with the existing new 
things in 1.6:


1) station variable (which may be of any type) with the attribute 
cf_role=”timeseries_id”, whose values uniquely identify the stations.

2) station variable with standard_name station_description
3) station variable with standard_name “station_wmo_id”

and propose clarification and extensions to that. The concrete proposal 
has come from Jeffery, so perhaps he wants to revise it based on 
feedback so far and propose another reversion?



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/24/2011 6:23 AM, Jonathan Gregory wrote:

Dear John


It seems to me it would be better to somehow denote the epoch
seperately, because its kind of silly keeping track of # millisecs
between two dates separated by 50 million years. plus its hard.

what about:

01-01-01 12:00 epoch 50m BCE

where the epoch 50m BCE is probably just carried along in the
string representation of the date.

I agree that that's a sensible way to do it for dates which are separated
by relatively short periods in an epoch which is a long time ago. However
it could be useful if the epoch wasn't just a string. If it was a longinteger
number of calendar years (relative to the normal year 1) you would be able
to interconvert dates expressed relative to different epochs. That is, you
could tell that 10 calendar Myears since 01-01-01 epoch 20M BCE means the
same as 0 calendar years since 01-01-01 epoch 10M BCE. This conversion would
be accurate provided the time unit in calendar years is stored as an integer.
Accuracy is lost if it is turned into floating-point milliseconds, I suppose.


I agree we should be able to calculate intervals between epochs. 
calendar years would also be my first choice as the unit of this.


since im going to propose some grammar that we will be stuck with for 
the next 50m years, id like your (or anyone's) thoughts on the wording.


currently i have thought of

01-01-01 12:00 epoch 50m BCE
01-01-01 12:00 reference 50m BCE


but im open to anything



The other kind of example of palaeoclimate is the
one where large periods of time are spanned by the time axis e.g. if you
had a timeseries whose time axis spanned all the geological periods since
600 Myear ago. (That would probably not be from a GCM! - but it could be from
geological data or other Earth system models.) That sort of axis would like
to use units of calendar Myear and they need to be stored accurately.


some possibilities that occur to me (not yet sure whats feasible in parser):

50 calendar Myears since 1980-01-01

50 Myears since 1980-01-01 (Myears == million calendar years)

50 Myears (assume reference is 0 CE)



About the multiples of calendar units: You have said they should be integers.
However, perhaps fractions could be rather useful. For example, we have the
recurrent question of where the time coordinate should be positioned in the
interval if the data actually applies to the interval rather than the instant.
It's a bit arbitrary and the usual advice is to put it in the middle. For this
it would be convenient to have time-bounds for a cell of e.g. 1 calendar month
since reference and 2 calendar months since reference, with the time-coord
of 1.5 calendar months since reference.


it seems like a more informative coordinate value would be, eg Feb 
2001 when the bounds are [1,2] calendar months since 2001-01-01 ?


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Hi Don:

Both java date and joda-time use a long to represent millisecs since ref 
date.


so min/max year is:

max = 292,278,994
min = -292,275,055

heres what im getting with your example in CDM 4.3 using joda-time:

5000 years since 1970-01-01 00:00:00Z == 50001928-10-07T01:30:00.064Z

5000 calendar years since 1970-01-01 00:00:00Z == 
50001970-01-01T00:00:00.000Z


calendar periods are integers, so you are limited to  +/- 2 Gyears.

It seems to me it would be better to somehow denote the epoch 
seperately, because its kind of silly keeping track of # millisecs 
between two dates separated by 50 million years. plus its hard.


what about:

01-01-01 12:00 epoch 50m BCE

where the epoch 50m BCE is probably just carried along in the string 
representation of the date.


(Im not sure why Myears is even accepted in Stu's example, since its a 
time unit, not a date unit).


john

On 8/22/2011 9:16 AM, Don Murray wrote:

John-

An example of how this could be handled (provided by Stuart Wier of 
UNAVCO) is available here:


http://geon.unavco.org/unavco/geodynamics/Lithgow-Bertelloni_Richards_Mesozoic_Cenozoic_Plate_Velocities.cdl 



described on the page:

http://geon.unavco.org/unavco/IDV_datasource_plates.html#c

Here, the time coordinate is listed as:

float time(time) ;
time:units = Myear ;
time:standard_name = time ;

with values of:

time = -170.0, -96.0, -94.0, -84.0, -74.0, -64.0, -56.0, -48.0, -43.0, 
-25.0, -10.0 ;


The problem is that udunits ends up computing times for -64 Myear as:

63998634-12-14 00:00:00 BCE

so you lose precision on the year.

Don


On 8/19/11 10:45 AM, John Caron wrote:



Regarding paleoclimate, a point I forgot is that some modellers may
wish to
have years which are very large negative numbers (many more than four
digits)
if they set up the model with the true date for the run. Although 
for

geological timescales you might say that this isn't necessary and you
might
as well choose an arbitrary year, there is a good reason for it in
Pleistocene
when you might be using the dates to relate to orbital forcing or
atmospheric
composition.


so the idea is that you are simulating some year, so you really need
time down to the hour or second. but the climate is from 5 million
years ago, so you need the year field to be able to handle that?


Im just thinking that fitting this into the ISO date format
500-01-01 12:00 seems awkward, esp as it indicates unwarranted
precision.

seems something like 01-01 12:00 reference 50m BCE would be better.
What do paleo modellers actually use, eg in the figures that they 
publish?

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/23/2011 6:13 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 22, 2011, at 6:36 PM, John Caron wrote:


On 8/22/2011 6:37 AM, Jonathan Gregory wrote:

Dear Chris


Perhaps there could be an attribute we could set that says whether we have 
accounted for leap seconds?  With the absence of such an attribute to be 
presumed as meaning leap seconds have been ignored.

Perhaps the real-world calendars with and without leap seconds should be
regarded as two different calendars, since they have different encodings
(meaning decoding/encoding as YMD HMS-   time-interval since reference-time).
The true real-world calendar is the one with leap seconds.

CF has a calendar
proleptic_gregorian

 A Gregorian calendar extended to dates before 1582-10-15. That is, a year 
is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is 
divisible by 400.

What if we clarified this calendar as not having leap seconds? Then it could
be used for real-world applications for recent dates meaning that it was just
like the real world except that it doesn't have leap seconds.

Model calendars, which are already idealised wrt length of year, don't have
leap seconds anyway, I am sure.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

I agree that a separate calendar is needed if we want to have leap
seconds. I think the common form is UTC (or TAI?). Chris, what does the
satellite community use?

Both UTC and TAI, actually.


Then i would propose adding those two calendars, if they are really 
used. OTOH, i dont have an implementation of them, although there are 
plans to add to java 8.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

On 8/19/2011 7:48 AM, Lynnes, Christopher S. (GSFC-6102) wrote:

On Aug 19, 2011, at 7:54 AM, Jon Blower wrote:


Hi  Chris,

Does the calendar system usually define whether leap-seconds are taken into 
account or not?

Generally speaking, I don't believe so, if only because calendar systems 
usually go down to the day granularity. Seconds-based doordinate time 
standards, such as UTC and TAI do, because they have to.  Julian date (NOT the 
day of year many associate with the name) is an exception, with its fractional 
dates.


In other words, given knowledge of which calendar system is in use, could a 
library make the correct calculation?  Or is other information needed too?




In my limited understanding, a calendar system might track leap seconds 
or not. If it does, then all calendar fields (except seconds and 
millisecs) would be variable length. But for any given calendar, the 
interval (in seconds) between two valid dates would be well defined. Of 
course, these intervals will differ between calendars.


Therefore, one can talk about a correct implementation of a calendar 
system, and one can talk about the difference between two calendar 
systems, one of which is the correct one to use in some context. 
Otherwise, im not sure there really is a correct calculation, or at 
least that might be a confusing way to state the issue.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Regarding paleoclimate, a point I forgot is that some modellers may 
wish to
have years which are very large negative numbers (many more than four 
digits)

if they set up the model with the true date for the run. Although for
geological timescales you might say that this isn't necessary and you 
might
as well choose an arbitrary year, there is a good reason for it in 
Pleistocene
when you might be using the dates to relate to orbital forcing or 
atmospheric

composition.


so the idea is that you are simulating some year, so you really need 
time down to the hour or second. but the climate is from 5 million 
years ago, so you need the year field to be able to handle that?


Im just thinking that fitting this into the ISO date format 
500-01-01 12:00 seems awkward, esp as it indicates unwarranted 
precision.


seems something like 01-01 12:00 reference 50m BCE would be better. 
What do paleo modellers actually use, eg in the figures that they publish?

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CDM calendar date handling

[John Caron]

Hi Jonathan:

On 8/18/2011 9:41 AM, Jonathan Gregory wrote:

Dear John


   http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html

If there's interest, I can propose as a CF convention. Otherwise it
can remain a CDM extension.

Thank you for doing this. I think it's an attractive idea to have calendar
handling in CF time coordinates. One issue to be dealt with would be the
need to be able to process these strings using any programming language
which might be used to process CF-netCDF data. Could the software be ported
to other languages in which the netCDF lib exists?


As Jon said, it should be straightforward to develop an extension of 
udunits (C library) to support this, once the specification is worked 
out. However, at the moment, we are not sure of who/how/if this will be 
done. A lot would depend on finding an open-source calendar libary in C 
that does the right thing. We may be able to leverage the ESMF library.



In order to do calculations
with times, which are often necessary, we need to be able to convert them into
a form which has a fixed-length unit since a reference time (like udunits),
even though that isn't the most convenient way to express them.
I think that given two calendar dates in the same calendar, there will 
be a well-defined # seconds between them. I think thats the essence of 
whats needed.


What you have proposed for the syntax looks very sensible to me. I have some
minor points:

* A bit related to Don's: it would be good to allow SI prefixes, like udunits
does. For instance, ms is the usual SI symbol for millisecond.
Im not sure I can actually implement prefixes for calendar fields, since 
in general that would violate the need for integers. millisecond makes 
sense, but millihour doesnt.


* What happens if n months or n years since a certain date is not a valid
date in the calendar? You probably have a rule to resolve that. Most safely,
it should be an error to encode such a time, I suppose (e.g. 1 year since
29 Feb 2008). Do you insist that the ref data is valid in the calendar?

Very good question.

1) There are some dates that are invalid in some calendars eg 29 Feb 
2009, and also apparently 1582-10-05 to 1582-10-14, as well as year 0 in 
the gregorian calendar. So if you try to use those in the 
calendar_field, you will get an error.


2) Then theres the question of 1 year since 29 Feb 2008. This is what 
the library im using does, which seems right to me:


 1 calendar years since 2008-02-29 00:00:00Z == 2009-02-28T00:00:00.000Z
 2 calendar years since 2008-02-29 00:00:00Z == 2010-02-28T00:00:00.000Z
 3 calendar years since 2008-02-29 00:00:00Z == 2011-02-28T00:00:00.000Z
 4 calendar years since 2008-02-29 00:00:00Z == 2012-02-29T00:00:00.000Z
 5 calendar years since 2008-02-29 00:00:00Z == 2013-02-28T00:00:00.000Z
 6 calendar years since 2008-02-29 00:00:00Z == 2014-02-28T00:00:00.000Z
 7 calendar years since 2008-02-29 00:00:00Z == 2015-02-28T00:00:00.000Z
 8 calendar years since 2008-02-29 00:00:00Z == 2016-02-29T00:00:00.000Z
 9 calendar years since 2008-02-29 00:00:00Z == 2017-02-28T00:00:00.000Z
10 calendar years since 2008-02-29 00:00:00Z == 2018-02-28T00:00:00.000Z
11 calendar years since 2008-02-29 00:00:00Z == 2019-02-28T00:00:00.000Z
12 calendar years since 2008-02-29 00:00:00Z == 2020-02-29T00:00:00.000Z
13 calendar years since 2008-02-29 00:00:00Z == 2021-02-28T00:00:00.000Z
14 calendar years since 2008-02-29 00:00:00Z == 2022-02-28T00:00:00.000Z

also:

 1 calendar months since 1930-01-31 00:00:00Z == 1930-02-28T00:00:00.000Z
 2 calendar months since 1930-01-31 00:00:00Z == 1930-03-31T00:00:00.000Z
 3 calendar months since 1930-01-31 00:00:00Z == 1930-04-30T00:00:00.000Z
 4 calendar months since 1930-01-31 00:00:00Z == 1930-05-31T00:00:00.000Z
 5 calendar months since 1930-01-31 00:00:00Z == 1930-06-30T00:00:00.000Z
 6 calendar months since 1930-01-31 00:00:00Z == 1930-07-31T00:00:00.000Z
 7 calendar months since 1930-01-31 00:00:00Z == 1930-08-31T00:00:00.000Z
 8 calendar months since 1930-01-31 00:00:00Z == 1930-09-30T00:00:00.000Z
 9 calendar months since 1930-01-31 00:00:00Z == 1930-10-31T00:00:00.000Z
10 calendar months since 1930-01-31 00:00:00Z == 1930-11-30T00:00:00.000Z
11 calendar months since 1930-01-31 00:00:00Z == 1930-12-31T00:00:00.000Z
12 calendar months since 1930-01-31 00:00:00Z == 1931-01-31T00:00:00.000Z




* The udunits ref time implies missing components e.g. 1 hour since 2011-1-1
means 1 hour since 00:00 on 2001-1-1. What do missing components of ISO strings
imply in your implementation? I think it has got to imply something, because
these datetimes are still complete time specifications, aren't they.

good point, i will add a sentence saying missing fields must be set to 0.


* For further udunits compatibility, it would be convenient to be able to omit
the time zone, which would imply Z. All global model data is in UTC, I should
think, and I suppose it must be the obvious choice for all global datasets.

ok, 

[CF-metadata] CDM calendar date handling

[John Caron]

In March I promised to do something with the udunits handling of fuzzy 
time units email conversation. I now have a preliminary implementation 
of extended date coordinates. docs are here:


  http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html 
http://www.unidata.ucar.edu/software/netcdf-java/CDM/DateTime.html


If there's interest, I can propose as a CF convention. Otherwise it can 
remain a CDM extension.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Is there a convention defining day offsets to use for monthly average time series?

[John Caron]

On 8/8/2011 3:43 PM, Jim Biard wrote:

Hi.

I have a time series of monthly averaged values.  I have an 
integer-valued time coordinate variable and an associated time_bounds 
variable.  Is it correct to use the 15th of February and the 16th of 
all the other months for my time centers, or should I use the 16th of 
every month?


Also, should I do anything differently if my data are climatological 
monthly averages (say, over 30 years of data)?  And, in this case, 
should the time coordinate values be day numbers from the beginning of 
the 30-year time interval, the end of the time interval, or something 
else entirely?


Grace and peace,

Jim Biard



At the moment, IMO the best that can be done in CF is to accurately 
record the date range (using the bounds attribute). The coordinate value 
should then be considered for labeling purposes only. Make a one line 
description and put into the long_name attribute. Make sure you have 
human readable documentation that explains whats going on in detail, and 
add a global attribute that references it. Set up a 24-hour hotline to 
answer questions, staffed by post-docs wearing beepers. Ok, maybe not 
the last ;^)

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] the need to store lat/lon coordinates in a CF metadata compliant netCDF file

[John Caron]

On 7/26/2011 4:24 AM, Jonathan Gregory wrote:

Dear all

For datasets which are intended for analysis by end-users I think it would be
undesirable to remove the requirement of providing explicit lat and lon
coords even if a grid_mapping is provided. I think it is unrealistic to expect
all software which someone might use to analyse netCDF files to be able to
recognise and act upon all possible values of the CF grid_mapping attribute,
and without the lat and lon information the user would have a problem. If the
issue is storage space in the file I think the much better choice is to store
the explicit coordinates in another file, by extending the CF convention to
allow datasets to be distributed over several linked files, as gridspec does
for example.

Steve appears to suggest that grid_mapping is required in some circumstances,
but I don't think it is at present. However, the text Steve quotes may not be
quite right:

/When the coordinate variables for a horizontal grid are not
longitude and latitude,*_it is required that the true latitude and
longitude coordinates be supplied_* via the coordinates attribute/.

The text should make it clear that this requirement applies when the data has a
geolocated horizontal grid. It doesn't necessarily apply to idealised cases.
We could clarify this with a defect ticket.

Without the grid_mapping, the lat and lon still make sense in the common case
(and original CF case) of GCM data, and in many other cases, the intended
usage of the data does not require precision about the figure of the Earth.
Although this metadata could be valuable if it can be defined, I think it would
be too onerous to require it.

CF is primarily intended to promote the processing and sharing of netCDF
files, it says in the Abstract. As John notes, it could be used for other
purposes, such as internal files in models. Do they have to be CF-compliant,
however, if they're not intended for analysis or for sharing?

Cheers

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




Storing coordinates in a separate file is a reasonable solution for 
large, managed archives, not so good for data exchange (eg extract a 
subset from large archive and give it to ArcGIS to read). I think we 
should provide a profile of CF which relaxes the requirement for 2D 
lat/lon coordinates, as long as grid_mapping is supplied.


We also may need a profile of CF for GIS data, eg to make the datum be 
required. Then we could say file is CF/GIS compliant without adding 
unneeded requirements for climate modelers.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] the need to store lat/lon coordinates in a CF-compliant netCDF file

[John Caron]

On 8/3/2011 8:19 AM, Jon Blower wrote:

Hi all,

I've been following this thread with great interest.  For me it boils down to 
this question:

  - Is the datum always known by the data provider?

If the answer is yes then I see no reason to omit the datum (and plenty of 
reasons to include it).  If there *are* situations where the datum is genuinely unknown 
or undefined then we need to express this clearly too so users can beware.

(Previous posts mentioned GCMs as possible situations where the datum is 
unknown - is this really true?  Surely there is always a sphere with a known 
radius on which coordinates are based?)

Cheers,
Jon

--
Dr Jon Blower
Technical Director, Reading e-Science Centre
Environmental Systems Science Centre
University of Reading, UK
Tel: +44 (0)118 378 5213
http://www.resc.reading.ac.uk


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


Just to add to the above, there are 2 issues we are discussing:

1) possible recommendations by CF to always include the datum, and 
making sure we have the right metadata to do so.


2) possible recommendations by CF as to what to do if the datum is not 
present, or is only partially specified.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] the need to store lat/lon coordinates in a CF metadata compliant netCDF file

[John Caron]

Hi all:

If I understand the situation, a GCM, whether global or regional, always 
has a datum. In the simple case, its just the radius of a spherical 
earth. However, for current global models its typical that this datum 
doesnt affect anything (presumably as long as its within the typical 
range of values). Is that correct?


It seems that modelers are wary of downstream users incorrectly assuming 
that the grid point locations have more precision than is warrented, and 
possibly drawing incorrect conclusions about accuracy etc. Is that the case?


In performing transformations, software has to use a datum, and so will 
make an assumption if the datum is missing. It would be nice if a human 
was made aware of that assumption, but in practice this mostly wont happen.


If the above is correct, I would advocate a sentence in section X 
something like:


/CF recommends that data providers always specify the datum. In the 
simplest case, this just means adding a _latitude_longitude_// grid 
mapping with the value of the spherical earth radius that was used in 
the model. However, data users are cautioned not to infer unwarranted 
precision of the model grid point locations when comparing to observations./


Or something like that, probably needs some tweaking.

John

On 7/28/2011 6:47 AM, David Blodgett wrote:

Dear Jonathan, All,

Coming from the perspective of a geographer, for CF to be a 
convention whereby people can provide accurate and complete metadata 
for their data. Datum specification would be required.


I understand that the normal practice in the climate science community 
is to throw out datum information from ingested ground based data as 
it is being applied to a model or summary product that essentially 
smears the measurement in space. So in fact, for strict accuracy of 
geolocation for course resolution low numerical accuracy applications, 
a datum may not add anything of value. Its clear I'm not going to win 
this argument by claiming that the status quo should not impose 
inaccuracy on the edge case.


I hope that software developers do throw warnings when data has an 
unknown piece of information that is required to perform a geographic 
transformation. This lack of required information for processing is 
really the crux of the issue and why I have pushed it this far.


In Chapter 4 (.1-.2) no recognition is given to the fact that latitude 
or longitude coordinates are essentially meaningless without the 
assumption of a datum, this recognition would go a long way to making 
me more comfortable with the spec. Something like the follows is may 
be warranted.


/Data consumers should be aware that latitude and longitude 
coordinates lacking description of the ellipsoid shape and prime 
meridian, or datum, must be assumed to lie on any arbitrary datum. It 
should be understood that it is up to end users of such data to assign 
this information according to their best judgement. We recommend, when 
accurate grid geolocation is appropriate, for data producers or 
publishers to use a /_/latitude_longitude/_/ grid mapping as described 
in chapter 5.6./

/
/
Cheers,

Dave


On Jul 28, 2011, at 5:00 AM, Jonathan Gregory wrote:


Dear Dave

I sympathise with your concerns about the consequences of missing 
datums but

I don't think CF can help a lot.

Are there any arguments against CF recommending a standard datum 
assumption when intersecting data without a datum specified with 
data that does have a datum specified?


I don't think that's the role of CF. CF is a convention whereby 
people can
provide accurate and complete metadata for their data. If the data 
provider
doesn't have a real-world datum that could be specified, it is not 
appropriate
for CF to suggest what it should be, I would argue. In fact, most 
parts of CF
are optional. People may choose not to describe their data with 
standard_names

for example. They are not mandatory, but of course it makes the data less
useful if they are not supplied. I think the best can be done, rather as
Balaji suggests, is for software that reads CF files to emit warnings 
about
metadata which could have been included but isn't, so that analysts 
are aware.


Best wishes

Jonathan



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF feature type trajectory (Ch. 9; May 10, 2011) and axis, attribute

[John Caron]

Hi Chris:

I think we agreed that this sentence in section 5 should be removed:

The|axis|attribute is not allowed for auxiliary coordinate variables.

If anyone has a better idea, let me know. otherwise i will submit a 
defect change.


John

On 7/20/2011 7:51 AM, Chris Paver wrote:

Dear list,

Was there any resolution to the issue I brought up?

Thanks,
Chris

 Original Message 
Date: Thu, 07 Jul 2011 11:59:09 -0700
From: Steve Hankin steven.c.han...@noaa.gov
Subject: Re: [CF-metadata] CF feature type trajectory (Ch. 9; May 10,
2011) and axis attribute
To: Jeffrey F. Painter paint...@llnl.gov
Cc: cf-metadata@cgd.ucar.edu
Message-ID: 4e1601fd.5020...@noaa.gov
Content-Type: text/plain; charset=ISO-8859-1; format=flowed



On 7/7/2011 11:53 AM, Jeffrey F. Painter wrote:
  It looks as though you've been reviewing the public draft of the
  future CF Conventions version 1.6.  It's great that someone has been
  looking at it!
 
  It seems to me that z in the examples of Appendix H (formerly
  designated A9) plays the role of an auxiliary coordinate variable,
  although technically it's not because there isn't (but should be) a
  'coordinates' attribute which lists auxiliary coordinate variables.
  I don't know why axes are banned for auxiliary coordinate 
variables in

  the case of trajectories.  Something needs to be clarified here, or
  the examples changed.
 
  This was ticket 37.  Are there any comments from Steve Hankin, the
  moderator, or any of the other people who contributed to writing
  Chapter 9, or anyone else who knows more than me?
stand by.   I think Chris has pointed out a needed correction to Chapter
5 that was discussed, but was not documented.  (thanks Chris)   But need
to confirm with others.

 - Steve



 
  Thanks to all!
  - Jeff Painter
 
  On 7/7/11 5:13 AM, Chris Paver wrote:
  Dear List,
  I have been reviewing the use of the axis attribute as defined in
  Chapter 5, and
  comparing it to the trajectory example in the Chapter 9 document at
  A9.4.  The
  Ch. 5 rules states that only a coordinate variable can have an axis
  attribute,
  whereas the auxiliary variables cannot.  However, in Ch. 9, the 
single

  trajectory dimension is time, but the variable z contains the axis
  attribute. Is
  this a change to the rule as stated in Ch. 5, or a typo?  I briefly
  looked over
  the other feature types in Ch. 9, and they all look to have the 
vertical

  variable listed with the axis attribute, so maybe it is a typo.
 
  Thanks,
  Chris



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-metadata Digest, Vol 99, Issue 8

[John Caron]

On 7/20/2011 3:43 AM, Comiskey, Glenn wrote:

Thanks for the reply John.

Just to confirm, the vertical coordinate is described by a number of
other metadata attributes such as axis, standard_name,
_CoordinateAxisType etc. and therefore wouldn't expect any problems with
applications discovering the vertical coordinate.
yes; the most important thing is that coordinates must be listed in a 
coordinates attribute (or are coordinate variables); so a data variable 
would not get listed and so would not be accidentally categorized as a 
coordinate.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Units: degrees - COARDS/CF Convention

[John Caron]

On 7/14/2011 8:48 AM, Comiskey, Glenn wrote:

Hi,
The use of the variable unit value degrees would appear to discount 
a data set from being both COARDS and CF compliant, at least according 
to the CF conventions document, i.e. v1.5, section 3.1 states The 
COARDS convention prohibits the unit degrees altogether while also 
stating this unit is not forbidden by the CF convention. Is this in 
fact true?
The COARDS convention does indeed state, when refering to UDUNITS, 
names therein...will be regarded as acceptable unit names for this 
standard with the following additions and deletions: degrees - 
deleted. However, it then goes on to say The unit degrees creates 
ambiguities when attempting to differentiate longitude and latitude 
coordinate variables; files must use degrees_east...and 
degrees_north.
Is it not a case that the COARDS convention is disallowing the use of 
the degrees value simply for coordinate variables, and subsequently 
mandating the use of degrees_east and degrees_north, to remove any 
potential ambiguity with regards to the position the data relates, 
i.e. COARDS requires degrees_east and degrees_north for 
coordinates variables, but allows uses of degrees for a data 
variable such wind direction (NOTE: I am aware that the CF standard 
canonical unit is degree).
Is there some convention (pun intended :-) to interpretating the 
COARDS convention as meaning that degrees is prohibited, period?


Hi glenn:

degrees is a valid udunit for things like wind direction. It is not 
allowed as a unit for lat or lon, period. I think that is how you are 
interpreting it above.


john



Kind regards,
Glenn Comiskey
Data System Administrator
Fugro Global Environment  Ocean Sciences
Fugro House, Hithercroft Road
Wallingford, Oxfordshire OX10 9RB, UK
Registration No: 2985431 / VAT No: GB 655475606
Tel: +44 (0) 1491 820500 (Switchboard) / Tel: +44 (0) 1491 820559 
(Direct) / Fax: +44 (0) 1491 820599
E-mail: _g.comiskey@geos.com_ mailto:g.comiskey@g.comis...@geos.com/ 
Website: _www.geos.com_ http://www.geos.com/

Please note:

   *
  The information contained in this message is privileged and
  confidential and intended only for the use of the addresses.  If
  you are not the intended recipient you should not read, copy,
  distribute or otherwise use the information;
   *
  Fugro Global Environment  Ocean Sciences accepts no
  responsibility for loss or damage of any kind arising from the
  use of this message;
   *
  If you have received this message in error, please note the
  sender immediately and delete the message.



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] the need to store lat/lon coordinates in a CF metadata compliant netCDF file

[John Caron]

On 6/30/2011 1:26 PM, Randy Horne wrote:

Paragraph 5.6 Coordinate Reference Systems, Grid
Mappings, and Projections, first sentence:

When the coordinate variables for a horizontal grid are not longitude and 
latitude, it is required that the true latitude and longitude coordinates be 
supplied via the coordinates attribute.

Are there any exceptions to this rule ?

The reason for asking is based on a scenario where the coordinate system and 
constituent locations for the data points associated with the gridded data set 
can be determined solely based on the grid mapping.

The reason for wanting to do this is to minimize the size of the netCDF file, 
and the hardware resources required to process, store, and distribute the file.



Hi Randy:

There has been no formal exception granted from this rule. The java 
netcdf library does not require lat/lon as long as a valid grid mapping 
is supplied. In that case, the lat/lon arrays are 2D, and can be a 
significant size. When writing CF compliant netcdf files (eg from 
netcdf subset service), the user is allowed to decide to include lat/lon 
or not. Technically if not included, the files are not fully CF compliant.


CF has mostly thought about how to make archival model data as 
self-contained as possible, thus the insistence on lat/lon arrays. I 
think there are other valid uses of netcdf-CF files, eg as an exchange 
format, and that the lat/lon arrays should be optional. There is 
currently no mechanism, eg a profile of CF to allow that.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] udunits time units question

[John Caron]

Thanks Cecelia!

 Is it written in C++ or Fortran90 or ?

John

On 4/18/2011 3:22 PM, Cecelia DeLuca wrote:

 Hi John, Chris, all:

Here's a more direct link to the ESMF utility for timekeeping than 
Brian provided:


http://www.earthsystemmodeling.org/esmf_releases/public/last/ESMF_refdoc/node5.html#SECTION0504 



There are a number of calendars supported (Proleptic Gregorian, 
no-leap,  Julian and
Julian day calendars, 360 day, custom calendars for planets, etc.).  
Proleptic Gregorian
is used already by some climate groups;  I would think its use would 
increase as modelers
try to answer questions that combine climate model output with other 
kinds of information,

and perform comparisons of model output with observations.

The length of days can change with the calendar, so they're not a 
basic time unit.


ESMF has time instant and time interval classes.  Their operators 
(+,-,,, ==, etc.) are

overloaded so you can write, for example:
time = init_time + timeinterval
or
timeinterval = init_timeinterval * 3

Increments can be in many different time units, including  months.  If 
you add three months

to a date like April 1 you will get July 1.  ESMF computes to the
same day that many months later.  There is a special end-of-the-month 
case so that

if you add three month to March 31 you will get June 30.
I think John was originally wondering if there was a lib that does 
this sort of thing.


Other methods it offers include getting things like middle of the 
month and
day of the week.  There are clock and alarm classes to manage model 
time, with clocks
that can run forward or backward in time.  Development priorities were 
set by modelers

so most of what's there was needed by somebody.

It's portable and pretty widely used in the modeling community.  If 
you think there's
something that could be done to make it more accessible or useful to 
the data community,

please write.

Note, one calendar that ESMF does not support is the mixed 
Gregorian/Julian calendar that
is the CF default.  For modelers having to use the mixed 
Gregorian/Julian calendar just

doesn't make sense.

-- Cecelia

On 4/1/2011 9:50 AM, Brian Eaton wrote:

Hi John,

You might have a look at the ESMF time handling utilies for some ideas
about what a library that supports modelling (i.e., non-standard 
calendars)

contains.

http://www.earthsystemmodeling.org/esmf_releases/public/last/ESMF_refdoc/ 



Brian


On Fri, Apr 01, 2011 at 05:04:42AM -0600, John Caron wrote:

Hi Martin and all:

Im wondering what essential methods a calendar library needs to have
to support, eg 360 day calendars? The only one that comes to mind is
to calculate the difference between two dates in, say, seconds? What
else?

John

On 4/1/2011 12:55 AM, Schultz, Martin wrote:

Hi Chris et al.,

  indeed it seems like some clarification is necessary about 
the use of different calendars in modelling. Your suggestion to 
map the 360 day calendar onto the Gregorian calendar in output 
files won't work: there would be no need for a 360 day calendar if 
it would. The idea behind fixed-length-year calendars is precisely 
the opposite: preserve seasonality without having to deal with leap 
years etc. Of course you need to adjust the solar cycle 
accordingly, that means your model will have a solar year that is 
precisely 360 days long (and not 365.25 as in reality). If you were 
to map these dates onto the gregorian calendar, you would have to 
interpolate the time axis so that days 1 to 360 fit into the range 
1 to 365 or 366, respectively. Yes - a converter 360-days (or 
365-days) to gregorian (and reverse) could be written, but you 
would redce the number of your friends if you insist that this is 
the only valid way to represent time of the model output ;-)


 You are right that a date such as 1960-01-31 is invalid in a 
360-day calendar, and indeed a smart tool should flag this as an 
error.


Hope this helps,

Martin



 

 


Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
 

 



Besuchen Sie uns auf unserem neuen Webauftritt unterwww.fz-juelich.de
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] physical vs dimensional units

[John Caron]

On 4/15/2011 10:29 AM, Steve Emmerson wrote:

John,

On 04/14/2011 08:09 PM, John Caron wrote:

Im thinking that ne wants a grammer that allows

moleX  / mole * mole/ moleX = 1

moleX  / mole * mole/ moleY= moleX/ moleY

and so on.

I wonder if this list could accumulate a list of such examples.

In my opinion, the semantics of such operations would be better handled
by a physical quantities package than a units package.


my thinking exactly. sorry that wasnt clear.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] physical vs dimensional units

[John Caron]

Hi Jonathan:

On 4/14/2011 2:14 AM, Jonathan Gregory wrote:

Dear John and Steve

I agree with Steve about this:


On 04/13/2011 02:25 PM, John Caron wrote:

the point im trying to make is that it would be better to understand
that mol mol-1 (canonical udunit = 1) is not the same as  m3 m-3
(canonical udunit = 1).

In my opinion, the distinction between mol/mol and m3/m3 is better
indicated by the name of the physical quantity being displayed rather
than by its unit attribute.

I would say mol mol-1 and m3 m-3 are actually the same unit. But that doesn't
mean the quantities with these units are the same. Similarly,
net_downward_longwave_flux_in_air and net_downward_shortwave_flux_in_air
are not the same quantity, even though they are both in W m-2.


yes, exactly.

A physical quantities library seperate from udunits could provide some 
assistance in detecting when two quantities can be added with unit 
convertsion, eg add m and feet but not add mol/mol and m3/m3.

The udunits Unix command doesn't accept garbage garbage-1 as a valid unit,
and nor should the Java version, I think.


Thats good. I think the java library might be my fault, because I asked 
Steve to allow unknown units as a half step towards a physical 
quantities library. in retrospect, best to separate the two.


Regards,
John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] physical vs dimensional units

[John Caron]

On 4/13/2011 2:52 PM, Steve Emmerson wrote:

On 04/13/2011 02:25 PM, John Caron wrote:

the point im trying to make is that it would be better to understand
that mol mol-1 (canonical udunit = 1) is not the same as  m3 m-3
(canonical udunit = 1).

In my opinion, the distinction between mol/mol and m3/m3 is better
indicated by the name of the physical quantity being displayed rather
than by its unit attribute. In the cases under consideration, the
respective physical quantities would be amount of substance fraction
and volume fraction.

See Table 12 ofhttp://physics.nist.gov/Pubs/SP811/sec08.html  for more
information.

For example, the Y-axis of a plot could be labeled Amount of Substance
Fraction of Carbon / 1e-9 to plot nanomoles of carbon per mole of whatever.

Regards,
Steve Emmerson
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


Hi Steve:

Thanks, that a useful reference. It may be that some extension to 
udunits following Table 12 could solve most of the issues.


Im thinking that ne wants a grammer that allows

mole X / mole * mole/ mole X = 1

mole X / mole * mole/ mole Y= mole X/ mole Y

and so on.

I wonder if this list could accumulate a list of such examples.

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata