Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

Hi Stephan:

I agree strongly with you, having seen similar issues from the POV of the
netcdf-java library. It seems to be very common to confuse the reference
library with the specification, whether it be file format, web service, OTW
protocol, etc. Its so easy to fix something in the library. Im sure Ive
done it myself. ;^(

Anyway, no sign it will happen here, but good reminder.

John

On Thu, Apr 21, 2016 at 10:43 PM, Stephan Hoyer  wrote:

> A custom attribute to identify netCDF4 files sounds perfectly reasonable
> as an iteration on the netCDF4 spec.
>
> But I'd like emphasize that it's important to continue to treat netCDF4 as
> a specification rather than merely whatever libnetcdf knows how to read and
> write. Doing so keeps you honest and forces more careful design.
>
> My project h5netcdf (https://github.com/shoyer/h5netcdf) is one such
> reimplementation of netCDF4 on top of an HDF5 API, which has exposed a
> number of bugs and edge cases in the netCDF-C and Python netCDF4 libraries.
>
> Cheers,
> Stephan
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

Here are the blogs:

http://www.unidata.ucar.edu/blogs/developer/en/entry/dimensions_scales
http://www.unidata.ucar.edu/blogs/developer/en/entry/dimension_scale2
http://www.unidata.ucar.edu/blogs/developer/en/entry/dimension_scales_part_3
http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf4_shared_dimensions
http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf4_use_of_dimension_scales

On Fri, Apr 22, 2016 at 7:57 AM, Pedro Vicente <
pedro.vice...@space-research.org> wrote:

> John
>
> >>>i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> >>>case you want details.
>
> yes, I am interested, I had the impression by looking at the code some
> years ago that netCDF writes some unique name attributes somewhere
>
> --
> Pedro Vicente
> pedro.vice...@space-research.org
> https://twitter.com/_pedro__vicente
> http://www.space-research.org/
>
>
>
>
> - Original Message -
> *From:* John Caron 
> *To:* Pedro Vicente 
> *Cc:* cf-metadata@cgd.ucar.edu ; Discussion forum for the NeXus data
> format  ; netcdfgr...@unidata.ucar.edu ; Dennis
> Heimbigner  ; NetCDF-Java community
> 
> *Sent:* Thursday, April 21, 2016 11:11 PM
> *Subject:* Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF
> versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>
> 1) I completely agree with the idea of adding system metadata that
> indicates the library version(s) that wrote the file.
>
> 2) the way shared dimensions are implemented by netcdf4 is a unique
> signature that would likely identify (100 - epsilon) % of real data files
> in the wild. One could add such detection to the netcdf4 and/or hdf5
> libraries, and/or write a utility program to detect.
>
> there are 2 variants:
>
> 2.1) one could write a netcdf4 file without shared dimensions, though im
> pretty sure no one does. but you could argue then that its fine to just
> treat it as an hdf5 file and read through hdf5 library
>
> 2.2) one could write a netcdf4 file with hdf5 library, if you knew what
> you are doing. i have heard of this happening. but then you could argue
> that its really a netcdf4 file and you should use netcdf library to read.
>
> i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> case you want details.
>
> On Thu, Apr 21, 2016 at 4:18 PM, Pedro Vicente <
> pedro.vice...@space-research.org> wrote:
>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>>
>>
>>
>> ok, thank you
>>
>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>>
>>
>>
>> Excellent idea , Dennis
>>
>> --
>> Pedro Vicente
>> pedro.vice...@space-research.org
>> https://twitter.com/_pedro__vicente
>> http://www.space-research.org/
>>
>>
>> - Original Message - From: 
>> To: "Pedro Vicente" ; <
>> cf-metadata@cgd.ucar.edu>; "Discussion forum for the NeXus data format" <
>> ne...@nexusformat.org>; 
>> Sent: Thursday, April 21, 2016 5:02 PM
>> Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 --
>> PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>>
>>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>> =Dennis Heimbigner
>>>   Unidata
>>>
>>>
>>> On 4/21/2016 2:17 PM, Pedro Vicente wrote:
>>>
>>>> Dennis
>>>>
>>>> I am in the process of adding a global attribute in the root group
>>>>>>>>
>>>>>>> that captures both the netcdf library version and the hdf5 library
>>>>> version
>>>>> whenever a netcdf file is created. The current  form is
>>>>> _NCProperties="version=...|netcdflibversion=...|hdflibversion=..."
>>>>>
>>>>
>>>>
>>>> ok, good to know, thank you
>>>>
>>>>
>>>> > 1. I am open to suggestio

Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

Hi Pedro:

there are 4 bl;ogs, starting with this one:


On Fri, Apr 22, 2016 at 7:57 AM, Pedro Vicente <
pedro.vice...@space-research.org> wrote:

> John
>
> >>>i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> >>>case you want details.
>
> yes, I am interested, I had the impression by looking at the code some
> years ago that netCDF writes some unique name attributes somewhere
>
> --
> Pedro Vicente
> pedro.vice...@space-research.org
> https://twitter.com/_pedro__vicente
> http://www.space-research.org/
>
>
>
>
> - Original Message -
> *From:* John Caron 
> *To:* Pedro Vicente 
> *Cc:* cf-metadata@cgd.ucar.edu ; Discussion forum for the NeXus data
> format  ; netcdfgr...@unidata.ucar.edu ; Dennis
> Heimbigner  ; NetCDF-Java community
> 
> *Sent:* Thursday, April 21, 2016 11:11 PM
> *Subject:* Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF
> versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>
> 1) I completely agree with the idea of adding system metadata that
> indicates the library version(s) that wrote the file.
>
> 2) the way shared dimensions are implemented by netcdf4 is a unique
> signature that would likely identify (100 - epsilon) % of real data files
> in the wild. One could add such detection to the netcdf4 and/or hdf5
> libraries, and/or write a utility program to detect.
>
> there are 2 variants:
>
> 2.1) one could write a netcdf4 file without shared dimensions, though im
> pretty sure no one does. but you could argue then that its fine to just
> treat it as an hdf5 file and read through hdf5 library
>
> 2.2) one could write a netcdf4 file with hdf5 library, if you knew what
> you are doing. i have heard of this happening. but then you could argue
> that its really a netcdf4 file and you should use netcdf library to read.
>
> i have written various blogs on the unidata site about why netcdf4 !=
> hdf5, and what the unique signature for shared dimensions looks like, in
> case you want details.
>
> On Thu, Apr 21, 2016 at 4:18 PM, Pedro Vicente <
> pedro.vice...@space-research.org> wrote:
>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>>
>>
>>
>> ok, thank you
>>
>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>>
>>
>>
>> Excellent idea , Dennis
>>
>> --
>> Pedro Vicente
>> pedro.vice...@space-research.org
>> https://twitter.com/_pedro__vicente
>> http://www.space-research.org/
>>
>>
>> - Original Message - From: 
>> To: "Pedro Vicente" ; <
>> cf-metadata@cgd.ucar.edu>; "Discussion forum for the NeXus data format" <
>> ne...@nexusformat.org>; 
>> Sent: Thursday, April 21, 2016 5:02 PM
>> Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 --
>> PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>>
>>
>> If you have hdf5 files that should be readable, then I will undertake to
>>> look at them and see what the problem is.
>>> WRT to old files:  We could produce a utility that would redef the file
>>> and insert the
>>>  _NCProperties attribute. This would allow someone to wholesale
>>>  mark old files.
>>> =Dennis Heimbigner
>>>   Unidata
>>>
>>>
>>> On 4/21/2016 2:17 PM, Pedro Vicente wrote:
>>>
>>>> Dennis
>>>>
>>>> I am in the process of adding a global attribute in the root group
>>>>>>>>
>>>>>>> that captures both the netcdf library version and the hdf5 library
>>>>> version
>>>>> whenever a netcdf file is created. The current  form is
>>>>> _NCProperties="version=...|netcdflibversion=...|hdflibversion=..."
>>>>>
>>>>
>>>>
>>>> ok, good to know, thank you
>>>>
>>>>
>>>> > 1. I am open to suggestions about changing the format or adding
>>>>>>> info > to it.
>>>>>>>
>>>>>>
>>>>
>>>> I personally don't care, anything that uniquely identifies a netCDF
>>>> file (HDF5 based) as such will work
>>>>
>>>>
>>>&

Re: [CF-metadata] [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS

[John Caron]

1) I completely agree with the idea of adding system metadata that
indicates the library version(s) that wrote the file.

2) the way shared dimensions are implemented by netcdf4 is a unique
signature that would likely identify (100 - epsilon) % of real data files
in the wild. One could add such detection to the netcdf4 and/or hdf5
libraries, and/or write a utility program to detect.

there are 2 variants:

2.1) one could write a netcdf4 file without shared dimensions, though im
pretty sure no one does. but you could argue then that its fine to just
treat it as an hdf5 file and read through hdf5 library

2.2) one could write a netcdf4 file with hdf5 library, if you knew what you
are doing. i have heard of this happening. but then you could argue that
its really a netcdf4 file and you should use netcdf library to read.

i have written various blogs on the unidata site about why netcdf4 != hdf5,
and what the unique signature for shared dimensions looks like, in case you
want details.

On Thu, Apr 21, 2016 at 4:18 PM, Pedro Vicente <
pedro.vice...@space-research.org> wrote:

> If you have hdf5 files that should be readable, then I will undertake to
>> look at them and see what the problem is.
>>
>
>
> ok, thank you
>
> WRT to old files:  We could produce a utility that would redef the file
>> and insert the
>>  _NCProperties attribute. This would allow someone to wholesale
>>  mark old files.
>>
>
>
> Excellent idea , Dennis
>
> --
> Pedro Vicente
> pedro.vice...@space-research.org
> https://twitter.com/_pedro__vicente
> http://www.space-research.org/
>
>
> - Original Message - From: 
> To: "Pedro Vicente" ; <
> cf-metadata@cgd.ucar.edu>; "Discussion forum for the NeXus data format" <
> ne...@nexusformat.org>; 
> Sent: Thursday, April 21, 2016 5:02 PM
> Subject: Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 --
> PROPOSED SOLUTIONS --REQUEST FOR COMMENTS
>
>
> If you have hdf5 files that should be readable, then I will undertake to
>> look at them and see what the problem is.
>> WRT to old files:  We could produce a utility that would redef the file
>> and insert the
>>  _NCProperties attribute. This would allow someone to wholesale
>>  mark old files.
>> =Dennis Heimbigner
>>   Unidata
>>
>>
>> On 4/21/2016 2:17 PM, Pedro Vicente wrote:
>>
>>> Dennis
>>>
>>> I am in the process of adding a global attribute in the root group
>>>
>> that captures both the netcdf library version and the hdf5 library
 version
 whenever a netcdf file is created. The current  form is
 _NCProperties="version=...|netcdflibversion=...|hdflibversion=..."

>>>
>>>
>>> ok, good to know, thank you
>>>
>>>
>>> > 1. I am open to suggestions about changing the format or adding
>> info > to it.
>>
>
>>>
>>> I personally don't care, anything that uniquely identifies a netCDF file
>>> (HDF5 based) as such will work
>>>
>>>
>>> 2. Of course this attribute will not exist in files written using older
>>
> versions of the netcdf library, but at least the process will have
 begun.

>>>
>>> yes
>>>
>>>
>>> 3. This technically does not address the original issue because there
 exist
  hdf5 files  not written by netcdf that are still compatible with
 and can be
  read by netcdf. Not sure this case is important or not.

>>>
>>> there will always be HDF5 files  not written by netcdf that netCDF will
>>> read as we are now.
>>>
>>> this is not really the issue, but you just made a further issue :-)
>>>
>>> the issue is that I would like an application that reads a netCDF (HDF5
>>> based) file to decide to use the netCDF or HDF5 API.
>>> your attribute writing will do , for future files.
>>> for older nertCDF files there may be  a way to detect the current
>>> attributes and data structures to see if we can make it "identify itself"
>>> as netCDF. A bit of debugging will confirm that, since Dimension Scales
>>> are used, that would be an (imperfect maybe) way to do it
>>>
>>> regarding the "further issue " above
>>>
>>> you could go one step further and for any HDF5 files  not written by
>>> netcdf , you could make netCDF reject the file reading,
>>> because it's not "netCDF compliant".
>>> Since having netCDF read pure HDF5 files is not a problem (at least for
>>> me), I don't know if you would want to do this, just an idea.
>>> In my mind taking complexity and ambiguities of problems is always a
>>> good thing
>>>
>>>
>>> ah, I forgot one thing, related to this
>>>
>>>
>>> In the past I have found several pure HDF5 files that netCDF failed in
>>> reading.
>>> Since netCDF is HDF5 binary compatible, one would expect that all HDF5
>>> files will be read by netCDF.
>>> Except if you specifically wrote something in the code that makes it to
>>> fail if some condition is not met,
>>> This was a while ago, I'll try to find those cases and I'll send a bug
>>> report to the bug report email
>>>
>>> --
>>> Ped

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

Hi David:

At the risk of giving more useful answers to the wrong question, i will say
that we could do something other than require ancillary or coordinate
variables to only have dimensions that the parent variable has. There just
must be a simple and explicit rule for mapping between parent and child
dimension.

We went down that path with DSG, in order to efficiently store ragged
arrays. Another possible use case is to represent contiguous bounds, where
lower_bounds(i+1) == upper_bounds(i), then one only needs n+1 values
instead of 2*n.

Anyway, i think the rule of only allowing dimensions that are in the parent
is a good one, but for compelling use cases, we could also allow "simple
and explicit" alternatives.

regards,
John

On Tue, Dec 22, 2015 at 4:07 AM, David Hassell 
wrote:

> Hello all,
>
> Many thanks for the replies. It seems there is only support for the
> case that ancillary variable dimensions must be a subset of their
> parent variable dimensions (treating scalar coordinate variables like
> size 1 coordinate variables for the sake of a snappier sentence - oh,
> that's blown it).
>
> This reminded of an old thread about this that Mark started
> (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056162.html). In
> this thread Jim convincingly argues for an ancillary variable which
> can have size-greater-than-one dimensions which the parent variable
> does not span
> (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056215.html):
>
>   "As an example, let's say that I have a variable a[X,Y,T], which
>contains the total energy deposited per day into the atmosphere by
>charged particles (from the solar wind & Van Allen belts) on a
>lon/lat grid as a function of time.  (Something I used to work on
>many years ago.)  Let's then say that I have another variable,
>b[X,Y,Z], which represents the model atmosphere density profile on
>the same lon/lat grid as a function of altitude.  This density
>profile was used in the calculation of the energy deposition values
>stored in variable a.  Variable b is clearly valid as an ancillary
>variable for a, isn't it?"
>
> Mark raised a defect ticket (#98) about this, but withdrew it in light
> of the lack of supprt from the mailing list discussion.
>
> However, variable b in Jim's example seems to me to be storing
> "sub-grid scale information", so if we are to allow that we perhaps we
> ought to allow coordinate variables which sub-grid values and which do
> not span their any of their variable dimensions. For example, we might
> want to store the longitude coordinates which contributed to a zonal
> mean. The size 1 dimension case I originally raised is merely a
> special case of this, I feel.
>
> But I do not think that there is any appetite for this at this time,
> and certainly not from me. John - your comment here was actually very
> useful!
>
> So, I now think that ancillary variable dimensions should be a subset
> of their asssociated data variable dimensions (which includes the
> possibility of ancillary variables have no dimensions). Perhaps, in
> the new year, ticket #98 could be reopened as an enhancement ...
>
> All the best,
>
> David
>
>
>  Original message from Karl Taylor (01PM 21 Dec 15)
>
> > Date: Mon, 21 Dec 2015 13:59:33 -0800
> > From: Karl Taylor 
> > To: cf-metadata@cgd.ucar.edu
> > Subject: Re: [CF-metadata] [CF Metadata] Question about ancillary
> >  variables/formula terms variables
> > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
> >  Gecko/20100101 Thunderbird/38.4.0
> >
> > Hi David,
> >
> > I can't think of anyone needing these constructs, although there are
> > cases when both the "parent" and ancillary or formula_terms variable
> > might *both* have a scalar dimension.
> >
> > Karl
> >
> > On 12/21/15 7:41 AM, David Hassell wrote:
> > >Hello,
> > >
> > >I was wondering if anyone has ever had use for an ancillary variable
> > >(or data variable pointed to from a formula_terms attribute), having a
> > >size 1 dimension which the "parent" field does *not* have. The
> > >convention do not say anything about this.
> > >
> > >For example:
> > >
> > >   dimensions:
> > > t = 1;
> > > x = 96;
> > > y = 73;
> > >   variables:
> > > float q(x, y) ;
> > >   q:standard_name = "specific_humidity" ;
> > >   q:units = "g/g" ;
> > >   q:ancillary_variables = "q_error_limit" ;
> > > float q_error_limit(t, x, y)
> > >   q_error_limit:standard_name = "specific_humidity standard_error"
> ;
> > >   q_error_limit:units = "g/g" ;
> > >Similary for a scalar coordinate variable, which is logically
> > >equivalent to the size 1 dimension case (this time shown on a formula
> > >terms variable):
> > >   dimensions:
> > > x = 96;
> > > y = 73;
> > > z = 19;
> > >   variables:
> > > float q(z, x, y) ;
> > >   q:standard_name = "specific_humidity" ;
> > >   q:units = "g/g" ;
> > > float z(z):
> > >   z:standard_name =

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

oops, sorry misread the question

On Mon, Dec 21, 2015 at 1:06 PM, Jim Biard  wrote:

> John,
>
> But ancillary variables aren't coordinates, are they?
>
> Jim
>
>
> On 12/21/15 1:08 PM, John Caron wrote:
>
> i think a crucial property of coordinates is that their dimensions are a
> subset of the variable's dimensions. otherwise one cant assign the correct
> coordinate.
>
> the length of a char valued coordinate is an exception, really char(n) is
> logically the same as string.
>
> while a dim=1 coordinate could be used, it would violate that simple
> principle for no obvious gain. scalar here is preffered i think.
>
> we added some language to CF recently to clarify this. not sure where it
> is.
>
> john
>
>
>
> On Mon, Dec 21, 2015 at 8:49 AM, Jim Biard  wrote:
>
>> David,
>>
>> I've not come across this particular case, but I've had cases where the
>> ancillary data that I wish to store has less dimensions than the
>> corresponding parent field. The Conventions seem to require that the
>> dimensions all match. I'm of the opinion that it should not be so
>> restrictive.
>>
>> In your particular instance, what is the significance of your time
>> variable?
>>
>> Grace and peace,
>>
>> Jim
>>
>>
>> On 12/21/15 10:41 AM, David Hassell wrote:
>>
>> Hello,
>>
>> I was wondering if anyone has ever had use for an ancillary variable
>> (or data variable pointed to from a formula_terms attribute), having a
>> size 1 dimension which the "parent" field does *not* have. The
>> convention do not say anything about this.
>>
>> For example:
>>
>>   dimensions:
>> t = 1;
>> x = 96;
>> y = 73;
>>
>>   variables:
>> float q(x, y) ;
>>   q:standard_name = "specific_humidity" ;
>>   q:units = "g/g" ;
>>   q:ancillary_variables = "q_error_limit" ;
>>
>> float q_error_limit(t, x, y)
>>   q_error_limit:standard_name = "specific_humidity standard_error" ;
>>   q_error_limit:units = "g/g" ;
>>
>>
>> Similary for a scalar coordinate variable, which is logically
>> equivalent to the size 1 dimension case (this time shown on a formula
>> terms variable):
>>
>>   dimensions:
>> x = 96;
>> y = 73;
>> z = 19;
>>
>>   variables:
>> float q(z, x, y) ;
>>   q:standard_name = "specific_humidity" ;
>>   q:units = "g/g" ;
>>
>> float z(z):
>>   z:standard_name = "atmosphere_hybrid_height_coordinate";
>>   z.formula_terms = "a: var1 b: var2 orog: var3"
>>
>> float var3(x, y):
>>   time:coordinates = "time";
>>
>> float time:
>>   time:standard_name = "time";
>>   time:units = "days since 2000-12-1";
>>
>>
>> Thanks for your help,
>>
>> All the best,
>>
>> David
>>
>>
>> --
>> David Hassell
>> National Centre for Atmospheric Science (NCAS)
>> Department of Meteorology, University of Reading,
>> Earley Gate, PO Box 243,
>> Reading RG6 6BB, U.K.
>>
>> Tel   : +44 118 3785613
>> E-mail: d.c.hass...@reading.ac.uk
>> ___
>> CF-metadata mailing 
>> listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>>
>>
>> --
>> [image: CICS-NC] <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA National Centers for Environmental Information
>> <http://ncdc.noaa.gov/>
>> *formerly NOAA’s National Climatic Data Center*
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbi...@cicsnc.org
>> o: +1 828 271 4900
>>
>> *Connect with us on Facebook for climate
>> <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
>> <https://www.facebook.com/NOAANCEIoceangeo> information, and follow us on
>> Twitter at @NOAANCEIclimate <https://twitter.com/NOAANCEIclimate> and
>> @NOAANCEIocngeo <https://twitter.com/NOAANCEIocngeo>. *
>>
>> ___
>> CF-metadata mailing list
>> CF-metadata@cgd.ucar.edu
>> 

Re: [CF-metadata] [CF Metadata] Question about ancillary variables/formula terms variables

[John Caron]

i think a crucial property of coordinates is that their dimensions are a
subset of the variable's dimensions. otherwise one cant assign the correct
coordinate.

the length of a char valued coordinate is an exception, really char(n) is
logically the same as string.

while a dim=1 coordinate could be used, it would violate that simple
principle for no obvious gain. scalar here is preffered i think.

we added some language to CF recently to clarify this. not sure where it is.

john



On Mon, Dec 21, 2015 at 8:49 AM, Jim Biard  wrote:

> David,
>
> I've not come across this particular case, but I've had cases where the
> ancillary data that I wish to store has less dimensions than the
> corresponding parent field. The Conventions seem to require that the
> dimensions all match. I'm of the opinion that it should not be so
> restrictive.
>
> In your particular instance, what is the significance of your time
> variable?
>
> Grace and peace,
>
> Jim
>
>
> On 12/21/15 10:41 AM, David Hassell wrote:
>
> Hello,
>
> I was wondering if anyone has ever had use for an ancillary variable
> (or data variable pointed to from a formula_terms attribute), having a
> size 1 dimension which the "parent" field does *not* have. The
> convention do not say anything about this.
>
> For example:
>
>   dimensions:
> t = 1;
> x = 96;
> y = 73;
>
>   variables:
> float q(x, y) ;
>   q:standard_name = "specific_humidity" ;
>   q:units = "g/g" ;
>   q:ancillary_variables = "q_error_limit" ;
>
> float q_error_limit(t, x, y)
>   q_error_limit:standard_name = "specific_humidity standard_error" ;
>   q_error_limit:units = "g/g" ;
>
>
> Similary for a scalar coordinate variable, which is logically
> equivalent to the size 1 dimension case (this time shown on a formula
> terms variable):
>
>   dimensions:
> x = 96;
> y = 73;
> z = 19;
>
>   variables:
> float q(z, x, y) ;
>   q:standard_name = "specific_humidity" ;
>   q:units = "g/g" ;
>
> float z(z):
>   z:standard_name = "atmosphere_hybrid_height_coordinate";
>   z.formula_terms = "a: var1 b: var2 orog: var3"
>
> float var3(x, y):
>   time:coordinates = "time";
>
> float time:
>   time:standard_name = "time";
>   time:units = "days since 2000-12-1";
>
>
> Thanks for your help,
>
> All the best,
>
> David
>
>
> --
> David Hassell
> National Centre for Atmospheric Science (NCAS)
> Department of Meteorology, University of Reading,
> Earley Gate, PO Box 243,
> Reading RG6 6BB, U.K.
>
> Tel   : +44 118 3785613
> E-mail: d.c.hass...@reading.ac.uk
> ___
> CF-metadata mailing 
> listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
> --
> [image: CICS-NC]  Visit us on
> Facebook  *Jim Biard*
> *Research Scholar*
> Cooperative Institute for Climate and Satellites NC 
> North Carolina State University 
> NOAA National Centers for Environmental Information
> 
> *formerly NOAA’s National Climatic Data Center*
> 151 Patton Ave, Asheville, NC 28801
> e: jbi...@cicsnc.org
> o: +1 828 271 4900
>
> *Connect with us on Facebook for climate
>  and ocean and geophysics
>  information, and follow us on
> Twitter at @NOAANCEIclimate  and
> @NOAANCEIocngeo . *
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Climatological Stats

[John Caron]

Hi all:

IMO, the problem stems from confusing datetime "2012-02-27T00:00:00Z" which
is not a unit, with time "secs", which is [1].Then, use a units package
like udunits to define datetime.

The problem is that it works most of the time, since in  the common case of
"real" time, datetime can be expressed as a time offset, and you can easily
form differences and manipulate them in udunits.

As Nan points out, this does not work for climatological times like
"months", and leads to confusion. In this case the "time" coordinate is
more than just a label, but not expressible as a fixed number of seconds
from a base date.

The CDM uses an extension of udunits syntax, eg "3 calendar months since
2012-02-27T00:00:00Z", which I think is a reasonable solution [2], in the
sense that its easy to implement, is a superset of udunits, and satisfies
known use cases.

John

[1] https://en.wikipedia.org/wiki/Units_of_measurement
[2]
http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/CDM/CalendarDateTime.html




On Thu, Sep 17, 2015 at 7:54 AM, Nan Galbraith  wrote:
>
> Hi Jim -
>
> The problem isn't human-readability, though. The problem is that when you
> generate a file that has, say, the mean temperature for each month,
sometimes
> over a period of years, there are no 'days' in the process. Any data that
represents
> February goes into the February bin, whether 28 or 29 days; although
March is always
> 31 days long, its mid-point is a different number of 'days since' the
beginning of
> the year.
>
> In the case of the file Ajay presented, time is a singleton, and its
value represents
> the center point of the first 3 months of the years 1955-2012. That can't
be accurately
> expressed as a number of days, only as months.
>
> Is there a trac ticket for climatology data? If not, do we need one?
>
> Cheers - Nan
>
>
> On 9/17/15 9:32 AM, Jim Biard wrote:
>
> Nan,
>
> The problem is, udunits defines a month as having a specific length of
year/12 = 30.44 days, so if you use udunits to convert to anything else,
you won't end up where you think you will. The better practice is to use
days. It's not as "human readable", but it's the only way to do proper
conversions between time bases.
>
> Grace and peace,
>
> Jim
>
> On 9/17/15 9:06 AM, Nan Galbraith wrote:
>
> While it's true that 'month is not a proper unit of measure',
> climatologies do in fact use months, not days, in calculating
> mean values. Adhering to udunits/CF in this regard could make
> the dates less easily understood.
>
> Regards - Nan
>
> On 9/11/15 1:34 PM, Karl Taylor wrote:
>
> Dear Ajay,
>
> Since "month" is not a proper unit of measure, convert your times to days
and use a unit "days since ...".
>
> Also, it is normally a bad idea to have your base time set to a date
before the switch from Julian to Gregorian calendar. I suggest using a base
time of "1955-01-01" (i.e., the beginning of your climatological period).
>
> I think the cell_methods should be:
> cell_methods="area: mean depth: mean time: mean within years time: mean
over years";
>
> The time bounds should be (expressed in date/time format):
>
> climatology_bounds = "1955-01-01", "2012-04-01"
>
> and you can choose your time coordinate value as you think most
appropriate, e.g.,
> the middle of the season of the first year of the climatology, or
> the beginning of the first month of the first year of the climatology, or
> the middle of the season of the middle year of the climatology, or
>
> ???
>
>
> Hope someone confirms this, as sometimes I make a mistakes.
>
> Karl
>
>
>
>
> On 9/11/15 9:54 AM, Ajay Krishnan - NOAA Affiliate wrote:
>
> Dear CF members,
>
> I would like your input on the way climatological stats are being
represented in a file that I am working on. I believe that I am not using
the time and the climatological_bounds properly:
>
> Seasonal SST
>
> Average seasonal temperature (Jan-Mar) for 6 decades (1955-2012)
>
> dimensions:
> time=1;
> nv=2;
> variables:
>
>
> double time(time);
> time:climatology="climatology_bounds";
> time:units="months since -01-01";
> double climatology_bounds(time,nv);
>
>
> double climatology_bounds(time,nv);
>
> climatology_bounds:comment=” This variable defines the bounds of the
climatological time period for each time” ;
>
> float t_mn(time,lat,lon,depth);
> t_mn:standard_name: “sea_water_temperature” ;
>
> t_mn:long_name: “Average of all unflagged interpolated values at each
standard depth level for sea_water_temperature in each grid-square which
contain at least one measurement.” ;
>
> t_mn:cell_methods: “area: mean depth: mean time: mean” ;
>
>
> data: // time coordinates translated to date/time format
> time= “1.5” ;
>
> climatology_bounds=”0.0”, “3.0” ;
>
>
> The CF examples are helpful but my case is different where in I have just
1-time co-ordinate in my file. In the above case, what is the best way to
record time and climatology bounds?
>
>
> Thanks,
>
> Ajay
>
>
>
>
> --
> 

[CF-metadata] Fwd: Geos grid mapping

[John Caron]

i think its just that CF 1.7 document has been stuck for going on 3 years
now.

-- Forwarded message --
From: Ethan Davis 
Date: Wed, May 6, 2015 at 3:29 PM
Subject: Re: [CF-metadata] Geos grid mapping
To: Randy Horne 
Cc: John Caron 


Hi Randy,

Sounds like you might have missed the last netCDF-Java support
message/email John sent, back in Sept 2014, on this topic. It said

We are releaseing 4.5.3 today that has this CF geostationary projection in
it. Please let us know if you see any problems.


Also, there is a related JIRA issue:

https://bugtracking.unidata.ucar.edu/browse/TDS-538


Sorry for any confusion.
Hope that helps,

Ethan

On Wed, May 6, 2015 at 2:12 PM, Randy Horne 
wrote:

> Martin et al:
>
>
> I think we ended up with the projection name “geostationary”.
>
> Also, I had some contact with the Unidata netcdf Java support team (RE:
> taking item netCDF #ZCD-209044]), and have not heard anything for over a
> year.
>
> I would have thought it would find its way into CF 1.7.
>
>
> very respectfully,
>
> randy
>
>
>
>
> > On May 6, 2015, at 3:41 PM, Martin Raspaud 
> wrote:
> >
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Hi all,
> >
> > A while ago (3 years ago), we submitted ticket #72 to add the "geos"
> > grid mapping to the conventions. However, it doesn't seem to go
> > anywhere. Is there anyone here that knows why, and could advise me how
> > I could help to make this happen ?
> >
> > Best regards,
> > Martin
> > -BEGIN PGP SIGNATURE-
> > Version: GnuPG v2.0.22 (GNU/Linux)
> >
> > iQEcBAEBAgAGBQJVSm5qAAoJEBdvyODiyJI4cxAH/33l7QKOgIWWIXYJEOqSvCRA
> > faivR8tNX21jPbJOLxg4dBhRRbg+lARClJjmapdLGn9a9FJU2OB/sTdSuZYyeDFh
> > ySVmPFscPzK4xFVa0WDMphytNPu6xkTmhciEmnSyP1ep+cgerrAwcOWFB+TTLpMa
> > mme5P+gBoMAWtwf/Lrwu0Ro3Q9aB98+frc1b3I6h/hmw3BCxdz1nqrlAAE2H9Xyb
> > 2+ddEMDRp3a/1VC2Emon7mwOB3yHspQdwbbmIfUTzjrj8695K3wS0yjAYd00pXvy
> > qJ9JCHcvtWzeNbv2ZHH7HwcYYLQWJqBmPzBjSxWuTBDs1uFDdNjMMcVHA8OSrfc=
> > =mz78
> > -END PGP SIGNATURE-
> > ___
> > CF-metadata mailing list
> > CF-metadata@cgd.ucar.edu
> > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
> 
>
> Randy C. Horne (rho...@excaliburlabs.com)
> Principal Engineer, Excalibur Laboratories Inc.
> voice & fax: (321) 952.5100
> url: http://www.excaliburlabs.com
>
>
>
>
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] geostationary vs vertical_perspective in CF 1.7.

[John Caron]

great, thanks for the clarification.

the reference in CF vertical perspective projections has this proj4
reference:

  http://www.remotesensing.org/geotiff/proj_list/geos.html

is it wrong/misleading?

is geostationary a special case of vertical perspective?

On Tue, Nov 18, 2014 at 6:28 PM, Randy Horne 
wrote:

> John:
>
> geostationary projection and vertical perspective projections are not the
> same.
>
> an example of a vertical projection image is one taken by a camera in
> space, which is not necessarily at the equator, and the data for the entire
> image is observed using a single exposure.
>
> the geostationary projection is that associated with scanning imagers from
> geostationary orbit where each image pixel covers a fixed number of
> radians/degrees in east/west and north/south relative to the camera’s
> aperture.
>
> very respectfully,
>
> randy
>
>
>
> On Nov 18, 2014, at 3:19 PM, John Caron  wrote:
>
> CF TRAC ticket 72 defines "geostationary projection", accepted for 1.7.
>
> 1) I _think_ that "Vertical Perspective" is an earlier version of this,
> and it should be deprecated in favor of "geostationary". Does anyone have a
> different opinion?
>
> 2) The draft 1.7 does not yet include it, so its been hard to refer users
> to it.
>
>
> http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/cf-conventions.html#appendix-grid-mappings
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
>
> 
>
> Randy C. Horne (rho...@excaliburlabs.com)
> Principal Engineer, Excalibur Laboratories Inc.
> voice & fax: (321) 952.5100
> url: http://www.excaliburlabs.com
>
>
>
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] geostationary vs vertical_perspective in CF 1.7.

[John Caron]

CF TRAC ticket 72 defines "geostationary projection", accepted for 1.7.

1) I _think_ that "Vertical Perspective" is an earlier version of this, and
it should be deprecated in favor of "geostationary". Does anyone have a
different opinion?

2) The draft 1.7 does not yet include it, so its been hard to refer users
to it.

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/cf-conventions.html#appendix-grid-mappings
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Correct mail group address?

[John Caron]

I see, so we should continue to use cf-metadata@cgd.ucar.edu. for normal
conversation?

BTW, the TRAC system is either down or slow enough to time out.

On Tue, Nov 18, 2014 at 12:57 PM, Jeffrey F. Painter 
wrote:

>  Use cf-metadata@cgd.ucar.edu.
> The one at llnl.gov is part of the system for connecting the Trac
> issue-tracker to the main mailing list at ucar.edu.
> - Jeff
>
>
> On 11/18/14, 11:52 AM, John Caron wrote:
>
> I have both
>
>  cf-metadata@cgd.ucar.edu
>
>  and
>
>  cf-metad...@lists.llnl.gov
>
>  it appears the many (most?) still use cf-metadata@cgd.ucar.edu, i guess its 
> being forwarded to cf-metad...@lists.llnl.gov
>
>
>
> ___
> CF-metadata mailing 
> listcf-metad...@cgd.ucar.eduhttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Correct mail group address?

[John Caron]

I have both

cf-metadata@cgd.ucar.edu


and


cf-metad...@lists.llnl.gov


it appears the many (most?) still use cf-metadata@cgd.ucar.edu, i
guess its being forwarded to cf-metad...@lists.llnl.gov
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] string valued coordinates

[John Caron]

My preference is that one explicitly puts in the units. For dimensionless,
"1" or "" is ok for udunits. If the units attribute isnt there, I assume
that the user forgot to specify it, so the units are unknown.

Im not sure what CF actually says, but it would be good to clarify.

John

On Thu, Oct 30, 2014 at 2:37 AM, Hedley, Mark 
wrote:

>  Hello CF
>
> > From: CF-metadata [cf-metadata-boun...@cgd.ucar.edu] on behalf of
> Jonathan Gregory [j.m.greg...@reading.ac.uk]
>
> > Yes, there are some standard names which imply string values, as Karl
> says. If the standard_name table says 1, that means the quantity is
> dimensionless, so it's also fine to omit the units, as Jim says.
>
> I would like to raise question about this statement.  Omitting the units
> and stating that the units are '1' are two very different things;
> dimensionless != no_unit
> is an important statement which should be clear to data consumers and
> producers.
>
> If the standard name table defines a canonical unit for a standard_name of
> '1' then I expect this quantity to be dimensionless, with a unit of '1' or
> some multiple there of.
> If the standard name states that the canonical unit for a standard_name is
> '' then I expect that quantity to have no unit stated.
> Any deviation from this behaviour is a break with the conventions.  I have
> code which explicitly checks this for data sets.
>
> Are people aware of examples of the pattern of use described by Jonathan,
> such as a categorical quantities identified by a standard_name with a
> canonical unit of '1'?
>
> thank you
> mark
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] string valued coordinates

[John Caron]

>I understand that netCDF coordinate variables have to be strictly
monotonic, and no-one wants to define what this means for the general case
of strings; that is fine.

in CDM, monontonicity is required to make the 1D coordinate maps
invertible. For string valued coordinates, the equivilent requirement is
uniqueness. Im not sure if CF data model states this or not.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[John Caron]

Hi all:

Yes, 1.10 would follow 1.9, if needed, on that "branch".

In software, we are used to major version increments meaning "API may have
changed". Since any CF file using features of the extended model will not
be fully visable (or even break) software that only understands the classic
model, it seems right to increment the major version number of the CF
Convention.

John



On Wed, Sep 24, 2014 at 2:54 AM,  wrote:

> Tim,
>
> The usual convention is that version numbers are not decimal but
> "."-separated integers.  Therefore, following 1.9 would be 1.10 and there
> is no necessity to merge into 2.0.
>
> On the subject of git and github.  I would thoroughly recommend embracing
> github as a collaborative tool.  The github ticket system is excellent and
> the freedom to experiment that comes with git branches and forks is truly
> revolutionary.  For me, the point about git is that all parties are in
> control -- no-one needs authorisation to try something new and the
> "official" branch does not loose control by allowing experimentation.  It
> is a technology which enables a true meritocracy.
>
> Having said that, I understand that it is a big culture shift which people
> don't "get" until they have had to use it for a while.
>
> My final point -- bear in mind that those of us who's job it is to archive
> data have no choice but to remain backwards compatible :-).
>
> Stephen.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
> -Original Message-
> From: Timothy Patterson [mailto:timothy.patter...@eumetsat.int]
> Sent: 24 September 2014 09:16
> To: 'Charlie Zender'; CF Metadata Mail List
> Subject: Re: [CF-metadata] CF-2.0 Convention discussion
>
> I'm wondering whether CF Convention 2.0 is the right naming convention to
> be using to refer to this new branch.
>
> Firstly, as the "classic" CF Convention has moved from 1.0 to 1.7 in 0.1
> increments, is the original branch then forced after CF 1.9 to either merge
> with CF 2.0 or to adopt a new naming convention (1.9, 1.91, 1.92...1.991,
> 1.992...)?
>
> Secondly, as a newcomer to the CF Convention, if I find documents called
> CF Convention 1.7 and CF Convention 2.0, I'm probably going to assume that
> CF 2.0 is the document I should be using.
>
> As an alternative, the new branch could be given a unique name e.g. " CF
> Conventions Enhanced 1.0" (to imply the use of netCDF-4 enhanced features).
> Then we could refer to CF Enhanced 1.0 and CF 1.7 or CF Classic 1.7 when we
> need to differentiate between the two branches.
>
> Regards,
>
> Tim
>
> -
>
> Dr. Timothy Patterson
> Instrument Data Simulation
> Product Format Specification
>
> EUMETSAT
> Eumetsat-Allee 1
> 64295 Darmstadt
> Germany
>
>
>
> -Original Message-
> From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
> Charlie Zender
> Sent: Friday, September 19, 2014 6:20 PM
> To: CF Metadata Mail List
> Subject: [CF-metadata] CF-2.0 Convention discussion
>
> I like the idea of a Google Doc to consolidate the issue list.
> Whether it will prove unwieldy is hard to guess.
>
> A 3-month period for listing use-cases seems reasonable.
> It seems more CF-like to be use-case driven than to presume the entire
> netCDF4 data model will be adopted.
> This has the advantage of parsimony--only implement what is necessary.
> From a technical point-of-view, however, it is advantageous to know already
> what netCDF4 can and cannot easily do. In my view the intersection of
> important use-cases and netCDF4 capabilities is the "sweet spot" for CF-2.0.
>
> Once the smoke clears I may summarize use-cases for handling
> ensembles/replicants and for handling large integer numbers. These could
> raise the issues of having groups and 64-bit ints in CF-2.0.
> But first let's settle the framework for discussions.
>
> Best,
> Charlie
> --
> Charlie Zender, Earth System Sci. & Computer Sci.
> University of California, Irvine 949-891-2429 )'(
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> --
> Scanned by iCritical.
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Why "surface_altitude" instead of "platform_altitude"?

[John Caron]

As I recall, the original proposal was for station_altitude. We decided to
change "station" to "platform". At the same time it was thought that the
existing standard name of "surface altitude" would be synonymous. I at
least was thinking of ground stations. So I think we make a mistake there
and "platform_altitude" would be the right correction.

An altitude of course needs a datum, and I think we have not been clear
enough on that. I think we should review our use or non-use of vertical
datum. A quick look seems to imply that "WGS 84" is assumed (?)

On Thu, Sep 18, 2014 at 1:42 PM, Signell, Richard  wrote:

> John,
> So then the surface needs to be defined relative to some known datum, no?
>
> Maybe we need platform_altitude_above_datum  and a specification of
> the vertical datum (EPSG:5701 (MSL), EPSG:5703 (NAVD88), etc)
>
> On Thu, Sep 18, 2014 at 1:47 PM, John Graybeal
>  wrote:
> > I assume surface_altitude is an important variable for providing the
> vertical location of measurements relative to a surface (as opposed to
> relative to a geoid -- notwithstanding the definition issue).
> >
> > John
> >
> > On Sep 18, 2014, at 08:30, Signell, Richard  wrote:
> >
> >> Maybe a simpler approach would be to just adopt "platform_altitude" as
> >> an alias for "surface_altitude" and suggest deprecating the use of
> >> "surface_altitude"?
> >>
> >> On Thu, Sep 18, 2014 at 11:15 AM, John Graybeal
> >>  wrote:
> >>> Interesting that there is so little discussion of this language in the
> mail list, only in John Caron's 2011.09.16 mail on standard names for
> stations (which refers to words already in draft 1.6, I think) -- which
> came at the tail end of a long thread on platform names/IDs.
> >>>
> >>> From those words, I infer that the original drafter thought
> surface_altitude was just as good for describing platform location, as it
> was for describing observation location. I suspect the assumption was that
> any corresponding observations were at the same location as the platform.
> >>>
> >>> Since this is not always true, I'm with you that there should be a
> term platform altitude, and it should be the one used in this sentence.
> >>>
> >>> I hereby propose the standard name platform_surface_altitude (m),
> "Standard names for platform describe the motion and orientation of the
> vehicle from which observations are made e.g. aeroplane, ship or satellite.
> >>> The surface called "surface" means the lower boundary of the
> atmosphere. Altitude is the (geometric) height above the horizontal
> reference surface."
> >>>
> >>> Note I've changed the standard wording of the _altitude definition,
> which generally says ".. above the geoid, which is the reference
> geopotential surface. The geoid is similar to mean sea level." This seems
> clearly in conflict with the definition of surface_altitude and this new
> term, and I think it should be changed in surface_altitude's definition too.
> >>>
> >>> I suppose if people agree with you and me, we need to do a Trac ticket
> for the corresponding change to the standard.
> >>>
> >>> John
> >>>
> >>>
> >>>
> >>> On Sep 18, 2014, at 06:40, Signell, Richard  wrote:
> >>>
>  In the CF-1.6 and CF-1.7 draft doc, in section H.2, we have:
> 
>  "It is recommended that there should be station variables with
>  standard_name attributes " platform_name ", " surface_altitude " and “
>  platform_id ” when applicable."
> 
>  Why is this "surface_altitude" instead of "platform_altitude"?
> 
>  In the ocean, we have lots of upward-looking Acoustic Doppler Current
>  Profilers (ADCP), where the instrument with transducer and other
>  sensors is located some distance below the ocean surface.   While
>  velocity and other properties are measured in vertical bins above the
>  instrument (timeSeriesProfile), other properties like pressure and
>  temperature are measured at the instrument.
> 
>  Since the instrument is not at the surface, it seems misleading to use
>  the standard_name "surface_altitude" instead of "platform_altitude",
>  particularly when we already have "platform_name" and "platform_id".
> 
>  In this example CF_1.6 ADCP dataset:
> 
> 
> http://geoport.whoi.edu/thredds/dodsC/usgs/data2/rsignell/data/adcp/7201adc-a_cf16.nc.html
> 
>  the variable "platform_altitude" has a value of -10.4522 m:
> 
> http://geoport.whoi.edu/thredds/dodsC/usgs/data2/rsignell/data/adcp/7201adc-a_cf16.nc.ascii?platform_altitude
> 
>  but we are forced to use a standard_name of "surface_altitude".
> 
>  Why not "platform_altitude"?
> 
>  Thanks,
>  Rich
> 
>  --
>  Dr. Richard P. Signell   (508) 457-2229
>  USGS, 384 Woods Hole Rd.
>  Woods Hole, MA 02543-1598
>  ___
>  CF-metadata mailing list
>  CF-metadata@cgd.ucar.edu
>  http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> >>>
> >

Re: [CF-metadata] CF Conventions and netCDF-4 enhanced model

[John Caron]

Hi Timothy:

>From my POV, groups are a way to, um, group things (variables, coord
systems, etc) together. They should be used for the same reason we use
directories in a file system, to prevent a big mess of things in one
namespace. On the surface, your layout above seems reasonable for those
reasons.

The main restriction from the CDM data model is that variables can only use
dimensions from their own group or a parent group.  the result is to keep
coordinate processing simple and close to the classic model. it does affect
the way you lay out your groups, which may be different from how you think
about your data. HDF-EOS tends to stick coordinates in a different group,
for example. But I havent seen a use case where it makes something
impossible that really is needed. You just need to push common dimensions
up into a common parent.

John

On Thu, Sep 11, 2014 at 4:27 AM, Timothy Patterson <
timothy.patter...@eumetsat.int> wrote:

>
> I spent an instructive evening reading through the previous discussions
> (thanks for the links, Corey) and the arguments for and against using
> hierarchal structures. I also re-read the CF conventions documents again
> (1.6 and 1.7 draft) and it seems the standard currently ignores groups
> rather than explicitly forbidding their use. It seems to me that a netCDF
> dataset with groups could still conform to the CF conventions as they are
> currently written, even with all the other restrictions that the standard
> imposes. I'd be interested in seeing and possibly helping with CF
> conventions for supporting the enhanced model.
>
> After reading the previous discussions, I thought it might be interesting
> to the list to explain our use of groups in netCDF products as it is
> somewhat different from the other cases that were discussed.
>
> Our netCDF datasets have to cope with a number of different needs from
> various parties - archive, end-users, higher-level processing,
> reprocessing, monitoring, etc. To keep things simple, we wanted a single
> format per instrument/processing level that is flexible enough to contain
> all the data or a subset of the data depending upon the consumer needs. To
> do this, we created a hierarchal data structure that encapsulates data in
> related, but independent groups. These groups can be present in or missing
> from the dataset as required by the needs of the consumer. So a level 2
> processing function might receive a product containing 20 instrument
> channels at 2 different resolutions, whereas the dissemination function
> might receive a product with just 5 of these channels at the lowest
> resolution. Both of these products are described by a single format
> specification.
>
> This model of including or omitting independent groups also supports other
> needs, for example being able to add data that is produced at irregular
> intervals but needs to be in the product when it is available. Also, by
> tagging groups with a specific attribute, we should also be able to have a
> single, generic method for end-users to be able to subset data retrieved
> from the archive without requiring specific knowledge of each netCDF
> product. They should be able to select only the tagged groups (which might
> correspond to instrument channels for instance) that they want in their
> retrieved datasets.
>
> This gives us a single, easily understood format definition that
> encompasses a wide variety of possible variations.
>
> Any feedback on the idiocy or genius of this (ab)use of the netCDF format
> is welcome.
>
> Thanks,
>
> Tim
>
> -
>
> Dr. Timothy Patterson
> Instrument Data Simulation
> Product Format Specification
>
> EUMETSAT
> Eumetsat-Allee 1
> 64295 Darmstadt
> Germany
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF Conventions and netCDF-4 enhanced model

[John Caron]

Hi Karl and all:

NetCDF-4 compression and chunking are transparent to the user, and are
compatible with the "classic data model".

I think we should be gathering experiences with  the enhanced data model,
and start a CF-2.X convention draft document that uses the enhanced model.
It would also be a good time to remove deprecated features and in general
not require backwards compatibility. Perhaps if there are 5-6 people we
could start a working group to discuss.

John


On Wed, Sep 10, 2014 at 10:35 AM, Corey Bettenhausen <
corey.bettenhau...@ssaihq.com> wrote:

> Tim,
> There was a discussion of this last year. See the archives:
> http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/author.html
>
> Particularly, the thread "Towards recognizing and exploiting hierarchical
> groups":
> http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2013/056827.html
>
> Cheers,
> -Corey
>
> On Sep 10, 2014, at 9:53 AM, Timothy Patterson wrote:
>
> > Is it correct to say that, although they don't explicitly state it, the
> CF conventions (1.6 and the draft 1.7) restrict compliant netCDF products
> to be either netCDF-3 or netCDF-4 in classic format? There are no
> conventions for the enhanced features such as groups and user-defined types
> like enumerated variables, and Section 2.2, as an example, bars the use of
> unsigned integer variables or string variables (which are even stated not
> to exist, again implying classic-model only).
> >
> > There are some features of the enhanced model we want to use for our
> future datasets (such as groups) and some features which would make life
> easier but could be worked around if it led to CF compliance (enumerated
> types, unsigned integers, string types, etc.). Are there any plans to
> introduce conventions for the use of these enhanced features at some point
> in the future or would non-classic model datasets always be seen as
> non-compliant?
> >
> > Thanks for your insights on this issue!
> >
> > Regards,
> >
> > Tim Patterson
> >
> >
> >
> > -
> >
> > Dr. Timothy Patterson
> > Instrument Data Simulation
> > Product Format Specification
> >
> > EUMETSAT, Eumetsatallee 1, D-64295 Darmstadt, Germany
> > ___
> > CF-metadata mailing list
> > CF-metadata@cgd.ucar.edu
> > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
> --
> Corey Bettenhausen
> Science Systems and Applications, Inc
> NASA Goddard Space Flight Center
> 301 614 5383
> corey.bettenhau...@ssaihq.com
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Date/time and leap seconds

[John Caron]

Hi Jim:

Thanks for explaining some of the details of TAI and UTC time standards.
Obviously Im not much of an expert on them. What I am 99 44/100 % sure of
is that udunits uses 60 seconds per minute no matter what, as does
joda-time, which the netcdf-java library uses. So thats what I mean by "udunits
does not support leap seconds". Im unclear whether that means udunits and
joda-time are not really UTC.

You are right that its all about the "conversion from time/date strings to
counts of seconds and back" . I have advocated that ISO 8601 datetime
strings be an optional way to specify time coordinates in CF, with the
argument that they are unambiguous and easy enough for even rocket
scientists to understand ;) That hasnt been accepted in CF, although
netcdf-java will allow them, and one can always use them
as auxiliary coordinates, in addition to the required udunits "secs since
epoch" coordinate.

I think, like Jonathan, an additional calendar option in CF could be the
way forward here. Java 8 has a new date/time package (based on joda-time)
that will allow for correct UTC leap-seconds. So I think I can get them
into netcdf-java (eventually) if that was needed.

What do you think?

Regards,
John




On Thu, Jul 17, 2014 at 7:14 AM, Jim Biard  wrote:

>  John,
>
> I'm afraid that your response has confused me quite a lot. If you don't
> mind, please clarify what you mean by "udunits does not support leap
> seconds". TAI does not use leap seconds. It is a running count of seconds
> since Jan 1, 1958 00:00:00 UTC. UTC time messes around with leap seconds in
> order to account for the fact that a day is not exactly 864000 seconds
> long, and that the rotation rate varies over time. The leap seconds are
> used to keep time/date "strings" synchronized to the diurnal cycle, much
> the same way that leap days are used to keep date strings synchronized to
> the Vernal Equinox.
>
> I think this problem may have more to do with the conversion from
> time/date strings to counts of seconds and back than anything about the
> counts of seconds themselves.
>
> Grace and peace,
>
> Jim
>
> A true count of seconds since an epoch matches the approach of TAI, not
> UTC.
>
> On 7/16/14, 5:58 PM, John Caron wrote:
>
> Hi Maik:
>
>  Unfortunately, CF references the udunits package which is no longer
> being developed, at least for date/time coordinates.
>
>  udunits does not support leap seconds.
>
>  your best bet is to add an auxiliary time coordinate which uses leap
> seconds, eg TAI. your specialized software can use that
>
>  also add a time coordinate which uses the standard non-leap seconds from
> udunits. visualization software will use this to display calendar dates, i
> think if you are careful this will only be wrong on a leap second.
>
>  You could propose adding a leap second-aware calendar to CF.
>
>
> On Wed, Jul 16, 2014 at 2:48 PM, Maik Riechert 
> wrote:
>
>> Hi Jim
>>
>> > I see where the potential for mismatches when using time values with
>> > "unexpected" encoding means that we should provide a way to declare what
>> > kind of time-since-epoch values are used in a particular time variable.
>>
>>  And that's exactly the problem. The CDF website summarizes it quite
>> nicely: "Science data should have well-defined times to enable more
>> accurate cross-comparison between missions and as documentation for
>> future archive use." netcdf-CF is basically CDF before introduction of
>> the CDF_TIME_TT2000 type.
>>
>> In the end, people use libraries to read datasets, and I'm pretty sure
>> that the netcdf4 library in Python is used a lot, and this one assumes a
>> certain time encoding, without actually having it specified within the
>> CF conventions. And I find this all a bit worrying, but many people
>> probably don't care as their time resolution is higher than 1 second.
>> Still, I think this issue should be addressed, better sooner than later.
>> Don't you think?
>>
>> Cheers
>> Maik
>>
>> >
>> > Grace and peace,
>> >
>> > Jim
>> >
>> > On 7/16/14, 4:06 AM, Maik Riechert wrote:
>> >> Hi,
>> >>
>> >> after reading
>> >> http://cdf.gsfc.nasa.gov/html/leapseconds_requirements.html I'm a bit
>> >> confused on how CF handles times for the netcdf world. It seems as if
>> >> the 'seconds since' convention is basically POSIX time in regards to
>> >> how leap seconds are handled. Quoting wiki " Due to its handling of
>> >> leap seconds, it is neither a linear representa

Re: [CF-metadata] Date/time and leap seconds

[John Caron]

Hi Maik:

Unfortunately, CF references the udunits package which is no longer being
developed, at least for date/time coordinates.

udunits does not support leap seconds.

your best bet is to add an auxiliary time coordinate which uses leap
seconds, eg TAI. your specialized software can use that

also add a time coordinate which uses the standard non-leap seconds from
udunits. visualization software will use this to display calendar dates, i
think if you are careful this will only be wrong on a leap second.

You could propose adding a leap second-aware calendar to CF.


On Wed, Jul 16, 2014 at 2:48 PM, Maik Riechert 
wrote:

> Hi Jim
>
> > I see where the potential for mismatches when using time values with
> > "unexpected" encoding means that we should provide a way to declare what
> > kind of time-since-epoch values are used in a particular time variable.
>
> And that's exactly the problem. The CDF website summarizes it quite
> nicely: "Science data should have well-defined times to enable more
> accurate cross-comparison between missions and as documentation for
> future archive use." netcdf-CF is basically CDF before introduction of
> the CDF_TIME_TT2000 type.
>
> In the end, people use libraries to read datasets, and I'm pretty sure
> that the netcdf4 library in Python is used a lot, and this one assumes a
> certain time encoding, without actually having it specified within the
> CF conventions. And I find this all a bit worrying, but many people
> probably don't care as their time resolution is higher than 1 second.
> Still, I think this issue should be addressed, better sooner than later.
> Don't you think?
>
> Cheers
> Maik
>
> >
> > Grace and peace,
> >
> > Jim
> >
> > On 7/16/14, 4:06 AM, Maik Riechert wrote:
> >> Hi,
> >>
> >> after reading
> >> http://cdf.gsfc.nasa.gov/html/leapseconds_requirements.html I'm a bit
> >> confused on how CF handles times for the netcdf world. It seems as if
> >> the 'seconds since' convention is basically POSIX time in regards to
> >> how leap seconds are handled. Quoting wiki " Due to its handling of
> >> leap seconds, it is neither a linear representation of time nor a true
> >> representation of UTC". This means that any dataset with <= 1 second
> >> resolution which covers the moment a leap second happens will have a
> >> problem encoding the time using the existing units convention.
> >>
> >> I first posted this issue on the netcdf4-python tracker:
> >> https://github.com/Unidata/netcdf4-python/issues/280
> >>
> >> Is this an issue which was discussed before and is there any solution?
> >>
> >> Cheers
> >> Maik
>
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Are ensembles a compelling use case for "group-aware" metadata? (CZ)

[john caron]

Hi all:

A few thoughts from my (possibly limited) POV.

1. Best strategy for storing very large collections of data in flat 
files like netcdf?


- store coherent chunks of the dataset in each file. Seems like a good 
file size these days is 100M - 1 Gbyte.
- choose "coherence" by deciding on the reader's most likely access 
pattern, assuming "write once - read many".
- build an external index of the entire dataset to make access as fast 
as possible. the index can be rebuilt as needed without modifying the 
data files.


2. Whats the best use for groups in a netcdf file?

I think its best as a mimic to heirarchical file systems.  when there 
are more than a (few?) hundred files in a directory, the listing become 
unwieldy and hard to navigate. also, a file directory is a "namespace" 
that allows the same filename to be used in different directories. i 
think groups should be used in the same way for the same reasons. from 
this POV, an array of groups i think is not a good idea.


Zip files are interesting in that they have internal directories, and 
tools like 7zip and Windows file explorer transparently let you browse 
the OS file heirarchy and the internal zip heirarchy.


In THREDDS, we have found a good use for groups as follows. Given a 
collection of GRIB files, if there are multiple horizontal grids, we 
create a seperate group for each horizontal grid, and put all the 
variables that use that horizontal grid into that group. This creates a 
dataset with multiple groups. Each group is actually self-contained, so 
the TDS, when serving the dataset out,  will break the dataset back into 
multiple datasets, one for each group. This simplifies things for 
software that is not group aware.


A netcdf file is itself a container that creates a namespace for the 
variables within it. So you dont usually need groups for a file that has 
less than N variables in it (N = 50? 200??). The case of multiple 
coordinate systems is a reasonable exception, because often you have 
exactly the same variables in each group, so you need a seperate namespace.


One could apply this argument to ensemble data, and i think its 
reasonable. But as others have pointd out, you likely have much more 
data than you want to place into a single file. So then the file becomes 
the namespace and internal groups are less compelling.


Suppose we are talking about the logical structure of a multi-file 
collection? One could decide that internal groups are a good way to keep 
things clear, even though you perhaps never have more than one group in 
a file. Then, when you "aggregate" the files into a logical dataset, the 
user sees a structure that makes it clear what things are. One would 
also want to be able to specify such a structure in an external 
configuration that describes the collection. No doubt this issue will 
come back when CF works on conventions for multifile collections.


3. What does the TDS currently do with groups?

The TDS data model (aka CDM) handles groups just fine. You can aggregate 
variables in groups across files, as long as they are identical. We are 
waiting for DAP4 to be able to serve these though OPeNDAP, but the other 
services (WMS, WCS, NCSS, cdmremote)  work now with groups. OTOH, we 
dont see them much used yet, except in HDF-EOS.


4. Things are hard to predict, especially in the future.

I think software will evolve to handle the extended data model. I think 
data providers should start to use it, and CF could start examining best 
practices for "extended data model" Conventions (Perhaps CF-2 ?). That 
is to say, providers cant wait for CF conventions, they need to try new 
stuff that CF eventually incorporates (or not). Jim Baird's satellite 
data seems to be a good example of this.




On 9/29/2013 9:46 PM, Charlie Zender wrote:

Hi Steve,

Thank you for your continued engagement and responses.
Looks like CF hasn't the appetite for group hierarchies anytime soon.
I'll report the lessons learned in this fruitful discussion to our
NASA WG next week. My "concluding" (for now) remarks are interleaved.

Best,
Charlie

Le 25/09/2013 12:33, Steve Hankin a écrit :>

On 9/24/2013 9:45 PM, Charlie Zender wrote:

It is not my place to determine whether there is a consensus, or how
close we are, but it's clear to me there is no consensus yet. Bryan
Lawrence, Steve Hankin, Jonathan Gregory, Karl Taylor, and Philip
Cameron-Smith are not "on board". I hope they will speak-up and say if
they concur that maintaining the status quo (flat files) is best
(period), or whether they do wish to extend CF to hierarchies
(starting now), or the additional information they would need to decide.

Hi Charlie et. al.,

Since you have asked   I have heard two points that seemed to
bolster Bryan's pov that the multi-model use case is  "great but not
compelling".  (See a more positive spin at the end.)

  1. file size.   Model outputs today are typically too large for even a
 single variable from a single model to be

Re: [CF-metadata] Are ensembles a compelling use case for "group-aware" metadata? (CZ)

[john caron]

On 9/19/2013 3:58 PM, Schultz, Martin wrote:

Now, here is another use case, which we haven't implemented yet - partly 
because we didn't see how it can be done in a CF consistent way:
While there has been a definition of a standard file layout for data from multiple 
stations (a contribution from Ben Domenico and Stefano Nativi if I am not mistaken), this 
concept cannot be applied to multiple aircraft flight data. The station data can be 
packaged together with help of a non-geophysical "station" coordinate, because 
all stations share the same time axis. With aircraft flights, the time axes often don't 
overlap, and forcing all data onto the superset of time would be a tremendous waste of 
space. Groups would seem as  the natural solution to this problem! Why not flat files? 
Because you might wish to retrieve all the aircraft data which were sampled in a given 
region during a specific period (a natural use case for a catalogue query it seems) in 
one entity, and not in N entities, where you cannot even predict N.

Hi Martin:

if i understand this example, you could use any of the DSG trajectory 
forms to have seperate time coordinates for each aircraft track, eg for 
multidimensional representation (4.1) :


 double time(trajectory, obs) ;

this one assumes you have (about) the same number of observation for 
each flight. The ragged array representation (4.3 and 4.4) would allow 
you to efficiently store any number of obs for each flight, and each obs 
would have its own time coordinate.


The DSG was indeed intended to enable spatial/temporal queries over the 
dataset. A natural organization might be to "time partition" the data 
(all flights for a day in one file). One either searches through the 
files for spatial subsets or employs an external  spatial index. The TDS 
will do spatial/temporal subsetting on time partitioned DSG files, 
though we dont yet have a spatial index.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi Sean:

Ive already made a fix in the 4.4.0 branch to preserve the global
attribute Conventions=CF.1-x if that exists in the source file. Easy to
do and its the right thing.

More difficult is to guarentee that all CF semantics get preserved in 
the translation. The subsetting process has to change the coordinate 
system, but we try to pass everything else through. Let us know any 
issues and we will try to fix. We are looking at the ones that Seth has 
reported.


The problem is that the damn software has to be told exactly what to do! 
"Read the CF Conventions document", I said. Sheesh.


John

On 8/30/2013 1:59 AM, Gaffney, Sean P. wrote:

Hi John and all,

My initial reaction was the same as Seth's in that I felt
uncomfortable that any aspect of the metadata was being altered.
However, if it can be confirmed that only change occurring will be
the alteration of the Conventions attribute back to 1.0, that is
something we can handle with a declaration to the user.

As it stands, I've inspected the variables and attributes for the
model data we've got at present once they've been run through THREDDS
and so far, there are no actual changes apart from the convention
being altered from 1.4 on the data we received, back to 1.0 in the
output. My concern is whether this situation could be guaranteed into
the future, as we at BODC receive more complex oceanographic model
data in CF 1.6.

Cheers

Sean

-Original Message- From: CF-metadata
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of John Caron
Sent: 29 August 2013 00:19 Cc: cf-metadata@cgd.ucar.edu Subject: Re:
[CF-metadata] thredds changing CF conventions version

Hi John and all:

For sure its a good idea to preserve the CF Conventions attribute if
it exists. I will add that to our issue tracker.

Remember that THREDDS is a general purpose subsetter, not specific
to CF. So we deal with lots of source datasets (eg GRIB) that arent
CF. The point is that we convert whatever we have to CF.

Up to this point, as CF versions have evolved, there hasnt been any
difference in the way that we write out these files. So im interested
if Ive missed something and there is some actual change in the file
structure that we should be tracking from Sean's POV. Other than the
actual version number.

John


On 8/28/2013 11:11 AM, John Graybeal wrote:

It isn't just about feature presentation, right?  On the assumption
that the latest version is always most current (so everyone will
use the latest version if they can), I use the version as a proxy
for how current/up-to-date/sophisticated the data provider is.

It also gives me a clue about which standard name table version
they might use, though that version info is often available
elsewhere.

If true, this may help explain why so few data providers seemed to
be using 1.6, anyway....

John

On Aug 28, 2013, at 08:56, John Caron 
wrote:


Hi Sean:

What feature of CF are you using that you need to preserve the
version?

John

"If you torture data long enough, it will confess." -- Economist
Ronald Coase



On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote:

Hi, Here at the British Oceanographic Data Centre we use
THREDDS to deliver and subset our numerical model data that we
hold in CF netcdf format. I've just been made aware that during
the delivery and subsetting process, THREDDS seems to be
converting the CF files we hold from whatever CF convention
they have been supplied in, to CF 1.0. Is this something that
the rest of the community are aware of, and if so, do people
consider it to be of major importance that the files outputted
by thredds are in a different convention to the source files
thredds interrogates? Cheers Sean Gaffney BODC

_   _ This message (and any
attachments) is for the recipient only. NERC is subject to the
Freedom of Information Act 2000 and the contents of this email
and any reply you make may be disclosed by NERC unless it is
exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.


___ CF-metadata
mailing list CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___ CF-metadata
mailing list CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


 John Graybeal Senior Data
Manager, Metadata and Semantics

T +1 (408) 675-5545 F +1 (408) 616-1626 skype: graybealski

Marinexplore 920 Stewart Drive Sunnyvale, CA



___ CF-metadata mailing
list CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

This message (and any attachments) is for the recipient only. NERC is
subject to the Freedom of Information Act 2000 and the contents of
this email and any reply you make may be disclosed by NERC unles

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi Seth:

On 8/28/2013 12:59 PM, Seth McGinnis wrote:

Hi Sean,

Personally, I would regard that as suspect behavior.

I'm of the opinion that it's best practice for a data service to
affect the data it operates on in very targeted and transparent
way, and that it should pass everything else through untouched.
If it doesn't do that, it's harder to trust the service, because
the more ways that it modifies the data, the more opportunities
there are for it to get something wrong, and the more likely it
is that the error will go unnoticed because it's hiding amongst a
bunch of other changes.


We take this philosophy also.



Case in point, I have recently discovered that when the THREDDS
server subsets CF data that uses a coordinate system other that
lat-lon, it mangles a bunch of the ancillary data in the file.  I
suspect this happens, ironically enough, precisely because it's
trying to make the data CF-1.0 conformant instead of just leaving
it as-is...


Let us know what the specifics are and we can try to make it better.



So yes, I think it's significant that the data coming out of
THREDDS is in a different convention than the source files, and
that it's cause for concern.


A famous story from NCAR data portal early days is that some users added 
the Conventions = "CF-1.0" attribute to their data because they had to 
make their data CF compliant. But thats all they did. Hey, it passed the 
compliance checker! ;)


Regards,
John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi John and all:

For sure its a good idea to preserve the CF Conventions attribute if it 
exists. I will add that to our issue tracker.


Remember that THREDDS is a general purpose subsetter, not specific to 
CF. So we deal with lots of source datasets (eg GRIB) that arent CF. The 
point is that we convert whatever we have to CF.


Up to this point, as CF versions have evolved, there hasnt been any 
difference in the way that we write out these files. So im interested if 
Ive missed something and there is some actual change in the file 
structure that we should be tracking from Sean's POV. Other than the 
actual version number.


John


On 8/28/2013 11:11 AM, John Graybeal wrote:

It isn't just about feature presentation, right?  On the assumption that the 
latest version is always most current (so everyone will use the latest version 
if they can), I use the version as a proxy for how 
current/up-to-date/sophisticated the data provider is.

It also gives me a clue about which standard name table version they might use, 
though that version info is often available elsewhere.

If true, this may help explain why so few data providers seemed to be using 
1.6, anyway

John

On Aug 28, 2013, at 08:56, John Caron  wrote:


Hi Sean:

What feature of CF are you using that you need to preserve the version?

John

"If you torture data long enough, it will confess."
  -- Economist Ronald Coase



On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote:

Hi,
Here at the British Oceanographic Data Centre we use THREDDS to deliver
and subset our numerical model data that we hold in CF netcdf format.
I’ve just been made aware that during the delivery and subsetting
process, THREDDS seems to be converting the CF files we hold from
whatever CF convention they have been supplied in, to CF 1.0.
Is this something that the rest of the community are aware of, and if
so, do people consider it to be of major importance that the files
outputted by thredds are in a different convention to the source files
thredds interrogates?
Cheers
Sean Gaffney
BODC

_   _
This message (and any attachments) is for the recipient only. NERC is
subject to the Freedom of Information Act 2000 and the contents of this
email and any reply you make may be disclosed by NERC unless it is
exempt from release under the Act. Any material supplied to NERC may be
stored in an electronic records management system.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



John Graybeal
Senior Data Manager, Metadata and Semantics

T +1 (408) 675-5545
F +1 (408) 616-1626
skype: graybealski

Marinexplore
920 Stewart Drive
Sunnyvale, CA



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] thredds changing CF conventions version

[John Caron]

Hi Sean:

What feature of CF are you using that you need to preserve the version?

John

"If you torture data long enough, it will confess."
  -- Economist Ronald Coase



On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote:

Hi,
Here at the British Oceanographic Data Centre we use THREDDS to deliver
and subset our numerical model data that we hold in CF netcdf format.
I’ve just been made aware that during the delivery and subsetting
process, THREDDS seems to be converting the CF files we hold from
whatever CF convention they have been supplied in, to CF 1.0.
Is this something that the rest of the community are aware of, and if
so, do people consider it to be of major importance that the files
outputted by thredds are in a different convention to the source files
thredds interrogates?
Cheers
Sean Gaffney
BODC

_   _
This message (and any attachments) is for the recipient only. NERC is
subject to the Freedom of Information Act 2000 and the contents of this
email and any reply you make may be disclosed by NERC unless it is
exempt from release under the Act. Any material supplied to NERC may be
stored in an electronic records management system.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series & lat/lon coordinates

[John Caron]

Hi Mark and Jonathan:

I will say straight up that I dont understand the implications of this 
particular discussion. But in case its helpful, I have found it 
important to separate the data model and the encoding of the model in 
the netCDF file. Probably you are both already doing that, but I wasnt sure.


John

On 6/7/2013 3:23 AM, Hedley, Mark wrote:

Hello Jonathan


As a result of this discussion, it seems me that for a DSG (which is indicated 
by the presence of featureType), scalar coordinate variables have to be 
interpreted as auxiliary coordinate variables of an omitted size-one instance 
dimension. That is what is implied by section 9.2. It's different from the 
interpretation that is implied by section 5.7,  which should exclude DSGs (and 
predates DSGs).


I don't think this approach holds together safely.  9.1 describes data 
variables and space-time (auxiliary) coordinate variables as varying with index 
dimensions.

The different feature types have different expectations for the relationship 
between dimensions, data and space-time coordinates.

9.2 describes the special case where the instance dimension is omitted and the 
space-time coordinates are scalar coordinate variables.

Interpreting sampling geometry scalar coordinates as a set of special cases, 
all different from the special cases of interpreting scalar coordinates in the 
absence of a featureType, is placing extensive complexity in the conventions 
and their interpretations.  I think this is asking too much of data producers 
and data consumers.

In many cases I think data producers have interpreted and will interpret the 
conventions (paraphrasing extensively) as 'these are just scalars'.

We also have to be mindful of the large selection of data sets with some 
sampling geometry like properties but which do not use the featureType 
attribute, e.g. from CF 1.5 or earlier.


I see no problem with having different interpretations for different purposes.


I see particular problems with this statement.  Treating the core types of the 
CF conventions differently depending on the context, such as by featureType and 
variable type introduces extensive complexity in data creation and 
interpretation, which I think CF should be very careful of. I do not think this 
is necessary.

Continuing and expanding the special casing for scalar coordinate variables in 
various contexts will cause significant issues for data consumers.

I am confident that scalar coordinate variable interpretation can be clearly 
defined in a manner consistent with the use of scalar coordinates in previous 
versions of the conventions.  I think that delivering a coherent definition of 
simple scalar coordinates for CF 1.7 will continue to support current usage and 
provide helpful clarification for future usage.

Examples 5.11, H4, H5 and H9 all use scalar coordinate variables in a way which 
I feel can be neatly supported by a simple definition of the scalar coordinate 
variable, distinct from vector coordinates.

Some more examples will help to clarify this for users of the conventions.

mark
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series & lat/lon coordinates

[John Caron]

Hi Jonathan:

If we use the time series featureType as example

(from 
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8307552)

AFAIU, the orthogonal multidimensional representation would be:

  float humidity(station,time)

not 

  float humidity(lat, lon, time)


Regards,
John


On 6/4/2013 11:33 AM, Jonathan Gregory wrote:
> Dear John C
> 
> I think the two questions are linked actually. There is nothing wrong with
> the size-1 dimensions in general, I would say. You could see it as a single
> point extracted from a larger gridded field (time,atl,lat,lon) in which all
> the dimensions were originally greater than one. It's legal in the orthogonal
> multidimensional representation, which is the one representation which has
> always existed in CF. When we added ch9, we brought that representation into
> the same chapter.
> 
> If the CDM doesn't like the orthogonal multidimensional representation for
> DSG, we could disallow featureType if that representation is used. Then the
> CDM wouldn't attempt to process it as a DSG. It would be a pity to lose the
> extra metadata that the featureType provides, though.
> 
> Best wishes
> 
> Jonathan
> 
> 
> - Forwarded message from John Caron  -
> 
>> Date: Tue, 4 Jun 2013 10:54:45 -0600
>> From: John Caron 
>> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509
>>  Thunderbird/17.0.6
>> To: cf-metadata@cgd.ucar.edu
>> Subject: Re: [CF-metadata] CF-1.6 DSG clarification: time series & lat/lon
>>  coordinates
>>
>> Hi Jonathan:
>>
>> On 6/4/2013 4:17 AM, Jonathan Gregory wrote:
>>> Dear John Caron and John Maurer
>>>
>>> I agree with John C that the problem arises when the coordinate variables
>>> are not size one. John M's example
>>>
>>>>>float lon(lon) ;
>>>>>float lat(lat) ;
>>>>>float alt(alt) ;
>>>>>float temp(time, alt, lat, lon) ;
>>>>>temp:standard_name = ?air_temperature? ;
>>>>>temp:units = "Celsius" ;
>>>>>temp:coordinates = "time lat lon alt" ;
>>>>>temp:_FillValue = -999.9;
>>>
>>> with alt=lat=lon=1 is legal in CF. In fact the coordinates attribute is not
>>> needed, because these are all (Unidata) coordinate variables (1D, with name
>>> of the variable equal to the name of the dimension). Ignoring the 
>>> coordinates
>>> attribute, this example is fine in COARDS as well. In the data model, lon 
>>> lat
>>> alt time are all dimension coordinate constructs with separate domain axes.
>>>
>>> But when there are *two* timeseries, you would not have alt=lat=lon=2. That
>>> would mean three independent size-2 dimensions. This would also be legal in
>>> CF and COARDS, but it means timeseries at a grid of 8 points, not two 
>>> points.
>>> To deal with this situation, we introduce an index dimension of size 2, and
>>> make alt lat lon auxiliary coordinate variables of this single dimension. In
>>> the data model, there is then only one domain axis for alt lat lon.
>>>
>>> Back to the case of one timeseries: Example H4 shows scalar coordinate
>>> variables for alt lat lon. That is, these size-1 dimensions have been 
>>> omitted.
>>> In this case, the coordinates attribute is needed; that's how scalar 
>>> coordinate
>>> variables are attached to the data variable. In the data model (in my 
>>> opinion)
>>> this is logically equivalent including the size-1 dimensions.
>>
>> Lets see, its currently not legal as a DSG file, according to the
>> spec. The CDM will barf on it, though I could put in a workaround.
>>
>> Should it be legal? That is, should people be "allowed" to put in
>> extraneous dimensions that only make sense for some values of the
>> dimension length (eg 1 but not >1) ? I think it would be a rather
>> ugly wart, and you would gain no new functionality.
>>
>> I also think it would confuse dimensions with coordinates (which has
>> already happened because the distinction isnt obvious). I think we
>> should try to be clear about the use of dimensions because it makes
>> the data model more powerful. So I would prefer not.
>>
>>>
>>> Maybe this question raises an issue for chapter 9 and Example H4. The 
>>> example
>>> is following Section 9.2:
>>>
>>> "If there is only a single feature t

Re: [CF-metadata] CF-1.6 DSG clarification: time series & lat/lon coordinates

[John Caron]

Hi Jonathan:

On 6/4/2013 4:17 AM, Jonathan Gregory wrote:

Dear John Caron and John Maurer

I agree with John C that the problem arises when the coordinate variables
are not size one. John M's example


   float lon(lon) ;
   float lat(lat) ;
   float alt(alt) ;
   float temp(time, alt, lat, lon) ;
   temp:standard_name = ?air_temperature? ;
   temp:units = "Celsius" ;
   temp:coordinates = "time lat lon alt" ;
   temp:_FillValue = -999.9;


with alt=lat=lon=1 is legal in CF. In fact the coordinates attribute is not
needed, because these are all (Unidata) coordinate variables (1D, with name
of the variable equal to the name of the dimension). Ignoring the coordinates
attribute, this example is fine in COARDS as well. In the data model, lon lat
alt time are all dimension coordinate constructs with separate domain axes.

But when there are *two* timeseries, you would not have alt=lat=lon=2. That
would mean three independent size-2 dimensions. This would also be legal in
CF and COARDS, but it means timeseries at a grid of 8 points, not two points.
To deal with this situation, we introduce an index dimension of size 2, and
make alt lat lon auxiliary coordinate variables of this single dimension. In
the data model, there is then only one domain axis for alt lat lon.

Back to the case of one timeseries: Example H4 shows scalar coordinate
variables for alt lat lon. That is, these size-1 dimensions have been omitted.
In this case, the coordinates attribute is needed; that's how scalar coordinate
variables are attached to the data variable. In the data model (in my opinion)
this is logically equivalent including the size-1 dimensions.


Lets see, its currently not legal as a DSG file, according to the spec. 
The CDM will barf on it, though I could put in a workaround.


Should it be legal? That is, should people be "allowed" to put in 
extraneous dimensions that only make sense for some values of the 
dimension length (eg 1 but not >1) ? I think it would be a rather ugly 
wart, and you would gain no new functionality.


I also think it would confuse dimensions with coordinates (which has 
already happened because the distinction isnt obvious). I think we 
should try to be clear about the use of dimensions because it makes the 
data model more powerful. So I would prefer not.




Maybe this question raises an issue for chapter 9 and Example H4. The example
is following Section 9.2:

"If there is only a single feature to be stored in a data variable, there is no
need for an instance dimension and it is permitted to omit it. The data will
then be one-dimensional, which is a special (degenerate) case of the
multidimensional array representation.  The instance variables will be scalar
coordinate variables; the data variable and other auxiliary coordinate
variables will have only an element dimension and not have an instance
dimension, e.g. data(o) and t(o) for a single timeSeries."

In the multidimensional array representation, featureType doesn't have to be
coded, because this representation has always existed in CF. We could say that
*if* you encode featureType, there *must* be an instance dimension (of size 1
if appropriate) and that alt lat lon must be auxiliary coordinate variables
with this dimension. That would be a restriction we don't have in CF 1.6, so
it would be a change to CF. What do you think, John C?


I think this is a seperate question yes?

From my POV, its just a practical question to make things clear between 
data producer and consumer. I think we allowed the scalar instance 
coordinates because it was a natural way for producers to think when 
there was only one feature in the file, ie "why do you want me to add 
this extra dimension?" As long as the "featureType" attribute is 
present, as well as the "coordinates" attribute I think the meaning is 
unambiguous. Requiring a dimension=1 is maybe a bit simpler, but i would 
still have to deal with the scalar case for versions before we change, 
so its not really better for me.


John Caron
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG clarification: time series & lat/lon coordinates

[John Caron]

Hi John:

1) The coordinates attribute is not new to DSG, it has been around since 
the beginning of CF. This defines the "auxiliary coordinate variables", 
that is, coordinate variables that do not follow "lat(lat)" template. I 
think it would be good if Grads et al could be upgraded to use them.


2) DSG data is 1 dimensional. The addition of extra dimensions of length 
1 seems harmless when you know you can ignore them, but is wrong when 
you want dimensions to represent independent coordinates of the 
sampling. To see that, ask yourself what the data variables would look 
like if the dimension had length 2.


  float humidity(sample) ->  float humidity(sample, alt, lat, lon);

  float humidity(sample, 1, 1, 1); // seems ok
  float humidity(sample, 2, 2, 2); // huh??

it looks like a grid, and its not. there are not 2 x 2 x 2 independent 
coordinates. point data is not degenerate grid data. it has a different 
connectedness between the indices, and its this connectedness that makes 
multidimensional arrays much faster than, say, an RDBM, so throw that 
away at your peril.


Im interested in hearing what Jonathan et al's data model would say 
about this. I can tell you that the CDM/TDS feature collection layer 
would break if you added the extraneous dimensions.


John


On 5/29/2013 12:35 PM, John Maurer wrote:

Hi all,
We ran into a glitch after converting one of our buoys to the new CF-1.6
discrete sampling geometries (DSG) format, and I'm looking for advice.
This dataset uses the single time series format, like the one provided
in the template of the CF document in Example H.4:
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8320208.
The problem we discovered is that the dataset is no longer readable in
software clients like GrADS and Dapper/DChart (likely others) because it
can't find the latitude and longitude coordinates anymore.

While the dataset previously included the lat/lon dimensions of the buoy
in NetCDF coordinate variables, CF-1.6 DSG no longer defines lat/lon as
coordinate variables. Instead, they are scalar variables which are
referenced by other variables (temp, salinity, etc.) via a new
variable-level "coordinates" attribute: e.g.

temp:coordinates = "time lat lon alt" ;

In order to allow the DSG datasets to continue working in existing
software clients that do not know yet to look for this new "coordinates"
variable attribute, does it break CF-1.6 DSG compliance to *also* change
the lat/lon scalars to coordinate variables like they used to operate?
Building on Example H.4, the modified dataset would look like this,
which is a melding of the old and new ways:

dimensions:
   time = 100233 ;
   name_strlen = 23 ;
   alt = 1 ;
   lat = 1 ;
   lon = 1 ;
variables:
   float lon(lon) ;
   lon:standard_name = "longitude";
   lon:long_name = "station longitude";
   lon:units = "degrees_east";
   float lat(lat) ;
   lat:standard_name = "latitude";
   lat:long_name = "station latitude" ;
   lat:units = "degrees_north" ;
   float alt(alt) ;
   alt:long_name = "vertical distance above the surface" ;
   alt:standard_name = "height" ;
   alt:units = "m";
   alt:positive = "up";
   alt:axis = "Z";
   char station_name(name_strlen) ;
   station_name:long_name = "station name" ;
   station_name:cf_role = "timeseries_id";

   double time(time) ;
   time:standard_name = "time";
   time:long_name = "time of measurement" ;
   time:units = "days since 1970-01-01 00:00:00" ;
   time:missing_value = -999.9;
   float humidity(time, alt, lat, lon) ;
   humidity:standard_name = “specific_humidity” ;
   humidity:coordinates = "time lat lon alt" ;
   humidity:_FillValue = -999.9;
   float temp(time, alt, lat, lon) ;
   temp:standard_name = “air_temperature” ;
   temp:units = "Celsius" ;
   temp:coordinates = "time lat lon alt" ;
   temp:_FillValue = -999.9;

attributes:
   :featureType = "timeSeries";



Thanks for any insights on this issue!
Cheers,
John Maurer
Pacific Islands Ocean Observing System (PacIOOS)
University of Hawaii at Manoa




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 DSG Trajectory files: clarification and advice

[John Caron]

Hi Derrick:

Some first reactions:

1. "the aggregation or collection will have two trajectories, the first 
is along a time axis "time" and the second is along a time axis "time_avg".


since they are both trajectories, then i think you dont have a problem 
with more than one feature type.


however, theres no convention which says how the two trajectories are 
related. they might be though of as being along the same trajectory, 
with different sampling points. this would be useful for CF coordinate 
systems to capture in a general way.


2. The THREDDS server is implementing with "feature collections". You 
have an interesting case here of creating essentially 2 feature 
collections, each collection in multiple files, but the 2 collections 
share the same files.


Maybe I can get some sample files and see how hard it would be to extend 
the TDS.


3. "is it legal to have a variable with a valid geometry AND a variable 
that is the result of a cell_method applied to that variable in the same 
file?"


i think its legal, but the question is, does any standard software able 
to understand it? Generally, from my POV, a main use of conventions is 
to allow software to understand these relationships.


John

On 5/21/2013 11:11 AM, Derrick Snowden - NOAA Federal wrote:

All,

We are attempting to create a valid CF 1.6 discrete sampling geometry
trajectory file to exchange data from underwater profiling gliders.
There is a schematic of the major sampling features for gliders at the
following web site
(https://github.com/IOOSProfilingGliders/Real-Time-File-Format/wiki/Glider-Types-and-Sampling#sampling-patterns).
There are a few variations of the files we are working on in the
following github repo
(https://github.com/IOOSProfilingGliders/Real-Time-File-Format/tree/master/examples/proposed_templates/trajectory_uv),
and"glider_trajectory_uv_template_v.0.0.cdl
"
is file in question.

We are grappling with the constraint in CF that each file must contain
one and only one feature type.  (Section 9.1 para 1:"/the features
contained within a collection must always be of the same type; and all
the collections in a CF file must be of the same feature type/." )  As
it is written, this constraint doesn't say anything about the shape or
dimensionality of the features, only that they are the same type.  Below
I'll try to describe our use case which differs from the examples in
chapter 9 in two ways.
1. The files only contain data from a single glider so the trajectory
variable is always a scalar (i.e. the variable called trajectory with
trajectory:cf_role=trajectory_id is usually just set to one in these tests).
2. The file only contains part of a complete deployment and we need to
aggregate them together into a complete deployment (one deployment may
contain hundreds or thousands of segments) but, each of these segments
still has trajectory=1.

As stated, our files will contain one "segment" as defined on the web
site above.  A segment is defined as the data collected between two
surface gps fixes of the glider.  When the glider submerges it is
collecting data (temp/sal etc) as a function of time.  But, underwater
there are no gps fixes so the lat/lon is estimated with a dead reckoning
algorithm.  So far we think we've encoded that correctly.  Our problem
is with data that is estimated as an average between gps fixes.

For example, when the glider surfaces at the end of a segment there are
two position fixes available (1: the gps position which is considered
true, and 2: the dead reckoning position which is estimated) The
difference between these two positions can be used to estimate the
depth/time averaged currents that pushed the glider off course during
it's trajectory underwater.  We'd like to encode these "depth averaged"
currents in the same file as the measured T/S etc but this data has a
different time axis and therefore a different lat/lon coordinate.  In
practice the depth averaged currents are assigned to the midpoint (time
and space) of the subsurface trajectory.  Our goal is for aggregations
of the segment files to represent an entire glider deployment and if
we're successful the aggregation or collection will have two
trajectories, the first is along a time axis "time" and the second is
along a time axis "time_avg".  They are both trajectories but they have
different time axes.  I'm not sure if THREDDS will allow this type of
aggregation so some changes may be necessary.

Does this violate the intent of the convention?  If so, can anyone
suggest a better encoding?  We're trying to keep these two data sets in
the same file but creating two files is possible if not desirable.

A similar related question could be, is it legal to have a variable with
a valid geometry AND a variable that is the result of a cell_method
applied to that variable in the same f

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi Karl:

Do you know of any specific "quick look" software that wouldnt work?

John

On 3/21/2013 10:04 AM, Karl Taylor wrote:

Hi John,

I'm probably repeating what others have said, but allowing strings as
actual (rather than ancillary) coordinate values would break lots of
"quick look" software which can't plot variables which are "functions"
of a string representation of the independent variable (i.e, time).   If
I'm correct about this, I think making the proposed change to CF would
be a mistake.

If the alternate string representation is stored as an ancillary
variable, a human reading the ncdump would immediately see what the
"dates" are, and the software would continue to work by reading the
actual coordinate values.

Karl

On 3/21/13 8:51 AM, John Caron wrote:

On 3/21/2013 9:41 AM, Steve Hankin wrote:


On 3/21/2013 8:25 AM, John Caron wrote:

Probably my proposal comes down to 2 parts, which are separable:

1. Find a suitable replacement for udunits as a reference library
for CF calendar dates. Unfortunately, udunits used a slightly
non-standard syntax, which we need to still be legal for backwards
compatibility. So its unlikely a standard library would work off the
shelf. Someone needs to commit to making a library that supports CF
as it evolves. Other libraries can test against it.

2. *Allow*


Hi John,

The crux of this debate has been whether to "allow" a new (for CF),
redundant encoding of time coordinates.  I *think* you are suggesting
in your bullet #2 to allow the new syntax only in ancillary variables
(metadata), but not as a representation of coordinates.  Can you clarify?

- Steve


a suitable string syntax for date/times, probably expressed as a
profile of ISO8601. Its a modest but worthwhile improvement IMO. The
syntax that is already used in udunits would be ok with me. However
this syntax has to be defined (it never was, unless you are yacc).
In my understanding, udunits is more or less a superset of ISO. that
is, its a bit more lenient in the syntax. However it adds a few
restrictions, like defaulting to UTC and filling non specified
fields with 0. So its not strictly a superset. The advantage of
using CF process to define the syntax ("clarify the profile of
ISO8601") is that we are less likely to miss some subtlety in syntax
and meaning.

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




Hi Steve:

Im proposing that time coordinates may be expressed as strings, in
addition to the current way of using time offsets from a base date
expressed as a string in the unit attribute. The two date string
syntaxes (standalone and in the unit attribute)  c/should be the same.

Presumably we want to do either this or Aleksandar's proposal ?

John


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

On 3/21/2013 11:17 AM, Chris Barker - NOAA Federal wrote:

On Tue, Mar 19, 2013 at 5:21 PM, Dave Allured - NOAA Affiliate
 wrote:


  You are making a set of technical
use specifications, with significant departures from the reference
standard ISO 8601.


If we do anything with this -- PLEASE, PLEASE, PLEASE simply use the
ISO standard! I"ve a bit lost track of where folks are proposing to
deviate, but what I have seen ( a different default time zone) seems
to be a case of  "my particular application currently uses this
standard, so I want CF to match that", which is a really bad idea.

If we really need to deviate, I'd say only do it by supporting a
subset of ISO (like requiring a TZ spec, for instance), but having the
exact same string mean something different in CF than ISO strikes me
as a really bad idea!


Hi all:

Ive always just worked with the "W3C profile of ISO8601"

  http://www.w3.org/TR/NOTE-datetime

So theres the question of supporting full ISO8601, or just a profile.

In fact, I really just rely on specialized libraries to deal with the 
subtleties. Unidata is not very interested in spending resources on a 
problem that has already general solutions but is complex in the 
corners. Its mostly the non-standard calendars to support models that 
prevents adopting general libraries.


If someone knows what the "departures from the reference standard ISO 
8601" that CF has already made, please post.


the only cases i know of where this might be true is:

  1) default timezone
  2) filling in non-specified fields with zeroes.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

On 3/21/2013 9:41 AM, Steve Hankin wrote:


On 3/21/2013 8:25 AM, John Caron wrote:

Probably my proposal comes down to 2 parts, which are separable:

1. Find a suitable replacement for udunits as a reference library for 
CF calendar dates. Unfortunately, udunits used a slightly 
non-standard syntax, which we need to still be legal for backwards 
compatibility. So its unlikely a standard library would work off the 
shelf. Someone needs to commit to making a library that supports CF 
as it evolves. Other libraries can test against it.


2. *Allow* 


Hi John,

The crux of this debate has been whether to "allow" a new (for CF), 
redundant encoding of time coordinates.  I *think* you are suggesting 
in your bullet #2 to allow the new syntax only in ancillary variables 
(metadata), but not as a representation of coordinates.  Can you clarify?


- Steve

a suitable string syntax for date/times, probably expressed as a 
profile of ISO8601. Its a modest but worthwhile improvement IMO. The 
syntax that is already used in udunits would be ok with me. However 
this syntax has to be defined (it never was, unless you are yacc). In 
my understanding, udunits is more or less a superset of ISO. that is, 
its a bit more lenient in the syntax. However it adds a few 
restrictions, like defaulting to UTC and filling non specified fields 
with 0. So its not strictly a superset. The advantage of using CF 
process to define the syntax ("clarify the profile of ISO8601") is 
that we are less likely to miss some subtlety in syntax and meaning.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




Hi Steve:

Im proposing that time coordinates may be expressed as strings, in 
addition to the current way of using time offsets from a base date 
expressed as a string in the unit attribute. The two date string 
syntaxes (standalone and in the unit attribute)  c/should be the same.


Presumably we want to do either this or Aleksandar's proposal ?

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Probably my proposal comes down to 2 parts, which are separable:

1. Find a suitable replacement for udunits as a reference library for CF 
calendar dates. Unfortunately, udunits used a slightly non-standard 
syntax, which we need to still be legal for backwards compatibility. So 
its unlikely a standard library would work off the shelf. Someone needs 
to commit to making a library that supports CF as it evolves. Other 
libraries can test against it.


2. Allow a suitable string syntax for date/times, probably expressed as 
a profile of ISO8601. Its a modest but worthwhile improvement IMO. The 
syntax that is already used in udunits would be ok with me. However this 
syntax has to be defined (it never was, unless you are yacc). In my 
understanding, udunits is more or less a superset of ISO. that is, its a 
bit more lenient in the syntax. However it adds a few restrictions, like 
defaulting to UTC and filling non specified fields with 0. So its not 
strictly a superset. The advantage of using CF process to define the 
syntax ("clarify the profile of ISO8601") is that we are less likely to 
miss some subtlety in syntax and meaning.


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

On 3/20/2013 9:41 AM, David Hassell wrote:

Hello,

My beer/coffee/perocet levels are too low to want to comment broadly on
this, so I'll just make one comment ...


Really the main advantage is that data writers are less likely to
make a mistake specifying

1952-08-15T00:00:00Z

than

2434567 days since -4713-01-01T00:00:00Z.

I'm not sure why the later would be any more prone to mistakes than
the former. Surely the conversion of

   

 to

   '2434567 days since -4713-01-01T00:00:00Z'

would be handled by a software library, just as would the conversion
of

   

 to

   '1952-08-15T00:00:00Z'

and its fair to assume, I think, that such libraries will do the right
thing.


I guess the point is that its not always fair to assume that, and the 
user wont know when it fails, esp for


'2434567 days since -4713-01-01T00:00:00Z'

unless she also computes

'1952-08-15T00:00:00Z'

which presumably she could do as a double check.


Which library would you use? Does it work for non-standard calendars?

How about if everyone who'g got nothing better to do try it out in the 
library of their choice and report the results?


John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi all:

I guess I started this messy discussion, then invited everyone to drink 
too much and say whatever they wanted. The conversation gets a bit loud 
and fuzzy. So now we've switched back to caffeine and the sober 
realities of deciding on grammars in the context of a specific proposal 
that seems to be acceptable to all.


Since there were a lot of "whats the problem" comments, I will just give 
my motivation for this topic.


There are two things you want to do with datetimes: 1) specify an 
instance on the datetime coordinate axis. 2) compute the time difference 
between two instances (a calendar is needed to compute time differences).


Our current "period since date" representation is better for the second 
case. A direct representation of the time instant (like ISO8601) is 
better for the first case. I suggest we allow both, as a modest improvement.


Mostly my POV is a library implementer, where the following issues are 
in play:


  1. It turned out to be a mistake to rely on udunit library for 
datetime processing. I think we no longer have an official reference 
library for this (?).


  2. Non-standard calendars are non-trivial.

  3. Theres little or no extra work to implement, because we already 
have to parse the date string in "period since date" representation.


  4. The time instant representation doesnt solve the non-standard 
calendar, since you will still want to be able to compute time differences.


Really the main advantage is that data writers are less likely to make a 
mistake specifying


1952-08-15T00:00:00Z

than

2434567 days since -4713-01-01T00:00:00Z.

and you shift the burden of calculating time differences onto the software.

Note that Im just advocating for adding this as an option, not replacing 
the current standard.


John

PS: Jonathan asks: "how can we reduce the possibility of these 
mistakes"? We need a reference library for datetime that reads CF files 
and calculates both instances and time differences, and spits them out 
in user-configurable ways so that data writers can double check that 
their files are written correctly. It has to handle all the non-standard 
calendars that are in CF.





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi all:

Another thing to consider is the relationship to the current udunit/CF 
standard for specifying dates in the unit string


  "period SINCE date"

The udunits documentation 
 
not being very clear, I wrote up my understanding of current practice here:


http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html#ISO_String

Note the paragraph towards the end starting with "with the following 
differences, to allow backwards compatibility with udunits:"


as this documents the differences AFAICT that current practice has with 
ISO 8601 standard.


It appears that this grammer is slightly more lenient than the one that 
Aleksandar proposes.


John



On 3/19/2013 6:21 PM, Dave Allured - NOAA Affiliate wrote:

Aleksandar,

I support this standard name proposal.  I like the restrictions on
syntax and use to a sensible subset for scientific applications.

Unlike most standard name proposals, this is more than a simple
definition of a physical meaning.  You are making a set of technical
use specifications, with significant departures from the reference
standard ISO 8601.  Will there be a reference archive for the
particular specifications?  Will this reside exclusively in the CF
standard name table, or in a new CF chapter or appendix, or something?
  I think there needs to be something better than "that CF list
discussion of January to March 2013".

Please be exact in the specification of the reference calendar.  Try
to eliminate all possible measurement ambiguities.  Leap seconds are a
problem.  There are no leap seconds in the standard calendars
currently defined by CF, and I was not able to locate exactly where
ISO8601 stands on this issue.  I would really hate to see a "Gregorian
calendar" or proleptic_gregorian implementation with interval
calculations that differ from the CF defined calendars.

I suggest a requirement that the calendar attribute be included on all
variables using datetime_iso8601.  If leap seconds are excluded, then
the correct attribute value should be "proleptic_gregorian".

You have "four-digit years".  Are you explicitly restricting the spec
to only years  through ?

Thank you for working on this!

--Dave

On Fri, Jan 11, 2013 at 10:00 AM, Aleksandar Jelenak - NOAA Affiliate
 wrote:

Dear All:

Here's the modified proposal for the datetime_iso8601 standard name:

standard_name: datetime_iso8601

Units: N/A

String representing date-time information according to the ISO
8601:2004(E) standard. Variables with this standard name cannot serve
as coordinate variables. Date-time information is in the Gregorian
calendar. For dates preceding the Gregorian calendar the date-time
information is in the proleptic Gregorian calendar. Possible date-time
string forms are:

 =  "T"   ;

 =  "-" MM "-" DD |  "-" DDD ;

 = hh | hh ":" mm | hh ":" mm ":" ss | hh ":" mm ":" ss "." S
   | hh ":" mm ":" ss "," S ;

 = "" | "Z" | "+" hh | "+" hh ":" mm | "-" hh | "-" hh ":" mm

Where:

* "" is a four-digit year (-).

* "MM" is a two-digit month of the year (01-12).

* "DD" is a two-digit day of the month (01-31).

* "DDD" is a three-digit ordinal day of the year (001-366).

* "hh" is a two-digit hour (00-23).

* "mm" is a two-digit minute (00-59)

* "ss" is a two-digit second (00-59).

* "S" is one or more digits representing a decimal fraction of the
  second.

* The value of any designator when not specified is zero.

* If  is ommitted the default value is "Z".


-Aleksandar

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi All:

Ok, its friday afternoon so ill bite on this, and wax philosophical even 
before the beer.


An insidious mistake is to think that problems should be fixed in 
software libraries. Actually the deep mistake is to mistake the 
reference software with the file encoding. Why bother fixing the 
encoding when a few lines in _your_ software can fix the problem 
transparently? Ive seen this occur in all three of the great western 
religious systems of our day: netCDF, HDF and OPeNDAP libraries.


Better is to do the encoding of information as cleanly as possible. 
Post-apocalyptic software engineers who have lost all knowledge of what 
netCDF and CF mean and are painstakingly uncovering climate archives 
with their whisk brooms will thank us.


"35246 hours since 1970-01-01" isnt just unreadable; it uses a calendar 
system that may be non-trivial. Calendars are hard; Java has got it 
wrong already twice, and is now trying for a 3rd time (with jsr 310 in 
Java 8, based on experience with joda-time).


"1974-01-08T14:00:00Z" ( == "35246 hours since 1970-01-01" in the 
standard calendar) is a better representation of that date. because at 
least you know what the user thought the damn date was.


The good argument for "35246 hours since 1970-01-01" representation, is 
that given two of them, at least you know what the user thought the damn 
time interval is between them.


Anyway, I think both are good, and should be allowed. Finish your beer 
and ill order another round.


John

PS: NEMO files use time units of "days since -4713-01-01T00:00:00Z". 
Dont know why, and no disrespect intended, but I cant say I like it.



On 3/15/2013 4:30 AM, Hattersley, Richard wrote:

Hi all,
I think Steve has got to the heart of the issue with his comment, "IMHO
it is the client libraries that hold the answer to this question."
However you choose to peer into your netCDF files you are seeing them
through the lens of a "client library". And it's worth noting that
date/times aren't special in this regard, this is just as true for
floating point numbers.
With the Iris library(*), we are working towards improving the
readability and usability of date/times because we recognise that simply
displaying "35246 hours since 1970-01-01" is essentially meaningless to
a human. From what I've seen, I think we can all agree on that! So we
want our users to be able to see date/times in a human-readable form,
but that certainly doesn't mean we will represent them that way internally.
The only real benefit I can see for the string representation is that
_some_ date/time values can be made more human-readable when viewed
through client libraries which don't support the
machine-readable-to-human-readable conversion. But if the library can't
do machine-to-human then it probably can't do human-to-machine. In which
case there's very little you can actually _do_ with the date/time values
(e.g. determine ordering or compute intervals). If the library is that
limited then adding the string representation to CF isn't really fixing
the right problem. If you'll excuse the analogy, it's like taking the
engine out of a car because the brakes don't work.
In short, I am against the addition of a string representation for
date/times.
*) Iris - a candidate reference implementation of CF implemented in
Python (http://scitools.org.uk/).
Richard Hattersley Iris Benevolent Dictator
*Met Office*  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Tel: +44 (0)1392 885702
Email: richard.hatters...@metoffice.gov.uk
<mailto:richard.hatters...@metoffice.gov.uk>  Web: www.metoffice.gov.uk
<http://www.metoffice.gov.uk/>

----
*From:* CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] *On Behalf
Of *Steve Hankin
*Sent:* 24 February 2013 19:07
*To:* John Caron
*Cc:* cf-metadata@cgd.ucar.edu
*Subject:* Re: [CF-metadata] New standard name: datetime_iso8601


On 2/23/2013 1:41 PM, John Caron wrote:

Hi Chris, and all:

On 1/11/2013 2:37 PM, Chris Barker - NOAA Federal wrote:

On Fri, Jan 11, 2013 at 9:00 AM, Aleksandar Jelenak - NOAA Affiliate
 wrote:


Here's the modified proposal for the datetime_iso8601 standard name:

...

String representing date-time information according to the ISO
8601:2004(E) standard.


I think we should NOT adopt a string option for datetime variables.

To quote Jonathan Gregory:

"""
In CF we have always applied the
principle that we only add to CF when there is a need to do so, i.e.
there is
a use-case for something which cannot already be represented in CF
"""

We already have a way to encode datetimes in CF-netcdf.


Yes, but  is not as good as  as an encoding.
The  is a result of cramming calendar handling into a
units package.

I would advocate both should be allowed.


Hi John,

Re: [CF-metadata] brand new user

[John Caron]

On 3/14/2013 6:39 AM, Izidor Pelajic wrote:

Hi all,
I'm new at netCDF, new at CF, and moderate MATLAB user. I'm new at
netCDF in a sense of days or week.
So my problem (beyond the fact that I do not have general knowledge)
is next. I was provided with a .nc file. In it there are 3 information
of interest: latitude matrix 160 by 22, longitude matrix 160 by 22 and
SST matrix also 160 by 22. File is created from NOAA satellite data
record (from hdf file, but not by me). I need to convert this custom
made nc file to netCDF CF file.
So far, I managed to open nc file in matlab, edit variables, make some
nice charts and save it as it is. But I do not know how to save in CF
convention. My problem is longitude and latitude that are of same
dimension as variable SST. How do I make sst( lon, lat ) if all lon
and lat elements are of different value? (it was written down in line
as satellite passed over an area)

Any idea, or example?
Izidor
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


Hi Izidor:

i presume that you need to  compute the 2D lat and lon coordinates and 
add them to the file.

If you think of renaming your dimensions like this, perhaps it is easier:

dimension
  scan = 160;
  xscan = 22

variables:
  float sst(scan,xscan);
sst:coordinate = "lat lon";

  float lat(scan,xscan);
lat:units= "degrees_north";

  float lon(scan,xscan);
lat:units= "degrees_east";


the

  sst:coordinate = "lat lon";

means add an attribute to the sst variable with name "coordinate" and 
value "lat lon". This is how CF associates the lat and lon coordinates 
to the data variable. The "units" attribute on lat and lon is how CF 
knows what they are (i.e. not by the variable name, which can be anything).


this is just the minimum, of course you might want to add standard_name 
attribute, etc.


Good luck!

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Seth:

Thanks for the description, its really helpful.

Im just wondering how we document this in the CF convention, so that 
implementers have something to check against. Is there a "reference 
implementation" we can use ?


John

On 3/6/2013 1:17 PM, Seth McGinnis wrote:

So there are an infinite number of ways to decompose a 3-D orientation into
three orthogonal rotations.  Specifying the projection in terms of the final
location of the north pole and an optional rotation around that point means
that CF is actually agnostic about exactly how you get there.

That said, given that it's being specified in that way, the decomposition that
maps most straightforwardly to those parameters is a Z-Y-X rotation sequence
using an extrinsic (space-fixed) coordinate system:

First rotate by angle phi around the north pole in space-fixed coordinates.
Then rotate by angle theta around 0N 90E in space-fixed coordinates.
Finally, rotate by angle psi around 0N 0E in space-fixed coordinates.

The location of the grid's north pole in the original lat-lon coordinate system
is now at longitude = phi, colatitude = minus theta.

(I'm not sure how psi relates to north_pole_grid_longitude mathematically; it
defaults to zero and I've never seen it in use, so I think it's included mostly
for completeness. I agree that we may well want to change it to something more
obvious like an azimuth angle.)

Switching to the GRIB2 convention where you use the south pole instead of the
north should be just a matter of adding 180 degrees to phi and negating theta.

Cheers,

--Seth


On Wed, 6 Mar 2013 18:29:47 +
  David Hassell  wrote:

Hello,

It is a bit of mess! As I understand it, the full rotation described
is a sequence of rotations about three different axes:

   Z  = 90S - 90N
   Y' = 90W' - 90E'
   X" = 0E" - 180E"

where it is understood that the definitions of the N-S and W-E axes
change after each rotation (hence the primes and double
primes). Therefore the order in which they are done matters.

I suspect that the usual and assumed order is Z, Y', X"?

>From the GRIB-2 stuff John posted the north_pole_grid_longitude gives
the rotation about the X" axis.

Something like John's "Angle of rotation" seems right to me.

The apparent lack of consistency between the parameter names irks
me. Perhaps one solution could be to:

i) Add some text to the conventions state the order (Z, Y', X", say)
   and direction of rotations.

ii) add three new, consistent, self-describing parameters (e.g.)

angle_of_rotation_z
angle_of_rotation_y
angle_of_rotation_x

iii) allow aliases for backwards compatibility

grid_north_pole_longitude <=> angle_of_rotation_z
grid_north_pole_latitude  <=> angle_of_rotation_y
north_pole_grid_longitude <=> angle_of_rotation_x

Or would that just obfuscate things even more?

All the best,

David

P.S. If you have a copy available, there are some nice descriptions in
 "Coordinate Systems and Map Projections" by D. H. Maling

--
David Hassell
National Centre for Atmospheric Science (NCAS)
Department of Meteorology, University of Reading,
Earley Gate, PO Box 243,
Reading RG6 6BB, U.K.

Tel   : 0118 3785613
Fax   : 0118 3788316
E-mail: d.c.hass...@reading.ac.uk
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Burkhardt:

Of course we cant change the existing conventions as there are files 
that already use it. But we need to document it so that we know when we 
are doing it right.


Im cleaning up my code and noticed that i have two implementations, one 
for grib and one for CF. So Id like to merge the two.


John

On 3/6/2013 12:32 PM, Burkhardt Rockel wrote:

John,

the rotated pole definition was requested by me in 2002 (see the thread 
"projections/default cell_methods"). In this thread the present description was 
chosen.
Please note: the rotated pole grid mapping is NOT a PROJECTION!  it is a 
TRANSFORMATION (therefore the GRIB description is misleading). This is 
probably the reason why you did not find it in Proj4.
The a, b, c in the GRIB definition description is fine. Whether  the rotated 
South Pole (grib) or North Pole (CF-Conventions) is used in the definition does 
not matter, the values can be easily converted to each other.

The rotated pole grid mapping as defined in the CF-Conventions document is 
already used widely as netCDF output standard for regional climate models in 
several international projects. It is also the standard in the IPCC-AR5 CORDEX 
model output for regional climate models. Therefore I am not in favor of 
changing the definition. However, a more precise description in the 
CF-Conventions would certainly be helpful for those not familiar with the 
rotated pole transformation.

Regards
Burkhardt

---
Dr. Burkhardt Rockel
Helmholtz-Zentrum Geesthacht
Institute of Coastal Research / Group Regional Atmospheric Modeling
Max-Planck-Strasse 1
D-21502 Geesthacht
Germany
Phone: +49 4152 87 1803
Fax: +49 4152 87 4 1803
Email: Burkhardt.Rockel (at) hzg.de
www: http://rockel.staff.coast.hzg.de
coordinates: 53.40579 N, 10.428647 E
---


Am 06.03.2013 um 17:09 schrieb John Caron :


Hi Heiko:

So

grid_north_pole_longitude = 
normalizeLongitude180(longitudeOfSouthernPoleInDegrees)
grid_north_pole_latitude = -1 * longitudeOfSouthernPoleInDegrees

?

where does one find documentation on proj4's ob_tran routine?

thanks,
John



On 3/6/2013 8:03 AM, Heiko Klein wrote:

Hi John,

all our models use rotated_latitute_longitude. And we have a consistent way of 
translating it between grib, proj and netcdf-java, i.e. publicly available at:
http://thredds.met.no/thredds/catalog/metno/proff4km/default/catalog.html


grid_mapping_name: rotated_latitude_longitude
grid_north_pole_longitude: 156.0
grid_north_pole_latitude: 23.5
earth_radius: 6367470.0

to proj:
proj4: +proj=ob_tran +o_proj=longlat +lon_0=-24 +o_lat_p=23.5 +a=6367470.0 
+no_defs

from grib:
lonRot = longitudeOfSouthernPoleInDegrees
latRot = latitudeOfSouthernPoleInDegrees

+proj=ob_tran +o_proj=longlat +lon_0=" << normalizeLongitude180(lonRot) << " 
+o_lat_p=" << (-1 * latRot);



This is the set of parameters which have been useful. I'm not sure if they are 
named in a useful way. proj4's ob_tran even allows for a o_lon_p parameter, but 
this doesn't seem to be used by grib or CF.

Best regards,

Heiko



On 2013-03-06 15:18, John Caron wrote:

The "Rotated Pole" projection here:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#appendix-grid-mappings


Rotated pole

grid_mapping_name = rotated_latitude_longitude


Map parameters:

  *

grid_north_pole_latitude

  *

grid_north_pole_longitude

  *

north_pole_grid_longitude - This parameter is option (default is 0).

Map coordinates:

The rotated latitude and longitude coordinates are identified by
the standard_name attribute values
grid_latitude and grid_longitude respectively.



is problematic. It has no reference to Proj.4, and is not in Snyder.

Its likely the same as the GRIB-2 GDS Template 3.1:

Grid Definition Template 3.1: Rotated Latitude/longitude (or
equidistant cylindrical, or Plate Carrée)

Octet No. Contents

   15-72 Same as Grid Definition Template 3.0 (see Note 1)

   73-76 Latitude of the southern pole of projection

77-80 Longitude of the southern pole of
projection

   81-84 Angle of rotation of projection

   85-nnList of number of points along
each meridian or parallel (These octets are only present for
quasi-regular grids as described in Note 3)

Notes:


(2) Three parameters define a general latitude/longitude
coordinate system, formed by a general rotation of the sphere. One
choice for these parameters is:

(a) The geographic latitude in degrees of the southern pole of
the coordinate system, θp for example.

(b) The geographic longitude in degrees of the southern pole of
the coordinate system, λp for example.

(c) The angle of rotation in degrees about the new polar axis
(measured clockwise when looking from the southern to the northern pole)
of the coordinate system, assuming t

Re: [CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

Hi Heiko:

So

grid_north_pole_longitude = 
normalizeLongitude180(longitudeOfSouthernPoleInDegrees)

grid_north_pole_latitude = -1 * longitudeOfSouthernPoleInDegrees

?

where does one find documentation on proj4's ob_tran routine?

thanks,
John



On 3/6/2013 8:03 AM, Heiko Klein wrote:

Hi John,

all our models use rotated_latitute_longitude. And we have a 
consistent way of translating it between grib, proj and netcdf-java, 
i.e. publicly available at:

http://thredds.met.no/thredds/catalog/metno/proff4km/default/catalog.html


grid_mapping_name: rotated_latitude_longitude
grid_north_pole_longitude: 156.0
grid_north_pole_latitude: 23.5
earth_radius: 6367470.0

to proj:
proj4: +proj=ob_tran +o_proj=longlat +lon_0=-24 +o_lat_p=23.5 
+a=6367470.0 +no_defs


from grib:
lonRot = longitudeOfSouthernPoleInDegrees
latRot = latitudeOfSouthernPoleInDegrees

+proj=ob_tran +o_proj=longlat +lon_0=" << 
normalizeLongitude180(lonRot) << " +o_lat_p=" << (-1 * latRot);




This is the set of parameters which have been useful. I'm not sure if 
they are named in a useful way. proj4's ob_tran even allows for a 
o_lon_p parameter, but this doesn't seem to be used by grib or CF.


Best regards,

Heiko



On 2013-03-06 15:18, John Caron wrote:

The "Rotated Pole" projection here:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#appendix-grid-mappings 




Rotated pole

grid_mapping_name = rotated_latitude_longitude


Map parameters:

  *

grid_north_pole_latitude

  *

grid_north_pole_longitude

  *

north_pole_grid_longitude - This parameter is option (default 
is 0).


Map coordinates:

The rotated latitude and longitude coordinates are identified by
the standard_name attribute values
grid_latitude and grid_longitude respectively.



is problematic. It has no reference to Proj.4, and is not in Snyder.

Its likely the same as the GRIB-2 GDS Template 3.1:

Grid Definition Template 3.1: Rotated Latitude/longitude (or
equidistant cylindrical, or Plate Carrée)

Octet No. Contents

   15-72 Same as Grid Definition Template 3.0 (see Note 1)

   73-76 Latitude of the southern pole of projection

77-80 Longitude of the southern pole of
projection

   81-84 Angle of rotation of projection

   85-nnList of number of points along
each meridian or parallel (These octets are only present for
quasi-regular grids as described in Note 3)

Notes:


(2) Three parameters define a general latitude/longitude
coordinate system, formed by a general rotation of the sphere. One
choice for these parameters is:

(a) The geographic latitude in degrees of the southern pole of
the coordinate system, θp for example.

(b) The geographic longitude in degrees of the southern pole of
the coordinate system, λp for example.

(c) The angle of rotation in degrees about the new polar axis
(measured clockwise when looking from the southern to the northern pole)
of the coordinate system, assuming the new axis to have been obtained by
first rotating the sphere through λp degrees about the geographic polar
axis, and then rotating through (90 + θp) degrees so that the southern
pole moved along the (previously rotated) Greenwich meridian.


Except note that GRIB-2 uses "southern pole of projection". Not sure how
that affects the algorithm.

The CF parameter "north_pole_grid_longitude" likely should be "Angle of
rotation".



Anyone using this or have some insight on it ??




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Rotated Pole projection possibly wrong

[John Caron]

The "Rotated Pole" projection here:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#appendix-grid-mappings


   Rotated pole

grid_mapping_name = rotated_latitude_longitude



Map parameters:

 *

   grid_north_pole_latitude

 *

   grid_north_pole_longitude

 *

   north_pole_grid_longitude - This parameter is option (default is 0).

Map coordinates:

   The rotated latitude and longitude coordinates are identified by
   the standard_name attribute values
   grid_latitude and grid_longitude respectively.



is problematic. It has no reference to Proj.4, and is not in Snyder.

Its likely the same as the GRIB-2 GDS Template 3.1:

Grid Definition Template 3.1: Rotated Latitude/longitude (or 
equidistant cylindrical, or Plate Carrée)


Octet No. Contents

  15-72 Same as Grid Definition Template 3.0 (see Note 1)

  73-76 Latitude of the southern pole of projection

77-80 Longitude of the southern pole of 
projection


  81-84 Angle of rotation of projection

  85-nnList of number of points along 
each meridian or parallel (These octets are only present for 
quasi-regular grids as described in Note 3)


Notes:


(2) Three parameters define a general latitude/longitude 
coordinate system, formed by a general rotation of the sphere. One 
choice for these parameters is:


(a) The geographic latitude in degrees of the southern pole of 
the coordinate system, ?p for example.


(b) The geographic longitude in degrees of the southern pole of 
the coordinate system, ?p for example.


(c) The angle of rotation in degrees about the new polar axis 
(measured clockwise when looking from the southern to the northern pole) 
of the coordinate system, assuming the new axis to have been obtained by 
first rotating the sphere through ?p degrees about the geographic polar 
axis, and then rotating through (90 + ?p) degrees so that the southern 
pole moved along the (previously rotated) Greenwich meridian.



Except note that GRIB-2 uses "southern pole of projection". Not sure how 
that affects the algorithm.


The CF parameter "north_pole_grid_longitude" likely should be "Angle of 
rotation".




Anyone using this or have some insight on it ??


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

Hi Chris, and all:

On 1/11/2013 2:37 PM, Chris Barker - NOAA Federal wrote:

On Fri, Jan 11, 2013 at 9:00 AM, Aleksandar Jelenak - NOAA Affiliate
 wrote:


Here's the modified proposal for the datetime_iso8601 standard name:

...

String representing date-time information according to the ISO
8601:2004(E) standard.


I think we should NOT adopt a string option for datetime variables.

To quote Jonathan Gregory:

"""
In CF we have always applied the
principle that we only add to CF when there is a need to do so, i.e. there is
a use-case for something which cannot already be represented in CF
"""

We already have a way to encode datetimes in CF-netcdf.


Yes, but  is not as good as  as an encoding. The 
 is a result of cramming calendar handling into a units 
package.


I would advocate both should be allowed.



I believe this proposal resulted from the discussion about adding a
more flexible approach to datetimes in the CF Data Model. I think
that's a good idea, but separate from what encoding is used in
CF-netcdf. ( see my recent note for more detail about the difference
between and encoding and a data model ).

1) Having multiple ways to encode the same data in file format adds
complication to all client code -- client code would need a way to
process both ISO strings and "time_unit since datetime"


client code already has to parse the "date" in "time since date". So 
theres no extra code involved.





2) Any client code that can process ISO strings is likely to need to
convert them to a client-specific datetime representation anyway, in
order to plot, calculate with, etc  them.

3) Any client library that can process ISO strings is very likely to
be able to also work with "time_unit since datetime" encoded data
anyway -- and it had better, as that encoding is part of the standard
anyway.

As a result, we would be complicating client code, and getting no new
functionality.


We get new functionality in that "date" is clearer than "time since 
date", and more likely to be correctly understood by non CF specific 
software and users of our data in 100 years when theres no more CF 
discussion group to help people out.


when you have non-standard calendars, the difficulty is compounded many 
times over. How many seconds since 1970 is April 3, 2045 at 1:13 am in 
the no-leap calendar? Are you sure? What if you could just put into your 
file "2045-04-03T01:13:00" ?? Even rocket scientists can do that ;^)




The one advantage I can see at the moment is that simple, non-CF-aware
clients, like ncdump, could easily present a nice human-readable
format. But I don't think that is worth the additional complication.


Ideally file encodings should be as independent as possible from 
libraries and applications. We have historically had an unfortunate 
dependence on the udunits reference library for date parsing. We are 
slowly unwinding that dependence. I think in this case widening the 
allowed encoding for datetimes is well worth the complication.


Regards,

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] New standard name: datetime_iso8601

[John Caron]

I think this is a good idea.

I would let it be used for datetime coordinate variables (what is 
argument against that?)


I would not restrict to Gregorian calendar - let it apply to whatever 
calendar is being used.


On 1/11/2013 10:00 AM, Aleksandar Jelenak - NOAA Affiliate wrote:

Dear All:

Here's the modified proposal for the datetime_iso8601 standard name:

standard_name: datetime_iso8601

Units: N/A

String representing date-time information according to the ISO
8601:2004(E) standard. Variables with this standard name cannot serve
as coordinate variables. Date-time information is in the Gregorian
calendar. For dates preceding the Gregorian calendar the date-time
information is in the proleptic Gregorian calendar. Possible date-time
string forms are:

 =  "T"   ;

 =  "-" MM "-" DD |  "-" DDD ;

 = hh | hh ":" mm | hh ":" mm ":" ss | hh ":" mm ":" ss "." S
   | hh ":" mm ":" ss "," S ;

 = "" | "Z" | "+" hh | "+" hh ":" mm | "-" hh | "-" hh ":" mm

Where:

* "" is a four-digit year (-).

* "MM" is a two-digit month of the year (01-12).

* "DD" is a two-digit day of the month (01-31).

* "DDD" is a three-digit ordinal day of the year (001-366).

* "hh" is a two-digit hour (00-23).

* "mm" is a two-digit minute (00-59)

* "ss" is a two-digit second (00-59).

* "S" is one or more digits representing a decimal fraction of the
  second.

* The value of any designator when not specified is zero.

* If  is ommitted the default value is "Z".


-Aleksandar
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Chris:

On 12/17/2012 4:50 PM, Chris Barker - NOAA Federal wrote:

I think a solution shouldn't break current files which followed what had
been a standard for a long time (however ill-advised the standard was). I
don't have a good sense of what might break if the standard changed in terms
of software so I can' speak for all users but I do know many people have
downloaded our mean files with 1-1-1 base dates


Hmm -- this seems to contradict Steve's "no known users".


there are no known users using mixed gregorian/julian who actually need 
to use it (cross the transition date line). if there are any such users, 
please speak up, so we can think about your use case.




BUt anyway, I wonder if folks currently using such files are actually
getting the "correct" results, when they do.


those using udunits get the correct result even when they cross the 
line. i suspect its correct because using a different implementation 
(joda time library) gets the same results for the small sample i have 
tested.




For instance, in my naiveté, I never thought about calendars when
using netcdf files -- I simple converted "something since some date"
to Python datetime objects using python's datetime package, which uses
"An idealized naive date, assuming the current Gregorian calendar
always was, and always will be, in effect"

So I would have gotten the "wrong" results.


it sounds like you were using what is called proplectic gregorian. so 
you would have gotten wrong results on the PSD dataset, which uses "time 
since 1-1-1". Thats the argument for not using mixed gregorian, that 
simple things arent simple.




I've lost track whether UDunits does the mixed calendar right, but I'm
sure not all libraries one might use do.

So is this a use case se need to preserve?


we have datasets out there who are using "time since 1-1-1" 
unfortunately. Theres no doubt we need to leave those "working"; and 
even that they can continue their dangerous ways if they want to 
explicitly include a calendar attribute in CF-1.7 convention file. The 
proposal is just to change the default in CF-1.7 to use a more common 
calendar, and to encourage people not to cross the time warp zone 
unecessarily.


John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Cathy:

I think that you are using "backwards compatible" in a different way.

The current proposal(s) would not change files that are written with 
:Conventions="CF-1.x", where x <= 6. Files with x > 6 could still use 
the (ill-advised) old way if they want to, by putting an explicit 
calendar attribute in. But if theres no explicit calendar attribute, 
then these new files will be interpreted in a way that is less likely to 
give incorrect dates.


So, im not sure why you keep saying "shouldn't break current files", 
since there is no such proposal on the table.


John

On 12/17/2012 4:13 PM, Cathy Smith (NOAA Affiliate) wrote:

Cecelia

I think a solution shouldn't break current files which followed what had
been a standard for a long time (however ill-advised the standard was).
I don't have a good sense of what might break if the standard changed in
terms of software so I can' speak for all users but I do know many
people have downloaded our mean files with 1-1-1 base dates (ignoring
the climatologies for now). While we can potentially change what we have
either by changing the dates and/or adding a calendar attribute,
changing the default calendar may create errors in reading dates for
users who already have those files (which are currently CF complaint ).
And,  they won't have the same ability to change the files and they
wouldn't necessarily know they needed to.  I think no default at all
would be problematic as what would software do?  So, I would support the
inclusion of a calendar attribute that would be used by software if
there but using the old default calendar if not. Also, I would support
making a calendar attribute for new files mandatory (with an updated CF
version) but I would keep backwards compatibility unless it were
reliably shown it was not an issue with users.
I'm not convinced that the users needs (as opposed to developers) have
been adequately researched.

Cathy



On 12/17/12 12:56 PM, Cecelia DeLuca - NOAA Affiliate wrote:

Cathy,

Of the other options, do you find some (or all) completely unacceptable?
Are some better than others?

- Cecelia

On 12/17/2012 10:59 AM, Cathy Smith (NOAA Affiliate) wrote:

Cecelia
I support 1) mostly for backward compatibility. I would also strongly
encourage but not demand that users change their base dates to after
1800
when it makes sense to do so.

And, I (again) want to make sure that LTMs and their time values are
addressed before any decisions are made as to negative times and
using base dates of 1-1-1 and the issue of what year to use for
climatologies. LTM dates are a problem when one needs to use a
calendar based on real dates.

Cathy


On 12/12/12 9:04 AM, Cecelia DeLuca - NOAA Affiliate wrote:

Hi Steve, Jonathan and all,

There are not that many options being discussed.

With respect to the default calendar:

1 keep the Julian-Gregorian calendar as default (no change)
2 remove the Julian-Gregorian calendar as default, and have no
default calendar (grid analogy)
3 replace the Julian-Gregorian calendar as default with the
proleptic Gregorian calendar
4 replace the Julian-Gregorian calendar as default with a strict
Gregorian calendar

Maybe it makes sense for people to cite (or rank)  their preference
at this point?

There were a couple other proposals, depending on which of above is
selected:
5 create a strict Gregorian calendar (optional for 1, 2, 3 and
needed for 4)
6 remove the Julian-Gregorian calendar (impossible for 1, optional
for 2, 3, 4)

Again, maybe worth it to see where people are after the round of
discussion?

Best,
Cecelia



On 12/10/2012 12:40 PM, Steve Hankin wrote:

Hi Jonathan,

I'm not sure if my remarks below conflict with your proposed
resolution.  But they do dispute the facts you assert, and these
waters are so muddy that agreeing on the facts seems an important
first step.

On 12/10/2012 1:21 AM, Jonathan Gregory wrote:

Dear Jon


Just to repeat a remark that Steve Hankin made whose implications have not been 
explored in this discussion: different countries adopted the Gregorian calendar 
at different times.  (Greece didn't adopt it till 1923!)  So what is considered 
a valid Gregorian date varies from country to country (and some of those 
countries don't even exist any more, or at least the boundaries have changed...)
2. The non-proleptic Gregorian calendar is extremely problematic for historical 
observations as well as for models (astronomers use the Julian calendar 
consistently for this reason).

Yes, that's right. Nonetheless I don't think we can abolish the real-world
calendar, despite its ambiguities, because*_it's the one we really use!_*


Are you sure this is true?  Evidence seems to suggest that our
community has _no use for the mixed Gregorian/Julian calendar at
all_, except the need to resolve the backwards compatibility mess
we have created for ourselves.

  * In everyday life we use is the modern Gregorian calendar, and
are not concerned with historical calendar changes.
  * In numeric

Re: [CF-metadata] calendar-based units of time

[John Caron]

Hi Jonathan:

On 12/17/2012 10:51 AM, Jonathan Gregory wrote:

Dear John


I have prototyped similar functionality in the CDM, documented here:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html

Thanks for this. I recall the earlier discussion and I think what you have
implemented would be a good extension to CF, namely

* Optional prefix "calendar" to the time-units means the corresponding
component of time is manipulated. You show only years and months. Do you
also accept the unproblematic units of days, hours, minutes and seconds
with "calendar" prefixed? Do you allow the SI prefixes to these units?


yes, the same periods are allowed that are in the udunits grammer:

period = "millisec" | "msec" | "second" | "sec" | "s" | "minute" | "min" | "hour" | "hr" | "day" | 
"week" | "month" | "mon" |
"year" | "yr"


we do not allow SI prefixes. we do allow plural forms, ie with an 
appended s.





* With calendar, only integer numbers of units are allowed. Are negative
numbers allowed?


yes



* Invalid dates caused by adding months and years are decremented in whole
days until a valid date is reached.

yes

regards,
John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] calendar-based units of time

[John Caron]

On 12/14/2012 10:35 AM, Jonathan Gregory wrote:





I think that calendar-based units of time cannot be introduced without a new
syntax for time units, and some rules about how to interpret the cases when
adding months to a reference date gives an impossible date. We could make such
changes to the convention.


I have prototyped similar functionality in the CDM, documented here:

http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html

John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi all:

Heres what I understand of the conversation:

1. Theres nothing to do about existing files CF-1.6 and before. we are 
stuck with the udunits mixed calendar.


2. Starting with the next version after acceptance, (1.7 currently), we 
can do something different. I agree that forcing people to put in a 
calendar attribute makes simple things not simple. So lets choose a 
reasonable default, either gregorian_propleptic or gregorian_strict is 
ok with me.


3. everyone agrees that in the unit "time since date", date is 
interpreted in the calendar that is specified, or the default if not 
specified.


4. i thing we should add some advice not to start counting from 0001 
when recording modern dates, no matter if you do it "right" or not.



John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] [netcdf-java] problem with times in PSD dataset

[John Caron]

Hi Cathy:

There no question that CF currently defaults to mixed gregorian 
calendar. The discussion is whether thats the best choice (probably 
not), and to advise users not to cross the discontinuity (eg store 
modern dates starting from 1-1-1).


Im curious as to how you generate the dates that you store? That is, how 
do you know that they are correct?


John

On 12/6/2012 4:34 PM, Cathy Smith (NOAA Affiliate) wrote:

John
There is some meteorological data that is available pre-Gregorian
calendar (paleo data, some temperature datasets) and of course there are
other scientific fields where data is pre-1500 (e.g. astronomy,
archeology) . Given that netCDF files with data dates spanning ~1550 probably 
already exist and the large number of preexisting files that
use the 1-1-1 base (we have over 2000), it doesn't seem reasonable to
request that files be changed to accommodate what is essentially a bug;
the date values we store are correct. We started using the 1-1-1 base in the mid
1990's (almost 20 years ago) as part of the COARDS (now CF) agreed upon 
standard.
It is reasonable for us to consider future changes but I don't think it 
reasonable for us to have to change our files because the Java interface is not 
backwards compatible.

Cathy Smith
NOAA/ESRL PSD

On 12/5/12 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian ("Mixed
Gregorian/Julian calendar") the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using "time since
1-1-1". Ive never seen a dataset where "time since 1-1-1" using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need "historical accuracy", then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. "time since
reference date" is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard libraries that
could be used. Anyone have candidate libraries in C or Python for
robust calendering>

In short, we should rely on clear encoding standards (eg ISO8601) with
reference software, rather than implementations like udunits that
eventually go away.

John

PS: I think ill cross post to cf, just to throw some gasoline on the
fire ;), and maybe some broader viewpoints.

On 12/5/2012 10:24 AM, Don Murray (NOAA Affiliate) wrote:

Hi Gerry-

On 12/5/12 9:42 AM, Gerry Creager - NOAA Affiliate wrote:

There are other datasets with reference to 1-1-1. I've seen them most
recently in some ocean models.


And the ESRL/PSD NCEP reanalysis datasets use it.

Don


On Wed, Dec 5, 2012 at 10:16 AM, Don Murray (NOAA Affiliate)
mailto:don.mur...@noaa.gov>> wrote:

 John-

 I meant to send this to support-netcdf-java, but perhaps  others on
 the list might have the same problem.


 On 12/4/12 4:51 PM, John Caron wrote:

 On 12/4/2012 4:09 PM, Don Murray (NOAA Affiliate) wrote:

 Hi-

 I was just trying to access the NOAA/ESRL/PSD  Outgoing
Longwave
 Radiation (OLR) data using netCDF-Java 4.3 ToolsUI and
 noticed that the
 times are wrong.  If you open:


dods://www.esrl.noaa.gov/psd/__thredds/dodsC/Datasets/__uninterp_OLR/olr.day.mean.nc



<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/uninterp_OLR/olr.day.mean.nc>





 in the ToolsUI grid viewer, the last time in the file is
 shown as
 2012-12-04 00Z.  However, the last time in the file is
actually
 2012-12-02 00Z.  Here is the time variable in that file:

  double time(time=3989);
:units = "hours since 1-1-1 00:00:0.0";
:long_name = "Time";
:actual_range = 1.7540448E7, 1.763616E7; // double
:delta_t = "-00-01 00:00:00";
:avg_period = "-00-01 00:00:00";
:standard_name = "time";
:axis = "T";

 netCDF-Java 4.2 and ncdump -t -v time (C version) show the
 correct
 date/times.


 hours from 1-1-1 is rather problematic, since you are crossing
the
 julian/gregorian weirdness line (i think thats the technical
term ;)

 Im guessing the trouble lies here:

 "Default calendar: for udunits, and therefore for CF, the
defau

[CF-metadata] CF calendars (was: problem with times in PSD dataset)

[John Caron]

Hi Steve, Cecelia:

I agree we should move to a better default calendar, probably 
proleptic_gregorian. We are stuck with mixed Gregorian for previous CF 
versions.


I think we just need a proposal that says "As of version X, the default 
calender is proleptic_gregorian. You should explicitly add the calendar 
attribute for clarity. Udunits is no longer considered to be the 
reference software for date/time."


It would be nice to add advice to the user about calendar tradeoffs and 
how to do historical date matching for modelers, assuming we have useful 
things to say.


I wonder if allowing ISO formatted strings in place of / in addition to 
the "time since reference date" form might be useful for historical time 
matching?


Arguably having calendar reference libraries in Python and Java would be 
sufficient, possibly Python is preferable even to one in C.


John

On 12/6/2012 1:45 PM, Cecelia DeLuca wrote:


Hi John and all,


Thanks for this information. A few questions:

* So you are not supporting "standard Gregorian calendar" even though
thats the CF default?

Correct, the climate modelers that we work with don't use it.  AFAIK the
decision of
CESM was to reject the CF default as unreasonable for modeling and use
proleptic
Gregorian.  GFDL might support it, I don't know.

We could probably add standard Gregorian as a new calendar if people on the
data side needed it.  However, we would be happier to join Steve in putting
a stake in its heart!


* Do modelers need to match "historical dates"? If so, what calendar
do they use?

I think the calendar used would depend on what was supported by the
model and
configuration, as well as the form of the date.  If you wanted to
represent the date
of something like a volcanic eruption you could probably make it work
with most
of the calendars.



* Is the time library suitable to be released seperate from the ESMF
code, eg as standalone C++?

The time library can be used apart from other ESMF interfaces, and some
modeling groups do use it that way.  I don't think we'd be willing to
extract that
part and support a separate build.   We did that years ago and it was a
pain
to maintain.  (We could try to isolate the documentation though, so users
didn't have to wade through the whole reference manual to find the API.)

With ESMP(ython), the scope of the interface is much smaller than
ESMF proper and I think Ryan could set time functions up nicely.  It might
make sense to represent it as a separate python package in that approach.
Would python work for you, or would you really prefer C?

- Cecelia



John

On 12/5/2012 3:01 PM, Cecelia DeLuca wrote:

Hi John and all,

ESMF has a mature time management library with calendars that are
commonly
used in climate/weather/related modeling. It supports the following: 360
day,
no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day,
and custom
(including custom calendars that may change the length of days). ESMF,
like CESM,
doesn't support the standard Gregorian calendar because it doesn't make
sense for modeling groups who may need to cross the Julian/Gregorian
weirdness line (we've never gotten a request to support standard
Gregorian from
modelers).  ESMF has calendar, time, time interval, clock, and alarm
constructs,
can run time forward or backward, and prints times and time intervals in
ISO8601
format. CESM/CAM, NASA, NOAA, Navy and other codes use it.

The most complete interface to the time lib is Fortran, and there are
some basic
time functions exposed in C (the lib is actually written in C++).
However, we could
wrap the time functions in Python and include them in ESMP, which is
currently a
set of ESMF regridding functions wrapped in Python. We could probably do
that
in the late/winter spring timeframe, with some guidance on what
functions were
desired and a pass through our prioritization board.

Best,
-- Cecelia


On 12/5/2012 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian ("Mixed
Gregorian/Julian calendar") the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using "time since
1-1-1". Ive never seen a dataset where "time since 1-1-1" using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need "historical accuracy", then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. "time since
reference date" is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in cale

Re: [CF-metadata] [netcdf-java] problem with times in PSD dataset

[John Caron]

Hi Cecelia:

Thanks for this information. A few questions:

* So you are not supporting "standard Gregorian calendar" even though 
thats the CF default?


* Do modelers need to match "historical dates"? If so, what calendar do 
they use?


* Is the time library suitable to be released seperate from the ESMF 
code, eg as standalone C++?


John

On 12/5/2012 3:01 PM, Cecelia DeLuca wrote:

Hi John and all,

ESMF has a mature time management library with calendars that are commonly
used in climate/weather/related modeling. It supports the following: 360
day,
no leap, Gregorian (proleptic), Julian, Julian day, modified Julian day,
and custom
(including custom calendars that may change the length of days). ESMF,
like CESM,
doesn't support the standard Gregorian calendar because it doesn't make
sense for modeling groups who may need to cross the Julian/Gregorian
weirdness line (we've never gotten a request to support standard
Gregorian from
modelers).  ESMF has calendar, time, time interval, clock, and alarm
constructs,
can run time forward or backward, and prints times and time intervals in
ISO8601
format. CESM/CAM, NASA, NOAA, Navy and other codes use it.

The most complete interface to the time lib is Fortran, and there are
some basic
time functions exposed in C (the lib is actually written in C++).
However, we could
wrap the time functions in Python and include them in ESMP, which is
currently a
set of ESMF regridding functions wrapped in Python. We could probably do
that
in the late/winter spring timeframe, with some guidance on what
functions were
desired and a pass through our prioritization board.

Best,
-- Cecelia


On 12/5/2012 12:25 PM, John Caron wrote:

Hi all:

Its probably the right thing to do to make gregorian ("Mixed
Gregorian/Julian calendar") the default calendar for COARDS/CF, for
backwards compatibility. However, CDM may leave proleptic_gregorian
(ISO8601) as its default.

And I would strongly suggest that data writers stop using "time since
1-1-1". Ive never seen a dataset where "time since 1-1-1" using the
mixed gregorian calendar was actually needed. If any one has a real
example, Id like to hear about it.

If you really need "historical accuracy", then put in an ISO8601
formatted string, and an explicit calendar attribute. CDM handles
those ok. CF should be upgraded to allow ISO strings also. "time since
reference date" is not good for very large ranges of time.

Ill just add that udunits never wanted to be a calendaring library,
and shouldnt be used anymore for that. Im relying on joda-time (and
its successor threeten) to be the expert in calendering, so i dont
have to. I think the netcdf-C library now uses some CDAT (?) code for
its calendaring, but Im sure theres other standard libraries that
could be used. Anyone have candidate libraries in C or Python for
robust calendering>

In short, we should rely on clear encoding standards (eg ISO8601) with
reference software, rather than implementations like udunits that
eventually go away.

John

PS: I think ill cross post to cf, just to throw some gasoline on the
fire ;), and maybe some broader viewpoints.

On 12/5/2012 10:24 AM, Don Murray (NOAA Affiliate) wrote:

Hi Gerry-

On 12/5/12 9:42 AM, Gerry Creager - NOAA Affiliate wrote:

There are other datasets with reference to 1-1-1. I've seen them most
recently in some ocean models.


And the ESRL/PSD NCEP reanalysis datasets use it.

Don


On Wed, Dec 5, 2012 at 10:16 AM, Don Murray (NOAA Affiliate)
mailto:don.mur...@noaa.gov>> wrote:

John-

I meant to send this to support-netcdf-java, but perhaps others on
the list might have the same problem.


On 12/4/12 4:51 PM, John Caron wrote:

On 12/4/2012 4:09 PM, Don Murray (NOAA Affiliate) wrote:

Hi-

I was just trying to access the NOAA/ESRL/PSD Outgoing
Longwave
Radiation (OLR) data using netCDF-Java 4.3 ToolsUI and
noticed that the
times are wrong.  If you open:


dods://www.esrl.noaa.gov/psd/__thredds/dodsC/Datasets/__uninterp_OLR/olr.day.mean.nc



<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/uninterp_OLR/olr.day.mean.nc>





in the ToolsUI grid viewer, the last time in the file is
shown as
2012-12-04 00Z.  However, the last time in the file is
actually
2012-12-02 00Z.  Here is the time variable in that file:

 double time(time=3989);
   :units = "hours since 1-1-1 00:00:0.0";
   :long_name = "Time";
   :actual_range = 1.7540448E7, 1.763616E7; // double
   :delta_t = "-00-01 00:00:00";
   :avg_period = "-00-01 00:00:00";
   :standard_name = "time";
   :axis = "T";

netCDF-Java 4.

Re: [CF-metadata] [netcdf-java] problem with times in PSD dataset

[John Caron]

Hi all:

Its probably the right thing to do to make gregorian ("Mixed 
Gregorian/Julian calendar") the default calendar for COARDS/CF, for 
backwards compatibility. However, CDM may leave proleptic_gregorian 
(ISO8601) as its default.


And I would strongly suggest that data writers stop using "time since 
1-1-1". Ive never seen a dataset where "time since 1-1-1" using the 
mixed gregorian calendar was actually needed. If any one has a real 
example, Id like to hear about it.


If you really need "historical accuracy", then put in an ISO8601 
formatted string, and an explicit calendar attribute. CDM handles those 
ok. CF should be upgraded to allow ISO strings also. "time since 
reference date" is not good for very large ranges of time.


Ill just add that udunits never wanted to be a calendaring library, and 
shouldnt be used anymore for that. Im relying on joda-time (and its 
successor threeten) to be the expert in calendering, so i dont have to. 
I think the netcdf-C library now uses some CDAT (?) code for its 
calendaring, but Im sure theres other standard libraries that could be 
used. Anyone have candidate libraries in C or Python for robust calendering>


In short, we should rely on clear encoding standards (eg ISO8601) with 
reference software, rather than implementations like udunits that 
eventually go away.


John

PS: I think ill cross post to cf, just to throw some gasoline on the 
fire ;), and maybe some broader viewpoints.


On 12/5/2012 10:24 AM, Don Murray (NOAA Affiliate) wrote:

Hi Gerry-

On 12/5/12 9:42 AM, Gerry Creager - NOAA Affiliate wrote:

There are other datasets with reference to 1-1-1. I've seen them most
recently in some ocean models.


And the ESRL/PSD NCEP reanalysis datasets use it.

Don


On Wed, Dec 5, 2012 at 10:16 AM, Don Murray (NOAA Affiliate)
mailto:don.mur...@noaa.gov>> wrote:

John-

I meant to send this to support-netcdf-java, but perhaps  others on
the list might have the same problem.


On 12/4/12 4:51 PM, John Caron wrote:

On 12/4/2012 4:09 PM, Don Murray (NOAA Affiliate) wrote:

Hi-

I was just trying to access the NOAA/ESRL/PSD  Outgoing
Longwave
Radiation (OLR) data using netCDF-Java 4.3 ToolsUI and
noticed that the
times are wrong.  If you open:


dods://www.esrl.noaa.gov/psd/__thredds/dodsC/Datasets/__uninterp_OLR/olr.day.mean.nc


<http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/uninterp_OLR/olr.day.mean.nc>




in the ToolsUI grid viewer, the last time in the file is
shown as
2012-12-04 00Z.  However, the last time in the file is
actually
2012-12-02 00Z.  Here is the time variable in that file:

 double time(time=3989);
   :units = "hours since 1-1-1 00:00:0.0";
   :long_name = "Time";
   :actual_range = 1.7540448E7, 1.763616E7; // double
   :delta_t = "-00-01 00:00:00";
   :avg_period = "-00-01 00:00:00";
   :standard_name = "time";
   :axis = "T";

netCDF-Java 4.2 and ncdump -t -v time (C version) show the
correct
date/times.


hours from 1-1-1 is rather problematic, since you are crossing
the
julian/gregorian weirdness line (i think thats the technical
term ;)

Im guessing the trouble lies here:

"Default calendar: for udunits, and therefore for CF, the default
calendar is gregorian ("Mixed Gregorian/Julian calendar"). For
CDM, the
default calendar is proleptic_gregorian (ISO8601 standard). This
only
matters for dates before 1582."


Joda time supports the GJ calendar (Historically accurate calendar
with Julian followed by Gregorian) which seems it would be backward
compatible with the CF/udunits.  Perhaps that should be the default
for backward compatibility.


I have to say relying uncritically on a calendar
implementation like
udunits is a mistake. putting the reference date unnecessarily to
include the problem is, um, unnecessary.


But it is historically accurate.  For climate datasets, this would
be important.


is there any way those files can be updated? specifying the
gregorian
calendar explicitly should do it, but changing to use a
reference date
after 1582 would be much better.


How's your FORTRAN? ;-)  I'm not sure why this was chosen
originally, but it doesn't seem reasonable to make people change
their datasets.

Does anyone else on the list know of datasets (other than
climatologies) that might use a reference of 1-1-1 that will be
affected by this change?



BTW, is there a

Re: [CF-metadata] Choice of fill value for unpacked data

[John Caron]

On 10/10/2012 6:36 AM, Jim Biard wrote:

John,

I agree that there is no ambiguity in the original dataset.  I was just
pointing out a scenario in which the difficulty mentioned by Phil in his
original post would manifest itself.  The methodology referenced by Russ
won't handle it.  If you (for reasons beyond your control) have a packed
fill value that is inside your valid range, then no guidance is provided
by CF to automated client software for how to represent the unpacked
fill value, leaving the user with a requirement to dig back through the
packed values, find the elements marked as fill, and "manually" mark the
unpacked values.


Its true CF is silent about this, so "automated client software" has to 
make their own Convention. The netcdf-JAVA package supplies a method


  boolean isMissing(double value);

for this purpose. Alternately, it will just set missing real values to 
NaNs, which works well in Java. I think NaNs are the right choice for 
missing values myself.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Choice of fill value for unpacked data

[John Caron]

Hi Jim:

_FillValue/missing_value refers to the packed value, so theres no 
ambiguity in the original dataset. It is best to make sure its outside 
the range of real values, but even if not, one just has to search for 
that exact bit pattern.


If someone rewrites the data, its their responsibility to choose a 
_FillValue/missing_value that is unambiguous.


If your use case is important, you could add _UnpackedFillValue, so that 
your software did the right thing. Dunno if it would be generally useful.


John

On 9/27/2012 7:00 AM, Jim Biard wrote:

Hi.

Assuming you have the luxury of specifying your _FillValue and/or
missing_value, I agree that this isn't a big deal.  However, I am
working with data where the project has defined fill/missing values that
are wholly within the range of possible values (NPP satellite data).
  The approach defined below fails in such cases.

Grace and peace,

Jim

Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave, Asheville, NC 28801-5001

jim.bi...@noaa.gov 
828-271-4900

On Sep 25, 2012, at 11:08 PM, Russ Rew mailto:r...@unidata.ucar.edu>> wrote:


Hi Phil,


The final para of section 2.5.1 of the CF conventions document describes
the use of the _FillValue (or missing_value) attribute in the case of
data packed using the scale-and-offset method.  What is not clear - at
least to me - is what the preferred application behaviour should be in
the case where the data is unpacked and then written out to a new netCDF
file. In particular, what fill value should be used for the unpacked
data variable?

I presume that one wouldn't normally want to use the original fill value
since that value (typically an 8- or 16-bit integer) is quite likely to
fall within the normal range of the unpacked data (typically a 32- or
64-bit float).

In the absence of explicitly setting a fill value attribute on the
unpacked data variable I assume that the netCDF default fill value will
be used for the data type in question. Which may not always be desirable
(certainly not for 32-bit floats, where the default fill value can give
rise to subtle precision-related problems).

With this in mind, I was wondering if there is any merit in defining a
new attribute called, say, _UnpackedFillValue (or
unpacked_missing_value)? If client software detected this attribute then
the associated value (same data type as the scale_factor and add_offset
attributes) would be used as the fill value for the unpacked data
variable.

Alternatively, the names _FillValueUnpacked (missing_value_unpacked)
might be preferable since they would then appear together pair-wise in
CDL-type listings, e.g.

short pkd_var(z, y, x) :
  ...
  pkd_var:_FillValue =3D -32768 ;
  pkd_var:_FillValueUnpacked =3D -1.0e30 ;
  pkd_var:add_offset =3D 42.0 ;
  pkd_var:scale_factor =3D 1234.0 ;
  ...


Any merit/mileage in this idea?


A more detailed recommendation for treating special values such as
_FillValue or missing_value for packed data is described, with
associated formulas, in the "Packed Data Values" section of a
best-practices document that we wrote a few years ago:

http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Packed%20Data%20Values

It provides a recommendation for ensuring the unpacked special value is
not in the range of other unpacked data values.  If that recommendation
is followed, I think there is no need for an additional
_FillValueUnpacked (or missing_value_unpacked) attribute.

If you agree that is an acceptable approach, perhaps we should add it to
CF ...

--Russ
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu 
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] cf4j - Java library for CF standard names

[John Caron]

cool, thanks!

On 7/31/2012 11:56 AM, Shane StClair wrote:

Hi all,

Just wanted to share a new Java library available on Maven Central. cf4j
(CF for Java) is a simple library that provides constants for all CF
standard names and feature types. The standard names are generated via XSLT
from the xml representation of the standard names[1] provided on the CF
Metadata site. The major version of cf4j corresponds to the revision number
of the CF standard names it provides (currently 19).

The project is hosted on GitHub here:
https://github.com/axiomalaska/cf4j

Project documentation is available here:
http://axiomalaska.github.com/cf4j/

...and JavaDocs here:
http://axiomalaska.github.com/cf4j/apidocs/

The most useful class is CFStandardNames, which provides all of the
standard name constants:
http://axiomalaska.github.com/cf4j/apidocs/com/axiomalaska/cf4j/CFStandardNames.html

The library will be updated as new versions of the CF names are produced. I
needed the library while working on a Sensor Observation Service project
and wasn't aware of any Java resources that provided versioned CF names.
Hopefully it will be useful for someone.

Best,
Shane



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Roms: ocean_s_coordinate_g1 and g2?

[John Caron]

Ive updated the docs at

http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html 



Thanks for the report.

On 7/17/2012 11:52 PM, Heiko Klein wrote:

Rich,

it must have been the netcdf-java package where I have seen these 
standard-names. The description on the netcdf-java pages are outdated, 
still using formula_terms a and b. I just checked the code, and that 
is up to date working with the formula_term -array C(k).


It'll be good to have an official version. Thanks for picking this up 
again.


Heiko


On 2012-07-17 17:13, Rich Signell wrote:

Heiko,

We worked with John Caron several years ago to get these into the CDM
in the NetCDF-Java library, so if you have g1 or g2 coordinates they
will work with codes that use NetCDF-java (like the Matlab NCTOOLBOX
and Unidata's IDV),  and I remember we drafted up some documentation
to  submit to CF, but looks like we dropped the ball.I'll pick it
up again.

-Rich

On Mon, Jul 16, 2012 at 10:24 AM, Heiko Klein  
wrote:

Hi,

I'm currently working with ROMS ocean model files. The model uses a
generalized ocean_s coordinate as defined by
https://www.myroms.org/wiki/index.php/Vertical_S-coordinate .

The same page says that their s_coordinate is covered by standard_names
ocean_s_coordinate_g1
ocean_s_coordinate_g2
and I believe I have seen at least the g1 case in a draft of a CF 
convention
some month ago. Both cases are also implemented in the netcdf-java 
package

with above standard_names:
http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html 



But I cannot find any of those names any longer, neither in CF-1.6 
nor in

the standard_names tables.

Does somebody remember what happened to these vertical coordinates?
According to the ROMS Wiki, the standard_names have been added in March
2009.



Best regards,

Heiko
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata








___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] You will get email from the CF Trac system.

[John Caron]

yes, i typically get 3 copies. it would be nice not to.


On 6/25/2012 4:29 PM, Seth McGinnis wrote:

I do want to see Trac messages, but since this update happened I'm getting two
or three copies of each one.  Is anyone else having this problem?

What do I need to do to get only one copy?

Thanks,

--Seth

On Thu, 21 Jun 2012 16:44:40 -0700
  "Jeffrey F. Painter"  wrote:

CF metadata issues get discussed both in this mailing list and in the CF
Metadata Trac system at https://cf-pcmdi.llnl.gov/trac/.  Some time in the
past, everyone on the cf-metadata list also got copies of the discussions in
the Trac system.  But the Trac system's mailing list hadn't been updated for
some years.  So today I updated it.  Everyone on this mailing list will get
messages from the CF Metadata Trac system.  There are rather few such
messages, so you may not notice the difference.

If you really don't want to see the Trac messages, but you want to stay on the
regular cf-metadata mailing list, you can opt out of only the Trac messages.
There is a special mailing list which acts as an intermediary between the
Trac system and cf-metad...@ugd.ucar.edu.  You can just unsubscribe to the
special mailing list as follows:  Send mail to majord...@lists.llnl.gov, from
the address at which you receive the mail.   The body of your message should
be the line "unsubscribe cf-metadata".  This will unsubscribe you only from
the special intermediary list, not the regular CF Metadata mailing list at
ucar.edu.

Thanks to Jonathan Gregory for suggesting that I address this issue.

- Jeff Painter

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Reverse-time trajectory

[John Caron]

On 5/11/2012 10:39 AM, Jonathan Gregory wrote:

Dear Richard

Auxiliary coordinate variables don't have to be monotonic. That's the usual
rule for them, and the reason for what it says in H.2.1. To require them to
be monotonic would be a new rule. I think it would be a specific requirement
for sect 9. That is, you could say, if it's a timeseries feature type, the time
coordinate must be monotonic, even if it's an aux coord var. However, I don't
think you'd want to require other aux coord vars to be monotonic, such as lat,
lon and alt in the same example. So this extra requirement of monotonicity
would be for particular coordinate types in particular feature types.

Is this necessary? Also, is it desirable? Currently, it is possible for the
times to be stored disordered in the index ragged array representation, for
instance. That might be convenient, since that representation is expected to
be used when data is written as it arrives, and it might not arrive in the
right order. Would you want to prohibit this by requiring time to be monotonic
in H.2.5, for example?

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


Yes, I agree. Writers cant always guarentee that time data is ordered; 
if they can they should use a coordinate variable. Otherwise, its up to 
higher level access services to add this kind of functionality, similar 
to an SQL SORT.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Accept geostationary projection

[John Caron]

If no one has any objections, i move that we accept this ticket:

"Adding the geostationary projection."

https://cf-pcmdi.llnl.gov/trac/ticket/72

the changes needed to the spec are at the bottom of the ticket, we didnt 
know how to change the top part.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Reverse-time trajectory

[John Caron]

On 5/1/2012 9:02 AM, Jonathan Gregory wrote:

Dear Richard


The CF definitions of discrete sampling geometries define the "trajectory" feature type as "a series of 
data points along a path through space with monotonically increasing times". This is a stricter stance than the 
usual CF coordinate definition of "ordered monotonically". What was the reason behind the addition of the 
"increasing" constraint, and can it be relaxed? We have data sets where a model is run backwards in time.

Good point. I would think that "monotonically" would be enough, not necessarily
increasing. I can't remember a reason for "increasing"; I assume it's just
because sect 9 was conceived for observations in the first place. However, John
Caron may well have a comment. I don't think anything prevents your storing
the data in the orthogonal multidimensional representation, which existed
before sect 9 did and doesn't require increasing coordinates.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


i agree, "increasing" is not needed, and could be changed with a defect 
ticket.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

On 4/30/2012 8:40 PM, andrew walsh wrote:

Hi John and CF-Metadata list,

Based on your earlier advice I decided using the Scalar way to represent
the coordinate lat, long and time rather than Vector way i.e lat(lat0, 
lon(lon), time(time)

mainly for reason of simplicity.


the correct other choice is lat(profile), lon(profile), time(profile). 
not sure if you caught that.


the scalar choice is fine for 1 profile per file. the other is used to 
store multiple profiles in one file.




I have run the Scalar sample through the BADC netCDF checker (*Note)
at http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl and I now get 
this error

on each of the lat, lon, time variables:

ERROR (4): Axis attribute is not allowed for auxillary coordinate 
variables.
I don't get this error with the Vector approach. It looks like the 
checker thinks
my scalar lat, lon, time are 'auxilliary coordinate variables' and 
axis attributes

are not allowed on these. Is the checker interpreting things correctly?


i think we recently clarified that axis is acceptable on auxiliary 
coordinates. OTOH, its unecessary, as long as you follow the other rules 
for identifying coordinates (chapter 4).


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

yes, good point.

if it is a multidimension representation, you would also need the 
profile dimension. if its a ragged array, you would need either the 
rowSize or parentIndex variable.


On 4/30/2012 7:22 AM, Jim Biard wrote:
Wouldn't you also need the profile dimension on the temperature and 
pressure variables in order to connect them all as part of the same 
profile collection?


Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave, Asheville, NC 28801-5001

jim.bi...@noaa.gov <mailto:jim.bi...@noaa.gov>
828-271-4900

On Apr 30, 2012, at 8:46 AM, John Caron wrote:


On 4/29/2012 5:33 PM, andrew walsh wrote:

Hi John,

My responses inline below.

Andrew


- Original Message ----- From: "John Caron" 
mailto:ca...@unidata.ucar.edu>>

To: mailto:cf-metadata@cgd.ucar.edu>>
Sent: Saturday, April 28, 2012 2:39 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi Andrew:

You can use a dimension=1 instead of a scalar.


OK, you mean if one used the  vector method.


yes





But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)


So, do you mean in the vector method have this? (simpler I guess to
have common profile dimension = 1)


yes, you cant use incorrect dimensions, even when they are all = 1.



dimensions:
profile=1
pressure = 57

variables:
lat(profile)
lon(profile)
time(profile)
pressure(pressure)
temperature(pressure)

Instead of this?

Dimensions:
time=1
lat=1
lon=1
pressure=57

Variables:
time(time)
lat(lat)
lon(lon)
pressure(pressure)
temperature(pressure)

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu <mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

On 4/29/2012 5:33 PM, andrew walsh wrote:

Hi John,

My responses inline below.

Andrew


- Original Message - From: "John Caron" 
To: 
Sent: Saturday, April 28, 2012 2:39 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi Andrew:

You can use a dimension=1 instead of a scalar.


OK, you mean if one used the  vector method.


yes





But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)


So, do you mean in the vector method have this? (simpler I guess to
have common profile dimension = 1)


yes, you cant use incorrect dimensions, even when they are all = 1.



dimensions:
profile=1
pressure = 57

variables:
lat(profile)
lon(profile)
time(profile)
pressure(pressure)
temperature(pressure)

Instead of this?

Dimensions:
time=1
lat=1
lon=1
pressure=57

Variables:
time(time)
lat(lat)
lon(lon)
pressure(pressure)
temperature(pressure)

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi Andrew:

You can use a dimension=1 instead of a scalar.

But you cant use

lat(lat)
lon(lon)
time(time)

the correct usage in the single profile case would be:

lat(profile)
lon(profile)
time(profile)

John

On 4/27/2012 12:03 AM, andrew walsh wrote:

Hi John,

Thanks for your advice.
For a single vertical profile and single valued time, lat, long.
it seems there are two acceptable ways i.e 1 element vector or single 
valued scalar.


Quoting from 'section 2.4 Dimensions' of CF standard:

1 element vector:

"Dimensions may be of any size, including unity. When a single value 
of some coordinate applies to all the values in a variable, the 
recommended means of attaching this information to the variable is by 
use of a dimension of size unity with a one-element coordinate variable.


and in the next sentence a scalar:

"It is also acceptable to use a scalar coordinate variable which 
eliminates the need for an associated size one dimension in the data 
variable."


1) ***Vector co-ordinate variables***

Dimensions:
time=1
pressure=56
lat=1
lon=1

Variables:
time(time)
lat(lat)
lon(lon)
temperature(pressure)

2) **Scalar co-ordinate varaibles

Dimensions:

pressure =56

Variables:
time
lat
lon
temperature(pressure)

The scalar approach you suggest as in section "H.3.3. Single profile" 
of the CF v1.6 standard

is simpler than the vector approach so we will take your advice.

Regards,

Andrew


- Original Message - From: "John Caron" 
To: "andrew walsh" 
Cc: ; "Luke Callcut" ; 


Sent: Thursday, April 26, 2012 10:48 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



Hi andrew:

The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a 
single profile. Assuming that, you would use the following template:


http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 



note that lat, lon and time coordinates all have to be scalars.

John

On 4/18/2012 8:24 PM, andrew walsh wrote:

Hi John,

We have now constructed a real netCDF file for you to check. QC flag
variables have been added in. The QC flagging method is based on the
proposed IODE scheme in (1) below.

Attached are the sample .nc file, text (ncdump) of same and a document
specifying the CDL.

Thanks and Regards,

Andrew Walsh


Ref.

(1) Konovalov et. al (March 2012), Proposal to adopt a quality flag 
scheme standard

for oceanographic and marine meteorological data, Version 1.2.

- Original Message - From: "John Caron" 


To: "andrew walsh" 
Sent: Thursday, April 05, 2012 10:44 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



ok

On 4/4/2012 6:42 PM, andrew walsh wrote:

Hi John,

Thank for your offer to check a sample netCDF CTD data file. At 
moment we don't have

some real .nc file but when we do I can send you a file for checking.

Andrew

- Original Message - From: John Caron
To: cf-metadata@cgd.ucar.edu
Sent: Wednesday, April 04, 2012 5:55 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions 
v1.6



Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I 
can check if it works in the CDM with the new 1.6 conventions, and 
maybe give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
"joinNew " and "joinExisting" NCML aggregation. I call it 
"syntactic aggregation" because it doesnt know what its 
aggregating, and the homogeneity requirements are strict. 2) 
"Feature Type collections" (aka "semantic aggregation") are the 
more recent development. These understand the coordinate 
information of the data, and so can handle variations in the 
representation, eg ragged arrays, scalar coordinates, etc. In this 
case, the CDM understands a dataset as a collection of "features 
objects", which can be stored in a collection of files.  The 
interfaces to these collections is still under development. Most 
current and future work in the CDM is in this category.


John

snip 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata








___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi andrew:

The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a 
single profile. Assuming that, you would use the following template:


http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696

note that lat, lon and time coordinates all have to be scalars.

John

On 4/18/2012 8:24 PM, andrew walsh wrote:

Hi John,

We have now constructed a real netCDF file for you to check. QC flag
variables have been added in. The QC flagging method is based on the
proposed IODE scheme in (1) below.

Attached are the sample .nc file, text (ncdump) of same and a document
specifying the CDL.

Thanks and Regards,

Andrew Walsh


Ref.

(1) Konovalov et. al (March 2012), Proposal to adopt a quality flag 
scheme standard

for oceanographic and marine meteorological data, Version 1.2.

- Original Message - From: "John Caron" 
To: "andrew walsh" 
Sent: Thursday, April 05, 2012 10:44 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6



ok

On 4/4/2012 6:42 PM, andrew walsh wrote:

Hi John,

Thank for your offer to check a sample netCDF CTD data file. At 
moment we don't have

some real .nc file but when we do I can send you a file for checking.

Andrew

- Original Message - From: John Caron
To: cf-metadata@cgd.ucar.edu
Sent: Wednesday, April 04, 2012 5:55 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6


Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I can 
check if it works in the CDM with the new 1.6 conventions, and maybe 
give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
"joinNew " and "joinExisting" NCML aggregation. I call it "syntactic 
aggregation" because it doesnt know what its aggregating, and the 
homogeneity requirements are strict. 2) "Feature Type collections" 
(aka "semantic aggregation") are the more recent development. These 
understand the coordinate information of the data, and so can handle 
variations in the representation, eg ragged arrays, scalar 
coordinates, etc. In this case, the CDM understands a dataset as a 
collection of "features objects", which can be stored in a 
collection of files.  The interfaces to these collections is still 
under development. Most current and future work in the CDM is in 
this category.


John

snip 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] identification of vector components

[John Caron]

On 4/24/2012 4:16 AM, Hedley, Mark wrote:

There are particular issue here with format interoperability and conversion 
with respect to phenomenon.  In GRIB2, for example, there are codes which 
identify vector components:
   Wind direction (from which blowing) degree true
   Wind speed ms
   u-component of wind ms
   v-component of wind
but no code for eastward_...

The current situation poses significant challenges for GRIB - CF 
interoperability.

On 4/24/2012 10:42 AM, Hedley, Mark wrote:

>  Can similar tests be used with Grib conversion? I assume that Grib must 
provide sufficient metadata to indicate whether the grid is lon-lat.

Yes and your assumption is correct, there is sufficient information to deliver 
interoperability, but implementing this is adding complexity which I feel is 
not necessary and may cause future issues maintaining the special cases.  One 
of the reasons I raise GRIB is that it treats vector component as one 
phenomenon, as I have described, not two as is done in CF currently.


A few comments on my favorite topic, GRIB:

AFAIU, like CF, GRIB does not associate, say, the u and v components of 
wind together, or identify them as vectors. You "just have to know" what 
the naming conventions are to make the association.


More generally, it seems that the crux of the issue is that one wants to 
know if the vector components of, say, wind, are reletive to the grid or 
to lat/lon.  GRIB does encode this info in the GDS, the equivalent of 
the CF projection. Again you just have to know if that information is 
actually pertinent to the GRIB record you are looking at. If you know 
this, one could use it in the correct standard CF name.


In terms of mapping between GRIB and CF standard_names, these is 
unfortunately no such up-to-date mapping, AFAIK. It would be very useful 
and quite challenging to do so. One reason its hard is that a GRIB 
"variable" must be discovered through hueristics that involve the 
parameter name, the level type, and a whole bunch of other things that 
you "just have to know". Some details for the foolhardy are at a URL you 
just have to know, but since I know it, here it is:


http://www.unidata.ucar.edu/blogs/developer/en/entry/dataset_schemas_are_lost_in

John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] code that does semantic checking of CF headers

[John Caron]

On 4/20/2012 4:54 AM, Gaffney, Sean P. wrote:

Hi everyone, Thanks for all your feedback.

It's made things clearer for me now. Just to summarise then, a suite of 
attributes that would provide a precise numeric description of the data within 
a variable could then be (ignoring things such as scaling factors and offsets)

Actual_min, actual_max or actual_range: which describe the actual variable 
values held in the file, their minimum and maximum (e.g. for 
sea_water_salinity, I could have data with actual_min of 25.732, actual_max of 
34.994)
Valid_min, valid_max and valid_range: these detail the feasible range of data, 
so for salinity they could range from 0 to 41.5 (if following the BODC 
parameter vocabulary)
_FillValue which then presets the value that should be used for absent data - 
in the case of salinity, if keeping to the BODC parameter vocabulary, a value 
of -1.
My own idiom is to use _FillValue only for values that are never 
written, which was its original intent. If I need missing values then I 
add an explicit attribute "missing_value"; if there are multiple missing 
values, the attribute can be a vector.


So, I never use _FillValue explicitly, nor valid_min or valid_max, and 
always use missing_value if needed.


I always code to write every value out, never relying on unwritten 
values == missing values. That means I can set fill to off, which can 
save a lot of time when writing (up to 2x)


As I say, thats just my own personal idiom for writing netcdf.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] code that does semantic checking of CF headers

[John Caron]

On 4/19/2012 2:11 PM, Dave Allured wrote:

Sean,

I run into this frequently, especially with files that do not come
from carefully crafted official archives.  I regard all flavors of
range attributes as frequently unreliable.  I think best practice is
to extract ranges directly from the coordinate values when plotting
data on the fly, and pay no attention to incorrect secondary metadata.
  This should be simple to code and have no performance penalty.

--Dave


In this case, its the data values that have to be read, in order to know 
what the data range is, which may be expensive.
As long as they are hints that can be modified by the user, seems safe 
to use.


OTOH, valid_min, valid_max are quite dangerous if they are not correct, 
as seth found out. Generic software really has no choice but to respect 
them. I recommend leaving them off if you dont actually need them.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] code that does semantic checking of CF headers

[John Caron]

On 4/19/2012 9:13 AM, Gaffney, Sean P. wrote:

Hi all,

My name is Sean Gaffney, from the British Oceanographic Data Centre, and I'm 
working on a project dealing with numerical model data that are in CF compliant 
NetCDF, so I thought I'd sign up to the community.

The project I am working on aims to develop a web-based delivery system for 
oceanographic numerical model data and has a module which allows visualisation 
of the data. We've been using test CF data to fine-tune some of the technical 
aspects of this visualisation.

I have a particular issue at the moment which I hope someone out there might be 
able to assist me with.

My problem started when I found that the test CF data were passing the BADC CF 
compliance checker, but not visualising properly. A check with the people who 
developed the visualisation module led to the discovery that, while the CF 
metadata were formatted correctly, the actual values within the metadata were 
incorrect e.g. the valid_min and valid_max attributes for both the latitude and 
longitude and dimensional variables had values which did not reflect the actual 
range of data in the file. The visualisation was setting itself up based on the 
values stored in the attributes and was therefore not displaying any data.


you should not use valid_min, valid_max, use actual_min, actual_max to 
document the actual range of data. Unnecessarily using valid_min, 
valid_max means that conforming software has to check each data value.


this hasnt made it into the docs yet, but you can see

https://cf-pcmdi.llnl.gov/trac/ticket/31



Has anyone in the CF community come across this sort of issue before and if so, 
what solutions would you recommend? My initial thoughts were that I'd have to 
develop some sort of code which interrogates the data file and compares the 
entries in the CF metadata header against the actual data values in the file, 
but I'd be interested to see what people think. Please bear in mind that I 
won't actually be generating model runs myself, but will be receiving data from 
people that have done so and need to know that I'm being given valid data and 
metadata.


In my experience, one can have a valid CF file, but the metadata is 
incorrect or insufficient to georeference data, since theres no 
requirement in CF that data actually be georeferenced.


The CDM validator gives feedback on what data it can georeference 
/visualize. Since we try to keep up with CF, its a good double check for 
this problem, a complement to the CF validator(s).


http://motherlode.ucar.edu:8080/cdmvalidator/validate.html

However, it would not have caught your problem. For that, you would want 
to visualize it and see that the picture was not what you would expect. 
You could automate that process in an integration testing framework.




Sorry for making my first message to the CF community so long.

Looking forward to your responses

Yours

Sean Gaffney
BODC



Sean Gaffney
Data Scientist
British Oceanographic Data Centre
Joseph Proudman Building
6 Brownlow Street
Liverpool
L3 5DA
UK
+44 (0)151 795 4950




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] identification of vector components

[John Caron]

Heres my two cents:

The meaning of the x_coordinate and y_coordinate is actually well 
defined. But it does not mean x=east and y=north. It means the input to 
the projection function proj(x,y) -> (lat,lon), which are defined in 
appendix F, with pointers to reference software. AFAIU, these 
coordinates are only meaningful in the context of this projection. Of 
course, for some projections it does mean east and north, but not always.


im not sure how that impacts this discussion on standard names, but i 
want to be clear that x and y are well defined. Its just that their 
properties vary depending on the projection.


John

On 4/18/2012 7:49 AM, Hedley, Mark wrote:

Hello Bryan


Sorry, silence doesn't mean consent.

I didn't think it did, but prodding that notion can encourage people to pitch 
in.

My reasoning is that I do not think it is the responsibility of the standard name to 
define what is meant by x.  The initial parts of the definition in the table: 
'"x" indicates a vector component along the grid x-axis' ... ', positive with 
increasing x'; say everything there is to say.

It is the responsibility of the coordinate and grid_mapping variables to define 
what 'x' means.

Rather than this we have the case now where significant metadata inspection on 
coordinate system and coordinate is required to determine the correct 
standard_name from two mutually exclusive choices when writing CF NetCDF.  This 
feels to me to be an unnecessary complication which delivers little benefit 
from a data and metadata definition perspective.


If you are importing something where x is used as the coordinate, and it is 
longitude, then why not put that in other metadata?

I would say that I have defined this explicitly, using the approach I propose.  
I define that the data variable is x-wind and I define that x is longitude, 
therefore I can infer that the x-wind data variable is eastward wind, with 
respect to the defined grid_mapping.  Forcing me to put it in the standard_name 
adds complexity to software which writes data and increases the opportunity for 
data to be written incorrectly.

For example, does the cf_checker cross reference the 'x' coordinate and any 
standard names to ensure that datasets defined with respect to a true longitude 
coordinate variable do not use standard names with the 'x' modifier?


The you say x, I say x, and we both mean different things, is what we need to 
avoid

This cannot be avoided, in almost all cases x means different things in 
different datasets. It can even mean different things in the same file.


in particular we must not change definitions of existing quantitities.

I don't think that it is safe to make that strong a statement on definition 
changes over time.  I can understand the desire to avoid invalidating datasets 
by narrowing definitions after they are defined; but I don't think that a 
constrained broadening of the definition of a modifier should be refused on 
principle.  Such changes sometimes need to take place to keep the standard as 
applicable to its community as possible.


That's not to say 'eastward' isn't a useful standard name: there is a good case 
for model intercomparison, as there is no guarantee that my 'x' is anything 
like your 'x' for a given dataset: we can agree to publish data as 'eastward' 
to allow quick and easy intercomparison.

even this becomes slightly problematic at small scales, as eastward is with 
respect to a coordinate reference system, so my east may be subtly different 
from yours.

many thanks
mark

-Original Message-
From: Bryan Lawrence [mailto:bryan.lawre...@ncas.ac.uk]
Sent: Wed 18/04/2012 11:34
To: cf-metadata@cgd.ucar.edu
Cc: Hedley, Mark
Subject: Re: [CF-metadata] identification of vector components

Hi Mark

Sorry, silence doesn't mean consent.

I think it is exactly the place of standard names to be completely proscriptive 
about what terms mean.

The you say x, I say x, and we both mean different things, is what we need to 
avoid, and in particular we must not change definitions of existing 
quantitities.

Admittedly, your change wouldn't strictly change anything retrospectively, 
since it's an inclusive change, but it's probably a dangerous thing to do. (My 
sense of deja vu tells me we've been here before, and I may even have been on 
the other side of the argument :-).

If you are importing something where x is used as the coordinate, and it is 
longitude, then why not put that in other metadata? The point of the CF 
standard is that it just that 

Bryan


There have not been any responses to this post in the last 10 days.

I know that this is a dangerous philosophy, but can I suggest that, in this 
case, silence equals consent?

If it is, I would like to see these amendments in the standard_name 
publications as soon as possible.  Would this cause concern?

many thanks
mark

-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu on behalf of Hedley, Mark
Sent: Thu 05/04/2012 17:35

[CF-metadata] need reference for rotated pole grid mapping

[John Caron]

This grid mapping has no reference implementation or any other 
documentation. I think we have to have something. Can anyone supply?



   Rotated pole

|grid_mapping_name|  = rotated_latitude_longitude



/Map parameters:/

 *

   |grid_north_pole_latitude|

 *

   |grid_north_pole_longitude|

 *

   |north_pole_grid_longitude|- This parameter is option (default
   is 0).

/Map coordinates:/

   The rotated latitude and longitude coordinates are identified by
   the|standard_name|attribute
   values|grid_latitude|and|grid_longitude|respectively.

/Notes:/


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Short guide to writing files using CF 1.6 discrete sampling features Conventions

[John Caron]

Im attempting to help data writers use the discrete sampling features 
Conventions in the CDM:


http://www.unidata.ucar.edu/software/netcdf-java/reference/FeatureDatasets/CFpointImplement.html

feedback appreciated!
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Using add_offset & scale_factor with coordinate variables

[John Caron]

On 4/11/2012 3:59 PM, Jonathan Gregory wrote:

Dear Randy

While I understand why this would be convenient for you, I'm inclined to think
that your convenience would be outweighed by the inconvenience to the users of
the data in this case. I don't expect there is any existing CF-aware analysis
software which would expect to unpack coordinate variables, so the likely
consequence is that the coordinates would be presented to the user as packed
values, which isn't what you intend.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Hi all:

Well, the netcdf-Java library will unpack coordinate variables, so any 
applications built with it will also.


If one already knows how to unpack data variables, its not any harder to 
unpack coordinate variables.


I guess the question is: what are the CF-aware analysis software that we 
know of?


John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] The use of CF cell constructs for point data

[John Caron]

On 4/9/2012 1:07 PM, Jonathan Gregory wrote:

Dear Randy


The GOES-R system will be producing a hemispheric lightning detection product.

It will be an array of lightning detection that occur within some number of 
seconds across the western hemisphere (i.e. it is not a gridded product).

Is is a timeseries of values, for probability or number of occurrence or
yes/no - something like that? If so, I don't think there's a featureType for
it yet. It's a timeseries which doesn't apply to a station. It would be
among the list of things not covered by chap 9 yet (see sect 9.1). But you
don't need to give it a featureType. Chap 9 is primarily for datasets which
need to use the techniques provided there for saving space.


hmm, my reading is that this would be an appropriate "point" feature 
type. Jonathan, not sure why you think not?





A lightning detection has a center location, and needs to be associated with an 
area and an interval of time.

(2) Use the cell "bounds" construct to capture the time interval.

Fine. cell_methods should say "time: sum", where time is the name of the time
dimension. From your description, I think "sum" is appropriate because this
quantity is extensive in time. The probability would be larger if the interval
were longer. See App E.


(3) Use the cell "cell_measures" construct to capture area (i.e. ":cell_measure 
= area: lightning_detection_area").

That's not quite right, I think. The entry is to indicate what kind of
statistic this is, and I would say that again this is extensive. If you
considered an area smaller than the W hemisphere, the probability would be
smaller, for instance. So I would suggest "area: sum".


if you are trying to describe the location bounds, a "bounds" on the 
lat, lon might be one way to do it?


or perhaps you just want to add some notion of error associated with the 
location?



John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6

[John Caron]

Hi all:

Lets see, I havent followed the entire conversation, but:

1) Andrew if you can send me a sample file (not just the CDL) I can 
check if it works in the CDM with the new 1.6 conventions, and maybe 
give you some advice from my POV.


2) Aggregation in the CDM comes in 2 flavors. 1) The original 
implementation simply appends multidimensional arrays together, eg 
"joinNew " and "joinExisting" NCML aggregation. I call it "syntactic 
aggregation" because it doesnt know what its aggregating, and the 
homogeneity requirements are strict.  2) "Feature Type collections" (aka 
"semantic aggregation") are the more recent development. These 
understand the coordinate information of the data, and so can handle 
variations in the representation, eg ragged arrays, scalar coordinates, 
etc. In this case, the CDM understands a dataset as a collection of 
"features objects", which can be stored in a collection of files.  The 
interfaces to these collections is still under development. Most current 
and future work in the CDM is in this category.


John

On 4/3/2012 12:48 PM, Upendra Dadi wrote:

Hi,
   I think Jim is talking about NCML (virtual) aggregation. THREDDS 
can aggregate even when the variable is a scalar using "joinNew". But 
I think THREDDS can aggregate only over regular arrays. That is, all 
the dimensions other than over which a variable is aggregated should 
be of same size across all the files. This is possible, for example, 
only if all the profiles have same depth( or pressure) levels. Which 
in general is not true. NCML aggregation I guess is of limited use 
when dealing with jagged arrays. But I agree with Jim in that ability 
to aggregate should be an important consideration when coming up with 
a representation. Physical aggregation is still possible. But I prefer 
virtual aggregation, since letting data to be stored in individual 
files in often operationally convenient, but users benefit from 
aggregated views. I wonder if NCML could be extended to be able to 
aggregate jagged arrays into incomplete multidimensional array 
representations.  I heard that ERDDAP has similar capability though I 
don't think it has anything like NCML where users can create views 
over remote data, not sure though.


Upendra



On 4/3/2012 1:27 PM, Jim Biard wrote:

Hi.

A colleague of mine has told me that if you use size-1 array for a 
variable, then data servers such as THREDDS can aggregate the 
variable over multiple files and deliver a single file in which the 
variables will be arrays of size > 1.  He said that if a variable is 
a scalar, THREDDS won't do this.  (I don't mess with THREDDS, so I am 
just parroting what he said.)


If this is correct, then you might want to consider this point when 
deciding which way to represent coordinates.


Grace and peace,

Jim

On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi > wrote:


Andrew,

However I have another small concern about all of this. Given
there is more than
1 way to treat dimensions does the netCDF plotting and
visualising sofware out
there understand both approaches. That is can we expect
something like
ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure
and work OK with BOTH approaches to coding the netCDF?

  Software like ncBrowse do not understand CF conventions. They
can read netCDF files, but wouldn't know how to interpret the
attributes like "coordinates". So if you want to plot the profile
location on a map, for example, it wouldn't be able to do it by
itself. It would need to be told how to interpret the
coordinates.  A CF based software wouldn't have to be supplied
with the additional semantics since it can read from the file by
itself. But if you want to plot salinity vs pressure,  software
like ncBrowse can already do it since lot of these software make
it easy to make plots based on a shared dimension.  And here
salinity and pressure share the same dimension.  So I guess, the
correct answer to your question is - it depends on what kind of
plot or task you want to do and if the software can understand CF
conventions.

Upendra




Thanks and Regards All,
Andrew Walsh

- Original Message -
*From:* Lowry, Roy K. 
*To:* Upendra Dadi  ; andrew
walsh 
*Cc:* Luke Callcut  ;
cf-metadata@cgd.ucar.edu  ;
sdn2-t...@seadatanet.org 
*Sent:* Tuesday, April 03, 2012 9:31 AM
*Subject:* RE: [CF-metadata] Ocean CTD data following CF
Conventions v1.6

Hi Upendra,
I like the idea of a station dimension.  It goes a long way
to resolving the issue raised in my response to Jim which
was based on the tunnel vision of having pre

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

To answer this concern, I would agree to modify the statement

"Applications are free to assume that data is missing where the auxiliary 
coordinates are missing"

to

"Applications should treat the data as missing where the auxiliary coordinates are 
missing"

My concern is that we shouldnt make a file "non CF compliant" just because the 
data provider would like to store data values where there arent coordinate values. But 
telling them that standard software _will_ ignore them seems good.



On 3/29/2012 9:47 AM, Rich Signell wrote:

Jonathan,

+1 on  your idea of only identifying variables as aux coordinate
variables once they have valid values at valid data locations.

-Rich

On Thu, Mar 29, 2012 at 11:32 AM, Jonathan Gregory
  wrote:

Dear Jim

We are discussing auxiliary coordinate variables. They do not have to be
1D or monotonic. Those requirements apply to coordinate variables in the
Unidata sense. CF distinguishes these two concepts in Sect 1.2.


The point is, the information in the variable *is* coordinate information,

I would say, if it's missing, it's not information.


What if we say something along the lines of, "Applications should treat the
data as missing where the auxiliary coordinates are missing when plotting
data."?  Would that resolve the problem?

Plotting is not the only thing that an application might wish to use it for.
If we said, more generally, "Applications should treat the data as missing for
all purposes where the aux coord variables are missing", it would be almost
the same as not allowing missing data in aux coord vars, since there would be
no point in providing a data value if it was not permitted to use it.

Although I am arguing one side, I could be convinced either way. But it does
feel unsafe to me at present.

Cheers

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata





___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/28/2012 10:49 AM, John Caron wrote:
I think we have a number of valid use cases for missing data in aux 
coordinates, and i would vote to allow that. 


sorry i didnt really answer jonathan's concern. I would suggest this 
wording:


Auxiliary coordinates do not have to be monotonic or have unique values, 
and may contain missing values. Data providers may choose to put valid 
data where the auxiliary coordinates are missing, but applications are 
free to assume that data is missing where the auxiliary coordinates are 
missing.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/28/2012 6:26 AM, Jonathan Gregory wrote:

Dear all

Brian has a good point that sect 5.3 allows that there might be missing data
in aux coord variables. Appendix A - which is equally ancient in the story of
CF :-) - is not consistent with this, because it didn't allow _FillValue or
missing_value atts for coordinates until sect 9 was introduced in version 1.6.
But never mind the history, the point is to clarify what we need now.

In Randy's, Rich's and Bert's examples, if I understand correctly, there are
non-existent points, at which both data and aux coord vars are missing values.
That is also what sect 9 requires. I don't see any problem with this.

Nan's example is different, because it has missing values in aux coord vars
at points where there is non-missing data. If we all agree that this is OK
too, then fine. Speaking for myself, I could agree to it, but I'm less happy,
because clearly the aux coord var is not doing its job of locating the data.

I think you said, Nan, that you might fill them in at some later stage. At
that stage, they should certainly be aux coord vars. Before they are filled
in, of course I am not saying they should be excluded from the file, but I
am asking if they should be regarded as data, rather than coordinates. A
pressure value which was destined to be an aux coord var, but is actually a
data variable measured by a sensor and has missing values in it, could be
named by the ancillary_variables attribute. It is really per-point metadata,
which is what ancillary_variables are
(http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#ancillary-data)
but you could argue it should not be regarded as an aux coord var if it can't
provide information for every point where there is data.

I'm just asking. I don't have a very strong opinion about this, but I'd like
to know if others have the same concern that I do.

My existing ticket (85) connected with this subject is a defect ticket. It
is only able to correct mistakes. It can't make a substantive change to the
convention, just clarify it. If we can decide easily how to clarify it, that
is fine. I'll amend the ticket if we have a clear decision. Otherwise, we
should use a different ticket.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


I think we have a number of valid use cases for missing data in aux 
coordinates, and i would vote to allow that.


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

Hi all:

From a CDM developer perspective, an auxiliary coordinate is "just as 
good" as a regular coordinate variable. The extra requirements on 
coordinate variables are helpful in knowing when to optimize, eg 
monotonicity allows one to efficiently find the index given the 
coordinate value.


John

On 3/26/2012 10:05 AM, Jim Biard wrote:
I am working with satellite data, and I, for example, have timestamps 
that arrive in the data stream along with sensor measurements.  I can 
have independent missing values in both my time variable and my 
measurement variables.  I want to preserve all the incoming data, and 
the restriction on "true" coordinate variables having no missing 
values prevents me from designating my time variable as a coordinate 
variable.  If I want to represent the relationship between the time 
variable and the measurement variables, the only recourse I have is to 
designate the time variable as an auxiliary coordinate variable in my 
measurement variables.


In the observational world, you cannot always be assured of meeting 
all the restrictions imposed on "true" coordinate variables.  In fact, 
other restrictions imposed on coordinate variables might not be met 
(monotonicity, for example), even though the contents of the variable 
do function properly as coordinate information for another variable. 
 The only mechanism that I am aware of within CF to document the 
relationship is the auxiliary coordinate attribute.


On Mon, Mar 26, 2012 at 11:20 AM, Nan Galbraith > wrote:


Hi Jonathan -

For underway CTD profiles (gliders and floats, too, I'd think) if
the pressure
sensor fails intermittently, you can approximate Z by
interpolating over
time, assuming there are good values along the track.  In "final"
data, we might
do that, but we might like to present raw data in CF files, too.
So, yes, this data
would be useful, with fill values here and there in the pressure
record.

Are we getting into a grey area that might be outside the scope of
the CF
standard -  judgements made about the usefulness of data?   Having
all your
coordinates seems like an excellent  NetCDF "best practice" and
could certainly
be a "rule" for many tools, but it could preempt the use of the CF
standard for
some kinds of observational data.

Cheers -
Nan


On 3/26/12 10:48 AM, Jonathan Gregory wrote:

Dear Nan and John

It's a good thing we're having this discussion! In my
understanding, we have
always prohibiting missing data in aux coord vars, and in
section 9 we
explicitly allowed for the first time. Evidently we should be
clear, one way
or the other (which was one of the intentions of the defect
ticket I opened).

The restriction on coord vars not having missing data is, I
think, hard to
avoid because they are required to be distinct and monotonic.

In Nan's case:

For something like towed CTD data, you might have a period
of time where
data from the pressure sensor is missing. If neither the
coordinate or aux
coordinate can contain null values, does this mean the
only options are
interpolating Z or removing that section of data?

When the sensor is missing, does that mean the data can't be
geolocated? As
you know, geolocation is one thing CF tries hard to do. Is the
data useful if
you don't know where it comes from? Perhaps you know in some
other way?

In John's case

Consider a geosynch satellite with lat/lon aux
coordinates. the
nature of the image is that there are many points around
the outside
of the image that are not located on the earth. i dont see
any good
choices other than to put missing values there for lat/lon..

Bert has coincidentally mentioned a similar case (not on the
list). I tend to
agree that if index space contains locations that simply do
not exist, you
could put missing data in aux coord vars, but I think this
needs a change to
the CF convention.

To add insult to injury, it seems possible that there are
valid data
values at these locations. Not sure about that however.
Can anyone
with geosynch data expertise comment?

As in Nan's case, I am curious about how the data can be
useful if it doesn't
have a location.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu 
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



-- 

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/26/2012 2:24 AM, Jonathan Gregory wrote:

Dear Ros

Regarding this requirement:


9.6 Where any auxiliary coordinate variable contains a missing value, all
other coordinate, auxiliary coordinate and data values corresponding to that
element should also contain missing values.

Appendix A notes that missing data is allowed in aux coord vars only in the
case of discrete sampling geometries. This means the checker could regard it as
an error also if it finds any missing data in aux coord vars, unless a valid
featureType att is present, arising from 9.6, in addition to the above.

However, I think the convention should be clear that this is not allowed except
for discrete sampling geometries, so I'll open a defect ticket for that.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


I think this may be more complicated.

Consider a geosynch satellite with lat/lon aux coordinates. the nature 
of the image is that there are many points around the outside of the 
image that are not located on the earth. i dont see any good choices 
other than to put missing values there for lat/lon..


To add insult to injury, it seems possible that there are valid data 
values at these locations. Not sure about that however. Can anyone with 
geosynch data expertise comment?


It seems at minimum this is another case where missing values in aux 
coordinates are needed.



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-1.6 Conformance Requirements/Recommendations

[John Caron]

On 3/23/2012 1:59 PM, Jim Biard wrote:

Hi.

Jonathan's reply contained the section:

9.6 Where any auxiliary coordinate variable contains a missing
value, all
other coordinate, auxiliary coordinate and data values
corresponding to that
element should also contain missing values.


I thought I understood that missing values were forbidden for true 
coordinate variables.  Has this changed?
No, coordinate variable cant have missing values but auxiliary 
coordinates can.


This requirement seems wrong to me anyway.  If the values in the data 
and auxiliary coordinate variables come from an external data source, 
it is entirely possible that you could have a measurement missing from 
one without it being missing from the other.  Why force a missing 
value into the data when you might, in fact, have a valid value?



An auxiliary coordinate is important to have for valid data, not some 
optional information. If it is optional, dont make it an auxiliary 
coordinate.


Allowing missing values in auxiliary coordinates is very useful, eg see 
the discrete sampling proposal, allowing you to use a rectangular array 
to store ragged arrays.



John
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Example 5.10. British National Grid has wrong parameter names

[John Caron]

Example 5.10. British National Grid

  int crs ;
crs:grid_mapping_name = "transverse_mercator";
crs:semi_major_axis = 6377563.396 ;
crs:semi_minor_axis = 6356256.910 ;
crs:inverse_flattening = 299.3249646 ;
crs:latitude_of_projection_origin = 49.0 ;
crs:longitude_of_projection_origin = -2.0 ;
crs:false_easting = 40.0 ;
crs:false_northing = -10.0 ;
crs:scale_factor_at_projection_origin = 0.9996012717 ;


but Appendix F has:

Transverse Mercator
grid_mapping_name = transverse_mercator

Map parameters:
  scale_factor_at_central_meridian
  longitude_of_central_meridian
  latitude_of_projection_origin
  false_easting
  false_northing

I think we can presume that

  scale_factor_at_central_meridian == scale_factor_at_projection_origin
  longitude_of_central_meridian == longitude_of_projection_origin

central_meridian is probably clearer; so maybe just update the example ??

or we can add this amendment to ticket #75= "fix documentation and 
definitions of 3 grid mapping definitions"

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] FW: GOES-R metadata prototype

[John Caron]

hi michael

1) There are a few typos (see attached) that made it invalid NcML. also 
".ncml" suffix is preferred over ".xml"
2) for cdm_data_type, i might not call this a grid, maybe "image" is 
better. the satellite group should probably come up with a feature type 
name for geostationary data.
3) dont use valid_min / valid_max unless you really have data stored 
outside that range that you want marked as missing.

use actual_min/actual_max if you want to document actual ranges.

if you can give me a real file with data and coordinates in it, i can 
test the projection parameters.


john


On 3/7/2012 3:30 PM, Carlomusto, Michael wrote:

_
*From:* Carlomusto, Michael
*Sent:* Wednesday, March 07, 2012 5:20 PM
*To:* 'CF-metadata@cgd.ucar.edu'
*Subject:* GOES-R metadata prototype
The attached NcML file has been vetted by a small group of people from 
the CF metadata list, especially with regard to the representation of 
the proposed GOES projection.

I would like to submit it for comments to the larger group.
I encourage a critical evaluation.
Thanks.
Mike Carlomusto
*/"This information is only intended for the use of the individual or 
entity named in this email. If you are not the intended recipient, you 
are hereby notified that any disclosure, copying, distribution, or use 
of the contents of this information is strictly prohibited. If you 
have received this email in error, please notify me and delete it 
immediately."/*

Michael Carlomusto
mcarl...@harris.com
Harris Corp.
Government Communications Systems Division (GCSD)
Melbourne, FL
(321) 309-7905


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2  http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd";
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

  
  

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

  
  
  
  
  

  
  








  

  








  

  
  






  

  
  




  

  
  











  

  











  

  
  












  

  












  

  
  














  

  
  










  

  















  

  
  








  

  









  

  











































  

  



















  

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] machine-accessible CF compliance checker?

[John Caron]

Hi all:

If the checker can read from opendap, then putting the files on an 
opendap server might be helpful, in

that only the metadata is read instead of the entire file.

There are a few difference among servers (Hyrax, TDS, GDS, etc) but im 
pretty sure this would work for
netcdf files well enough for the checker's needs. could be worth finding 
out.


john

On 2/29/2012 7:55 AM, Bentley, Philip wrote:

Hi Stephen,

I may well be wrong, but I suspect that one of the reasons - perhaps the
main reason - why folks would like access to a stand-alone cfchecker
utility is the large size of their netcdf files. (It's certainly the
reason we use a local version.)

As you know only too well (from CMIP5), most of the netcdf files we're
dealing with these days run into the high 100s of MB or low GBs. It's
not really practical to upload such files in order to perform a CF
compliance check. At least not on more than a few files.

I wonder, therefore, if a possible solution is to develop an alternative
checker utility that operates not on the netcdf file itself, but rather
on the header info output by the ncdump -h command. Generating and
uploading those would be trivial by comparison.

Off the top of my head (and having given it almost zero thought!) the
only test that I can think of which examines the actual data values is
the check for coord vars being monotonic. But I suspect there are others
I'm unaware of.

If nothing else, such a utility might have some value as a 'first pass'
checker.

Any mileage in this? Apologies if it's already done the rounds!

Regards,
Phil



-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
stephen.pas...@stfc.ac.uk
Sent: 29 February 2012 11:39
To: christopher.s.lyn...@nasa.gov; cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] machine-accessible CF compliance checker?

Hi Christopher,

Are you looking for a way to run the cfchecker locally or
submit checking requests to one of the websites in a script?

I think both would be possible with a little help.  The
cfchecker is now available on the Python Package Index at
http://pypi.python.org/pypi/cfchecker.  It needs a little
help to work out of the box -- you need to configure UDUNITS
and the location of several CF XML files.  It's on my
long-term TODO list to make this configuration easier.

Also the CF-checker at BADC and NCAS-CMS has a very simple
HTTP interface that could be scripted.

Cheers,
Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK


-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
Lynnes, Christopher S. (GSFC-6102)
Sent: 28 February 2012 16:03
To: cf-metadata@cgd.ucar.edu
Subject: [CF-metadata] machine-accessible CF compliance checker?

I know about the web sites where you use a form to upload
files to check CF compliance, but is there a service
somewhere that can be accessed by a script?
--
Dr. Christopher Lynnes NASA/GSFC, Code 610.2phone:
301-614-5185


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
Scanned by iCritical.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] "threshold" variable in the coordinates attribute

[John Caron]

in example 7.11:

variables:
  float n1(lat,lon);
n1:standard_name="number_of_days_with_air_temperature_below_threshold";
n1:coordinates="threshold time";
n1:cell_methods="time: minimum within days time: sum over days";
  float n2(lat,lon);

n2:standard_name="spell_length_of_days_with_air_temperature_below_threshold";
n2:coordinates="threshold time";
n2:cell_methods="time: minimum within days time: maximum over days";
  float threshold;
threshold:standard_name="air_temperature";
threshold:units="degC";
  double time;
time:climatology="climatology_bounds";
time:units="days since 2000-6-1";
  double climatology_bounds(time,nv);
data: // time coordinates translated to date/time format
  time="2008-1-16 6:00";
  climatology_bounds="2007-12-1 6:00", "2000-8-2 6:00";
  threshold=0.;


im somewhat surprised by seeing the "threshold" variable in the 
coordinates attribute. I didnt see a discussion of this at


https://cf-pcmdi.llnl.gov/trac/ticket/33

can anyone explain what this means?
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] defects in example 7.11, 7.12

[John Caron]

Possible defects:

1) Example 7.11. Extreme statistics and spell-lengths

 climatology_bounds="2007-12-1 6:00", "2000-8-2 6:00";

should be

 climatology_bounds="2007-12-1 6:00", "2008-3-1 6:00";

perhaps clearer as

 climatology_bounds="2007-12-1", "2008-3-1";

2) flip order of 7.11 and 7.12, since 7.12 refers to 7.10.

3) 7.12 Temperature for each hour of the typical climatological day

 temperature:cell_methods="time: mean within days ",
  "time: mean over days time: mean over years";

may be confused with array of strings, perhaps should be:

 temperature:cell_methods="time: mean within days time: mean over days 
time: mean over years";



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] How to represent "6-hourly diurnal monthly means" in CF ?

[John Caron]

Thanks, Jonathan.

On 2/21/2012 11:18 AM, Jonathan Gregory wrote:

Dear John


However, im not sure what this sentence in 7.4 means: "When
considering intervals within days, if the earlier time of day is
equal to the later time of day, then the method is applied to a full
24 hour day."

It doesn't matter in the instantaneous case, but it does matter for statistics
that apply to the whole cell, like means and accumulations, such as


 precipitation:cell_methods="time: sum within days time: mean over days";

If the sums were for 24-hour intervals, the bounds might be


   climatology_bounds="2008-08-01T00:00Z", "2008-09-01T00:00Z",

Here, the time of day is the same (00:00Z) at both ends. That means it
applies to full days. That's what the bit quoted is talking about. However,
this doesn't arise in your six-hourly case.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] How to represent "6-hourly diurnal monthly means" in CF ?

[John Caron]

On 2/21/2012 7:48 AM, Jim Biard wrote:

John,

Is this actually climatology data?  If it were climatology data, it 
would usually be the average for a given hour, day, or whatever, where 
the average is taken over a decade or so.  I think you may just need 
to apply regular bounds.


You might be right, Im trying to get clarification on this. Thanks for 
your comments.


John



Grace and peace,

Jim


On 2/20/2012 12:27 PM, John Caron wrote:
Im working with NOAA's CFSR dataset, reading it into the CDM, which 
essentially means converting GRIB into CF/netCDF.


ref:
http://nomads.ncdc.noaa.gov/data.php#cfs-reanal-data
http://dss.ucar.edu/datasets/ds093.2/

from NCAR's DSS page:

 "CFSR monthly atmospheric, oceanic and land surface output products 
are available as 6-hourly diurnal monthly means (00Z, 06Z, 12Z, and 18Z)"


If I understand this correctly, the monthly mean data is as follows:

  for each month:
  for each day of the month:
compute separate averages of the data at the 0,6,12, and 18Z 
time coordinate.


So one ends up with 4 time coordinates per month, and the data 
represents the average for that month at that time. The time 
coordinate is not a real calendar date, for example for the month 
2008-08, the time coordinate is not really


   2008-08-01T00:00:00.000Z
   2008-08-01T06:00:00.000Z
   2008-08-01T12:00:00.000Z
   2008-08-01T18:00:00.000Z

but more like:

   2008-08-*T00:00:00.000Z
   2008-08-*T06:00:00.000Z
   2008-08-*T12:00:00.000Z
   2008-08-*T18:00:00.000Z

where the * indicates an average over the days in that month.

From

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#climatological-statistics

It appears that CF would represent this as:
dimensions:
   time=4;
   nv=2;

  variables:
   float temperature(time,lat,lon);
 temperature:long_name="surface air temperature";
 temperature:cell_methods="time: point within days time: mean over days";
 temperature:units="K";

  double time(time);
 time:climatology="climatology_bounds";
 time:units="days since 2008-01-01";

  double climatology_bounds(time,nv);

data: 
   time="2008-08-01T00:00Z","2008-08-01T06:00Z","2008-08-01T12:00Z","2008-08-01T18:00Z" ;


   climatology_bounds="2008-08-01T00:00Z", "2008-08-31T00:00Z",
  "2008-08-01T06:00Z", "2008-08-31T06:00Z",
  "2008-08-01T12:00Z", "2008-08-31T12:00Z",
  "2008-08-01T18:00Z", "2008-08-31T18:00Z" ;
Using "point within days" because  we have an instantaneous quantity.

However, im not sure what this sentence in 7.4 means: "When 
considering intervals within days, if the earlier time of day is 
equal to the later time of day, then the method is applied to a full 
24 hour day."



Another case is when the data are on time intervals, for example 
"Total Precipitation Accumulation". In this case, the original data 
are on time intervals which i represent as (start,end) dates:


2008-08-01T00:00:00.000Z, 2008-08-01T00:00:06.000Z
2008-08-01T06:00:00.000Z, 2008-08-01T00:00:12.000Z
2008-08-01T12:00:00.000Z, 2008-08-01T00:00:18.000Z
2008-08-01T18:00:00.000Z, 2008-08-01T00:00:24.000Z
2008-08-02T00:00:00.000Z, 2008-08-02T00:00:06.000Z
2008-08-02T06:00:00.000Z, 2008-08-02T00:00:12.000Z
2008-08-02T12:00:00.000Z, 2008-08-02T00:00:18.000Z
2008-08-02T18:00:00.000Z, 2008-08-02T00:00:24.000Z
2008-08-03T00:00:00.000Z, 2008-08-03T00:00:06.000Z
...

which are are averaged over days, giving 4 averages for each month, 
which i denote as:


2008-08-*T00:00:00.000Z, 2008-08-*T00:00:06.000Z
2008-08-*T06:00:00.000Z, 2008-08-*T00:00:12.000Z
2008-08-*T12:00:00.000Z, 2008-08-*T00:00:18.000Z
2008-08-*T18:00:00.000Z, 2008-08-*T00:00:24.000Z

So perhaps this is:
dimensions:
   time=4;
   nv=2;

  variables:
   float precipitation(time,lat,lon);
 precipitation:long_name="surface air temperature";
 precipitation:cell_methods="time: sum within days time: mean over days";
 precipitation:units="K";

  double time(time);
 time:climatology="climatology_bounds";
 time:units="days since 2008-01-01";

  double climatology_bounds(time,nv);

data: 
   time="2008-08-01T03:00Z","2008-08-01T09:00Z","2008-08-01T15:00Z","2008-08-01T21:00Z" ;


   climatology_bounds="2008-08-01T00:00Z", "2008-08-31T06:00Z",
  "2008-08-01T06:00Z", "2008-08-31T12:00Z",
  "2008-08-01T12:00Z", "2008-08-31T18:00Z",
  "2008-08-01T18:00Z", "2008-08-31T24:00Z" ;
??




thanks for any insights,
 John




___
CF-metadata mailing l

[CF-metadata] How to represent "6-hourly diurnal monthly means" in CF ?

[John Caron]

Im working with NOAA's CFSR dataset, reading it into the CDM, which 
essentially means converting GRIB into CF/netCDF.


ref:
http://nomads.ncdc.noaa.gov/data.php#cfs-reanal-data
http://dss.ucar.edu/datasets/ds093.2/

from NCAR's DSS page:

 "CFSR monthly atmospheric, oceanic and land surface output products 
are available as 6-hourly diurnal monthly means (00Z, 06Z, 12Z, and 18Z)"


If I understand this correctly, the monthly mean data is as follows:

  for each month:
  for each day of the month:
compute separate averages of the data at the 0,6,12, and 18Z 
time coordinate.


So one ends up with 4 time coordinates per month, and the data 
represents the average for that month at that time. The time coordinate 
is not a real calendar date, for example for the month 2008-08, the time 
coordinate is not really


   2008-08-01T00:00:00.000Z
   2008-08-01T06:00:00.000Z
   2008-08-01T12:00:00.000Z
   2008-08-01T18:00:00.000Z

but more like:

   2008-08-*T00:00:00.000Z
   2008-08-*T06:00:00.000Z
   2008-08-*T12:00:00.000Z
   2008-08-*T18:00:00.000Z

where the * indicates an average over the days in that month.

From

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#climatological-statistics

It appears that CF would represent this as:

dimensions:
  time=4;
  nv=2;

 variables:
  float temperature(time,lat,lon);
temperature:long_name="surface air temperature";
temperature:cell_methods="time: point within days time: mean over days";
temperature:units="K";

 double time(time);
time:climatology="climatology_bounds";
time:units="days since 2008-01-01";

 double climatology_bounds(time,nv);

data: 
  time="2008-08-01T00:00Z","2008-08-01T06:00Z","2008-08-01T12:00Z","2008-08-01T18:00Z" ;


  climatology_bounds="2008-08-01T00:00Z", "2008-08-31T00:00Z",
 "2008-08-01T06:00Z", "2008-08-31T06:00Z",
 "2008-08-01T12:00Z", "2008-08-31T12:00Z",
 "2008-08-01T18:00Z", "2008-08-31T18:00Z" ;

Using "point within days" because  we have an instantaneous quantity.

However, im not sure what this sentence in 7.4 means: "When considering 
intervals within days, if the earlier time of day is equal to the later 
time of day, then the method is applied to a full 24 hour day."



Another case is when the data are on time intervals, for example "Total 
Precipitation Accumulation". In this case, the original data are on time 
intervals which i represent as (start,end) dates:


2008-08-01T00:00:00.000Z, 2008-08-01T00:00:06.000Z
2008-08-01T06:00:00.000Z, 2008-08-01T00:00:12.000Z
2008-08-01T12:00:00.000Z, 2008-08-01T00:00:18.000Z
2008-08-01T18:00:00.000Z, 2008-08-01T00:00:24.000Z
2008-08-02T00:00:00.000Z, 2008-08-02T00:00:06.000Z
2008-08-02T06:00:00.000Z, 2008-08-02T00:00:12.000Z
2008-08-02T12:00:00.000Z, 2008-08-02T00:00:18.000Z
2008-08-02T18:00:00.000Z, 2008-08-02T00:00:24.000Z
2008-08-03T00:00:00.000Z, 2008-08-03T00:00:06.000Z
...

which are are averaged over days, giving 4 averages for each month, 
which i denote as:


2008-08-*T00:00:00.000Z, 2008-08-*T00:00:06.000Z
2008-08-*T06:00:00.000Z, 2008-08-*T00:00:12.000Z
2008-08-*T12:00:00.000Z, 2008-08-*T00:00:18.000Z
2008-08-*T18:00:00.000Z, 2008-08-*T00:00:24.000Z

So perhaps this is:

dimensions:
  time=4;
  nv=2;

 variables:
  float precipitation(time,lat,lon);
precipitation:long_name="surface air temperature";
precipitation:cell_methods="time: sum within days time: mean over days";
precipitation:units="K";

 double time(time);
time:climatology="climatology_bounds";
time:units="days since 2008-01-01";

 double climatology_bounds(time,nv);

data: 
  time="2008-08-01T03:00Z","2008-08-01T09:00Z","2008-08-01T15:00Z","2008-08-01T21:00Z" ;


  climatology_bounds="2008-08-01T00:00Z", "2008-08-31T06:00Z",
 "2008-08-01T06:00Z", "2008-08-31T12:00Z",
 "2008-08-01T12:00Z", "2008-08-31T18:00Z",
 "2008-08-01T18:00Z", "2008-08-31T24:00Z" ;

??




thanks for any insights,
 John



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] When are the units attribute mandatory?

[John Caron]

Hi Egil:

My first thought is that its not a dimensional quantity. Why do you 
think it is?


John

On 2/16/2012 8:00 AM, Egil Støren wrote:

From the CF 1.6 Convention, chapter 3.1 Units:

"The units attribute is required for all variables that represent 
dimensional quantities (except for boundary variables defined in 
Section 7.1, “Cell Boundaries” and climatology variables defined in 
Section 7.4, “Climatological Statistics”)."


In the same document, in appendix H.3.4. Contiguous ragged array 
representation of profiles, the following example appears:


  int profile(profile) ;
  profile:cf_role = "profile_id";

This array obviously contains identification numbers. It is also a 
dimensional quantity and accordinly the units attribute is mandatory 
(none of the listed exceptions apply). Are there any unit acceptable 
by udunits that can be used in this case? The best I have found is 
"count" which I think is misleading.


Or should this and similar variables also be included in the list of 
exceptions above?


Best regards,

   Egil Støren
   met.no
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] MSG CPP standard names, projection and WMS service

[John Caron]

On 1/24/2012 3:12 AM, plieger wrote:

Hi John and Martin,

John: Thanks a lot for the information. I put the MSGnavigation properties in 
the file, and the
projection now fits much better than the vertical perspective projection. 
Attached you will see a
screenshot of the longitude variable with this projection. At the edges of the 
globe I still see a
small offset of 0-0.1 degrees, but this is changing more or less where you 
query.
I tinkered a little bit with the height_from_earth_center and the semi_major 
and semi_minor axis,
but did not result in a noticable change of the projection. The 2D latitude and 
longitude fields are
always provided with these files so the projection can be verified with these 
fields.


char projection
   projection:proj4_params = "+proj=geos +lon_0=0.00 +lat_0=0 +h=35807.414063 
+a=6378.169 +b=6356.5838" ;
   projection:grid_mapping_name = "MSGnavigation" ;
   projection:longitude_of_projection_origin = 0.f ;
   projection:latitude_of_projection_origin = 0.f ;
   projection:semi_major_axis = 6356584.f ;
   projection:semi_minor_axis = 6378169.f ;
   projection:height_from_earth_center = 4.216397e+07f ;
   projection:scale_x = 35785.83f ;
   projection:scale_y = -35785.83f ;




Hi Maarten:

Yes, it seems as good as Ive seen.

How did you generate your projection coordinates and your lat/lon 
coordinates?


John




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Convention attribute

[john caron]

On 12/22/2011 2:11 AM, Jonathan Gregory wrote:

Dear all

The existing Unidata recommendation is OK and we could incorporate it into
CF but it would help to be more precise, for instance: If the Conventions att
includes no commas, it is interpreted as a blank-separated list of conventions;
if it contains at least one comma, it is interpreted as a comma-separated list.
Blank-separated lists are more CF-like - many CF attributes use that syntax -
but obviously we can't insist that other conventions don't have blanks in their
names, and it would be simpler therefore to use a comma-separated list for
this attribute, despite the Unidata recommendation.

I see no problem with allowing multiple conventions except the important
proviso that if the file follows more that one convention it is the
responsibility of the data-writer to ensure there is no inconsistency between
the metadata following these conventions. That is, they must serve
complementary purposes. It would be impossible to check this automatically so
we have to depend on the data-writer doing it correctly.

Best wishes

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

I think this would solve the problem:

"If the Conventions att
includes no commas, it is interpreted as a blank-separated list of conventions;
if it contains at least one comma, it is interpreted as a comma-separated list."

I could also point out that reading software has a list of conventions it 
recognizes, so in practice one takes the result of this parsing and compares to 
a known list. also, the netcdf-4 data model allows attribute values to be a 1-d 
array of Strings.

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] inconsistencies in translating projection parameters OGC WKT <-> CF-1.5

[John Caron]

On 12/15/2011 8:03 AM, Etienne Tourigny wrote:

John, thanks for your answers.

Would anyone be kind enough to point me to real files that use the
projections mentioned in this thread?

Answers below, but here is a quick summary of my recommendations

1) LCC:  calculate the missing scale_factor from other parameters (no
changes to CF)

2) CEA: drop the scale_factor_at_projection_origin variant all
together, or add a scale_factor parameter to that variant.

3) PS: add an optional parameter (Scale factor at natural origin) that
defaults to 1 if absent

On Wed, Dec 14, 2011 at 2:21 PM, John Caron  wrote:

Hi Etienne:

Generally, CF has followed proj4 conventions as much as we could, and John
Snyder's reference book is often underneath:

http://pubs.usgs.gov/pp/1395/report.pdf

OGC specs tended to come from experts using the EPSG, and generally we are
less familiar with that.

some specific comments below, based on my own understandings.


On 12/13/2011 2:21 PM, Etienne Tourigny wrote:

Hi all,

I am opening a seperate thread at this deals merely with the problems
encountered in translating projection parameters to/from OGC WKT and
CF-1.5.

I would appreciate any guidance in resolving these issues.

In summary:

1) LCC (Lambert conformal): should include a scale_factor parameter in
the 1 standard parallel case
2) CEA (Lambert Cylindrical Equal Area): how to convert the
'scale_factor_at_projection_origin' alternative to standard parallel
to OGC WKT?
3) PS (Polar stereographic):This projection's parameters in CF-1 are
significantly different from OGC/PROJ.4 ,  CF-1 standard parameters
'Either standard_parallel or scale_factor_at_projection_origin' is
incompatible with OGC WKT, which requires both

More detailed information (reproduced here) can be found at
http://trac.osgeo.org/gdal/wiki/NetCDF_ProjectionTestingStatus

1) LCC 1SP (Lambert conformal)

The OGC WKT 'Lambert_Conformal_Conic_2SP' maps clearly to LCC in CF-1
with 2 std parallels.

The CF-1 spec claims that a 1SP version can be used, but it doesn't
include a 'scale_factor' as does the OGC WKT
'Lambert_Conformal_Conic_1SP'.

For import: Given discussion and example on #3324, and also support
from NARCCAP project: it seems the CF-1 LCC 1SP projection is a
'degenerate' LCC-2SP with both SP's at the same location. Thus seems
we can read with both 'std_parallel' (1 value) and
'latitude_of_projection_origin' mapping to latitude_of_origin (WKT)
and setting scale_factor=1? This is the code's current behaviour
(though perhaps could do more error-checking).

yes, we handle both 1  and 2 standard parallels in the same projection. the
1 parallel can be thought of as a degenerate case of the 2 parellels.

AFAIU, the scale factor is determined by the standard parallels, so is
reduntant to specify, see Snyder, equation 15-4.

As you follow proj4 closely, it could make more sense to use the proj4
string for LCC1SP, as described in [1]

  +proj=lcc   +lat_1=Latitude of natural origin
  +lon_0=Longitude of natural origin
  +k_0=Scale factor at natural origin
  +x_0=False Origin Easting
  +y_0=False Origin Northing

But this is another issue, I think that we can manage the CF-WKT conversion.


i agree that the CF parameters are misnamed



Given the way CF stores LCC1-SP parameters, I see how to calculate k_0
(scale factor at natural origin), using eq. 15-4 with CF
standard_parallel1 and latitude_of_projection_origin .




However, when converting a OGC WKT to CF grid_mapping,  I don't see
how to do the inverse calculation , which requires to  obtain
latitude_of_projection_origin from proj4 parameters above.

The example given in
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#lambert-conformal-projection
is trivial, as standard_parallel1=latitude_of_projection_origin=25.0 ,
therefore k_0 is 1, and the inverse is trivial

It is my understanding from [1] that the 1SP is usually considered to
have a scale factor of 1, and when it is not the 2SP form is preferred
  (with a scale factor of 1 for each SP) .

Therefore, would it make sense to do the following?

1)
CF->WKT : calculate scale_factor using eq. 15-4 and use the 1SP form
WKT->CF : set standard_parallel1=latitude_of_projection_origin, but
return an error if scale_factor != 1

2)
Another option would be that, if a CF definition contains 1SP and the
computed scale_factor is not 1 (because standard_parallel1 !=
latitude_of_projection_origin), then use the 2SP form in WKT.
A conversion back to CF would then work (in a 2SP form), although the
SP would probably not have integer values of degrees or degrees and
minutes (see [1]).

I prefer the second option as it closes the loop.  Any examples of 1SP
with standard_parallel1 != latitude_of_projection_origin out there?

[1] 
http://www.remotesensing.org/geotiff/proj_list/lambert_conic_

Re: [CF-metadata] inconsistencies in translating projection parameters OGC WKT <-> CF-1.5

[John Caron]

On 12/15/2011 3:26 PM, Etienne Tourigny wrote:

An update on LCC-1SP:

I have modified the GDAL driver to compute scale_factor from CF
parameters ( std_parallel and latitude_of_projection_origin), and keep
the std_parallel1 (which is not OGC WKT for LCC-1SP), so it exports
back to CF without a hitch.

The only exception is translating from WKT to CF, with scale_factor !=
1 and no  std_parallel1 (a valid OGC WKT), there is no valid CF
definition (because I can't see how to get std_parallel from
scale_factor and latitude_of_projection_origin). In this case, the 2SP
variant is preferred (but I don't know how to compute it). On the
positive side, at least GDAL is able to read it back so no information
is lost.

For anyone interested here is the trac entry:
http://trac.osgeo.org/gdal/ticket/3324#comment:13

Regards,
Etienne

Hi Etienne :

I agree thats the best to do, unless we run across the code to translate 
the scale factor with 1SP to 2SP.


John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] inconsistencies in translating projection parameters OGC WKT <-> CF-1.5

[John Caron]

Hi Etienne:

Generally, CF has followed proj4 conventions as much as we could, and 
John Snyder's reference book is often underneath:


http://pubs.usgs.gov/pp/1395/report.pdf

OGC specs tended to come from experts using the EPSG, and generally we 
are less familiar with that.


some specific comments below, based on my own understandings.

On 12/13/2011 2:21 PM, Etienne Tourigny wrote:

Hi all,

I am opening a seperate thread at this deals merely with the problems
encountered in translating projection parameters to/from OGC WKT and
CF-1.5.

I would appreciate any guidance in resolving these issues.

In summary:

1) LCC (Lambert conformal): should include a scale_factor parameter in
the 1 standard parallel case
2) CEA (Lambert Cylindrical Equal Area): how to convert the
'scale_factor_at_projection_origin' alternative to standard parallel
to OGC WKT?
3) PS (Polar stereographic):This projection's parameters in CF-1 are
significantly different from OGC/PROJ.4 ,  CF-1 standard parameters
'Either standard_parallel or scale_factor_at_projection_origin' is
incompatible with OGC WKT, which requires both

More detailed information (reproduced here) can be found at
http://trac.osgeo.org/gdal/wiki/NetCDF_ProjectionTestingStatus

1) LCC 1SP (Lambert conformal)

The OGC WKT 'Lambert_Conformal_Conic_2SP' maps clearly to LCC in CF-1
with 2 std parallels.

The CF-1 spec claims that a 1SP version can be used, but it doesn't
include a 'scale_factor' as does the OGC WKT
'Lambert_Conformal_Conic_1SP'.

For import: Given discussion and example on #3324, and also support
from NARCCAP project: it seems the CF-1 LCC 1SP projection is a
'degenerate' LCC-2SP with both SP's at the same location. Thus seems
we can read with both 'std_parallel' (1 value) and
'latitude_of_projection_origin' mapping to latitude_of_origin (WKT)
and setting scale_factor=1? This is the code's current behaviour
(though perhaps could do more error-checking).
yes, we handle both 1  and 2 standard parallels in the same projection. 
the 1 parallel can be thought of as a degenerate case of the 2 parellels.


AFAIU, the scale factor is determined by the standard parallels, so is 
reduntant to specify, see Snyder, equation 15-4.


its possible that the more general case is to use another constant to 
multiply the one given in 15-4, but i have never seen that.


Im not sure what #3324  refers to.



For export: given the LCC-1SP can specifically include a scale_factor
to control the projection (see
http://www.remotesensing.org/geotiff/proj_list/lambert_conic_conformal_1sp.html),
it seems this can't be directly exported to NetCDF without throwing
away the scale factor.
Perhaps the reasonable thing for now is to export the scale_factor to
NetCDF, so it can at least be read back in. Otherwise, we'd have to
try to translate the LCC-1SP to a LCC-2SP on the fly.


2) CEA (Lambert Cylindrical Equal Area):

The CF-1 conventions claim this can be encoded with a
'scale_factor_at_projection_origin' alternative to standard parallel -
how would this conversion be done from/to OGC WKT (which requires
central_meridian and standard_parallel_1)?


It does appear at first glance that there should be a seperate scale 
factor. I am less familiar since I havent coded this one.



3) PS (Polar stereographic)

This projection's parameters in CF-1 are significantly different from
http://www.remotesensing.org/geotiff/proj_list/polar_stereographic.html.
In particular the CF-1 equivalent for 'latitude of natural origin'
seems to be 'standard_parallel'.


i 
agree, i dont know where these CF names came from.




CF-1 defines either standard_parallel or
scale_factor_at_projection_origin, which is incompatible with OGC WKT
(and OGR) that requires both. Note: we have verified the current
approach we take to mapping is the same used by at least one other
NetCDF CF-1 user in the UCAR NARCCAP project.

Current implementation (as of Dec. 13) uses the following rules:
- WKT scale_factor maps to CF scale_factor_at_projection_origin
- WKT central_meridian maps to CF straight_vertical_longitude_from_pole
- WKT latitude_of_origin maps to WKT standard_parallel
- WKT to CF: if WKT latitude_of_origin is positive then CF
latitude_of_projection_origin is +90, else it is -90 (North vs. South
Pole)
- CF to WKT: unsupported if standard_parallel is not defined, as
OGRSpatialReference::SetPS() requires both CF
scale_factor_at_projection_origin
Im guessing these rules may have to be specific to a projection, rather 
than assuming you can do a global mapping of the attribute names.


I would like to incorporate your findings into the CDM library when I 
can. Do you have any official documentation describing this mapping?




Thanks
Etienne
___


The question for CF is whether we should start to follow the WKT spec 
closer, and add aliases where needed.  I would prefer this since most of 
us arent exp

Re: [CF-metadata] Proposal for better handling vector quantities in CF

[John Caron]

On 12/9/2011 11:37 AM, Jonathan Gregory wrote:

Dear John

I prefer the idea that Thomas has put forward of an umbrella, rather than
containing the vector/tensor components in one data variable, because

* I really don't like the idea of units being mixed within a data variable.
I think things with different units must certainly be different quantities
and could not be regarded as the same field. You can get away with it if they
are all m s-1, for instance, but not so easily if the vertical one is orders
of magnitude different from the horizontal, and not at all if the vector is
expressed in polar coordinates.


I think the common case is that the vector components have the same 
unit. One could restrict to that case.




* I think it would be very inconvenient, and would break a lot of existing
software, if the coordinates were not what they appeared to be, because an
offset had to be added. Also, in general, the component fields of a staggered
grid do not have the same dimensions, as well as differing in the coordinates.

Im not sure what "an offset had to be added" means.

I think the common case of staggered grids could be handled with a 
convention defining the staggering, rather than seperate dimensions. I 
pull out the one Rich Signell and I cam up with a long time ago, for its 
own sake.




* It avoids the need to define a convention for labelling vector/tensor
components.
I think this convention would be about as complex as the one you will 
need for Thomas' proposal.




* It is completely backwards-compatible as regards the component fields, which
are exactly as before; we're just adding some separate information linking
them. This seems neat to me.


I agree thats a strong reason for Thomas' method.

OTOH, if we start thinking in terms of the extended model, a Structure 
("compound type" in HDF5 parlance) might be useful. What do you think 
about starting to think about possible uses of extended data model?


Thanks for your thoughts, as always, interesting.

John

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata