Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Dear Andrew ERROR (4): Axis attribute is not allowed for auxillary coordinate variables. I don't get this error with the Vector approach. It looks like the checker thinks my scalar lat, lon, time are 'auxilliary coordinate variables' and axis attributes are not allowed on these. Is the checker interpreting things correctly? The scalar coordinate vars are formally aux coord vars, because they are named by the coordinates attribute. There has been confusion for a long time on whether the axis attribute is allowed for aux coord vars, but we eventually decided that it should be. This has been agreed on trac ticket 62 https://cf-pcmdi.llnl.gov/trac/ticket/62 which should go in the next version of CF. Therefore you could ignore these errors. The checker will be updated when CF is. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
On 4/30/2012 8:40 PM, andrew walsh wrote: Hi John and CF-Metadata list, Based on your earlier advice I decided using the Scalar way to represent the coordinate lat, long and time rather than Vector way i.e lat(lat0, lon(lon), time(time) mainly for reason of simplicity. the correct other choice is lat(profile), lon(profile), time(profile). not sure if you caught that. the scalar choice is fine for 1 profile per file. the other is used to store multiple profiles in one file. I have run the Scalar sample through the BADC netCDF checker (*Note) at http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl and I now get this error on each of the lat, lon, time variables: ERROR (4): Axis attribute is not allowed for auxillary coordinate variables. I don't get this error with the Vector approach. It looks like the checker thinks my scalar lat, lon, time are 'auxilliary coordinate variables' and axis attributes are not allowed on these. Is the checker interpreting things correctly? i think we recently clarified that axis is acceptable on auxiliary coordinates. OTOH, its unecessary, as long as you follow the other rules for identifying coordinates (chapter 4). ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Jim, Yes, thats correct. If you see the attached CDL in earlier email we have used coordinates attributes on measured variables e.g temperature:coordinates = time latitude longitude pressure Andrew - Original Message - From: Jim Biard To: cf-metadata@cgd.ucar.edu Sent: Wednesday, May 02, 2012 4:51 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi. If someone chooses the scalar approach, isn't it necessary to declare the coordinates via the coordinates attribute on the measurement variables? Grace and peace, Jim Jim Biard Research Scholar Cooperative Institute for Climate and Satellites Remote Sensing and Applications Division National Climatic Data Center 151 Patton Ave, Asheville, NC 28801-5001 jim.bi...@noaa.gov 828-271-4900 On May 1, 2012, at 12:21 PM, John Caron wrote: On 4/30/2012 8:40 PM, andrew walsh wrote: Hi John and CF-Metadata list, Based on your earlier advice I decided using the Scalar way to represent the coordinate lat, long and time rather than Vector way i.e lat(lat0, lon(lon), time(time) mainly for reason of simplicity. the correct other choice is lat(profile), lon(profile), time(profile). not sure if you caught that. the scalar choice is fine for 1 profile per file. the other is used to store multiple profiles in one file. I have run the Scalar sample through the BADC netCDF checker (*Note) at http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl and I now get this error on each of the lat, lon, time variables: ERROR (4): Axis attribute is not allowed for auxillary coordinate variables. I don't get this error with the Vector approach. It looks like the checker thinks my scalar lat, lon, time are 'auxilliary coordinate variables' and axis attributes are not allowed on these. Is the checker interpreting things correctly? i think we recently clarified that axis is acceptable on auxiliary coordinates. OTOH, its unecessary, as long as you follow the other rules for identifying coordinates (chapter 4). ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
On 4/29/2012 5:33 PM, andrew walsh wrote: Hi John, My responses inline below. Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: cf-metadata@cgd.ucar.edu Sent: Saturday, April 28, 2012 2:39 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew: You can use a dimension=1 instead of a scalar. OK, you mean if one used the vector method. yes But you cant use lat(lat) lon(lon) time(time) the correct usage in the single profile case would be: lat(profile) lon(profile) time(profile) So, do you mean in the vector method have this? (simpler I guess to have common profile dimension = 1) yes, you cant use incorrect dimensions, even when they are all = 1. dimensions: profile=1 pressure = 57 variables: lat(profile) lon(profile) time(profile) pressure(pressure) temperature(pressure) Instead of this? Dimensions: time=1 lat=1 lon=1 pressure=57 Variables: time(time) lat(lat) lon(lon) pressure(pressure) temperature(pressure) ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Wouldn't you also need the profile dimension on the temperature and pressure variables in order to connect them all as part of the same profile collection? Jim Biard Research Scholar Cooperative Institute for Climate and Satellites Remote Sensing and Applications Division National Climatic Data Center 151 Patton Ave, Asheville, NC 28801-5001 jim.bi...@noaa.gov 828-271-4900 On Apr 30, 2012, at 8:46 AM, John Caron wrote: On 4/29/2012 5:33 PM, andrew walsh wrote: Hi John, My responses inline below. Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: cf-metadata@cgd.ucar.edu Sent: Saturday, April 28, 2012 2:39 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew: You can use a dimension=1 instead of a scalar. OK, you mean if one used the vector method. yes But you cant use lat(lat) lon(lon) time(time) the correct usage in the single profile case would be: lat(profile) lon(profile) time(profile) So, do you mean in the vector method have this? (simpler I guess to have common profile dimension = 1) yes, you cant use incorrect dimensions, even when they are all = 1. dimensions: profile=1 pressure = 57 variables: lat(profile) lon(profile) time(profile) pressure(pressure) temperature(pressure) Instead of this? Dimensions: time=1 lat=1 lon=1 pressure=57 Variables: time(time) lat(lat) lon(lon) pressure(pressure) temperature(pressure) ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
yes, good point. if it is a multidimension representation, you would also need the profile dimension. if its a ragged array, you would need either the rowSize or parentIndex variable. On 4/30/2012 7:22 AM, Jim Biard wrote: Wouldn't you also need the profile dimension on the temperature and pressure variables in order to connect them all as part of the same profile collection? Jim Biard Research Scholar Cooperative Institute for Climate and Satellites Remote Sensing and Applications Division National Climatic Data Center 151 Patton Ave, Asheville, NC 28801-5001 jim.bi...@noaa.gov mailto:jim.bi...@noaa.gov 828-271-4900 On Apr 30, 2012, at 8:46 AM, John Caron wrote: On 4/29/2012 5:33 PM, andrew walsh wrote: Hi John, My responses inline below. Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu mailto:ca...@unidata.ucar.edu To: cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu Sent: Saturday, April 28, 2012 2:39 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew: You can use a dimension=1 instead of a scalar. OK, you mean if one used the vector method. yes But you cant use lat(lat) lon(lon) time(time) the correct usage in the single profile case would be: lat(profile) lon(profile) time(profile) So, do you mean in the vector method have this? (simpler I guess to have common profile dimension = 1) yes, you cant use incorrect dimensions, even when they are all = 1. dimensions: profile=1 pressure = 57 variables: lat(profile) lon(profile) time(profile) pressure(pressure) temperature(pressure) Instead of this? Dimensions: time=1 lat=1 lon=1 pressure=57 Variables: time(time) lat(lat) lon(lon) pressure(pressure) temperature(pressure) ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu mailto:CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi John and CF-Metadata list,
Based on your earlier advice I decided using the Scalar way to represent
the coordinate lat, long and time rather than Vector way i.e lat(lat0, lon(lon),
time(time)
mainly for reason of simplicity.
I have run the Scalar sample through the BADC netCDF checker (*Note)
at http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl and I now get this error
on each of the lat, lon, time variables:
ERROR (4): Axis attribute is not allowed for auxillary coordinate variables.
I don't get this error with the Vector approach. It looks like the checker
thinks
my scalar lat, lon, time are 'auxilliary coordinate variables' and axis
attributes
are not allowed on these. Is the checker interpreting things correctly?
If the scalar approach is breaking the CF rules then I would prefer to go
back to the vector approach which is what I proposed in the
first email to this topic on April 2.
For reference attached are netCDF files for the Scalar and Vector alternative
and
corresponding text files.
Andrew
(*Note: I set the 'Conventions' global attribute temporarily to CF-1.5 in the
netCDF file
because the checker doesn't recognise CF-1.6 yet, only works up to 1.5.
The 'Conventions' attribute will be CF-1.6 in the final file.)
- Original Message -
From: John Caron ca...@unidata.ucar.edu
To: andrew walsh awa...@metoc.gov.au
Cc: cf-metadata@cgd.ucar.edu
Sent: Monday, April 30, 2012 10:46 PM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
On 4/29/2012 5:33 PM, andrew walsh wrote:
Hi John,
My responses inline below.
Andrew
- Original Message - From: John Caron ca...@unidata.ucar.edu
To: cf-metadata@cgd.ucar.edu
Sent: Saturday, April 28, 2012 2:39 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Andrew:
You can use a dimension=1 instead of a scalar.
OK, you mean if one used the vector method.
yes
But you cant use
lat(lat)
lon(lon)
time(time)
the correct usage in the single profile case would be:
lat(profile)
lon(profile)
time(profile)
So, do you mean in the vector method have this? (simpler I guess to
have common profile dimension = 1)
yes, you cant use incorrect dimensions, even when they are all = 1.
dimensions:
profile=1
pressure = 57
variables:
lat(profile)
lon(profile)
time(profile)
pressure(pressure)
temperature(pressure)
Instead of this?
Dimensions:
time=1
lat=1
lon=1
pressure=57
Variables:
time(time)
lat(lat)
lon(lon)
pressure(pressure)
temperature(pressure)
netcdf Vector1.5 {
dimensions:
time = 1 ;
pressure = UNLIMITED ; // (9 currently)
latitude = 1 ;
longitude = 1 ;
variables:
double time(time) ;
time:standard_name = time ;
time:units = days since 1950-01-01 00:00:00 ;
time:axis = T ;
time:valid_min = 0 ;
time:valid_max = 99 ;
byte time_qc_flag ;
time_qc_flag:long_name = quality control flag for time (primary
Level 1 flag) ;
time_qc_flag:quality_control_convention = Proposed IODE qc scheme
March 2012 ;
time_qc_flag:valid_min = 1 ;
time_qc_flag:valid_max = 9 ;
time_qc_flag:flag_values = 1b, 2b, 3b, 4b, 9b ;
time_qc_flag:flag_meanings = good not_evaluated_or_unknown suspect
bad missing ;
double latitude(latitude) ;
latitude:standard_name = latitude ;
latitude:units = degrees_north ;
latitude:axis = Y ;
latitude:valid_min = -90 ;
latitude:valid_max = 90 ;
double longitude(longitude) ;
longitude:standard_name = longitude ;
longitude:units = degrees_east ;
longitude:axis = X ;
longitude:valid_min = -180 ;
longitude:valid_max = 180 ;
byte position_qc_flag ;
position_qc_flag:long_name = quality control flag for position
(primary Level 1 flag) ;
position_qc_flag:quality_control_convention = Proposed IODE qc
scheme March 2012 ;
position_qc_flag:valid_min = 1 ;
position_qc_flag:valid_max = 9 ;
position_qc_flag:flag_values = 1b, 2b, 3b, 4b, 9b ;
position_qc_flag:flag_meanings = good not_evaluated_or_unknown
suspect bad missing ;
double pressure(pressure) ;
pressure:standard_name = sea_water_pressure ;
pressure:units = decibars ;
pressure:axis = Z ;
pressure:valid_min = 0 ;
pressure:valid_max = 12000 ;
pressure:positive = down ;
double temperature(pressure) ;
temperature:_FillValue = -99.99 ;
temperature:standard_name = sea_water_temperature ;
temperature:units = degrees_C ;
temperature:valid_min = -2
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi John, My responses inline below. Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: cf-metadata@cgd.ucar.edu Sent: Saturday, April 28, 2012 2:39 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew: You can use a dimension=1 instead of a scalar. OK, you mean if one used the vector method. But you cant use lat(lat) lon(lon) time(time) the correct usage in the single profile case would be: lat(profile) lon(profile) time(profile) So, do you mean in the vector method have this? (simpler I guess to have common profile dimension = 1) dimensions: profile=1 pressure = 57 variables: lat(profile) lon(profile) time(profile) pressure(pressure) temperature(pressure) Instead of this? Dimensions: time=1 lat=1 lon=1 pressure=57 Variables: time(time) lat(lat) lon(lon) pressure(pressure) temperature(pressure) John On 4/27/2012 12:03 AM, andrew walsh wrote: Hi John, Thanks for your advice. For a single vertical profile and single valued time, lat, long. it seems there are two acceptable ways i.e 1 element vector or single valued scalar. Quoting from 'section 2.4 Dimensions' of CF standard: 1 element vector: Dimensions may be of any size, including unity. When a single value of some coordinate applies to all the values in a variable, the recommended means of attaching this information to the variable is by use of a dimension of size unity with a one-element coordinate variable. and in the next sentence a scalar: It is also acceptable to use a scalar coordinate variable which eliminates the need for an associated size one dimension in the data variable. 1) ***Vector co-ordinate variables*** Dimensions: time=1 pressure=56 lat=1 lon=1 Variables: time(time) lat(lat) lon(lon) temperature(pressure) 2) **Scalar co-ordinate varaibles Dimensions: pressure =56 Variables: time lat lon temperature(pressure) The scalar approach you suggest as in section H.3.3. Single profile of the CF v1.6 standard is simpler than the vector approach so we will take your advice. Regards, Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Cc: cf-metadata@cgd.ucar.edu; Luke Callcut l...@metoc.gov.au; g...@metoc.gov.au Sent: Thursday, April 26, 2012 10:48 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi andrew: The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a single profile. Assuming that, you would use the following template: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 note that lat, lon and time coordinates all have to be scalars. John On 4/18/2012 8:24 PM, andrew walsh wrote: Hi John, We have now constructed a real netCDF file for you to check. QC flag variables have been added in. The QC flagging method is based on the proposed IODE scheme in (1) below. Attached are the sample .nc file, text (ncdump) of same and a document specifying the CDL. Thanks and Regards, Andrew Walsh Ref. (1) Konovalov et. al (March 2012), Proposal to adopt a quality flag scheme standard for oceanographic and marine meteorological data, Version 1.2. - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Sent: Thursday, April 05, 2012 10:44 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 ok On 4/4/2012 6:42 PM, andrew walsh wrote: Hi John, Thank for your offer to check a sample netCDF CTD data file. At moment we don't have some real .nc file but when we do I can send you a file for checking. Andrew - Original Message - From: John Caron To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 5:55 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John snip ___ CF-metadata mailing
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi John, Thanks for your advice. For a single vertical profile and single valued time, lat, long. it seems there are two acceptable ways i.e 1 element vector or single valued scalar. Quoting from 'section 2.4 Dimensions' of CF standard: 1 element vector: Dimensions may be of any size, including unity. When a single value of some coordinate applies to all the values in a variable, the recommended means of attaching this information to the variable is by use of a dimension of size unity with a one-element coordinate variable. and in the next sentence a scalar: It is also acceptable to use a scalar coordinate variable which eliminates the need for an associated size one dimension in the data variable. 1) ***Vector co-ordinate variables*** Dimensions: time=1 pressure=56 lat=1 lon=1 Variables: time(time) lat(lat) lon(lon) temperature(pressure) 2) **Scalar co-ordinate varaibles Dimensions: pressure =56 Variables: time lat lon temperature(pressure) The scalar approach you suggest as in section H.3.3. Single profile of the CF v1.6 standard is simpler than the vector approach so we will take your advice. Regards, Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Cc: cf-metadata@cgd.ucar.edu; Luke Callcut l...@metoc.gov.au; g...@metoc.gov.au Sent: Thursday, April 26, 2012 10:48 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi andrew: The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a single profile. Assuming that, you would use the following template: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 note that lat, lon and time coordinates all have to be scalars. John On 4/18/2012 8:24 PM, andrew walsh wrote: Hi John, We have now constructed a real netCDF file for you to check. QC flag variables have been added in. The QC flagging method is based on the proposed IODE scheme in (1) below. Attached are the sample .nc file, text (ncdump) of same and a document specifying the CDL. Thanks and Regards, Andrew Walsh Ref. (1) Konovalov et. al (March 2012), Proposal to adopt a quality flag scheme standard for oceanographic and marine meteorological data, Version 1.2. - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Sent: Thursday, April 05, 2012 10:44 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 ok On 4/4/2012 6:42 PM, andrew walsh wrote: Hi John, Thank for your offer to check a sample netCDF CTD data file. At moment we don't have some real .nc file but when we do I can send you a file for checking. Andrew - Original Message - From: John Caron To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 5:55 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John snip ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Andrew: You can use a dimension=1 instead of a scalar. But you cant use lat(lat) lon(lon) time(time) the correct usage in the single profile case would be: lat(profile) lon(profile) time(profile) John On 4/27/2012 12:03 AM, andrew walsh wrote: Hi John, Thanks for your advice. For a single vertical profile and single valued time, lat, long. it seems there are two acceptable ways i.e 1 element vector or single valued scalar. Quoting from 'section 2.4 Dimensions' of CF standard: 1 element vector: Dimensions may be of any size, including unity. When a single value of some coordinate applies to all the values in a variable, the recommended means of attaching this information to the variable is by use of a dimension of size unity with a one-element coordinate variable. and in the next sentence a scalar: It is also acceptable to use a scalar coordinate variable which eliminates the need for an associated size one dimension in the data variable. 1) ***Vector co-ordinate variables*** Dimensions: time=1 pressure=56 lat=1 lon=1 Variables: time(time) lat(lat) lon(lon) temperature(pressure) 2) **Scalar co-ordinate varaibles Dimensions: pressure =56 Variables: time lat lon temperature(pressure) The scalar approach you suggest as in section H.3.3. Single profile of the CF v1.6 standard is simpler than the vector approach so we will take your advice. Regards, Andrew - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Cc: cf-metadata@cgd.ucar.edu; Luke Callcut l...@metoc.gov.au; g...@metoc.gov.au Sent: Thursday, April 26, 2012 10:48 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi andrew: The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a single profile. Assuming that, you would use the following template: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 note that lat, lon and time coordinates all have to be scalars. John On 4/18/2012 8:24 PM, andrew walsh wrote: Hi John, We have now constructed a real netCDF file for you to check. QC flag variables have been added in. The QC flagging method is based on the proposed IODE scheme in (1) below. Attached are the sample .nc file, text (ncdump) of same and a document specifying the CDL. Thanks and Regards, Andrew Walsh Ref. (1) Konovalov et. al (March 2012), Proposal to adopt a quality flag scheme standard for oceanographic and marine meteorological data, Version 1.2. - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Sent: Thursday, April 05, 2012 10:44 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 ok On 4/4/2012 6:42 PM, andrew walsh wrote: Hi John, Thank for your offer to check a sample netCDF CTD data file. At moment we don't have some real .nc file but when we do I can send you a file for checking. Andrew - Original Message - From: John Caron To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 5:55 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John snip ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi andrew: The file you sent (20110818T001140Z_M_HI504BEN.nc) appears to be a single profile. Assuming that, you would use the following template: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp8363696 note that lat, lon and time coordinates all have to be scalars. John On 4/18/2012 8:24 PM, andrew walsh wrote: Hi John, We have now constructed a real netCDF file for you to check. QC flag variables have been added in. The QC flagging method is based on the proposed IODE scheme in (1) below. Attached are the sample .nc file, text (ncdump) of same and a document specifying the CDL. Thanks and Regards, Andrew Walsh Ref. (1) Konovalov et. al (March 2012), Proposal to adopt a quality flag scheme standard for oceanographic and marine meteorological data, Version 1.2. - Original Message - From: John Caron ca...@unidata.ucar.edu To: andrew walsh awa...@metoc.gov.au Sent: Thursday, April 05, 2012 10:44 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 ok On 4/4/2012 6:42 PM, andrew walsh wrote: Hi John, Thank for your offer to check a sample netCDF CTD data file. At moment we don't have some real .nc file but when we do I can send you a file for checking. Andrew - Original Message - From: John Caron To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 5:55 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John snip ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi John,
We have now constructed a real netCDF file for you to check. QC flag
variables have been added in. The QC flagging method is based on the
proposed IODE scheme in (1) below.
Attached are the sample .nc file, text (ncdump) of same and a document
specifying the CDL.
Thanks and Regards,
Andrew Walsh
Ref.
(1) Konovalov et. al (March 2012), Proposal to adopt a quality flag scheme
standard
for oceanographic and marine meteorological data, Version 1.2.
- Original Message -
From: John Caron ca...@unidata.ucar.edu
To: andrew walsh awa...@metoc.gov.au
Sent: Thursday, April 05, 2012 10:44 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
ok
On 4/4/2012 6:42 PM, andrew walsh wrote:
Hi John,
Thank for your offer to check a sample netCDF CTD data file. At moment we
don't have
some real .nc file but when we do I can send you a file for checking.
Andrew
- Original Message - From: John Caron
To: cf-metadata@cgd.ucar.edu
Sent: Wednesday, April 04, 2012 5:55 AM
Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi all:
Lets see, I havent followed the entire conversation, but:
1) Andrew if you can send me a sample file (not just the CDL) I can check if
it works in the CDM with the new 1.6 conventions, and maybe give you some
advice from my POV.
2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation
simply appends multidimensional arrays together, eg joinNew and
joinExisting NCML aggregation. I call it syntactic aggregation because it
doesnt know what its aggregating, and the homogeneity requirements are
strict. 2) Feature Type collections (aka semantic aggregation) are the
more recent development. These understand the coordinate information of the
data, and so can handle variations in the representation, eg ragged arrays,
scalar coordinates, etc. In this case, the CDM understands a dataset as a
collection of features objects, which can be stored in a collection of
files. The interfaces to these collections is still under development. Most
current and future work in the CDM is in this category.
John
snip
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
netcdf 20110818T001140Z_M_HI504BEN {
dimensions:
time = 1 ;
pressure = UNLIMITED ; // (9 currently)
latitude = 1 ;
longitude = 1 ;
variables:
double time(time) ;
time:standard_name = time ;
time:units = days since 1950-01-01 00:00:00Z ;
time:axis = T ;
time:valid_min = 0 ;
time:valid_max = 99 ;
byte time_qc_flag ;
time_qc_flag:long_name = quality control flag for time (primary
Level 1 flag) ;
time_qc_flag:quality_control_convention = Proposed IODE qc scheme
March 2012 ;
time_qc_flag:valid_min = 1 ;
time_qc_flag:valid_max = 9 ;
time_qc_flag:flag_values = 1b, 2b, 3b, 4b, 9b ;
time_qc_flag:flag_meanings = good not_evaluated_or_unknown suspect
bad missing ;
double latitude(latitude) ;
latitude:standard_name = latitude ;
latitude:units = degrees_north ;
latitude:axis = Y ;
latitude:valid_min = -90 ;
latitude:valid_max = 90 ;
double longitude(longitude) ;
longitude:standard_name = longitude ;
longitude:units = degrees_east ;
longitude:axis = X ;
longitude:valid_min = -180 ;
longitude:valid_max = 180 ;
byte position_qc_flag ;
position_qc_flag:long_name = quality control flag for position
(primary Level 1 flag) ;
position_qc_flag:quality_control_convention = Proposed IODE qc
scheme March 2012 ;
position_qc_flag:valid_min = 1 ;
position_qc_flag:valid_max = 9 ;
position_qc_flag:flag_values = 1b, 2b, 3b, 4b, 9b ;
position_qc_flag:flag_meanings = good not_evaluated_or_unknown
suspect bad missing ;
double pressure(pressure) ;
pressure:standard_name = sea_water_pressure ;
pressure:units = decibars ;
pressure:axis = Z ;
pressure:valid_min = 0 ;
pressure:valid_max = 12000 ;
pressure:positive = down ;
double temperature(pressure) ;
temperature:_FillValue = -99.99 ;
temperature:standard_name = sea_water_temperature ;
temperature:units = degrees_C ;
temperature:valid_min = -2 ;
temperature:valid_max = 40 ;
temperature:ancillary_variables = temperature_whole_profile_flag
temperature_qc_flag temperature_sd_test
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Dear Andrew Thanks for your email. 1) What distinguishes an 'auxilliary coordinate variable' from a plain old 'coordinate variable'? A plain old coordinate variable is as defined by the Unidata netCDF user's guide. It is 1D, monotonic, and its name is the same as the name of its dimension. An auxiliary coordinate variable is a CF concept. It can be multi-D and it is not necessarily monotonic. Auxiliary coordinate variables are linked to the data variable by being named by the coordinates attribute; that is the function of this attribute. Coordinate variables are implicitly associated with the data variable through the name of their dimension. 2) Is it permissable for a 'auxilliary coordinate variable' to have either the scalar form e.g float lat or the array form lat(lat) with lat=1 declared as a dimension? lat(lat) is a (Unidata) coordinate variable. It is permissible (but not required or recommended) for it to be listed in the coordinates attribute. lat (scalar) must be named in the coordinates attribute; otherwise, it would not be associated with the data variable. Hence, it is an auxiliary coord var. CF calls a scalar auxiliary coordinate variable a scalar coordinate variable. 3) Do we need the coordinates attribute to look-up the corresponding auxilliary cordinates variable? For example if we had a variable declared as salinity(z) then to look up its lat, long and time coordinates for say display purpose would the salinity:coordinates = lat long time attribute used for the lookup. Yes. That is the only way you could associate lon, lat and time with the salinity variable, since it doesn't have dimensions of lon, lat or time. 4) However if we had the salinity variable be declared as salinity(lat,long,time,z) then the application would lookup the co-ordinates from the co-ordinates variables declaration names and we wouldn't need the salinity:coordinates attribute? Exactly. I do notice that a lot of the netCDF CTD data out there doesn't use the coordinates attribute. Is this a feature that came in later with CF v1.5 or v1.6? No, it's existed since the start of CF. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi John, Thank for your offer to check a sample netCDF CTD data file. At moment we don't have some real .nc file but when we do I can send you a file for checking. Andrew - Original Message - From: John Caron To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 5:55 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John snip ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hello again, It has been pointed out to me that in the current SeaDataNet format specification (written and updated by me!) that the initial position of insisting upon depth for the z co-ordinate was relaxed to allow either pressure or depth. So, I guess we need to be relaxed and leave z co-ordinate harmonisation to the the application tools. Yet another 'senior moment'.. Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Lowry, Roy K. [r...@bodc.ac.uk] Sent: 03 April 2012 06:46 To: andrew walsh Cc: cf-metadata@cgd.ucar.edu; sdn2-t...@seadatanet.org; g...@metoc.gov.au; Luke Callcut; Upendra Dadi Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi again, The reason that I prefer depth to pressure as the dimension for CTD is that the information required for pressure to depth conversion is virtually always present in the CTD data. However, in other oceanographic profiles - say nutrient bottle data - it is quite possible to have depth present without temperature and salinity, making the conversion to pressure impossible. Calculating depth from pressure at the time of standardised formatting of CTD data is as you say is trivial. So, why not make the CTD data more compatible with bottle data for very little cost? SeaDataNet already mandate inclusion of depth in their other data formats for CTD data delivery and so I can't see them switching to pressure for the dimension in NetCDF. Cheers, Roy. From: andrew walsh [awa...@metoc.gov.au] Sent: 03 April 2012 06:15 To: Lowry, Roy K. Cc: Jim Biard; Upendra Dadi; cf-metadata@cgd.ucar.edu; Luke Callcut; g...@metoc.gov.au; sdn2-t...@seadatanet.org Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Roy/All, Agree totally it would be good to get this CTD netCDF done in an interoperable way. Regarding having pressure vs. depth. We liked to use pressure because: 1) It is the thing that is measured, let us store just that and calculate depth if needed. 2) Depth can later be easily be calculated on the fly using the 'Pressure to Depth' algorithm in UNESCO (1983) Techinical Papers in Marine Science 44, Algorithms for Computation of fundamental properties of Seawater. One can use the python seawater library (see http://packages.python.org/seawater/ ) and seawater.csiro.depth(p, lat) to get depth from pressure and seawater.csiro.pres(depth, lat) to get pressure from depth. 3) I noted that the ARGO project (1's CTD like profiles) and others like CSIRO Oceanography in Aust. make data available with just pressure. 4) It makes our processing and QC a whole lot simpler. We don't have to worry about calculating and managing the extra 'depth' variable. Is there any problem with having the pressure as a co-ordinate which isn't really a dimensional quantity like depth (z) in the 4-D sense i.e x,y,z,t ? However I note that pressure (decibar) is allowed as a vertical axis e.g see section 4.3. Vertical (Height or Depth) Coordinate of CF v1.6 conventions document which says: A vertical coordinate will be identifiable by: . units of pressure; or . the presence of the positive attribute with a value of up or down (case insensitive). AND section 4.3.1. Dimensional Vertical Coordinate which says: The units attribute for dimensional coordinates will be a string formatted as per the udunits.dat3 file. The acceptable units for vertical (depth or height) coordinate variables are: . units of pressure as listed in the file udunits.dat. For vertical axes the most commonly used of these include include bar, millibar, decibar, atmosphere (atm), pascal (Pa), and hPa. ... ...¶ Regards, Andrew - Original Message - From: Lowry, Roy K. To: andrew walsh Cc: Jim Biard ; Upendra Dadi ; cf-metadata@cgd.ucar.edu ; Luke Callcut ; g...@metoc.gov.au ; sdn2-t...@seadatanet.org Sent: Tuesday, April 03, 2012 2:16 PM Subject: RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, SeaDataNet will very soon be considering how it is going to encode data, including single CTD casts, in CF-compliant NetCDF and so I think the time is ripe for agreeing how the significant numbers of us who indulge in this practice for whatever reason do it. That way we'll end up with interoperable data. I think there are a number of people on this list who have already encoded single CTDs into NetCDF and so it would be useful to start by asking for descriptions (like Andrew's examples) of how this has been done and what tools are dependent upon that encoding. The z co-ordinate parameter (pressure/depth) is also an issue worth resolving. Whilst interconversions are relatively straightforward, agreement would make life much easier. My preference leans dowards having depth as the dimension with pressure as an optional variable. That way we interoperate with other kinds
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Andrew - I am not sure if I should have TEMPERATURE and SALINITY arrays with 4 dimensions like TEMPERATURE(TIME,LATITUDE,LONGITUDE,PRESSURE) or just 1 dimension like I have above i.e. TEMPERATURE(PRESSURE). ? While we don't store profile data in OceanSITES, we are working on our coordinates to make them conform more to published standards, so I've been reading the docs on this lately. So, it's fresh in my mind - fresh, if not exactly clear. The NetCDF Best Practices page (unidata.ucar.edu/software/netcdf/docs/BestPractices.html) recommends: Make coordinate variables for every dimension possible (except for string length dimensions). And the CF manual has this recommendation about the order of the coordinates and the use of scalar coordinates as dimensions, in section 2.4: If any or all of the dimensions of a variable have the interpretations of date or time (|T|), height or depth (|Z|), latitude (|Y|), or longitude (|X|) then we recommend, but do not require (see Section 1.4, Relationship to the COARDS Conventions http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coards-relationship), those dimensions to appear in the relative order |T|, then |Z|, then |Y|, then |X| in the CDL definition corresponding to the file. All other dimensions should, whenever possible, be placed to the left of the spatiotemporal dimensions. Dimensions may be of any size, including unity. When a single value of some coordinate applies to all the values in a variable, the recommended means of attaching this information to the variable is by use of a dimension of size unity with a one-element coordinate variable. These 2 recommendations would lead me to suggest that, *IF* your pressure is monotonic, you'd want to use: TEMPERATURE(TIME,PRESSURE,LATITUDE,LONGITUDE) If your pressure isn't monotonic, and so can't be a coordinate variable, I don't know whether it's preferable to demote ALL the coordinates to auxiliary coordinate variable status, TEMPERATURE: coordinates = TIME,PRESSURE,LATITUDE,LONGITUDE ; Cheers - Nan On 4/3/12 1:15 AM, andrew walsh wrote: Hi Roy/All, Agree totally it would be good to get this CTD netCDF done in an interoperable way. Regarding having pressure vs. depth. We liked to use pressure because: 1) It is the thing that is measured, let us store just that and calculate depth if needed. 2) Depth can later be easily be calculated on the fly using the 'Pressure to Depth' algorithm in UNESCO (1983) Techinical Papers in Marine Science 44, Algorithms for Computation of fundamental properties of Seawater. One can use the python seawater library (see http://packages.python.org/seawater/ ) and seawater.csiro.depth(p, lat) to get depth from pressure and seawater.csiro.pres(depth, lat) to get pressure from depth. 3) I noted that the ARGO project (1's CTD like profiles) and others like CSIRO Oceanography in Aust. make data available with just pressure. 4) It makes our processing and QC a whole lot simpler. We don't have to worry about calculating and managing the extra 'depth' variable. Is there any problem with having the pressure as a co-ordinate which isn't really a dimensional quantity like depth (z) in the 4-D sense i.e x,y,z,t ? However I note that pressure (decibar) is allowed as a vertical axis e.g see section 4.3. Vertical (Height or Depth) Coordinate of CF v1.6 conventions document which says: A vertical coordinate will be identifiable by: . units of pressure; or . the presence of the positive attribute with a value of up or down (case insensitive). AND section 4.3.1. Dimensional Vertical Coordinate which says: The units attribute for dimensional coordinates will be a string formatted as per the udunits.dat3 file. The acceptable units for vertical (depth or height) coordinate variables are: . units of pressure as listed in the file udunits.dat. For vertical axes the most commonly used of these include include bar, millibar, decibar, atmosphere (atm), pascal (Pa), and hPa. ... ...¶ Regards, Andrew - Original Message - From: Lowry, Roy K. To: andrew walsh Cc: Jim Biard ; Upendra Dadi ; cf-metadata@cgd.ucar.edu ; Luke Callcut ; g...@metoc.gov.au ; sdn2-t...@seadatanet.org Sent: Tuesday, April 03, 2012 2:16 PM Subject: RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, SeaDataNet will very soon be considering how it is going to encode data, including single CTD casts, in CF-compliant NetCDF and so I think the time is ripe for agreeing how the significant numbers of us who indulge in this practice for whatever reason do it. That way we'll end up with interoperable data. I think there are a number of people on this list who have already encoded single CTDs into NetCDF and so it would be useful to start by asking for descriptions (like Andrew's examples) of how this has been done and what tools are dependent upon that encoding. The z co-ordinate parameter
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Dear Andrew and Nan If your pressure isn't monotonic, and so can't be a coordinate variable, I don't know whether it's preferable to demote ALL the coordinates to auxiliary coordinate variable status, TEMPERATURE: coordinates = TIME,PRESSURE,LATITUDE,LONGITUDE ; I don't think there's a recommendation to treat the coordinates in the same way. If size-1 coordinates are listed in the coordinates attribute, it is also permissible to make them scalar variables and not define the size-1 dimension. These are CF scalar coordinate variables. e.g. float lon; float lat; float time; float temperature(pressure); coordinates=lat lon time; where all the variables should have units and standard_names as usual. It doesn't matter what order is used for the coordinates attribute. Note that it's a blank-separated list (not comma-separated). Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi. A colleague of mine has told me that if you use size-1 array for a variable, then data servers such as THREDDS can aggregate the variable over multiple files and deliver a single file in which the variables will be arrays of size 1. He said that if a variable is a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am just parroting what he said.) If this is correct, then you might want to consider this point when deciding which way to represent coordinates. Grace and peace, Jim On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi upendra.d...@noaa.gov wrote: Andrew, However I have another small concern about all of this. Given there is more than 1 way to treat dimensions does the netCDF plotting and visualising sofware out there understand both approaches. That is can we expect something like ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure and work OK with BOTH approaches to coding the netCDF? Software like ncBrowse do not understand CF conventions. They can read netCDF files, but wouldn't know how to interpret the attributes like coordinates. So if you want to plot the profile location on a map, for example, it wouldn't be able to do it by itself. It would need to be told how to interpret the coordinates. A CF based software wouldn't have to be supplied with the additional semantics since it can read from the file by itself. But if you want to plot salinity vs pressure, software like ncBrowse can already do it since lot of these software make it easy to make plots based on a shared dimension. And here salinity and pressure share the same dimension. So I guess, the correct answer to your question is - it depends on what kind of plot or task you want to do and if the software can understand CF conventions. Upendra Thanks and Regards All, Andrew Walsh - Original Message - *From:* Lowry, Roy K. r...@bodc.ac.uk *To:* Upendra Dadi upendra.d...@noaa.gov ; andrew walshawa...@metoc.gov.au *Cc:* Luke Callcut l...@metoc.gov.au ; cf-metadata@cgd.ucar.edu ; sdn2-t...@seadatanet.org *Sent:* Tuesday, April 03, 2012 9:31 AM *Subject:* RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Upendra, I like the idea of a station dimension. It goes a long way to resolving the issue raised in my response to Jim which was based on the tunnel vision of having pressure/depth as a dimension. I have yet to look at the recently published NODC NetCDF templates. Is this CTD encoding included in them? If so, I'll bump up looking at them on my 'todo' list. I'd also recommend that Andrew and my colleagues in SeaDataNet take a look. Cheers, Roy. -- *From:* cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Upendra Dadi [upendra.d...@noaa.gov] *Sent:* 02 April 2012 17:21 *To:* andrew walsh *Cc:* Luke Callcut; cf-metadata@cgd.ucar.edu *Subject:* Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, Either way it should be okay as far as CF compliance is concerned. But the dimensions - latitude, longitude and time are not really required. If it is required to indicate that there is only one station(profile) in the file, there could be a dimension for number of stations instead, with a value of 1. Also, using a station dimension is the way to go if storing a collection of profiles in a single file. Here at NODC, we took the approach that we would use the same consistent representation whether there is a single instance or a collection in a file. Upendra On Mon, Apr 2, 2012 at 2:51 AM, andrew walsh awa...@metoc.gov.au wrote: Hi CF lis We are working on coding up some 1000's netCDF files off CTD instruments and want to make usre we are following the latest netCDF conventions (v1.6) OK. As background the CTD records a profile pressure, temperature and salinity. Here is a summarised CDL version (not all attributes+variables+qc flags there, just majors for now) of what we propose: dimensions: TIME=1 PRESSURE=729 LATITUDE=1 LONGITUDE=1 variables: double TIME(TIME) ; TIME:standard_name = time ; TIME:units = days since 1950-01-01 00:00:00Z ; TIME:axis = T ; TIME:valid_min = 0. ; TIME:valid_max = 99. ; double LATITUDE(LATITUDE) ; LATITUDE:standard_name = latitude ; LATITUDE:units = degrees_north ; LATITUDE:axis = Y ; LATITUDE:valid_min = -90. ; LATITUDE:valid_max = 90. ; double LONGITUDE(LONGITUDE) ; LONGITUDE:standard_name = longitude ; LONGITUDE:units = degrees_east ; LONGITUDE:axis = X ; LONGITUDE:valid_min = -180. ; LONGITUDE:valid_max = 180. ; double PRESSURE(PRESSURE) ; PRESSURE:standard_name = sea_water_pressure ; PRESSURE:units = decibars ; PRESSURE:axis = Z ; PRESSURE:valid_min = 0. ; PRESSURE:valid_max = 12000. ; PRESSURE:positive = down ; double TEMPERATURE(PRESSURE) ; TEMPERATURE:standard_name = sea_water_temperature ; TEMPERATURE:units
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi, I think Jim is talking about NCML (virtual) aggregation. THREDDS can aggregate even when the variable is a scalar using joinNew. But I think THREDDS can aggregate only over regular arrays. That is, all the dimensions other than over which a variable is aggregated should be of same size across all the files. This is possible, for example, only if all the profiles have same depth( or pressure) levels. Which in general is not true. NCML aggregation I guess is of limited use when dealing with jagged arrays. But I agree with Jim in that ability to aggregate should be an important consideration when coming up with a representation. Physical aggregation is still possible. But I prefer virtual aggregation, since letting data to be stored in individual files in often operationally convenient, but users benefit from aggregated views. I wonder if NCML could be extended to be able to aggregate jagged arrays into incomplete multidimensional array representations. I heard that ERDDAP has similar capability though I don't think it has anything like NCML where users can create views over remote data, not sure though. Upendra On 4/3/2012 1:27 PM, Jim Biard wrote: Hi. A colleague of mine has told me that if you use size-1 array for a variable, then data servers such as THREDDS can aggregate the variable over multiple files and deliver a single file in which the variables will be arrays of size 1. He said that if a variable is a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am just parroting what he said.) If this is correct, then you might want to consider this point when deciding which way to represent coordinates. Grace and peace, Jim On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi upendra.d...@noaa.gov mailto:upendra.d...@noaa.gov wrote: Andrew, However I have another small concern about all of this. Given there is more than 1 way to treat dimensions does the netCDF plotting and visualising sofware out there understand both approaches. That is can we expect something like ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure and work OK with BOTH approaches to coding the netCDF? Software like ncBrowse do not understand CF conventions. They can read netCDF files, but wouldn't know how to interpret the attributes like coordinates. So if you want to plot the profile location on a map, for example, it wouldn't be able to do it by itself. It would need to be told how to interpret the coordinates. A CF based software wouldn't have to be supplied with the additional semantics since it can read from the file by itself. But if you want to plot salinity vs pressure, software like ncBrowse can already do it since lot of these software make it easy to make plots based on a shared dimension. And here salinity and pressure share the same dimension. So I guess, the correct answer to your question is - it depends on what kind of plot or task you want to do and if the software can understand CF conventions. Upendra Thanks and Regards All, Andrew Walsh - Original Message - *From:* Lowry, Roy K. mailto:r...@bodc.ac.uk *To:* Upendra Dadi mailto:upendra.d...@noaa.gov ; andrew walsh mailto:awa...@metoc.gov.au *Cc:* Luke Callcut mailto:l...@metoc.gov.au ; cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu ; sdn2-t...@seadatanet.org mailto:sdn2-t...@seadatanet.org *Sent:* Tuesday, April 03, 2012 9:31 AM *Subject:* RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Upendra, I like the idea of a station dimension. It goes a long way to resolving the issue raised in my response to Jim which was based on the tunnel vision of having pressure/depth as a dimension. I have yet to look at the recently published NODC NetCDF templates. Is this CTD encoding included in them? If so, I'll bump up looking at them on my 'todo' list. I'd also recommend that Andrew and my colleagues in SeaDataNet take a look. Cheers, Roy. *From:* cf-metadata-boun...@cgd.ucar.edu mailto:cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Upendra Dadi [upendra.d...@noaa.gov mailto:upendra.d...@noaa.gov] *Sent:* 02 April 2012 17:21 *To:* andrew walsh *Cc:* Luke Callcut; cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu *Subject:* Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, Either way it should be okay as far as CF compliance is concerned. But the dimensions - latitude, longitude and time
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi all: Lets see, I havent followed the entire conversation, but: 1) Andrew if you can send me a sample file (not just the CDL) I can check if it works in the CDM with the new 1.6 conventions, and maybe give you some advice from my POV. 2) Aggregation in the CDM comes in 2 flavors. 1) The original implementation simply appends multidimensional arrays together, eg joinNew and joinExisting NCML aggregation. I call it syntactic aggregation because it doesnt know what its aggregating, and the homogeneity requirements are strict. 2) Feature Type collections (aka semantic aggregation) are the more recent development. These understand the coordinate information of the data, and so can handle variations in the representation, eg ragged arrays, scalar coordinates, etc. In this case, the CDM understands a dataset as a collection of features objects, which can be stored in a collection of files. The interfaces to these collections is still under development. Most current and future work in the CDM is in this category. John On 4/3/2012 12:48 PM, Upendra Dadi wrote: Hi, I think Jim is talking about NCML (virtual) aggregation. THREDDS can aggregate even when the variable is a scalar using joinNew. But I think THREDDS can aggregate only over regular arrays. That is, all the dimensions other than over which a variable is aggregated should be of same size across all the files. This is possible, for example, only if all the profiles have same depth( or pressure) levels. Which in general is not true. NCML aggregation I guess is of limited use when dealing with jagged arrays. But I agree with Jim in that ability to aggregate should be an important consideration when coming up with a representation. Physical aggregation is still possible. But I prefer virtual aggregation, since letting data to be stored in individual files in often operationally convenient, but users benefit from aggregated views. I wonder if NCML could be extended to be able to aggregate jagged arrays into incomplete multidimensional array representations. I heard that ERDDAP has similar capability though I don't think it has anything like NCML where users can create views over remote data, not sure though. Upendra On 4/3/2012 1:27 PM, Jim Biard wrote: Hi. A colleague of mine has told me that if you use size-1 array for a variable, then data servers such as THREDDS can aggregate the variable over multiple files and deliver a single file in which the variables will be arrays of size 1. He said that if a variable is a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am just parroting what he said.) If this is correct, then you might want to consider this point when deciding which way to represent coordinates. Grace and peace, Jim On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi upendra.d...@noaa.gov mailto:upendra.d...@noaa.gov wrote: Andrew, However I have another small concern about all of this. Given there is more than 1 way to treat dimensions does the netCDF plotting and visualising sofware out there understand both approaches. That is can we expect something like ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure and work OK with BOTH approaches to coding the netCDF? Software like ncBrowse do not understand CF conventions. They can read netCDF files, but wouldn't know how to interpret the attributes like coordinates. So if you want to plot the profile location on a map, for example, it wouldn't be able to do it by itself. It would need to be told how to interpret the coordinates. A CF based software wouldn't have to be supplied with the additional semantics since it can read from the file by itself. But if you want to plot salinity vs pressure, software like ncBrowse can already do it since lot of these software make it easy to make plots based on a shared dimension. And here salinity and pressure share the same dimension. So I guess, the correct answer to your question is - it depends on what kind of plot or task you want to do and if the software can understand CF conventions. Upendra Thanks and Regards All, Andrew Walsh - Original Message - *From:* Lowry, Roy K. mailto:r...@bodc.ac.uk *To:* Upendra Dadi mailto:upendra.d...@noaa.gov ; andrew walsh mailto:awa...@metoc.gov.au *Cc:* Luke Callcut mailto:l...@metoc.gov.au ; cf-metadata@cgd.ucar.edu mailto:cf-metadata@cgd.ucar.edu ; sdn2-t...@seadatanet.org mailto:sdn2-t...@seadatanet.org *Sent:* Tuesday, April 03, 2012 9:31 AM *Subject:* RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Upendra, I like the idea of a station dimension. It goes a long way to resolving the issue raised in my response to Jim which was based on the tunnel vision of having pressure
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Thierry, Thank you for this information. These netCDF examples and documentation are excellent. I was able to visualise (vertical cross section of Temperature ,Salinity) the Pheidippides glider netCDF data with the ncBrowse tool OK. Regards, Andrew - Original Message - From: Thierry Carval thierry.car...@ifremer.fr To: 'Lowry, Roy K.' r...@bodc.ac.uk Cc: 'Luke Callcut' l...@metoc.gov.au; cf-metadata@cgd.ucar.edu; 'Upendra Dadi' upendra.d...@noaa.gov; sdn2-t...@seadatanet.org; g...@metoc.gov.au; 'andrew walsh' awa...@metoc.gov.au Sent: Wednesday, April 04, 2012 4:31 AM Subject: TR: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hello Roy and All, I am involved in Argo data-management, OceanSITES and the MyOcean EU project. These 3 projects are intensive users of NetCDF CTD data files. Within these communities : · The natural vertical reference for a CTD is pressure. This is also true for Argo floats, gliders or sea-mammals. · The natural vertical reference for XBTs, moorings and buoys is depth. So the NetCDF files vertical reference can be either pressure or depth, but not both. If another community wants an homogeneous vertical reference, a profile with additional data (pressure or depth) could be provided, as a product. BUT, this adds complexity; I feel more comfortable with data reported as naturally as possible : · If you add a vertical reference, you have to decide which one is the Z axis (PRES:axis = Z or DEPH:axis = Z attribute). · All the QC flags assigned to pressure should be transferred to depth; but this is not straightforward. If the salinity is bad or spiky, from a good pressure you will have a bad depth. Here are links to different NetCDF user's manual managing vertical profiles with pressure or depth (but not both) vertical axes: · Argo documentation http://www.argodatamgt.org/Documentation , the user's manual http://www.argodatamgt.org/content/download/12096/80327/file/argo-dm-user-m anual.pdf : we are heading toward 1 million profiles · OceanSITES http://www.oceansites.org/data/index.html , the user's manual http://www.oceansites.org/docs/oceansites_user_manual_version1.2.pdf : 1400 long time-series from mooring sites The EU MyOcean project http://www.myocean.eu.org/ adopted the OceanSITES implementation of NetCDF CF for oceanographic in-situ data. The last 3 years of in-situ observations are available in OceanSITES NetCDF CF files (see attached map of 3.3 million vertical profiles). This format is also used for EGO gliders http://ego.dt.insu.cnrs.fr/dokuwiki/doku.php?id=start , where pressure is the vertical axis. Example : Pheidippides glider data http://www.ifremer.fr/co/ego/ego/pheidippides/ (NetCDF OceanSITES vertical profiles) operating now around Cyprus island. The above user's manuals for NetCDF-CF CTDs (and others) files may be enhanced with this ongoing v1.6 proposal. Best regards, Thierry -Message d'origine- De : cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] De la part de Lowry, Roy K. Envoyé : mardi 3 avril 2012 12:56 À : Lowry, Roy K.; andrew walsh Cc : Luke Callcut; cf-metadata@cgd.ucar.edu; Upendra Dadi; sdn2-t...@seadatanet.org; g...@metoc.gov.au Objet : Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hello again, It has been pointed out to me that in the current SeaDataNet format specification (written and updated by me!) that the initial position of insisting upon depth for the z co-ordinate was relaxed to allow either pressure or depth. So, I guess we need to be relaxed and leave z co-ordinate harmonisation to the the application tools. Yet another 'senior moment'.. Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Lowry, Roy K. [r...@bodc.ac.uk] Sent: 03 April 2012 06:46 To: andrew walsh Cc: cf-metadata@cgd.ucar.edu; sdn2-t...@seadatanet.org; g...@metoc.gov.au; Luke Callcut; Upendra Dadi Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi again, The reason that I prefer depth to pressure as the dimension for CTD is that the information required for pressure to depth conversion is virtually always present in the CTD data. However, in other oceanographic profiles - say nutrient bottle data - it is quite possible to have depth present without temperature and salinity, making the conversion to pressure impossible. Calculating depth from pressure at the time of standardised formatting of CTD data is as you say is trivial. So, why not make the CTD data more compatible with bottle data for very little cost? SeaDataNet already mandate inclusion of depth in their other data formats for CTD data delivery and so I can't see them switching to pressure for the dimension in NetCDF. Cheers, Roy. From: andrew
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Jonathan, Thanks, for your reply and from others in the CF community. I certainly have had lots of responses on this topic, all useful information. I appreciate your deep knowledge of the CF standard which often clarifies the best way to do things. Actually I am not 100% clear about use of auxilliary co-ordinate variables and the coordinates attribute. Some questions: 1) What distinguishes an 'auxilliary coordinate variable' from a plain old 'coordinate variable'? 2) Is it permissable for a 'auxilliary coordinate variable' to have either the scalar form e.g float lat or the array form lat(lat) with lat=1 declared as a dimension? 3) Do we need the coordinates attribute to look-up the corresponding auxilliary cordinates variable? For example if we had a variable declared as salinity(z) then to look up its lat, long and time coordinates for say display purpose would the salinity:coordinates = lat long time attribute used for the lookup. 4) However if we had the salinity variable be declared as salinity(lat,long,time,z) then the application would lookup the co-ordinates from the co-ordinates variables declaration names and we wouldn't need the salinity:coordinates attribute? I do notice that a lot of the netCDF CTD data out there doesn't use the coordinates attribute. Is this a feature that came in later with CF v1.5 or v1.6? Regards, Andrew PS: To clarify, the netCDF CTD data we produce will have monotonic pressures. The raw data we get in may go up/down due the motion of the CTD instrument. - Original Message - From: Jonathan Gregory j.m.greg...@reading.ac.uk To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 12:47 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Dear Andrew and Nan If your pressure isn't monotonic, and so can't be a coordinate variable, I don't know whether it's preferable to demote ALL the coordinates to auxiliary coordinate variable status, TEMPERATURE: coordinates = TIME,PRESSURE,LATITUDE,LONGITUDE ; I don't think there's a recommendation to treat the coordinates in the same way. If size-1 coordinates are listed in the coordinates attribute, it is also permissible to make them scalar variables and not define the size-1 dimension. These are CF scalar coordinate variables. e.g. float lon; float lat; float time; float temperature(pressure); coordinates=lat lon time; where all the variables should have units and standard_names as usual. It doesn't matter what order is used for the coordinates attribute. Note that it's a blank-separated list (not comma-separated). Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Upendra and Jim, Thanks for answering my questions about capabilities of the viewing sofware e.g ncBrowse and also the ideas around aggregation. A NCML enhancement to allow aggregation of different sized depth/pressure arrays could be very useful for ocean profile data from e.g. CTD, ARGO, Glider and XBT instruments. Regards, Andrew - Original Message - From: Upendra Dadi To: cf-metadata@cgd.ucar.edu Sent: Wednesday, April 04, 2012 4:48 AM Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi, I think Jim is talking about NCML (virtual) aggregation. THREDDS can aggregate even when the variable is a scalar using joinNew. But I think THREDDS can aggregate only over regular arrays. That is, all the dimensions other than over which a variable is aggregated should be of same size across all the files. This is possible, for example, only if all the profiles have same depth( or pressure) levels. Which in general is not true. NCML aggregation I guess is of limited use when dealing with jagged arrays. But I agree with Jim in that ability to aggregate should be an important consideration when coming up with a representation. Physical aggregation is still possible. But I prefer virtual aggregation, since letting data to be stored in individual files in often operationally convenient, but users benefit from aggregated views. I wonder if NCML could be extended to be able to aggregate jagged arrays into incomplete multidimensional array representations. I heard that ERDDAP has similar capability though I don't think it has anything like NCML where users can create views over remote data, not sure though. Upendra On 4/3/2012 1:27 PM, Jim Biard wrote: Hi. A colleague of mine has told me that if you use size-1 array for a variable, then data servers such as THREDDS can aggregate the variable over multiple files and deliver a single file in which the variables will be arrays of size 1. He said that if a variable is a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am just parroting what he said.) If this is correct, then you might want to consider this point when deciding which way to represent coordinates. Grace and peace, Jim On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi upendra.d...@noaa.gov wrote: Andrew, However I have another small concern about all of this. Given there is more than 1 way to treat dimensions does the netCDF plotting and visualising sofware out there understand both approaches. That is can we expect something like ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure and work OK with BOTH approaches to coding the netCDF? Software like ncBrowse do not understand CF conventions. They can read netCDF files, but wouldn't know how to interpret the attributes like coordinates. So if you want to plot the profile location on a map, for example, it wouldn't be able to do it by itself. It would need to be told how to interpret the coordinates. A CF based software wouldn't have to be supplied with the additional semantics since it can read from the file by itself. But if you want to plot salinity vs pressure, software like ncBrowse can already do it since lot of these software make it easy to make plots based on a shared dimension. And here salinity and pressure share the same dimension. So I guess, the correct answer to your question is - it depends on what kind of plot or task you want to do and if the software can understand CF conventions. Upendra Thanks and Regards All, Andrew Walsh - Original Message - From: Lowry, Roy K. To: Upendra Dadi ; andrew walsh Cc: Luke Callcut ; cf-metadata@cgd.ucar.edu ; sdn2-t...@seadatanet.org Sent: Tuesday, April 03, 2012 9:31 AM Subject: RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Upendra, I like the idea of a station dimension. It goes a long way to resolving the issue raised in my response to Jim which was based on the tunnel vision of having pressure/depth as a dimension. I have yet to look at the recently published NODC NetCDF templates. Is this CTD encoding included in them? If so, I'll bump up looking at them on my 'todo' list. I'd also recommend that Andrew and my colleagues in SeaDataNet take a look. Cheers, Roy. From: cf-metadata-boun...@cgd.ucar.edu [cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Upendra Dadi [upendra.d...@noaa.gov] Sent: 02 April 2012 17:21 To: andrew walsh Cc: Luke Callcut; cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, Either way it should be okay as far as CF compliance is concerned. But the dimensions - latitude, longitude and time are not really required. If it is required to indicate that there is only one station(profile) in the file, there could be a dimension for number of stations instead, with a value of 1. Also, using a station dimension is the way to go if storing a collection
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi. Here's my best understanding (which could be flawed, so I look forward to finding out if I have missed something). Assuming that your time, latitude, and longitude are all proper coordinate variables (no fill values), either choice for the shape of your salinity variable works. If you use the more complex shape, then you don't need to have the coordinates attribute, as the relationships are declared through the shape. Speaking of that, the coordinates attribute doesn't need to have PRESSURE in it. Doesn't hurt anything, but it isn't necessary. Having said all that, I find myself wondering why you are storing each salinity profile in a separate file. Would you mind sharing what went into deciding to take this approach? Grace and peace, Jim On Mon, Apr 2, 2012 at 2:51 AM, andrew walsh awa...@metoc.gov.au wrote: Hi CF list, We are working on coding up some 1000's netCDF files off CTD instruments and want to make usre we are following the latest netCDF conventions (v1.6) OK. As background the CTD records a profile pressure, temperature and salinity. Here is a summarised CDL version (not all attributes+variables+qc flags there, just majors for now) of what we propose: dimensions: TIME=1 PRESSURE=729 LATITUDE=1 LONGITUDE=1 variables: double TIME(TIME) ; TIME:standard_name = time ; TIME:units = days since 1950-01-01 00:00:00Z ; TIME:axis = T ; TIME:valid_min = 0. ; TIME:valid_max = 99. ; double LATITUDE(LATITUDE) ; LATITUDE:standard_name = latitude ; LATITUDE:units = degrees_north ; LATITUDE:axis = Y ; LATITUDE:valid_min = -90. ; LATITUDE:valid_max = 90. ; double LONGITUDE(LONGITUDE) ; LONGITUDE:standard_name = longitude ; LONGITUDE:units = degrees_east ; LONGITUDE:axis = X ; LONGITUDE:valid_min = -180. ; LONGITUDE:valid_max = 180. ; double PRESSURE(PRESSURE) ; PRESSURE:standard_name = sea_water_pressure ; PRESSURE:units = decibars ; PRESSURE:axis = Z ; PRESSURE:valid_min = 0. ; PRESSURE:valid_max = 12000. ; PRESSURE:positive = down ; double TEMPERATURE(PRESSURE) ; TEMPERATURE:standard_name = sea_water_temperature ; TEMPERATURE:units = degrees_C ; TEMPERATURE:_FillValue = -99.99 ; TEMPERATURE:valid_min = -2. ; TEMPERATURE:valid_max = 40. ; TEMPERATURE:coordinates=TIME LATITUDE LONGITUDE PRESSURE double SALINITY(PRESSURE) ; SALINITY:standard_name = sea_water_salinity ; SALINITY:units = psu ; SALINITY:_FillValue = -99.99 ; SALINITY:valid_min = 0. ; SALINITY:valid_max = 40. ; SALINITY:coordinates=TIME LATITUDE LONGITUDE PRESSURE // global attributes: :conventions = CF-1.6 ; :featureType = profile :cdm_data_type = profile + several other attributes later for ISO19115 metadata generation I am not sure if I should have TEMPERATURE and SALINITY arrays with 4 dimensions like TEMPERATURE(TIME,LATITUDE,**LONGITUDE,PRESSURE) or just 1 dimension like I have above i.e. TEMPERATURE(PRESSURE). ? Any feedback on the above is greatly appreciated. Andrew Walsh __**_ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/**mailman/listinfo/cf-metadatahttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- Jim Biard Research Scholar Cooperative Institute for Climate and Satellites Remote Sensing and Applications Division National Climatic Data Center 151 Patton Ave, Asheville, NC 28801-5001 jim.bi...@noaa.gov 828-271-4900 ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Andrew, Either way it should be okay as far as CF compliance is concerned. But the dimensions - latitude, longitude and time are not really required. If it is required to indicate that there is only one station(profile) in the file, there could be a dimension for number of stations instead, with a value of 1. Also, using a station dimension is the way to go if storing a collection of profiles in a single file. Here at NODC, we took the approach that we would use the same consistent representation whether there is a single instance or a collection in a file. Upendra On Mon, Apr 2, 2012 at 2:51 AM, andrew walsh awa...@metoc.gov.au wrote: Hi CF lis We are working on coding up some 1000's netCDF files off CTD instruments and want to make usre we are following the latest netCDF conventions (v1.6) OK. As background the CTD records a profile pressure, temperature and salinity. Here is a summarised CDL version (not all attributes+variables+qc flags there, just majors for now) of what we propose: dimensions: TIME=1 PRESSURE=729 LATITUDE=1 LONGITUDE=1 variables: double TIME(TIME) ; TIME:standard_name = time ; TIME:units = days since 1950-01-01 00:00:00Z ; TIME:axis = T ; TIME:valid_min = 0. ; TIME:valid_max = 99. ; double LATITUDE(LATITUDE) ; LATITUDE:standard_name = latitude ; LATITUDE:units = degrees_north ; LATITUDE:axis = Y ; LATITUDE:valid_min = -90. ; LATITUDE:valid_max = 90. ; double LONGITUDE(LONGITUDE) ; LONGITUDE:standard_name = longitude ; LONGITUDE:units = degrees_east ; LONGITUDE:axis = X ; LONGITUDE:valid_min = -180. ; LONGITUDE:valid_max = 180. ; double PRESSURE(PRESSURE) ; PRESSURE:standard_name = sea_water_pressure ; PRESSURE:units = decibars ; PRESSURE:axis = Z ; PRESSURE:valid_min = 0. ; PRESSURE:valid_max = 12000. ; PRESSURE:positive = down ; double TEMPERATURE(PRESSURE) ; TEMPERATURE:standard_name = sea_water_temperature ; TEMPERATURE:units = degrees_C ; TEMPERATURE:_FillValue = -99.99 ; TEMPERATURE:valid_min = -2. ; TEMPERATURE:valid_max = 40. ; TEMPERATURE:coordinates=TIME LATITUDE LONGITUDE PRESSURE double SALINITY(PRESSURE) ; SALINITY:standard_name = sea_water_salinity ; SALINITY:units = psu ; SALINITY:_FillValue = -99.99 ; SALINITY:valid_min = 0. ; SALINITY:valid_max = 40. ; SALINITY:coordinates=TIME LATITUDE LONGITUDE PRESSURE // global attributes: :conventions = CF-1.6 ; :featureType = profile :cdm_data_type = profile + several other attributes later for ISO19115 metadata generation I am not sure if I should have TEMPERATURE and SALINITY arrays with 4 dimensions like TEMPERATURE(TIME,LATITUDE,**LONGITUDE,PRESSURE) or just 1 dimension like I have above i.e. TEMPERATURE(PRESSURE). ? Any feedback on the above is greatly appreciated. Andrew Walsh __**_ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/**mailman/listinfo/cf-metadatahttp://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi Roy/All, Agree totally it would be good to get this CTD netCDF done in an interoperable way. Regarding having pressure vs. depth. We liked to use pressure because: 1) It is the thing that is measured, let us store just that and calculate depth if needed. 2) Depth can later be easily be calculated on the fly using the 'Pressure to Depth' algorithm in UNESCO (1983) Techinical Papers in Marine Science 44, Algorithms for Computation of fundamental properties of Seawater. One can use the python seawater library (see http://packages.python.org/seawater/ ) and seawater.csiro.depth(p, lat) to get depth from pressure and seawater.csiro.pres(depth, lat) to get pressure from depth. 3) I noted that the ARGO project (1's CTD like profiles) and others like CSIRO Oceanography in Aust. make data available with just pressure. 4) It makes our processing and QC a whole lot simpler. We don't have to worry about calculating and managing the extra 'depth' variable. Is there any problem with having the pressure as a co-ordinate which isn't really a dimensional quantity like depth (z) in the 4-D sense i.e x,y,z,t ? However I note that pressure (decibar) is allowed as a vertical axis e.g see section 4.3. Vertical (Height or Depth) Coordinate of CF v1.6 conventions document which says: A vertical coordinate will be identifiable by: . units of pressure; or . the presence of the positive attribute with a value of up or down (case insensitive). AND section 4.3.1. Dimensional Vertical Coordinate which says: The units attribute for dimensional coordinates will be a string formatted as per the udunits.dat3 file. The acceptable units for vertical (depth or height) coordinate variables are: . units of pressure as listed in the file udunits.dat. For vertical axes the most commonly used of these include include bar, millibar, decibar, atmosphere (atm), pascal (Pa), and hPa. ... ...¶ Regards, Andrew - Original Message - From: Lowry, Roy K. To: andrew walsh Cc: Jim Biard ; Upendra Dadi ; cf-metadata@cgd.ucar.edu ; Luke Callcut ; g...@metoc.gov.au ; sdn2-t...@seadatanet.org Sent: Tuesday, April 03, 2012 2:16 PM Subject: RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, SeaDataNet will very soon be considering how it is going to encode data, including single CTD casts, in CF-compliant NetCDF and so I think the time is ripe for agreeing how the significant numbers of us who indulge in this practice for whatever reason do it. That way we'll end up with interoperable data. I think there are a number of people on this list who have already encoded single CTDs into NetCDF and so it would be useful to start by asking for descriptions (like Andrew's examples) of how this has been done and what tools are dependent upon that encoding. The z co-ordinate parameter (pressure/depth) is also an issue worth resolving. Whilst interconversions are relatively straightforward, agreement would make life much easier. My preference leans dowards having depth as the dimension with pressure as an optional variable. That way we interoperate with other kinds of oceanographic profile data such as bottle data. If we can get that far, we can then look at how to aggregate multiple CTDs into a file according to the CF point data conventions. Cheers, Roy. From: andrew walsh [awa...@metoc.gov.au] Sent: 03 April 2012 04:39 To: Lowry, Roy K. Cc: Jim Biard; Upendra Dadi; cf-metadata@cgd.ucar.edu; Luke Callcut; g...@metoc.gov.au Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hello Roy, Upendra, Jim and CF list, Thanks for all your feedbacks. My proposal relates to single CTD profile data (a vertical profile of pressure, Temp. Salinity) not a trajectory in x,y,z,t and I have put :featureType = profile as recommended in the global attributes section. As Roy has mentioned to Jim CTD data is usually processed and QC'ed as a single profile per netCDF file so that's why I am doing it this way. Aggregations using multiple CTD profiles per netCDF file may be constructed later at say national/international data centres and these aggregrations would follow the CF conventions v1.6, Chapter 9 - Discrete Sampling geometries and also the new netCDF templates provided by NODC, thanks NODC -:) Roy, The recent NODC netCDF templates don't have an aggregation example for CTD however the Profile/World Ocean Database Observed Level example comes close (see http://www.nodc.noaa.gov/data/formats/netcdf/) and click on Profile/World Ocean Database Observed Level link. This example appears to be for ocean station/bottle samples with vertical dimension of depth (z) (m) rather than case of CTD which would use pressure (dbar) as the vertical (z) dimension. It would be useful I think to have a NODC netCDF template for an aggregation of CTD casts. Upendra, Based on your responses and what I have seen at NODC and other places it seems there are 2 methods to do
Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6
Hi again, The reason that I prefer depth to pressure as the dimension for CTD is that the information required for pressure to depth conversion is virtually always present in the CTD data. However, in other oceanographic profiles - say nutrient bottle data - it is quite possible to have depth present without temperature and salinity, making the conversion to pressure impossible. Calculating depth from pressure at the time of standardised formatting of CTD data is as you say is trivial. So, why not make the CTD data more compatible with bottle data for very little cost? SeaDataNet already mandate inclusion of depth in their other data formats for CTD data delivery and so I can't see them switching to pressure for the dimension in NetCDF. Cheers, Roy. From: andrew walsh [awa...@metoc.gov.au] Sent: 03 April 2012 06:15 To: Lowry, Roy K. Cc: Jim Biard; Upendra Dadi; cf-metadata@cgd.ucar.edu; Luke Callcut; g...@metoc.gov.au; sdn2-t...@seadatanet.org Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Roy/All, Agree totally it would be good to get this CTD netCDF done in an interoperable way. Regarding having pressure vs. depth. We liked to use pressure because: 1) It is the thing that is measured, let us store just that and calculate depth if needed. 2) Depth can later be easily be calculated on the fly using the 'Pressure to Depth' algorithm in UNESCO (1983) Techinical Papers in Marine Science 44, Algorithms for Computation of fundamental properties of Seawater. One can use the python seawater library (see http://packages.python.org/seawater/ ) and seawater.csiro.depth(p, lat) to get depth from pressure and seawater.csiro.pres(depth, lat) to get pressure from depth. 3) I noted that the ARGO project (1's CTD like profiles) and others like CSIRO Oceanography in Aust. make data available with just pressure. 4) It makes our processing and QC a whole lot simpler. We don't have to worry about calculating and managing the extra 'depth' variable. Is there any problem with having the pressure as a co-ordinate which isn't really a dimensional quantity like depth (z) in the 4-D sense i.e x,y,z,t ? However I note that pressure (decibar) is allowed as a vertical axis e.g see section 4.3. Vertical (Height or Depth) Coordinate of CF v1.6 conventions document which says: A vertical coordinate will be identifiable by: . units of pressure; or . the presence of the positive attribute with a value of up or down (case insensitive). AND section 4.3.1. Dimensional Vertical Coordinate which says: The units attribute for dimensional coordinates will be a string formatted as per the udunits.dat3 file. The acceptable units for vertical (depth or height) coordinate variables are: . units of pressure as listed in the file udunits.dat. For vertical axes the most commonly used of these include include bar, millibar, decibar, atmosphere (atm), pascal (Pa), and hPa. ... ...¶ Regards, Andrew - Original Message - From: Lowry, Roy K. To: andrew walsh Cc: Jim Biard ; Upendra Dadi ; cf-metadata@cgd.ucar.edu ; Luke Callcut ; g...@metoc.gov.au ; sdn2-t...@seadatanet.org Sent: Tuesday, April 03, 2012 2:16 PM Subject: RE: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hi Andrew, SeaDataNet will very soon be considering how it is going to encode data, including single CTD casts, in CF-compliant NetCDF and so I think the time is ripe for agreeing how the significant numbers of us who indulge in this practice for whatever reason do it. That way we'll end up with interoperable data. I think there are a number of people on this list who have already encoded single CTDs into NetCDF and so it would be useful to start by asking for descriptions (like Andrew's examples) of how this has been done and what tools are dependent upon that encoding. The z co-ordinate parameter (pressure/depth) is also an issue worth resolving. Whilst interconversions are relatively straightforward, agreement would make life much easier. My preference leans dowards having depth as the dimension with pressure as an optional variable. That way we interoperate with other kinds of oceanographic profile data such as bottle data. If we can get that far, we can then look at how to aggregate multiple CTDs into a file according to the CF point data conventions. Cheers, Roy. From: andrew walsh [awa...@metoc.gov.au] Sent: 03 April 2012 04:39 To: Lowry, Roy K. Cc: Jim Biard; Upendra Dadi; cf-metadata@cgd.ucar.edu; Luke Callcut; g...@metoc.gov.au Subject: Re: [CF-metadata] Ocean CTD data following CF Conventions v1.6 Hello Roy, Upendra, Jim and CF list, Thanks for all your feedbacks. My proposal relates to single CTD profile data (a vertical profile of pressure, Temp. Salinity) not a trajectory in x,y,z,t and I have put :featureType = profile as recommended in the global attributes section. As Roy has mentioned to Jim CTD data is usually processed and QC'ed
