Hi all:
Lets see, I havent followed the entire conversation, but:
1) Andrew if you can send me a sample file (not just the CDL) I can
check if it works in the CDM with the new 1.6 conventions, and maybe
give you some advice from my POV.
2) Aggregation in the CDM comes in 2 flavors. 1) The original
implementation simply appends multidimensional arrays together, eg
"joinNew " and "joinExisting" NCML aggregation. I call it "syntactic
aggregation" because it doesnt know what its aggregating, and the
homogeneity requirements are strict. 2) "Feature Type collections" (aka
"semantic aggregation") are the more recent development. These
understand the coordinate information of the data, and so can handle
variations in the representation, eg ragged arrays, scalar coordinates,
etc. In this case, the CDM understands a dataset as a collection of
"features objects", which can be stored in a collection of files. The
interfaces to these collections is still under development. Most current
and future work in the CDM is in this category.
John
On 4/3/2012 12:48 PM, Upendra Dadi wrote:
Hi,
I think Jim is talking about NCML (virtual) aggregation. THREDDS
can aggregate even when the variable is a scalar using "joinNew". But
I think THREDDS can aggregate only over regular arrays. That is, all
the dimensions other than over which a variable is aggregated should
be of same size across all the files. This is possible, for example,
only if all the profiles have same depth( or pressure) levels. Which
in general is not true. NCML aggregation I guess is of limited use
when dealing with jagged arrays. But I agree with Jim in that ability
to aggregate should be an important consideration when coming up with
a representation. Physical aggregation is still possible. But I prefer
virtual aggregation, since letting data to be stored in individual
files in often operationally convenient, but users benefit from
aggregated views. I wonder if NCML could be extended to be able to
aggregate jagged arrays into incomplete multidimensional array
representations. I heard that ERDDAP has similar capability though I
don't think it has anything like NCML where users can create views
over remote data, not sure though.
Upendra
On 4/3/2012 1:27 PM, Jim Biard wrote:
Hi.
A colleague of mine has told me that if you use size-1 array for a
variable, then data servers such as THREDDS can aggregate the
variable over multiple files and deliver a single file in which the
variables will be arrays of size > 1. He said that if a variable is
a scalar, THREDDS won't do this. (I don't mess with THREDDS, so I am
just parroting what he said.)
If this is correct, then you might want to consider this point when
deciding which way to represent coordinates.
Grace and peace,
Jim
On Tue, Apr 3, 2012 at 1:02 PM, Upendra Dadi <upendra.d...@noaa.gov
<mailto:upendra.d...@noaa.gov>> wrote:
Andrew,
However I have another small concern about all of this. Given
there is more than
1 way to treat dimensions does the netCDF plotting and
visualising sofware out
there understand both approaches. That is can we expect
something like
ncBrowse, IDV or ODV? give me a plot of say salinity vs. pressure
and work OK with BOTH approaches to coding the netCDF?
Software like ncBrowse do not understand CF conventions. They
can read netCDF files, but wouldn't know how to interpret the
attributes like "coordinates". So if you want to plot the profile
location on a map, for example, it wouldn't be able to do it by
itself. It would need to be told how to interpret the
coordinates. A CF based software wouldn't have to be supplied
with the additional semantics since it can read from the file by
itself. But if you want to plot salinity vs pressure, software
like ncBrowse can already do it since lot of these software make
it easy to make plots based on a shared dimension. And here
salinity and pressure share the same dimension. So I guess, the
correct answer to your question is - it depends on what kind of
plot or task you want to do and if the software can understand CF
conventions.
Upendra
Thanks and Regards All,
Andrew Walsh
----- Original Message -----
*From:* Lowry, Roy K. <mailto:r...@bodc.ac.uk>
*To:* Upendra Dadi <mailto:upendra.d...@noaa.gov> ; andrew
walsh <mailto:awa...@metoc.gov.au>
*Cc:* Luke Callcut <mailto:l...@metoc.gov.au> ;
cf-metadata@cgd.ucar.edu <mailto:cf-metadata@cgd.ucar.edu> ;
sdn2-t...@seadatanet.org <mailto:sdn2-t...@seadatanet.org>
*Sent:* Tuesday, April 03, 2012 9:31 AM
*Subject:* RE: [CF-metadata] Ocean CTD data following CF
Conventions v1.6
Hi Upendra,
I like the idea of a station dimension. It goes a long way
to resolving the issue raised in my response to Jim which
was based on the tunnel vision of having pressure/depth as a
dimension. I have yet to look at the recently published
NODC NetCDF templates. Is this CTD encoding included in
them? If so, I'll bump up looking at them on my 'todo'
list. I'd also recommend that Andrew and my colleagues in
SeaDataNet take a look.
Cheers, Roy.
------------------------------------------------------------------------
*From:* cf-metadata-boun...@cgd.ucar.edu
<mailto:cf-metadata-boun...@cgd.ucar.edu>
[cf-metadata-boun...@cgd.ucar.edu
<mailto:cf-metadata-boun...@cgd.ucar.edu>] On Behalf Of
Upendra Dadi [upendra.d...@noaa.gov
<mailto:upendra.d...@noaa.gov>]
*Sent:* 02 April 2012 17:21
*To:* andrew walsh
*Cc:* Luke Callcut; cf-metadata@cgd.ucar.edu
<mailto:cf-metadata@cgd.ucar.edu>
*Subject:* Re: [CF-metadata] Ocean CTD data following CF
Conventions v1.6
Hi Andrew,
Either way it should be okay as far as CF compliance is
concerned. But the dimensions - latitude, longitude and time
are not really required. If it is required to indicate that
there is only one station(profile) in the file, there could
be a dimension for number of stations instead, with a value
of 1. Also, using a station dimension is the way to go if
storing a collection of profiles in a single file. Here at
NODC, we took the approach that we would use the same
consistent representation whether there is a single instance
or a collection in a file.
Upendra
On Mon, Apr 2, 2012 at 2:51 AM, andrew walsh
<awa...@metoc.gov.au <mailto:awa...@metoc.gov.au>> wrote:
Hi CF lis
We are working on coding up some 1000's netCDF files off CTD
instruments and want to make usre we are following the
latest netCDF conventions (v1.6) OK. As background the
CTD records
a profile pressure, temperature and salinity.
Here is a summarised CDL version (not all
attributes+variables+qc flags there, just majors for now)
of what we propose:
dimensions:
TIME=1
PRESSURE=729
LATITUDE=1
LONGITUDE=1
variables:
double TIME(TIME) ;
TIME:standard_name = "time" ;
TIME:units = "days since 1950-01-01 00:00:00Z" ;
TIME:axis = "T" ;
TIME:valid_min = 0. ;
TIME:valid_max = 999999. ;
double LATITUDE(LATITUDE) ;
LATITUDE:standard_name = "latitude" ;
LATITUDE:units = "degrees_north" ;
LATITUDE:axis = "Y" ;
LATITUDE:valid_min = -90. ;
LATITUDE:valid_max = 90. ;
double LONGITUDE(LONGITUDE) ;
LONGITUDE:standard_name = "longitude" ;
LONGITUDE:units = "degrees_east" ;
LONGITUDE:axis = "X" ;
LONGITUDE:valid_min = -180. ;
LONGITUDE:valid_max = 180. ;
double PRESSURE(PRESSURE) ;
PRESSURE:standard_name = "sea_water_pressure" ;
PRESSURE:units = "decibars" ;
PRESSURE:axis = "Z" ;
PRESSURE:valid_min = 0. ;
PRESSURE:valid_max = 12000. ;
PRESSURE:positive = "down" ;
double TEMPERATURE(PRESSURE) ;
TEMPERATURE:standard_name = "sea_water_temperature" ;
TEMPERATURE:units = "degrees_C" ;
TEMPERATURE:_FillValue = -99.99 ;
TEMPERATURE:valid_min = -2. ;
TEMPERATURE:valid_max = 40. ;
TEMPERATURE:coordinates="TIME LATITUDE LONGITUDE PRESSURE"
double SALINITY(PRESSURE) ;
SALINITY:standard_name = "sea_water_salinity" ;
SALINITY:units = "psu" ;
SALINITY:_FillValue = -99.99 ;
SALINITY:valid_min = 0. ;
SALINITY:valid_max = 40. ;
SALINITY:coordinates="TIME LATITUDE LONGITUDE PRESSURE"
// global attributes:
:conventions = "CF-1.6" ;
:featureType = "profile"
:cdm_data_type = "profile"
+ several other attributes later for ISO19115 metadata
generation
I am not sure if I should have TEMPERATURE and SALINITY
arrays with 4 dimensions
like TEMPERATURE(TIME,LATITUDE,LONGITUDE,PRESSURE) or
just 1 dimension
like I have above i.e. TEMPERATURE(PRESSURE). ?
Any feedback on the above is greatly appreciated.
Andrew Walsh
_______________________________________________
CF-metadata mailing list
CF-metadata@cgd.ucar.edu <mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
This message (and any attachments) is for the recipient
only. NERC
is subject to the Freedom of Information Act 2000 and the
contents
of this email and any reply you make may be disclosed by
NERC unless
it is exempt from release under the Act. Any material
supplied to
NERC may be stored in an electronic records management system.
_______________________________________________
CF-metadata mailing list
CF-metadata@cgd.ucar.edu <mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
--
Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites
Remote Sensing and Applications Division
National Climatic Data Center
151 Patton Ave, Asheville, NC 28801-5001
jim.bi...@noaa.gov <mailto:jim.bi...@noaa.gov>
828-271-4900
_______________________________________________
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
_______________________________________________
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
_______________________________________________
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
