Re: [CF-metadata] Sea water transparency and reflectance.
Hi Frederic,
Thank you for your input.
Yes there is already a reflectance quantity in CF, but it is the TOA one.
Anyway to me the definition of reflectance in general is probably broader than
in your needs, then I think introducing
ratio_of_upwelling_radiance_emerging_from_sea_water_to_downwelling_radiative_flux_in_air
(sr^-1)
as you suggested, is a good idea because usually CF standard names have to
match exactly the geophysical content of one variable. And, moreover, I think
it's better to build a new standard name from existing ones.
Concerning the geometry of observation, we could introduce angle_of_incidence
as suggested by Jonathan, plus probably something like azimuth_angle?
And in CF there is already quantities such as solar_azimuth_angle and
solar_elevation_angle to describe the solar part in the geometry of
observation in ocean colour products, if necessary.
Best regards,
Olivier.
-Message d'origine-
De : Frederic MELIN [mailto:frederic.me...@jrc.ec.europa.eu]
Envoyé : jeudi 22 octobre 2009 10:16
À : 'Jonathan Gregory'; olivier lauret
Cc : 'Laurence Crosnier'; cf-metadata@cgd.ucar.edu; 'Andrei LINTU';
cristina.tronc...@artov.isac.cnr.it; 'Philippe Garnesson'; 'Ludovic BOURG';
'Yann BARZIC'
Objet : RE: [CF-metadata] Sea water transparency and reflectance.
Dear all,
a (late, sorry!) input on the definition of sea water reflectance, as an
ocean color product.
I have some doubts for surface_bidirectional_reflectance.
For the sake of completeness, in marine optics, 'reflectance' is used mainly
in two ways:
the irradiance reflectance, Eu/Ed (upwelling irradiance / downwelling
irradiance), usually measured at 0- (just below water surface).
the radiance reflectance, Lu/Ed (upwelling radiance / downwelling
irradiance), at 0- or 0+ (just above water).
Often, the radiance reflectance is given at 0+ and then noted Lw/Ed (water
leaving radiance / downwelling irradiance), and is also termed remote
sensing reflectance, because it is a standard satellite product (Eu/Ed is
not).
This product has a bidirectional dependence (on geometry of illumination and
observation).
Actually, looking at the existing CF list, some quantities relevant to the
discussion are there, and might be helpful.
I would say that the ratio of
surface_upwelling_spectral_radiative_flux_in_sea_water
and
surface_downwelling_spectral_radiative_flux_in_sea_water
is Eu/Ed at 0- (irradiance reflectance).
and the ratio of:
surface_upwelling_spectral_radiance_in_air_emerging_from_sea_water
and
surface_downwelling_spectral_radiative_flux_in_air
is Lw/Ed at 0+ , i.e. (remote sensing) radiance reflectance (the term
'emerging_from_sea_water' is important).
So following the CF definition of a ratio, the remote sensing reflectance
would be something like:
ratio_of_upwelling_radiance_emerging_from_sea_water_to_
downwelling_radiative_flux_in_air , in sr^-1.
If the term surface_bidirectional_reflectance is fully equivalent to this
(long!) definition, it is fine. Part of my doubts come from the fact that I
weren't able to find 'reflectance' in the CF list.
surface_bidirectional_reflectance implies that it is a radiance
reflectance (bidirectional). On the other hand, it should specify the
vertical coordinate ('in_air' or 'in_sea_water'). For 0+ (Lw), it should say
something about 'emerging_from_sea_water' because this is the satellite
product of interest.
Being bidirectional, it should contain the value of 'angle_of_incidence' and
other angles if possible (in the present case the term 'angle_of_reflection'
is not appropriate because we are not describing a reflection - the geometry
of observation should be defined with a viewing angle and a relative
azimuth, but I don't know the names for these). These angles are 0
(vertical) if the remote sensing reflectance is 'normalized'. In optical
remote sensing, 'normalized' usually means corrected for bidirectional
effects (as if sun and observer were directly ahead). The satellite product
then becomes an intrinsic property of the water, independent of conditions
of observation and illumination. This is at least valid for the NASA
products. This might be too complicated to be included in the name, and
anyway the actual result depends on the models used for these corrections. I
guess that these aspects should be described in the product description.
I hope this helps,
Best,
frederic
-Original Message-
From: cf-metadata-boun...@cgd.ucar.edu
[mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory
Sent: Monday, 19 October, 2009 19:04
To: olivier lauret
Cc: Laurence Crosnier; cf-metadata@cgd.ucar.edu; Andrei LINTU;
cristina.tronc...@artov.isac.cnr.it; Philippe Garnesson; Ludovic BOURG; Yann
BARZIC
Subject: [CF-metadata] Sea water transparency and reflectance.
Dear Olivier
*About reflectance, the quantity that is usually provided in ocean
colour products is the bidirectional reflectance; albedo is of the
same nature than reflectance, except
Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf
Thanks for the responses. Seth is right, I was looking for an alternative to having a gigantic flag_meanings attribute. I have tried Seth's approach, but can’t get it past the cf-checker. There appears to be a character set restriction: I can get the first 547 flag_meaning strings accepted if I remove all the -, . and , characters. I'm reasonably sure there are no control characters in the following strings which could be responsible for the error message that I get (ERROR (3.5): Invalid syntax for 'flag_meanings' attribute) when I increase the number of flag_meanings to 550. Could it be something to do with the string length (16472 characters)? I could try abbreviating the flag_meanings, but this would either be tedious work by hand or liable to create confusion if done automatically. If you could point me to some information about exactly what string format is acceptable, that would be very helpful. To answer John's question: I'm currently using netcdf 3, in IDL -- I'm not sure what the options for upgrading to netcdf 4 are. Cheers, Matin -Original Message- From: Seth McGinnis [mailto:mcgin...@ucar.edu] Sent: 26 October 2009 23:13 To: John Graybeal; Juckes, Martin (STFC,RAL,SSTD) Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf On Mon, 26 Oct 2009 12:16:17 -0700 John Graybeal grayb...@marinemetadata.org wrote: Love the question! :- Are you saying every map location has one index value, which is one of the numbers from 1 to 827? So these are mutually exclusive? Or are there 827 flag values assigned independently for each map location? It sounds to me like it's the former case, and the problem is that concatenating 827 flag values into a single string attribute makes something that is easy for a machine to interpret, but impenetrable to the human reader. As I read section 3.5, there are no other options for encoding the information according to spec; a gigantic flag_meanings attribute is the right answer. But if the issue is really usability, I think the answer is not to do it differently, but just to add an extra attribute that presents the same information in a more user-friendly format. You could add an attribute named something descriptive, like category_legend, and pair the values meanings up, one per line. Then you'd end up with something that looks like this: float landtype(yc, xc) ; landtype:long_name =Land cover type ; landtype:standard_name =land_cover ; landtype:flag_values = 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f, 10.f, 11.f, 12.f, 13.f, 14.f, 15.f, 16.f, 17.f, 18.f, 19.f, 20.f, 21.f, 22.f, 23.f, 24 .f; landtype:flag_meanings =Urban_and_Built-up_Land Dryland_Cropland_and_Pasture Irrigated_Cropland_and_Pasture Mixed_Dryland/Irrigated_Cropland_and_Pasture Cropland/Grassland_Mosaic Cropland/Woodland_Mosaic Grassland Shrubland Mixed_Shrubland/Grassland Savanna Deciduous_Broadleaf_Forest Deciduous_Needleleaf_Forest Evergreen_Broadleaf Evergreen_Needleleaf Mixed_Forest Water_Bodies Herbaceous_Wetland Wooden_Wetland Barren_or_Sparsely_Vegetated Herbaceous_Tundra Wooded_Tundra Mixed_Tundra Bare_Ground_Tundra Snow_or_Ice ; landtype:category_legend =\n, 1: Urban and Built-up Land \n, 2: Dryland Cropland and Pasture \n, 3: Irrigated Cropland and Pasture \n, 4: Mixed Dryland/Irrigated Cropland and Pasture \n, 5: Cropland/Grassland Mosaic \n, 6: Cropland/Woodland Mosaic \n, 7: Grassland \n, 8: Shrubland \n, 9: Mixed Shrubland/Grassland \n, 10: Savanna \n, 11: Deciduous Broadleaf Forest \n, 12: Deciduous Needleleaf Forest \n, 13: Evergreen Broadleaf \n, 14: Evergreen Needleleaf \n, 15: Mixed Forest \n, 16: Water Bodies \n, 17: Herbaceous Wetland \n, 18: Wooden Wetland \n, 19: Barren or Sparsely Vegetated \n, 20: Herbaceous Tundra \n, 21: Wooded Tundra \n, 22: Mixed Tundra \n, 23: Bare Ground Tundra \n, 24: Snow or Ice ; (Sorry for the length, but it gives a real sense of what I mean.) The flag_values and flag_meanings attributes are still a mess to look at, but they're easy to deal with programmatically, while end-users can easily look at category_legend and figure out what cover type 22 is if that's all they're after. Fundamentally, this is the same issue as the question of how to record the grid mapping, and the underlying problem is that while attributes allow you to associate key-value pairs with a variable, there's no way to nest another associative array within an attribute; the best you can do is reference a
Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf
Hello Martin, There is another possible solution to your problem, which we are looking at for dealing with a data source flag to be used with the GEBCO bathymetric grid. This is to put a URI base into an attribute that when concatenated with a flag values gives the flag definition from a vocabulary server. For example, having a flag_definition attribute like: URI_base='http://vocab.ndg.nerc.ac.uk/term/GGS1/current/' For a flag value of 1, the resulting URL is: http://vocab.ndg.nerc.ac.uk/term/GGS1/current/1 which is a live NERC Vocabulary Server URL returning the flag definition attributes in a SKOS document. I appreciate that this moves away from fully self-contained CF files and that a standardised syntax for the URI base encoding is required, but I think this provides a scalable solution to the flag definition problem. Cheers, Roy. -Original Message- From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of martin.juc...@stfc.ac.uk Sent: 27 October 2009 10:45 To: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf Thanks for the responses. Seth is right, I was looking for an alternative to having a gigantic flag_meanings attribute. I have tried Seth's approach, but can’t get it past the cf-checker. There appears to be a character set restriction: I can get the first 547 flag_meaning strings accepted if I remove all the -, . and , characters. I'm reasonably sure there are no control characters in the following strings which could be responsible for the error message that I get (ERROR (3.5): Invalid syntax for 'flag_meanings' attribute) when I increase the number of flag_meanings to 550. Could it be something to do with the string length (16472 characters)? I could try abbreviating the flag_meanings, but this would either be tedious work by hand or liable to create confusion if done automatically. If you could point me to some information about exactly what string format is acceptable, that would be very helpful. To answer John's question: I'm currently using netcdf 3, in IDL -- I'm not sure what the options for upgrading to netcdf 4 are. Cheers, Matin -Original Message- From: Seth McGinnis [mailto:mcgin...@ucar.edu] Sent: 26 October 2009 23:13 To: John Graybeal; Juckes, Martin (STFC,RAL,SSTD) Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf On Mon, 26 Oct 2009 12:16:17 -0700 John Graybeal grayb...@marinemetadata.org wrote: Love the question! :- Are you saying every map location has one index value, which is one of the numbers from 1 to 827? So these are mutually exclusive? Or are there 827 flag values assigned independently for each map location? It sounds to me like it's the former case, and the problem is that concatenating 827 flag values into a single string attribute makes something that is easy for a machine to interpret, but impenetrable to the human reader. As I read section 3.5, there are no other options for encoding the information according to spec; a gigantic flag_meanings attribute is the right answer. But if the issue is really usability, I think the answer is not to do it differently, but just to add an extra attribute that presents the same information in a more user-friendly format. You could add an attribute named something descriptive, like category_legend, and pair the values meanings up, one per line. Then you'd end up with something that looks like this: float landtype(yc, xc) ; landtype:long_name =Land cover type ; landtype:standard_name =land_cover ; landtype:flag_values = 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f, 10.f, 11.f, 12.f, 13.f, 14.f, 15.f, 16.f, 17.f, 18.f, 19.f, 20.f, 21.f, 22.f, 23.f, 24 .f; landtype:flag_meanings =Urban_and_Built-up_Land Dryland_Cropland_and_Pasture Irrigated_Cropland_and_Pasture Mixed_Dryland/Irrigated_Cropland_and_Pasture Cropland/Grassland_Mosaic Cropland/Woodland_Mosaic Grassland Shrubland Mixed_Shrubland/Grassland Savanna Deciduous_Broadleaf_Forest Deciduous_Needleleaf_Forest Evergreen_Broadleaf Evergreen_Needleleaf Mixed_Forest Water_Bodies Herbaceous_Wetland Wooden_Wetland Barren_or_Sparsely_Vegetated Herbaceous_Tundra Wooded_Tundra Mixed_Tundra Bare_Ground_Tundra Snow_or_Ice ; landtype:category_legend =\n, 1: Urban and Built-up Land \n, 2: Dryland Cropland and Pasture \n, 3: Irrigated Cropland and Pasture \n, 4: Mixed Dryland/Irrigated Cropland and Pasture \n, 5: Cropland/Grassland Mosaic \n, 6: Cropland/Woodland Mosaic \n, 7: Grassland \n, 8: Shrubland \n, 9: Mixed Shrubland/Grassland \n, 10: Savanna \n, 11: Deciduous Broadleaf Forest \n,
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
Hello Roy, That is an interesting idea. There are definitions of these areas on a WWF site, of the form http://www.worldwildlife.org/wildworld/profiles/terrestrial/nt/nt0908_full.html where nt0908 is a tag associated with a particular flag value. My first thought was: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meaning]_full.html; with nt/nt0908 in the flag_meanings attribute, but it appears that the cf-checker does not like having a / in flag_meanings. This means we need to either have a more complex syntax, or set up our definition URLs along the lines you suggest. A more complex syntax might be: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meaning][0:2]/[flag_meaning]_full.html; Since the above returns html rather than skos/xml it should perhaps be complemented by a uri_content_type attribute, with value 'html/text'. I'll think about the option of setting up something like your vocab server -- or possibly putting some more definitions in the ndg vocab server? Regards, Martin -Original Message- From: Lowry, Roy K [mailto:r...@bodc.ac.uk] Sent: 27 October 2009 11:07 To: Juckes, Martin (STFC,RAL,SSTD); cf-metadata@cgd.ucar.edu Cc: Weatherall, Pauline Subject: RE: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf Hello Martin, There is another possible solution to your problem, which we are looking at for dealing with a data source flag to be used with the GEBCO bathymetric grid. This is to put a URI base into an attribute that when concatenated with a flag values gives the flag definition from a vocabulary server. For example, having a flag_definition attribute like: URI_base='http://vocab.ndg.nerc.ac.uk/term/GGS1/current/' For a flag value of 1, the resulting URL is: http://vocab.ndg.nerc.ac.uk/term/GGS1/current/1 which is a live NERC Vocabulary Server URL returning the flag definition attributes in a SKOS document. I appreciate that this moves away from fully self-contained CF files and that a standardised syntax for the URI base encoding is required, but I think this provides a scalable solution to the flag definition problem. Cheers, Roy. -Original Message- From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata- boun...@cgd.ucar.edu] On Behalf Of martin.juc...@stfc.ac.uk Sent: 27 October 2009 10:45 To: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf Thanks for the responses. Seth is right, I was looking for an alternative to having a gigantic flag_meanings attribute. I have tried Seth's approach, but can’t get it past the cf-checker. There appears to be a character set restriction: I can get the first 547 flag_meaning strings accepted if I remove all the -, . and , characters. I'm reasonably sure there are no control characters in the following strings which could be responsible for the error message that I get (ERROR (3.5): Invalid syntax for 'flag_meanings' attribute) when I increase the number of flag_meanings to 550. Could it be something to do with the string length (16472 characters)? I could try abbreviating the flag_meanings, but this would either be tedious work by hand or liable to create confusion if done automatically. If you could point me to some information about exactly what string format is acceptable, that would be very helpful. To answer John's question: I'm currently using netcdf 3, in IDL -- I'm not sure what the options for upgrading to netcdf 4 are. Cheers, Matin -Original Message- From: Seth McGinnis [mailto:mcgin...@ucar.edu] Sent: 26 October 2009 23:13 To: John Graybeal; Juckes, Martin (STFC,RAL,SSTD) Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf On Mon, 26 Oct 2009 12:16:17 -0700 John Graybeal grayb...@marinemetadata.org wrote: Love the question! :- Are you saying every map location has one index value, which is one of the numbers from 1 to 827? So these are mutually exclusive? Or are there 827 flag values assigned independently for each map location? It sounds to me like it's the former case, and the problem is that concatenating 827 flag values into a single string attribute makes something that is easy for a machine to interpret, but impenetrable to the human reader. As I read section 3.5, there are no other options for encoding the information according to spec; a gigantic flag_meanings attribute is the right answer. But if the issue is really usability, I think the answer is not to do it differently, but just to add an extra attribute that presents the same information in a more user-friendly format. You could add an attribute named something descriptive, like category_legend, and pair the values meanings up, one per line. Then you'd end up with something that looks like this:
Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf
martin.juc...@stfc.ac.uk wrote: Thanks for the responses. Seth is right, I was looking for an alternative to having a gigantic flag_meanings attribute. I have tried Seth's approach, but can’t get it past the cf-checker. There appears to be a character set restriction: I can get the first 547 flag_meaning strings accepted if I remove all the -, . and , characters. I'm reasonably sure there are no control characters in the following strings which could be responsible for the error message that I get (ERROR (3.5): Invalid syntax for 'flag_meanings' attribute) when I increase the number of flag_meanings to 550. Could it be something to do with the string length (16472 characters)? That seems to be an error in the cf-checker (?) I think attribute values should be any UTF-8 encoded Unicode characters (aka UTF-8 string). I could try abbreviating the flag_meanings, but this would either be tedious work by hand or liable to create confusion if done automatically. If you could point me to some information about exactly what string format is acceptable, that would be very helpful. To answer John's question: I'm currently using netcdf 3, in IDL -- I'm not sure what the options for upgrading to netcdf 4 are. I mention netcdf-4 because it has enum types - which would be a nice solution for this class of problems, at least when you want to include the meanings of the flags in the file. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf
Lowry, Roy K wrote: Hello Martin, There is another possible solution to your problem, which we are looking at for dealing with a data source flag to be used with the GEBCO bathymetric grid. This is to put a URI base into an attribute that when concatenated with a flag values gives the flag definition from a vocabulary server. For example, having a flag_definition attribute like: URI_base='http://vocab.ndg.nerc.ac.uk/term/GGS1/current/' For a flag value of 1, the resulting URL is: http://vocab.ndg.nerc.ac.uk/term/GGS1/current/1 which is a live NERC Vocabulary Server URL returning the flag definition attributes in a SKOS document. I appreciate that this moves away from fully self-contained CF files and that a standardised syntax for the URI base encoding is required, but I think this provides a scalable solution to the flag definition problem. Cheers, Roy. Hi Roy: If I was an application trying to use this feature, i would like to be able to download all the possible values for this flag enumeration with one call to your server. Is that possible? I guess the other concern with external tables is ensuring that the writer and the reader really have the same table. Is there some way to ensure that? ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf
Hi John, Yes, there is a URL of the form: http://vocab.ndg.nerc.ac.uk/list/GGS1/current/ that returns all the flags (currently only three as we have yet to fully populate the vocabulary). Note the term 'current' in the URL. This specifies that the current version should be served. However, if you want to be absolutely sure the external definitions and the internal reference match, then you can explicitly specify the version number. Compare and contrast two version of the GEBCO flag list using: http://vocab.ndg.nerc.ac.uk/list/GGS1/1/ and http://vocab.ndg.nerc.ac.uk/list/GGS1/2/ Cheers, Roy. -Original Message- From: John Caron [mailto:ca...@unidata.ucar.edu] Sent: 27 October 2009 13:17 To: Lowry, Roy K Cc: martin.juc...@stfc.ac.uk; cf-metadata@cgd.ucar.edu; Weatherall, Pauline Subject: Re: [CF-metadata] Dealing with large numbers of flag values innetcdf cf Lowry, Roy K wrote: Hello Martin, There is another possible solution to your problem, which we are looking at for dealing with a data source flag to be used with the GEBCO bathymetric grid. This is to put a URI base into an attribute that when concatenated with a flag values gives the flag definition from a vocabulary server. For example, having a flag_definition attribute like: URI_base='http://vocab.ndg.nerc.ac.uk/term/GGS1/current/' For a flag value of 1, the resulting URL is: http://vocab.ndg.nerc.ac.uk/term/GGS1/current/1 which is a live NERC Vocabulary Server URL returning the flag definition attributes in a SKOS document. I appreciate that this moves away from fully self-contained CF files and that a standardised syntax for the URI base encoding is required, but I think this provides a scalable solution to the flag definition problem. Cheers, Roy. Hi Roy: If I was an application trying to use this feature, i would like to be able to download all the possible values for this flag enumeration with one call to your server. Is that possible? I guess the other concern with external tables is ensuring that the writer and the reader really have the same table. Is there some way to ensure that? -- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
martin.juc...@stfc.ac.uk writes: Hello Roy, That is an interesting idea. There are definitions of these areas on a WWF site, of the form http://www.worldwildlife.org/wildworld/profiles/terrestrial/nt/nt0908_full.html where nt0908 is a tag associated with a particular flag value. My first thought was: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meaning]_full.html; with nt/nt0908 in the flag_meanings attribute, but it appears that the cf-checker does not like having a / in flag_meanings. This means we need to either have a more complex syntax, or set up our definition URLs along the lines you suggest. A more complex syntax might be: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meaning][0:2]/[flag_meaning]_full.html; Since the above returns html rather than skos/xml it should perhaps be complemented by a uri_content_type attribute, with value 'html/text'. I'll think about the option of setting up something like your vocab server -- or possibly putting some more definitions in the ndg vocab server? Regards, Martin Howdy all! I really don't think you should store metadata for a file on the web. Just put it in the file. Most scientific data are around a lot longer than most URLs. What happens in 15 years when someone wants to understand this data, and the URL has vanished? It the metadata were stored in the file, then it would be there for that programmer of the future. (And that programmer may well be you! insert scary Halloween laughter here) If the metadata seem too large to put in the file, consider storing them in variables, and then use netCDF-4 with compression turned on for those variables. Thanks, Ed -- Ed Hartnett -- e...@unidata.ucar.edu ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
This is a matter of philosophy and practice, I imagine. One of the reasons I recommend semantic URIs is that the URI represents a clue, if by some chance the web is not available or the information is not served at that URI any more. I think some semantic description should be included in the netCDF file -- on this I agree with Ed. But I agree with Roy on the bigger picture -- we are building a system that is intended to be long-lived, and in fact much of the web is moving to systems that assume URLs are long-lived. There are a lot of problems that will result if vocabulary servers like Roy's go away. And the fact is, you just can't store all the metadata in the file -- there is an expanding web of information associated with every observation, and much of that metadata can evolve as our knowledge evolves about the relevant instruments, best calibration values, algorithms, and so on. So we need to allow the ability to reference additional metadata, not just bring it all into every netCDF file (and then update all the files when additional metadata is obtained). John On Oct 27, 2009, at 0837, Ed Hartnett wrote: martin.juc...@stfc.ac.uk writes: Hello Roy, That is an interesting idea. There are definitions of these areas on a WWF site, of the form http://www.worldwildlife.org/wildworld/profiles/terrestrial/nt/nt0908_full.html where nt0908 is a tag associated with a particular flag value. My first thought was: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/ [flag_meaning]_full.html with nt/nt0908 in the flag_meanings attribute, but it appears that the cf-checker does not like having a / in flag_meanings. This means we need to either have a more complex syntax, or set up our definition URLs along the lines you suggest. A more complex syntax might be: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/ [flag_meaning][0:2]/[flag_meaning]_full.html Since the above returns html rather than skos/xml it should perhaps be complemented by a uri_content_type attribute, with value 'html/ text'. I'll think about the option of setting up something like your vocab server -- or possibly putting some more definitions in the ndg vocab server? Regards, Martin Howdy all! I really don't think you should store metadata for a file on the web. Just put it in the file. Most scientific data are around a lot longer than most URLs. What happens in 15 years when someone wants to understand this data, and the URL has vanished? It the metadata were stored in the file, then it would be there for that programmer of the future. (And that programmer may well be you! insert scary Halloween laughter here) If the metadata seem too large to put in the file, consider storing them in variables, and then use netCDF-4 with compression turned on for those variables. Thanks, Ed -- Ed Hartnett -- e...@unidata.ucar.edu ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- I have my new work email address: jgrayb...@ucsd.edu -- John Graybeal mailto:jgrayb...@ucsd.edu phone: 858-534-2162 Development Manager Ocean Observatories Initiative Cyberinfrastructure Project: http://ci.oceanobservatories.org Marine Metadata Interoperability Project: http://marinemetadata.org ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
Hello Ed, To Roy's comment I'll add that I would like to complement in-file metadata with additional online info, not replace it. E.g. for the tag NT0908 I have a name Paraná flooded savanna which can go in the file, but there is a web page giving the full characterisation of what is meant by Paraná flooded savanna, and this includes an extended description, references and photographs. It would be unreasonable to include all these photos etc in the file, so a convenient means of referring to them would be helpful. Regards, Martin -Original Message- From: Lowry, Roy K [mailto:r...@bodc.ac.uk] Sent: 27 October 2009 15:45 To: Ed Hartnett; Juckes, Martin (STFC,RAL,SSTD) Cc: cf-metadata@cgd.ucar.edu; Weatherall, Pauline Subject: RE: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf Hi Ed, Establishing on-line metadata resources with permanence is the name of my game. If I didn't have confidence it would be around in 15 years I wouldn't be promoting it. Cheers, Roy. -Original Message- From: Ed Hartnett [mailto:e...@unidata.ucar.edu] Sent: 27 October 2009 15:37 To: martin.juc...@stfc.ac.uk Cc: Lowry, Roy K; cf-metadata@cgd.ucar.edu; Weatherall, Pauline Subject: Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf martin.juc...@stfc.ac.uk writes: Hello Roy, That is an interesting idea. There are definitions of these areas on a WWF site, of the form http://www.worldwildlife.org/wildworld/profiles/terrestrial/nt/nt0908_f ull.html where nt0908 is a tag associated with a particular flag value. My first thought was: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meani ng]_full.html with nt/nt0908 in the flag_meanings attribute, but it appears that the cf-checker does not like having a / in flag_meanings. This means we need to either have a more complex syntax, or set up our definition URLs along the lines you suggest. A more complex syntax might be: URI= http://www.worldwildlife.org/wildworld/profiles/terrestrial/[flag_meani ng][0:2]/[flag_meaning]_full.html Since the above returns html rather than skos/xml it should perhaps be complemented by a uri_content_type attribute, with value 'html/text'. I'll think about the option of setting up something like your vocab server -- or possibly putting some more definitions in the ndg vocab server? Regards, Martin Howdy all! I really don't think you should store metadata for a file on the web. Just put it in the file. Most scientific data are around a lot longer than most URLs. What happens in 15 years when someone wants to understand this data, and the URL has vanished? It the metadata were stored in the file, then it would be there for that programmer of the future. (And that programmer may well be you! insert scary Halloween laughter here) If the metadata seem too large to put in the file, consider storing them in variables, and then use netCDF-4 with compression turned on for those variables. Thanks, Ed -- Ed Hartnett -- e...@unidata.ucar.edu -- This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. -- Scanned by iCritical. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
On Tuesday 27 October 2009 15:45:17 Lowry, Roy K wrote: Hi Ed, Establishing on-line metadata resources with permanence is the name of my game. If I didn't have confidence it would be around in 15 years I wouldn't be promoting it. My job is to persist data indefinitely, and I'm relying on vocabulary servers ... (and in particular, Roy's ... ) somehow we will persist the vocab.ndg.nerc.ac.uk address, regardless of nerc's future, and of ndg's future, although if ac.uk went away we might be in trouble :-). Cheers Bryan -- Bryan Lawrence Director of Environmental Archival and Associated Research (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC) STFC, Rutherford Appleton Laboratory Phone +44 1235 445012; Fax ... 5848; Web: home.badc.rl.ac.uk/lawrence ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Dealing with large numbers of flag valuesinnetcdf cf
Bryan Lawrence writes: My job is to persist data indefinitely, and I'm relying on vocabulary servers ... (and in particular, Roy's ... ) somehow we will persist the vocab.ndg.nerc.ac.uk address, regardless of nerc's future, and of ndg's future, although if ac.uk went away we might be in trouble :-). Could happen... http://www.guardian.co.uk/politics/2009/oct/18/conservatives-defence I see nerc.co.uk coming... -- V. Balaji Office: +1-609-452-6516 Head, Modeling Systems Group, GFDL Home:+1-212-253-6662 Princeton UniversityEmail: v.bal...@noaa.gov ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] CF point observation Conventions ready for review
I have complete a new version of the CF point observation Conventions at: https://cf-pcmdi.llnl.gov/trac/wiki/PointObservationConventions Discussion is at: https://cf-pcmdi.llnl.gov/trac/ticket/37 I have incorporated various feedback from the past year, and made a preliminary implementation to be sure that a generic application can distinguish the various cases without human intervention. I have tried to simplify, esp in the station profile and section feature types, where the combinations of options got too complex. The document is now explicit about all possible representations. The use of missing values is also clarified. While the document is rather long, if you manage to wade through it you'll see that the patterns of use keep repeating and are mostly regular. Showing full examples I think is the best way to prevent misunderstandings. I did make enough changes that anyone who wrote files using the previous version should check to see whats changed. Apologies for that. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
