Re: [CF-metadata] thredds changing CF conventions version
Hi Seth: On 8/28/2013 12:59 PM, Seth McGinnis wrote: Hi Sean, Personally, I would regard that as suspect behavior. I'm of the opinion that it's best practice for a data service to affect the data it operates on in very targeted and transparent way, and that it should pass everything else through untouched. If it doesn't do that, it's harder to trust the service, because the more ways that it modifies the data, the more opportunities there are for it to get something wrong, and the more likely it is that the error will go unnoticed because it's hiding amongst a bunch of other changes. We take this philosophy also. Case in point, I have recently discovered that when the THREDDS server subsets CF data that uses a coordinate system other that lat-lon, it mangles a bunch of the ancillary data in the file. I suspect this happens, ironically enough, precisely because it's trying to make the data CF-1.0 conformant instead of just leaving it as-is... Let us know what the specifics are and we can try to make it better. So yes, I think it's significant that the data coming out of THREDDS is in a different convention than the source files, and that it's cause for concern. A famous story from NCAR data portal early days is that some users added the Conventions = "CF-1.0" attribute to their data because they had to make their data CF compliant. But thats all they did. Hey, it passed the compliance checker! ;) Regards, John ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] thredds changing CF conventions version
Hi John and all: For sure its a good idea to preserve the CF Conventions attribute if it exists. I will add that to our issue tracker. Remember that THREDDS is a general purpose subsetter, not specific to CF. So we deal with lots of source datasets (eg GRIB) that arent CF. The point is that we convert whatever we have to CF. Up to this point, as CF versions have evolved, there hasnt been any difference in the way that we write out these files. So im interested if Ive missed something and there is some actual change in the file structure that we should be tracking from Sean's POV. Other than the actual version number. John On 8/28/2013 11:11 AM, John Graybeal wrote: It isn't just about feature presentation, right? On the assumption that the latest version is always most current (so everyone will use the latest version if they can), I use the version as a proxy for how current/up-to-date/sophisticated the data provider is. It also gives me a clue about which standard name table version they might use, though that version info is often available elsewhere. If true, this may help explain why so few data providers seemed to be using 1.6, anyway John On Aug 28, 2013, at 08:56, John Caron wrote: Hi Sean: What feature of CF are you using that you need to preserve the version? John "If you torture data long enough, it will confess." -- Economist Ronald Coase On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote: Hi, Here at the British Oceanographic Data Centre we use THREDDS to deliver and subset our numerical model data that we hold in CF netcdf format. I’ve just been made aware that during the delivery and subsetting process, THREDDS seems to be converting the CF files we hold from whatever CF convention they have been supplied in, to CF 1.0. Is this something that the rest of the community are aware of, and if so, do people consider it to be of major importance that the files outputted by thredds are in a different convention to the source files thredds interrogates? Cheers Sean Gaffney BODC _ _ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata John Graybeal Senior Data Manager, Metadata and Semantics T +1 (408) 675-5545 F +1 (408) 616-1626 skype: graybealski Marinexplore 920 Stewart Drive Sunnyvale, CA ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] thredds changing CF conventions version
Hi Sean, Personally, I would regard that as suspect behavior. I'm of the opinion that it's best practice for a data service to affect the data it operates on in very targeted and transparent way, and that it should pass everything else through untouched. If it doesn't do that, it's harder to trust the service, because the more ways that it modifies the data, the more opportunities there are for it to get something wrong, and the more likely it is that the error will go unnoticed because it's hiding amongst a bunch of other changes. Case in point, I have recently discovered that when the THREDDS server subsets CF data that uses a coordinate system other that lat-lon, it mangles a bunch of the ancillary data in the file. I suspect this happens, ironically enough, precisely because it's trying to make the data CF-1.0 conformant instead of just leaving it as-is... So yes, I think it's significant that the data coming out of THREDDS is in a different convention than the source files, and that it's cause for concern. Cheers, --Seth Seth McGinnis NARCCAP Data Manager NCAR - IMAGe On Wed, 28 Aug 2013 10:11:10 -0700 John Graybeal wrote: >It isn't just about feature presentation, right? On the assumption that the >latest version is always most current (so everyone will use the latest version >if they can), I use the version as a proxy for how >current/up-to-date/sophisticated the data provider is. > >It also gives me a clue about which standard name table version they might >use, though that version info is often available elsewhere. > >If true, this may help explain why so few data providers seemed to be using >1.6, anyway > >John > >On Aug 28, 2013, at 08:56, John Caron wrote: > >> Hi Sean: >> >> What feature of CF are you using that you need to preserve the version? >> >> John >> >> "If you torture data long enough, it will confess." >> -- Economist Ronald Coase >> >> >> >> On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote: >>> Hi, >>> Here at the British Oceanographic Data Centre we use THREDDS to deliver >>> and subset our numerical model data that we hold in CF netcdf format. >>> I’ve just been made aware that during the delivery and subsetting >>> process, THREDDS seems to be converting the CF files we hold from >>> whatever CF convention they have been supplied in, to CF 1.0. >>> Is this something that the rest of the community are aware of, and if >>> so, do people consider it to be of major importance that the files >>> outputted by thredds are in a different convention to the source files >>> thredds interrogates? >>> Cheers >>> Sean Gaffney >>> BODC >>> >>> _ _ >>> This message (and any attachments) is for the recipient only. NERC is >>> subject to the Freedom of Information Act 2000 and the contents of this >>> email and any reply you make may be disclosed by NERC unless it is >>> exempt from release under the Act. Any material supplied to NERC may be >>> stored in an electronic records management system. >>> >>> >>> ___ >>> CF-metadata mailing list >>> CF-metadata@cgd.ucar.edu >>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata >>> >> ___ >> CF-metadata mailing list >> CF-metadata@cgd.ucar.edu >> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > >John Graybeal >Senior Data Manager, Metadata and Semantics > >T +1 (408) 675-5545 >F +1 (408) 616-1626 >skype: graybealski > >Marinexplore >920 Stewart Drive >Sunnyvale, CA > > > >___ >CF-metadata mailing list >CF-metadata@cgd.ucar.edu >http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] thredds changing CF conventions version
It isn't just about feature presentation, right? On the assumption that the latest version is always most current (so everyone will use the latest version if they can), I use the version as a proxy for how current/up-to-date/sophisticated the data provider is. It also gives me a clue about which standard name table version they might use, though that version info is often available elsewhere. If true, this may help explain why so few data providers seemed to be using 1.6, anyway John On Aug 28, 2013, at 08:56, John Caron wrote: > Hi Sean: > > What feature of CF are you using that you need to preserve the version? > > John > > "If you torture data long enough, it will confess." > -- Economist Ronald Coase > > > > On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote: >> Hi, >> Here at the British Oceanographic Data Centre we use THREDDS to deliver >> and subset our numerical model data that we hold in CF netcdf format. >> I’ve just been made aware that during the delivery and subsetting >> process, THREDDS seems to be converting the CF files we hold from >> whatever CF convention they have been supplied in, to CF 1.0. >> Is this something that the rest of the community are aware of, and if >> so, do people consider it to be of major importance that the files >> outputted by thredds are in a different convention to the source files >> thredds interrogates? >> Cheers >> Sean Gaffney >> BODC >> >> _ _ >> This message (and any attachments) is for the recipient only. NERC is >> subject to the Freedom of Information Act 2000 and the contents of this >> email and any reply you make may be disclosed by NERC unless it is >> exempt from release under the Act. Any material supplied to NERC may be >> stored in an electronic records management system. >> >> >> ___ >> CF-metadata mailing list >> CF-metadata@cgd.ucar.edu >> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata >> > ___ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata John Graybeal Senior Data Manager, Metadata and Semantics T +1 (408) 675-5545 F +1 (408) 616-1626 skype: graybealski Marinexplore 920 Stewart Drive Sunnyvale, CA ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] thredds changing CF conventions version
Hi Sean: What feature of CF are you using that you need to preserve the version? John "If you torture data long enough, it will confess." -- Economist Ronald Coase On 8/28/2013 3:28 AM, Gaffney, Sean P. wrote: Hi, Here at the British Oceanographic Data Centre we use THREDDS to deliver and subset our numerical model data that we hold in CF netcdf format. I’ve just been made aware that during the delivery and subsetting process, THREDDS seems to be converting the CF files we hold from whatever CF convention they have been supplied in, to CF 1.0. Is this something that the rest of the community are aware of, and if so, do people consider it to be of major importance that the files outputted by thredds are in a different convention to the source files thredds interrogates? Cheers Sean Gaffney BODC _ _ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] thredds changing CF conventions version
Hi, Here at the British Oceanographic Data Centre we use THREDDS to deliver and subset our numerical model data that we hold in CF netcdf format. I've just been made aware that during the delivery and subsetting process, THREDDS seems to be converting the CF files we hold from whatever CF convention they have been supplied in, to CF 1.0. Is this something that the rest of the community are aware of, and if so, do people consider it to be of major importance that the files outputted by thredds are in a different convention to the source files thredds interrogates? Cheers Sean Gaffney BODC This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
