Re: [CF-metadata] Reverse-time trajectory
So somewhat contrary to my previous statement, my current thinking is table 9.1 should keep the monotonically and ordered descriptions, and these should be clarified by further rules stating their ordered nature even when expressed as an aux coord var. I think it would be possible make such a restriction if you use the featureType attribute, which indicates that the rules of sect 9 apply. But the use of auxiliary coordinate variables for time already existed before sect 9 was written, in what sect 9 refers to as the orthogonal multidimensional representation, and that can be used without the featureType attribute. Why not sort the data into monotonic order of time before using it? From one perspective I don't really mind either approach - the logical view presented by software can be in time order irrespective of the encoding. But having to sort the data wastes time and makes the file less human-readable. Why put the onus on the consumer if there are very few/no producers who would find it hard to comply with the in-order-within-a-single-time-series constraint? It's analogous to the strictly-monotonic constraint on coordinate variables. CF could just state that they have to be logically monotonic and leave it to the consumer to sort. I'd love to see more feedback from the people who write real-time data-logging systems... Richard Hattersley AVD Iris Technical Lead Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44 (0)1392 885702 Fax: +44 (0)1392 885681 Email: richard.hatters...@metoffice.gov.uk Website: www.metoffice.gov.uk ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Dear Jonathan and John, What does 9.1 actually mean by monotonically increasing if the data can be stored in non-monotonic order? Ah yes, good point! Perhaps we should delete with monotonically increasing times in Table 9.1, and also ordered in the trajectoryProfile definition. Does CF really need to support unordered elements *within* a single[1] time series? e.g. Are there a significant number of real-time situations where the data from a single stream need to be written out of order to a CF file? So somewhat contrary to my previous statement, my current thinking is table 9.1 should keep the monotonically and ordered descriptions, and these should be clarified by further rules stating their ordered nature even when expressed as an aux coord var. [1] It's clear how it can help to support unordered elements *across* multiple time series - but that's taken care of by the indexed ragged array representation (9.3.4). Regards, Richard Hattersley AVD Iris Technical Lead Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44 (0)1392 885702 Fax: +44 (0)1392 885681 Email: richard.hatters...@metoffice.gov.uk Website: www.metoffice.gov.uk ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Dear Richard Does CF really need to support unordered elements *within* a single[1] time series? e.g. Are there a significant number of real-time situations where the data from a single stream need to be written out of order to a CF file? I think that's a reason for having the ragged representation, which mentions real-time data as a canonical use case. Data can arrive in the wrong order, I expect, even if it doesn't happen often. So somewhat contrary to my previous statement, my current thinking is table 9.1 should keep the monotonically and ordered descriptions, and these should be clarified by further rules stating their ordered nature even when expressed as an aux coord var. I think it would be possible make such a restriction if you use the featureType attribute, which indicates that the rules of sect 9 apply. But the use of auxiliary coordinate variables for time already existed before sect 9 was written, in what sect 9 refers to as the orthogonal multidimensional representation, and that can be used without the featureType attribute. Why not sort the data into monotonic order of time before using it? Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Dear Richard It's not my intention to call for a new coordinate variant. I'm just trying to reconcile the description of time series given in 9.1: a series ... with monotonically increasing times with the prose in H.2.1. What does 9.1 actually mean by monotonically increasing if the data can be stored in non-monotonic order? Ah yes, good point! Perhaps we should delete with monotonically increasing times in Table 9.1, and also ordered in the trajectoryProfile definition. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
At the risk of getting boring... Section H.2.1 contains the sentence (emphasis added): This has either a one-dimensional coordinate variable, time(time), provided the time values are ordered monotonically, or a one-dimensional auxiliary coordinate variable, time(o), where o is the element dimension. This seems to imply that it's possible for the time values *not* to be ordered monotonically, which is counter to the definition of a time series. My understanding is you can have either time(time) or time(o), but in both cases the values must be ordered monotonically. If that's the case I suggest simply dropping the conditional clause from the sentence to leave: This has either a one-dimensional coordinate variable, time(time), or a one-dimensional auxiliary coordinate variable, time(o), where o is the element dimension. In addition, we could add the following at the end of the paragraph: In both cases, the time values must be ordered monotonically. Thanks for your patience! Richard Hattersley AVD Iris Technical Lead Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44 (0)1392 885702 Fax: +44 (0)1392 885681 Email: richard.hatters...@metoffice.gov.uk Website: www.metoffice.gov.uk -Original Message- From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Jonathan Gregory Sent: 10 May 2012 16:41 To: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Reverse-time trajectory Dear Richard For timeseries I'd apply the same argument as before, that they already existed before sect 9 and weren't prohibited from having reversed time axes. By analogy it seems fine to me to allow reversed time in any case. Cheers Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Hi all, Thanks for the considered responses. It looks like we have a consensus on a change to remove increasing from the trajectory feature type. As things stand, I plan to raise a defect ticket in a couple of days. You may also have spotted the increasing requirement in the descriptions for the timeSeries and timeSeriesProfile feature types. I don't have a use case which might suggest the removal of increasing from these feature types, but perhaps it should be done anyway (and at this relatively early stage in the lifecycle of section 9) to keep them consistent? Regards, Richard Hattersley AVD Iris Technical Lead Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44 (0)1392 885702 Fax: +44 (0)1392 885681 Email: richard.hatters...@metoffice.gov.uk Website: www.metoffice.gov.uk -Original Message- From: cf-metadata-boun...@cgd.ucar.edu [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Steve Hankin Sent: 02 May 2012 05:25 To: Jonathan Gregory Cc: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Reverse-time trajectory Jonathan, David, I probably sounded more negative than I intended. I was just raising the concerns that should be balanced against an easy yes. In public, group discussions the forces of no are inherently weakened by the social dynamics. That, and imho CF has long had an imbalance between the perspectives of those creating data, and the perspectives of those writing the applications that will read it. You convinced me -- yes is probably the right answer in this case. - Steve (the Blue Meanie) On 5/1/2012 2:23 PM, Jonathan Gregory wrote: Dear Steve As the famed Henning piece on CORBA stated -- in standards committees no is a preferable answer to yes all other things considered. More generality can often lead to less interoperability in CF or other data standards. I think that's too negative myself. CF is successful partly because it tries to accommodate what people want to do, by and large, usually with existing mechanisms, sometimes with new ones. This is more effective for encouraging take-up of a standard than it would be to tell people what they have to do. I agree that No is the starting-point when we have a proposal to add or change conventions; there has to be a good reason for adding more complexity, and even more for breaking backward compatibility. However, what is already permitted by the standard is surely OK, isn't it, even if it is unusual. CF has always permitted coordinate axes to run in either sense, and never said anything special about time in that respect. I see no reason why a time coordinate axis shouldn't run backwards, just as a latitude axis could be N-S or S-N. Apart from Richard's example, paleoclimate timeseries often have backward time axes. Richard asked about discrete sampling geometries in particular. Here, there is a question whether we really need to require time to run forwards in the new representations (orthogonal multidimensional and ragged representations). It would be interesting to know what John thinks, since his software is probably the most widely used implementation of sect 9. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Jonathan, David, I probably sounded more negative than I intended. I was just raising the concerns that should be balanced against an easy yes. In public, group discussions the forces of no are inherently weakened by the social dynamics. That, and imho CF has long had an imbalance between the perspectives of those creating data, and the perspectives of those writing the applications that will read it. You convinced me -- yes is probably the right answer in this case. - Steve (the Blue Meanie) On 5/1/2012 2:23 PM, Jonathan Gregory wrote: Dear Steve As the famed Henning piece on CORBA stated -- in standards committees no is a preferable answer to yes all other things considered. More generality can often lead to less interoperability in CF or other data standards. I think that's too negative myself. CF is successful partly because it tries to accommodate what people want to do, by and large, usually with existing mechanisms, sometimes with new ones. This is more effective for encouraging take-up of a standard than it would be to tell people what they have to do. I agree that No is the starting-point when we have a proposal to add or change conventions; there has to be a good reason for adding more complexity, and even more for breaking backward compatibility. However, what is already permitted by the standard is surely OK, isn't it, even if it is unusual. CF has always permitted coordinate axes to run in either sense, and never said anything special about time in that respect. I see no reason why a time coordinate axis shouldn't run backwards, just as a latitude axis could be N-S or S-N. Apart from Richard's example, paleoclimate timeseries often have backward time axes. Richard asked about discrete sampling geometries in particular. Here, there is a question whether we really need to require time to run forwards in the new representations (orthogonal multidimensional and ragged representations). It would be interesting to know what John thinks, since his software is probably the most widely used implementation of sect 9. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] Reverse-time trajectory
Dear Richard The CF definitions of discrete sampling geometries define the trajectory feature type as a series of data points along a path through space with monotonically increasing times. This is a stricter stance than the usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time. Good point. I would think that monotonically would be enough, not necessarily increasing. I can't remember a reason for increasing; I assume it's just because sect 9 was conceived for observations in the first place. However, John Caron may well have a comment. I don't think anything prevents your storing the data in the orthogonal multidimensional representation, which existed before sect 9 did and doesn't require increasing coordinates. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Richard, Jonathan, et. al., As the famed Henning piece on CORBA stated -- in standards committees no is a preferable answer to yes all other things considered. More generality can often lead to less interoperability in CF or other data standards. Having CF time axes that run backwards will break a lot of software. It will be a net disruption to interoperability. So the question: is there a sufficiently compelling use case for admitting it?, where we interpret compelling in the context of balancing the general loss of interoperability against the gains to particular datasets from adding this bit of flexibility. - Steve On 5/1/2012 8:02 AM, Jonathan Gregory wrote: Dear Richard The CF definitions of discrete sampling geometries define the trajectory feature type as a series of data points along a path through space with monotonically increasing times. This is a stricter stance than the usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time. Good point. I would think that monotonically would be enough, not necessarily increasing. I can't remember a reason for increasing; I assume it's just because sect 9 was conceived for observations in the first place. However, John Caron may well have a comment. I don't think anything prevents your storing the data in the orthogonal multidimensional representation, which existed before sect 9 did and doesn't require increasing coordinates. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
On 5/1/2012 9:02 AM, Jonathan Gregory wrote: Dear Richard The CF definitions of discrete sampling geometries define the trajectory feature type as a series of data points along a path through space with monotonically increasing times. This is a stricter stance than the usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time. Good point. I would think that monotonically would be enough, not necessarily increasing. I can't remember a reason for increasing; I assume it's just because sect 9 was conceived for observations in the first place. However, John Caron may well have a comment. I don't think anything prevents your storing the data in the orthogonal multidimensional representation, which existed before sect 9 did and doesn't require increasing coordinates. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata i agree, increasing is not needed, and could be changed with a defect ticket. ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Dear Steve, Having CF time axes that run backwards will break a lot of software. It will be a net disruption to interoperability. It seems to me that it will only break the software if that software is used, and that CF-netCDF is being excluded from a valid use. I wonder if it is better to use CF and cherry pick your software than otherwise? Afterall, I suspect that discrete sampling geometries themselves will also break a lot of current software ... :-) All the best, David Original message from Steve Hankin (11AM 01 May 12) Date: Tue, 1 May 2012 11:15:54 -0700 From: Steve Hankin steven.c.han...@noaa.gov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120420 Thunderbird/12.0 To: Jonathan Gregory j.m.greg...@reading.ac.uk CC: cf-metadata@cgd.ucar.edu Subject: Re: [CF-metadata] Reverse-time trajectory Richard, Jonathan, et. al., As the famed Henning piece on CORBA stated -- in standards committees no is a preferable answer to yes all other things considered. More generality can often lead to less interoperability in CF or other data standards. Having CF time axes that run backwards will break a lot of software. It will be a net disruption to interoperability. So the question: is there a sufficiently compelling use case for admitting it?, where we interpret compelling in the context of balancing the general loss of interoperability against the gains to particular datasets from adding this bit of flexibility. - Steve On 5/1/2012 8:02 AM, Jonathan Gregory wrote: Dear Richard The CF definitions of discrete sampling geometries define the trajectory feature type as a series of data points along a path through space with monotonically increasing times. This is a stricter stance than the usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time. Good point. I would think that monotonically would be enough, not necessarily increasing. I can't remember a reason for increasing; I assume it's just because sect 9 was conceived for observations in the first place. However, John Caron may well have a comment. I don't think anything prevents your storing the data in the orthogonal multidimensional representation, which existed before sect 9 did and doesn't require increasing coordinates. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata -- David Hassell National Centre for Atmospheric Science (NCAS) Department of Meteorology, University of Reading, Earley Gate, PO Box 243, Reading RG6 6BB, U.K. Tel : 0118 3785613 Fax : 0118 3788316 E-mail: d.c.hass...@reading.ac.uk ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
Re: [CF-metadata] Reverse-time trajectory
Dear Steve As the famed Henning piece on CORBA stated -- in standards committees no is a preferable answer to yes all other things considered. More generality can often lead to less interoperability in CF or other data standards. I think that's too negative myself. CF is successful partly because it tries to accommodate what people want to do, by and large, usually with existing mechanisms, sometimes with new ones. This is more effective for encouraging take-up of a standard than it would be to tell people what they have to do. I agree that No is the starting-point when we have a proposal to add or change conventions; there has to be a good reason for adding more complexity, and even more for breaking backward compatibility. However, what is already permitted by the standard is surely OK, isn't it, even if it is unusual. CF has always permitted coordinate axes to run in either sense, and never said anything special about time in that respect. I see no reason why a time coordinate axis shouldn't run backwards, just as a latitude axis could be N-S or S-N. Apart from Richard's example, paleoclimate timeseries often have backward time axes. Richard asked about discrete sampling geometries in particular. Here, there is a question whether we really need to require time to run forwards in the new representations (orthogonal multidimensional and ragged representations). It would be interesting to know what John thinks, since his software is probably the most widely used implementation of sect 9. Best wishes Jonathan ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
[CF-metadata] Reverse-time trajectory
Hi all, The CF definitions of discrete sampling geometries define the trajectory feature type as a series of data points along a path through space with monotonically increasing times. This is a stricter stance than the usual CF coordinate definition of ordered monotonically. What was the reason behind the addition of the increasing constraint, and can it be relaxed? We have data sets where a model is run backwards in time. Thanks, Richard Hattersley AVD Iris Technical Lead Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44 (0)1392 885702 Fax: +44 (0)1392 885681 Email: richard.hatters...@metoffice.gov.uk Website: www.metoffice.gov.uk ___ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
