Re: [CF-metadata] CF-2.0 Convention discussion

[John Caron]

Hi all:

Yes, 1.10 would follow 1.9, if needed, on that "branch".

In software, we are used to major version increments meaning "API may have
changed". Since any CF file using features of the extended model will not
be fully visable (or even break) software that only understands the classic
model, it seems right to increment the major version number of the CF
Convention.

John



On Wed, Sep 24, 2014 at 2:54 AM,  wrote:

> Tim,
>
> The usual convention is that version numbers are not decimal but
> "."-separated integers.  Therefore, following 1.9 would be 1.10 and there
> is no necessity to merge into 2.0.
>
> On the subject of git and github.  I would thoroughly recommend embracing
> github as a collaborative tool.  The github ticket system is excellent and
> the freedom to experiment that comes with git branches and forks is truly
> revolutionary.  For me, the point about git is that all parties are in
> control -- no-one needs authorisation to try something new and the
> "official" branch does not loose control by allowing experimentation.  It
> is a technology which enables a true meritocracy.
>
> Having said that, I understand that it is a big culture shift which people
> don't "get" until they have had to use it for a while.
>
> My final point -- bear in mind that those of us who's job it is to archive
> data have no choice but to remain backwards compatible :-).
>
> Stephen.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
> -Original Message-
> From: Timothy Patterson [mailto:timothy.patter...@eumetsat.int]
> Sent: 24 September 2014 09:16
> To: 'Charlie Zender'; CF Metadata Mail List
> Subject: Re: [CF-metadata] CF-2.0 Convention discussion
>
> I'm wondering whether CF Convention 2.0 is the right naming convention to
> be using to refer to this new branch.
>
> Firstly, as the "classic" CF Convention has moved from 1.0 to 1.7 in 0.1
> increments, is the original branch then forced after CF 1.9 to either merge
> with CF 2.0 or to adopt a new naming convention (1.9, 1.91, 1.92...1.991,
> 1.992...)?
>
> Secondly, as a newcomer to the CF Convention, if I find documents called
> CF Convention 1.7 and CF Convention 2.0, I'm probably going to assume that
> CF 2.0 is the document I should be using.
>
> As an alternative, the new branch could be given a unique name e.g. " CF
> Conventions Enhanced 1.0" (to imply the use of netCDF-4 enhanced features).
> Then we could refer to CF Enhanced 1.0 and CF 1.7 or CF Classic 1.7 when we
> need to differentiate between the two branches.
>
> Regards,
>
> Tim
>
> -
>
> Dr. Timothy Patterson
> Instrument Data Simulation
> Product Format Specification
>
> EUMETSAT
> Eumetsat-Allee 1
> 64295 Darmstadt
> Germany
>
>
>
> -Original Message-
> From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of
> Charlie Zender
> Sent: Friday, September 19, 2014 6:20 PM
> To: CF Metadata Mail List
> Subject: [CF-metadata] CF-2.0 Convention discussion
>
> I like the idea of a Google Doc to consolidate the issue list.
> Whether it will prove unwieldy is hard to guess.
>
> A 3-month period for listing use-cases seems reasonable.
> It seems more CF-like to be use-case driven than to presume the entire
> netCDF4 data model will be adopted.
> This has the advantage of parsimony--only implement what is necessary.
> From a technical point-of-view, however, it is advantageous to know already
> what netCDF4 can and cannot easily do. In my view the intersection of
> important use-cases and netCDF4 capabilities is the "sweet spot" for CF-2.0.
>
> Once the smoke clears I may summarize use-cases for handling
> ensembles/replicants and for handling large integer numbers. These could
> raise the issues of having groups and 64-bit ints in CF-2.0.
> But first let's settle the framework for discussions.
>
> Best,
> Charlie
> --
> Charlie Zender, Earth System Sci. & Computer Sci.
> University of California, Irvine 949-891-2429 )'(
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
> --
> Scanned by iCritical.
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[stephen.pascoe]

Tim,

The usual convention is that version numbers are not decimal but "."-separated 
integers.  Therefore, following 1.9 would be 1.10 and there is no necessity to 
merge into 2.0.

On the subject of git and github.  I would thoroughly recommend embracing 
github as a collaborative tool.  The github ticket system is excellent and the 
freedom to experiment that comes with git branches and forks is truly 
revolutionary.  For me, the point about git is that all parties are in control 
-- no-one needs authorisation to try something new and the "official" branch 
does not loose control by allowing experimentation.  It is a technology which 
enables a true meritocracy.  

Having said that, I understand that it is a big culture shift which people 
don't "get" until they have had to use it for a while.

My final point -- bear in mind that those of us who's job it is to archive data 
have no choice but to remain backwards compatible :-).

Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK


-Original Message-
From: Timothy Patterson [mailto:timothy.patter...@eumetsat.int] 
Sent: 24 September 2014 09:16
To: 'Charlie Zender'; CF Metadata Mail List
Subject: Re: [CF-metadata] CF-2.0 Convention discussion

I'm wondering whether CF Convention 2.0 is the right naming convention to be 
using to refer to this new branch.

Firstly, as the "classic" CF Convention has moved from 1.0 to 1.7 in 0.1 
increments, is the original branch then forced after CF 1.9 to either merge 
with CF 2.0 or to adopt a new naming convention (1.9, 1.91, 1.92...1.991, 
1.992...)?

Secondly, as a newcomer to the CF Convention, if I find documents called CF 
Convention 1.7 and CF Convention 2.0, I'm probably going to assume that CF 2.0 
is the document I should be using.

As an alternative, the new branch could be given a unique name e.g. " CF 
Conventions Enhanced 1.0" (to imply the use of netCDF-4 enhanced features). 
Then we could refer to CF Enhanced 1.0 and CF 1.7 or CF Classic 1.7 when we 
need to differentiate between the two branches.

Regards,

Tim 

-

Dr. Timothy Patterson
Instrument Data Simulation
Product Format Specification

EUMETSAT
Eumetsat-Allee 1
64295 Darmstadt
Germany



-Original Message-
From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of 
Charlie Zender
Sent: Friday, September 19, 2014 6:20 PM
To: CF Metadata Mail List
Subject: [CF-metadata] CF-2.0 Convention discussion

I like the idea of a Google Doc to consolidate the issue list.
Whether it will prove unwieldy is hard to guess.

A 3-month period for listing use-cases seems reasonable.
It seems more CF-like to be use-case driven than to presume the entire netCDF4 
data model will be adopted.
This has the advantage of parsimony--only implement what is necessary. From a 
technical point-of-view, however, it is advantageous to know already what 
netCDF4 can and cannot easily do. In my view the intersection of important 
use-cases and netCDF4 capabilities is the "sweet spot" for CF-2.0.

Once the smoke clears I may summarize use-cases for handling 
ensembles/replicants and for handling large integer numbers. These could raise 
the issues of having groups and 64-bit ints in CF-2.0.
But first let's settle the framework for discussions.

Best,
Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'( 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
-- 
Scanned by iCritical.
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[Timothy Patterson]

I'm wondering whether CF Convention 2.0 is the right naming convention to be 
using to refer to this new branch.

Firstly, as the "classic" CF Convention has moved from 1.0 to 1.7 in 0.1 
increments, is the original branch then forced after CF 1.9 to either merge 
with CF 2.0 or to adopt a new naming convention (1.9, 1.91, 1.92...1.991, 
1.992...)?

Secondly, as a newcomer to the CF Convention, if I find documents called CF 
Convention 1.7 and CF Convention 2.0, I'm probably going to assume that CF 2.0 
is the document I should be using.

As an alternative, the new branch could be given a unique name e.g. " CF 
Conventions Enhanced 1.0" (to imply the use of netCDF-4 enhanced features). 
Then we could refer to CF Enhanced 1.0 and CF 1.7 or CF Classic 1.7 when we 
need to differentiate between the two branches.

Regards,

Tim 

-

Dr. Timothy Patterson
Instrument Data Simulation
Product Format Specification

EUMETSAT
Eumetsat-Allee 1
64295 Darmstadt
Germany



-Original Message-
From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of 
Charlie Zender
Sent: Friday, September 19, 2014 6:20 PM
To: CF Metadata Mail List
Subject: [CF-metadata] CF-2.0 Convention discussion

I like the idea of a Google Doc to consolidate the issue list.
Whether it will prove unwieldy is hard to guess.

A 3-month period for listing use-cases seems reasonable.
It seems more CF-like to be use-case driven than to presume the entire netCDF4 
data model will be adopted.
This has the advantage of parsimony--only implement what is necessary. From a 
technical point-of-view, however, it is advantageous to know already what 
netCDF4 can and cannot easily do. In my view the intersection of important 
use-cases and netCDF4 capabilities is the "sweet spot" for CF-2.0.

Once the smoke clears I may summarize use-cases for handling 
ensembles/replicants and for handling large integer numbers. These could raise 
the issues of having groups and 64-bit ints in CF-2.0.
But first let's settle the framework for discussions.

Best,
Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'( 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[Derrick Snowden - NOAA Federal]

In an earlier, and different, thread Rich Signell mentioned the Semantic
Versioning concept of x.y.z notation, meaning MAJOR.MINOR.PATCH.  See
http://semver.org/ and in particular items 6-8.  I agree with Rich in that
semantic versioning is a good approach and something CF should consider
explicitly adopting.

If we do adopt it, then by considering a 2.0.0 (note the extra zero)
development effort, we are admitting backwards incompatibility into the
discussion.  I personally don't believe we can be infinitely backwards
compatible but I do subscribe to the somewhat conservative use case driven
approach toward evolution.  If CF is unwilling to admit backwards
incompatibilty into the discussion then we're really talking about 1.8.z
not 2.0.0 requirements.

NOTE I don't think that embracing the full feature set of netCDF4/HDF5
necessarily implies we are allowing the existence of technology to drive
the evolution of the standard.  We can still have a use case driven plan to
force us to decide between 1.8.z and 2.0.0.

I think Heiko's suggestion of strict vs. transitional subsets is
interesting, even if it complicates the management of the standard
artifacts.  I *think* this idea is supported by semver, in particular see
item 11, but I'd like someone more seasoned than me to verify that
statement.

If we fully embrace the semantic versioning ideas, then it's worth further
exploring the proposal to move to the github model of fork/branch/pull
request/merge to support management of collaborations, branches, and
explorations.  Git-flow ( eg. [1]

and [2]  ) allows
for the selective inclusion of branches into the trunk.  The existence of
an explorative branch doesn't necessarily imply that it will be merged into
the trunk, but git does support the management of the branch in a
consistent way.  It get's hard to adhere to semver item 3 without a mature
set of tools like git.

Further, this model can support the inclusion of some of the "workgroups"
into the CF standard more explicitly.  For example, I've never fully
understood the relationship of groups like cf-satellite or cf-radials to
the core CF standard.  It seems to me that they are somewhat orphans.


I look forward to the discussion!
Derrick

On Mon, Sep 22, 2014 at 3:51 AM, Heiko Klein  wrote:

> Dear Jonathan,
>
> removal/backward incompatibility is a difficult issue, and having several
> ways to do things just because of backward compatibility will make CF hard
> to understand. It is not only new features in CF which introduces too many
> ways to do things, also the existing document has plenty of examples, e.g.
> at least 6 different ways to describe a latitude axis (and that even
> without counting standard_name and axis attributes).
>
> While I think maintaining two sets of CF (1.x, 2.x) will be very hard, I
> think acceptance of a CF-2.x series can only be achieved with enough
> compatibility.
>
> I liked the way of the HTML-4 group to allow for a "transitional" and a
> "strict" set of features. This should make it easy to adapt existing
> datasets to CF-2-transitional, while new datasets should be tested against
> a 'strict' CF-checker.
>
>
> I agree with your comments about the way of discussion, history and open
> communications is import.
>
>
> Best regards,
>
> Heiko
>
>
>
>
> On 2014-09-19 16:11, Jonathan Gregory wrote:
>
>> Dear John
>>
>>  So I propose we have a 3 month discussion period where we clarify what
>>> the
>>> issues are and possible improvements or changes. We wont try too hard
>>> during that period to create final wording, and divergences of opinion
>>> will
>>> be recorded rather than resolved. After that we will have another 3 month
>>> period where we try to write a document and make decisions.
>>>
>>
>> It is good to have a timescale for discussion. That might not be long
>> enough,
>> but let's see!
>>
>>  I propose these ground-rules for what we discuss:
>>>1. Changes needed to use the extended model.
>>>2. Possible things to deprecate or remove.
>>>3. Possible new functionality esp if it comes from using the extended
>>> model.
>>>4. Backwards compatibility is desirable, but not required if there is
>>> a
>>> substantial advantage in simplicity and clarity. So respect precedent and
>>> dont constrain important innovation.
>>>
>>
>> By "extended model" do you mean the netCDF4 model? I don't think we should
>> begin with an *aim* to use the netCDF4 model. The question is, what do we
>> need
>> to do which we *can't* do in the classic model, or could be done much more
>> easily in the extended model than in the classic model? That is, changes
>> to CF
>> should be driven by use-cases, not technology. So I would combine 3 and 1
>> as
>>
>> (1) What use-cases cannot be met with the classic model, or could be met
>> much
>> more easily with the extended model?
>>
>> Since "remove

Re: [CF-metadata] CF-2.0 Convention discussion

[Heiko Klein]

Dear Jonathan,

removal/backward incompatibility is a difficult issue, and having 
several ways to do things just because of backward compatibility will 
make CF hard to understand. It is not only new features in CF which 
introduces too many ways to do things, also the existing document has 
plenty of examples, e.g. at least 6 different ways to describe a 
latitude axis (and that even without counting standard_name and axis 
attributes).


While I think maintaining two sets of CF (1.x, 2.x) will be very hard, I 
think acceptance of a CF-2.x series can only be achieved with enough 
compatibility.


I liked the way of the HTML-4 group to allow for a "transitional" and a 
"strict" set of features. This should make it easy to adapt existing 
datasets to CF-2-transitional, while new datasets should be tested 
against a 'strict' CF-checker.



I agree with your comments about the way of discussion, history and open 
communications is import.



Best regards,

Heiko



On 2014-09-19 16:11, Jonathan Gregory wrote:

Dear John


So I propose we have a 3 month discussion period where we clarify what the
issues are and possible improvements or changes. We wont try too hard
during that period to create final wording, and divergences of opinion will
be recorded rather than resolved. After that we will have another 3 month
period where we try to write a document and make decisions.


It is good to have a timescale for discussion. That might not be long enough,
but let's see!


I propose these ground-rules for what we discuss:
   1. Changes needed to use the extended model.
   2. Possible things to deprecate or remove.
   3. Possible new functionality esp if it comes from using the extended
model.
   4. Backwards compatibility is desirable, but not required if there is a
substantial advantage in simplicity and clarity. So respect precedent and
dont constrain important innovation.


By "extended model" do you mean the netCDF4 model? I don't think we should
begin with an *aim* to use the netCDF4 model. The question is, what do we need
to do which we *can't* do in the classic model, or could be done much more
easily in the extended model than in the classic model? That is, changes to CF
should be driven by use-cases, not technology. So I would combine 3 and 1 as

(1) What use-cases cannot be met with the classic model, or could be met much
more easily with the extended model?

Since "remove" implies "backward-incompatible", I would put (2) as

(2) Possible things to deprecate (because there is a better way to do it
- I assume - are there other reasons?)

Then

(3) Possible things to remove, or to require if they are currently only
optional. Such changes mean backwards-incompatibility. I think that the
default should be backwards compatibility. There has to be a strong advantage
for making incompatible changes. (NB I am not saying that backward
incompatibility is out of the question!)

I do not think we should introduce a new way of doing something because
it might look nicer, unless it is clearly functionally much better than the
existing way. Also, if we introduce a new way to do something, we should
carefully consider removing the old way, because it is easier for users of
the convention if there is only one way to do something.

Whatever way the discussion is done, I think it's important that
- it should be public to read and contribute to
- it should have a history, so we can all see what we and others said earlier.

Best wishes and thanks

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



--
Dr. Heiko Klein  Tel. + 47 22 96 32 58
Development Section / IT Department  Fax. + 47 22 69 63 55
Norwegian Meteorological Institute   http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] CF-2.0 Convention discussion

[Timothy Patterson]

Whether or not the entire netCDF4 enhanced data model is adopted, can I suggest 
that all features of the model are acknowledged and those that are not 
compatible with the 2.0 conventions are explicitly identified.  

As an example, I currently find nothing in the current conventions to say that 
the use of groups is incompatible with the conventions, yet from the 
discussions here, there is an inherent assumption that groups are not allowed. 
This makes the specification hard to interpret for a CF convention neophyte 
like myself.

Regards,

Tim Patterson


-

Dr. Timothy Patterson
Instrument Data Simulation
Product Format Specification

EUMETSAT
Eumetsat-Allee 1
64295 Darmstadt
Germany

-Original Message-
From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of 
Charlie Zender
Sent: Friday, September 19, 2014 6:20 PM
To: CF Metadata Mail List
Subject: [CF-metadata] CF-2.0 Convention discussion

I like the idea of a Google Doc to consolidate the issue list.
Whether it will prove unwieldy is hard to guess.

A 3-month period for listing use-cases seems reasonable.
It seems more CF-like to be use-case driven than to presume the entire netCDF4 
data model will be adopted.
This has the advantage of parsimony--only implement what is necessary. From a 
technical point-of-view, however, it is advantageous to know already what 
netCDF4 can and cannot easily do. In my view the intersection of important 
use-cases and netCDF4 capabilities is the "sweet spot" for CF-2.0.

Once the smoke clears I may summarize use-cases for handling 
ensembles/replicants and for handling large integer numbers. These could raise 
the issues of having groups and 64-bit ints in CF-2.0.
But first let's settle the framework for discussions.

Best,
Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'( 
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] CF-2.0 Convention discussion

[Charlie Zender]

I like the idea of a Google Doc to consolidate the issue list.
Whether it will prove unwieldy is hard to guess.

A 3-month period for listing use-cases seems reasonable.
It seems more CF-like to be use-case driven than to
presume the entire netCDF4 data model will be adopted.
This has the advantage of parsimony--only implement
what is necessary. From a technical point-of-view,
however, it is advantageous to know already what netCDF4
can and cannot easily do. In my view the intersection of
important use-cases and netCDF4 capabilities is the
"sweet spot" for CF-2.0.

Once the smoke clears I may summarize use-cases for
handling ensembles/replicants and for handling
large integer numbers. These could raise the issues
of having groups and 64-bit ints in CF-2.0.
But first let's settle the framework for discussions.

Best,
Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] CF-2.0 Convention discussion

[Jonathan Gregory]

Dear John

> So I propose we have a 3 month discussion period where we clarify what the
> issues are and possible improvements or changes. We wont try too hard
> during that period to create final wording, and divergences of opinion will
> be recorded rather than resolved. After that we will have another 3 month
> period where we try to write a document and make decisions.

It is good to have a timescale for discussion. That might not be long enough,
but let's see!

> I propose these ground-rules for what we discuss:
>   1. Changes needed to use the extended model.
>   2. Possible things to deprecate or remove.
>   3. Possible new functionality esp if it comes from using the extended
> model.
>   4. Backwards compatibility is desirable, but not required if there is a
> substantial advantage in simplicity and clarity. So respect precedent and
> dont constrain important innovation.

By "extended model" do you mean the netCDF4 model? I don't think we should
begin with an *aim* to use the netCDF4 model. The question is, what do we need
to do which we *can't* do in the classic model, or could be done much more
easily in the extended model than in the classic model? That is, changes to CF
should be driven by use-cases, not technology. So I would combine 3 and 1 as

(1) What use-cases cannot be met with the classic model, or could be met much
more easily with the extended model?

Since "remove" implies "backward-incompatible", I would put (2) as

(2) Possible things to deprecate (because there is a better way to do it
- I assume - are there other reasons?)

Then

(3) Possible things to remove, or to require if they are currently only
optional. Such changes mean backwards-incompatibility. I think that the
default should be backwards compatibility. There has to be a strong advantage
for making incompatible changes. (NB I am not saying that backward
incompatibility is out of the question!)

I do not think we should introduce a new way of doing something because
it might look nicer, unless it is clearly functionally much better than the
existing way. Also, if we introduce a new way to do something, we should
carefully consider removing the old way, because it is easier for users of
the convention if there is only one way to do something.

Whatever way the discussion is done, I think it's important that
- it should be public to read and contribute to
- it should have a history, so we can all see what we and others said earlier.

Best wishes and thanks

Jonathan
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata