Re: [CF-metadata] PID to external description of Quality and Status states

[Kehoe, Kenneth E.]

I agree with Johnathan that the individual flag descriptions would be 
documented in the flag_meanings attribute. If the description of the 
flag or the process to generate the flags is too long to put in the 
flag_meanings (which is what I would assume for many cases) then the 
full description can be linked. I would assume using "references" 
attribute could point to a DOI, URL or paper citation. That is how I 
would solve this though experiment.

I don't see the need to link each flag meaning to an external source but 
if that was needed a new attribute of flag_references could be added to 
providing a linking method.

Ken



On 2019-7-25 12:36, Martin Juckes - UKRI STFC wrote:
> Dear Daniel,
>
>
> I agree with Jonathan's interpretation: "as self-describing as possible" can 
> only go so far. I don't see any reason in principle not to include pointers 
> to external resources. However, the problem is always in the detail.
>
>
> The problem appears to be that you consider the amount of information which 
> can be put in a valid flag_meanings word is too limited in some cases, which 
> I find plausible.
>
>
> You can place extra information in the "comment" attribute, but this is not 
> semantically tied to the flags, so there is a plausible case for having a 
> specific attribute to provide additional information about the flags.
>
> There is a precedent for referring to external definitions of terms in the 
> "land_cover_lccs" standard name, which refers to the UN Land Cover 
> Classification System. Not a great precedent, as the reference given is a web 
> page from 2000 explaining the system and motivation behind the terms, but 
> getting at the terms requires running the database on a computer with windows 
> 95/98 or NT, apparently. It appears that LCCS has evolved to become a 
> software tool and the terms the community is now using are in the Land Cover 
> Meta Language (LCML), which is an ISO standard. This suggests, I think, we 
> need to be cautious about how external references are introduced.
>
> regards,
> Martin
>
>
>
>
>
>
> 
> From: CF-metadata  on behalf of Jonathan 
> Gregory 
> Sent: 24 July 2019 16:58
> To: cf-metadata@cgd.ucar.edu
> Subject: [CF-metadata] PID to external description of Quality and Status 
> states
>
> Dear Daniel
>
> I assume you mean a data variable which has strings describing quality or
> status states, probably encoded with flag_values and flag_meanings. Would
> the flag_meanings be meaningful to a human reading them? If so, you could
> regard the file as self-describing. Self-describing doesn't have to mean
> self-documenting, I would say. Standard names are self-explanatory to some
> extent, and make the file self-describing, but they also have definitions
> which aren't in the file, and papers in the peer-reviewed literature about
> the quantities. If the flag_meanings are self-explanatory, I think that's OK,
> but I'm not in favour of putting codes in the CF-netCDF file.
>
> Best wishes
>
> Jonathan
>
> - Forwarded message from Daniel Neumann  -
>
>> Date: Wed, 24 Jul 2019 14:45:33 +0200
>> From: Daniel Neumann 
>> To: cf-metadata@cgd.ucar.edu
>> Subject: [CF-metadata] PID to external description of Quality and Status
>> states
>> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101
>> Thunderbird/60.8.0
>>
>> Dear list,
>>
>> A CF-conformal NetCDF file should be as self-describing as possible.
>> URIs/PIDs poitining to external data sources are problematic in this
>> context. However, IDs pointing to external data sources were already
>> discussed for taxa in the CF Trac Ticket #99 "Taxon Names and
>> Identifiers" (https://cf-trac.llnl.gov/trac/ticket/99).
>>
>> Let's assume we have a long description of quality control measures
>> performed resulting in N possible quality states for a NetCDF
>> variable. The full description could be part of a peer-reviewed
>> paper or grey literature (with assigned PID). However, it is too
>> long to be included in the attribute "description" of the
>> quality/status flag variable. One solution would be to add a brief
>> description of the quality states to the quality_flag/status_flag
>> variable and point to the external full description via it's PID.
>>
>> What is your opinion on this?
>>
>> This is not a request for a feature but rather meant to collect some
>> opinions on this topic. It is somehow related to the request of a
>> global PID attribute in CF GitHub Issue 160 "Add attribute
>> citation_id"
>> (https://github.com/cf-convention/cf-conventions/issues/160).
>>
>> Regards,
>> Daniel
>>
>>
>> --
>> Dr. Daniel Neumann
>> Abteilung Datenmanagement
>>
>> Deutsches Klimarechenzentrum GmbH (DKRZ)
>> Bundesstraße 45 a • D-20146 Hamburg • Germany
>>
>> Phone: +49 40 460094-120
>> Email: daniel.neum...@dkrz.de
>> URL:  www.dkrz.de
>>
>> Geschäftsführer: Prof. Dr. Thomas Ludwig
>> Sitz der Gesellschaft: Hamburg
>> 

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

John,

I don't see status_flag excluding someone from providing information about 
status of instruments or equipment. It would be information about data if it is 
information about the equipment. I think leaving status_flag as a general catch 
all right now is good. My concern with trying to fix that is where do we stop? 
It could require 10's to 100's of new names to cover every possible case.

Ken



On 2019-7-26 12:03, Andrew Barna wrote:
Hi John,

The examples provided in the CF Document are almost all about instrument state: 
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#flags<https://urldefense.proofpoint.com/v2/url?u=http-3A__cfconventions.org_Data_cf-2Dconventions_cf-2Dconventions-2D1.7_cf-2Dconventions.html-23flags=DwMFaQ=qKdtBuuu6dQK9MsRUVJ2DPXW6oayO8fu4TfEHS8sGNk=Vm7o2ZGxPkkqRuPs8nVMVQ=Px7ouMltX0p3H_tfw4fe1Zgt601CKzQFegJtw97j5_Q=uwRsYLzUWtcu5B-t7Kmd6yyXMf-BS3XXcRptynmDhPU=>
 so it seems to support your use case.

-Barna

On Wed, Jul 24, 2019 at 11:40 AM John Graybeal 
mailto:jgrayb...@stanford.edu>> wrote:
I support the point about defining 'status' and 'quality'. Yes, there are cases 
when we define terms that are re-used, but I don't think these terms are 
reused, they appear only in these flags. Just defining the standard name should 
do.

Ken, I did like the qualifying text about status_flag but maybe that's because 
I always thought status_flag could be used that way, as a status of 
instruments. Looking at the definition 
(http://mmisw.org/cfsn/#/search/status<https://urldefense.proofpoint.com/v2/url?u=http-3A__mmisw.org_cfsn_-23_search_status=DwMFaQ=qKdtBuuu6dQK9MsRUVJ2DPXW6oayO8fu4TfEHS8sGNk=Vm7o2ZGxPkkqRuPs8nVMVQ=Px7ouMltX0p3H_tfw4fe1Zgt601CKzQFegJtw97j5_Q=Ao72mQ_pT52B8ySc00OurGMGGBCcVejKyaPZrvBpcDQ=>)
 it doesn't say that, does it?  It's all about the data. I even searched the 
archives, I was so sure people talked about it in another way, but I can't find 
any evidence of that.

So I conclude equipment status is not included in the model currently supported 
by status flag, and we shouldn't try to fix that here. What do you think?

John

On Jul 24, 2019, at 10:34 AM, Kehoe, Kenneth E. 
mailto:kke...@ou.edu>> wrote:

Daniel,

Thanks for the information. At some point we should chat about how our two 
organizations think about and perform quality analysis.

Martin,

I'm confused about your suggestion to include definitions of status and 
quality. I guess we could define those terms better in the general standard 
name table, but that is not my intention. My concern is that the definition of 
those terms is larger than the scope of what I wanted to propose. I would 
prefer to just work on the definitions of the status_flag and quality_flag.

Looking at your suggestion to numerically order the values suggests I think we 
have a different notion of how to use quality_flag. A quality_flag is not 
intend to indicate severity or ranking of tests. It is just a state field. My 
program had discussions to do something like that in the past and it did not 
end well.

If we want to add terminology along the lines of "The variable with standard 
name quality_flag refers to an assessed quality of the corresponding data." 
that is OK with me. Your expanded definition of status does not help me to 
better understand status. I think it's the statement of "may" that confuses me. 
I see a definition needing to be more definitive.

I don't see the addition of quality_flag as changing status_flag. I see 
quality_flag as a more narrow sub-class of status_flag. I would prefer to not 
change much with status_flag since it has such a long history with CF.

I think we have these definitions:

status_flag: A variable with the standard name of status_flag contains an 
indication of quality or other status of another data variable. The linkage 
between the data variable and the variable with the standard name of 
status_flag is achieved using the ancillary_variables attribute. A variable 
which contains purely quality information may use the standard name of 
quality_flag to provided an assessed quality of the corresponding data.

quality_flag = A variable with the standard name of quality_flag contains an 
indication of assessed quality information of another data variable. The 
linkage between the data variable and the variable or variables with the 
standard_name of quality_flag is achieved using the ancillary_variables 
attribute.

Thanks,

Ken





On 2019-7-24 03:40, Daniel Neumann wrote:
Dear Ken, Martin, John, Roy and Barna,

I/we thought about submitting a similar proposal to add some extended model 
quality information to netCDF files. The suggested description of 
"quality_flag" and the modified description of "status_flag" fit well into our 
project.

I am just writing this to show that there are more people in the community who 
are interested in this.

Cheers,
Daniel


Am 24.07.2019 um 10:49 sc

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Daniel,

Thanks for the information. At some point we should chat about how our two 
organizations think about and perform quality analysis.

Martin,

I'm confused about your suggestion to include definitions of status and 
quality. I guess we could define those terms better in the general standard 
name table, but that is not my intention. My concern is that the definition of 
those terms is larger than the scope of what I wanted to propose. I would 
prefer to just work on the definitions of the status_flag and quality_flag.

Looking at your suggestion to numerically order the values suggests I think we 
have a different notion of how to use quality_flag. A quality_flag is not 
intend to indicate severity or ranking of tests. It is just a state field. My 
program had discussions to do something like that in the past and it did not 
end well.

If we want to add terminology along the lines of "The variable with standard 
name quality_flag refers to an assessed quality of the corresponding data." 
that is OK with me. Your expanded definition of status does not help me to 
better understand status. I think it's the statement of "may" that confuses me. 
I see a definition needing to be more definitive.

I don't see the addition of quality_flag as changing status_flag. I see 
quality_flag as a more narrow sub-class of status_flag. I would prefer to not 
change much with status_flag since it has such a long history with CF.

I think we have these definitions:

status_flag: A variable with the standard name of status_flag contains an 
indication of quality or other status of another data variable. The linkage 
between the data variable and the variable with the standard name of 
status_flag is achieved using the ancillary_variables attribute. A variable 
which contains purely quality information may use the standard name of 
quality_flag to provided an assessed quality of the corresponding data.

quality_flag = A variable with the standard name of quality_flag contains an 
indication of assessed quality information of another data variable. The 
linkage between the data variable and the variable or variables with the 
standard_name of quality_flag is achieved using the ancillary_variables 
attribute.

Thanks,

Ken





On 2019-7-24 03:40, Daniel Neumann wrote:
Dear Ken, Martin, John, Roy and Barna,

I/we thought about submitting a similar proposal to add some extended model 
quality information to netCDF files. The suggested description of 
"quality_flag" and the modified description of "status_flag" fit well into our 
project.

I am just writing this to show that there are more people in the community who 
are interested in this.

Cheers,
Daniel


Am 24.07.2019 um 10:49 schrieb Martin Juckes - UKRI STFC:
Dear John, Roy,


OK, I'm happy to drop the line about ordering of quality flags if it doesn't 
work. This is consistent with Roy's suggested definitions (posted 2 minutes 
before John's reply), which also drop this sentence, and add a broader 
description of valid usage of the status flag (I've copied them her to get the 
discussion back in a single thread):


Status: The value of a variable with standard name status_flag may refer to the 
status of the instrument or process which generated the corresponding data, or 
it may refer to the data itself. This may include information about data 
quality, particularly in legacy data sets. 'quality_flag' should be used if 
data quality is the only type of information contained in the variable.

Quality: The value of a variable with standard name quality_flag refers to an 
assessed quality of the corresponding data.


regards,

Martin


From: John Graybeal <mailto:jgrayb...@stanford.edu>
Sent: 24 July 2019 09:20
To: Juckes, Martin (STFC,RAL,RALSP)
Cc: Andrew Barna; Kehoe, Kenneth E.; CF Metadata List
Subject: Re: [CF-metadata] New standard_name of quality_flag for corresponding 
quality control variables

+1 Martin, just what I was thinking also, it creates the opening but does not 
preclude mixing status and quality flags in a single status_flag, which I think 
is important.

Um, I don't think you can dictate that "Numeric values of the quality flag 
should be ordered, such the lowest value corresponds to the poorest quality and 
the highest value to the best quality." Some people will be documenting their 
own flags which are whatever they are.

Responding to an earlier possible misconception, I want to emphasize (read: 
confirm) these are the standard names, which are used to characterize the 
attributes. They are not the variable names, so you can have multiple different 
variables that express different status_flags or different quality_flags.

John

On Jul 24, 2019, at 12:46 AM, Martin Juckes - UKRI STFC 
mailto:martin.juc...@stfc.ac.uk><mailto:martin.juc...@stfc.ac.uk><mailto:martin.juc...@stfc.ac.uk>>
 wrote:

Dear Ken, Barna,


I agree that we shoul

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Barna,

OK your definition is fine. I suggest one small change, drop the word 
subjective.

status_flag: A variable with the standard name of status_flag contains an 
indication of quality or other status of another data variable. The linkage 
between the data variable and the variable with the standard_name of 
status_flag is achieved using the ancillary_variables attribute. A variable 
which contains purely quality information may use the standard_name of 
quality_flag.

Ken


On 2019-7-23 15:28, Andrew Barna wrote:
Ken,

I think I'm confused by the text of the proposed change to the definition of 
status_flag.

In your proposed change the "quality" wording of the status_flag definition was 
dropped. Here is the first sentence of each:
Current: A variable with the standard name of status_flag contains an 
indication of quality or other status of another data variable.
Proposed: A variable with the standard name of status_flag contains an 
indication of status of another data variable.

Perhaps the following for "status_flag":
A variable with the standard name of status_flag contains an indication of 
quality or other status of another data variable. The linkage between the data 
variable and the variable with the standard_name of status_flag is achieved 
using the ancillary_variables attribute. A variable which contains purely 
subjective quality information may use the standard_name of quality_flag.

That is, keep the current definition, but also inform of a more restrictive 
option. I don't see any way around not reading the flag_meanings with any of 
these options.

-Barna


On Tue, Jul 23, 2019 at 1:03 PM Kehoe, Kenneth E. 
mailto:kke...@ou.edu>> wrote:
Barna,

I see this as an optional addition to narrow the standard. It does not
prohibit someone from using status_flag (as a standard_name or a
standard_name modifier) from a previous convention version
implementation nor invalidate that use from a previous convention
version. In your example the use of status_flag is a mixture of state
and quality. I see this new name as a way to improve things going
forward. Since the historical WOCE example uses state and quality with
some additional rules not listed in the CF standard it would be up to
the user to understand how to use the variable. Without seeing the WOCE
data I can't make a specific suggestion.

I don't know about any rules regarding a restriction. I think the
general concept of CF is to set the minimum rules. Additional rules
applied by another group on top of CF is allowed. For example my
organization uses additional attributes not defined in CF. I see
quality_flag as a narrowing of the rules of status_flag not replace it.
status_flag can still have a mixture of state and quality if the data
provider prefers to do it that way. quality_flag can only have quality
information. The determination of what is quality information is
actually up to the data provider to decide.

Ken



On 2019-7-23 13:33, Andrew Barna wrote:
> Ken,
>
> Ok I see how this can be useful. Two more questions:
> * How would you deal with "legacy" flag schemes which mix "status" and
> "quality" already? I'm thinking of WOCE CTD as an example where "7"
> means Despiked (a status) and "3" means Questionable measurement (a
> quality). The way my seagoing group have dealt with both is by having
> the "quality" override "status" if the quality is anything other than
> "good", e.g. a questionable measurement which has been despiked gets
> flag 3.
>
> * Are there rules in CF regarding restricting an existing definition?
> I imagine there are many datasets already using the "status_flag" name
> as either a stand alone standard name or a standard name modifier.
> This change seems to be "breaking" in that previously compliant
> datasets would now have quality information in a purely status field.
>
> Thanks
> -Barna
>
> On Tue, Jul 23, 2019 at 10:08 AM Kehoe, Kenneth E. 
> mailto:kke...@ou.edu>> wrote:
>> Martin,
>>
>> Thanks for your reply. I would prefer to keep the proposal simple. My 
>> example of a weighted mean was just one I created off the top of my head. I 
>> don't see it as something to actually look into implementing.
>>
>> I need a way to indicate a variable is a quality status field. The 
>> distinction that the status field only contains quality information is the 
>> important distinction. The variable indicated with quality_flag will need to 
>> also use flag_meanings, same as status_flag. Hence my reason for choosing 
>> quality_flag to follow a similar naming pattern.
>>
>> Barna,
>>
>> Without a distinction that the entire variable is a quality variable the 
>> user is forced to parse the flag_meanings to see if the variable

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Barna,

I see this as an optional addition to narrow the standard. It does not 
prohibit someone from using status_flag (as a standard_name or a 
standard_name modifier) from a previous convention version 
implementation nor invalidate that use from a previous convention 
version. In your example the use of status_flag is a mixture of state 
and quality. I see this new name as a way to improve things going 
forward. Since the historical WOCE example uses state and quality with 
some additional rules not listed in the CF standard it would be up to 
the user to understand how to use the variable. Without seeing the WOCE 
data I can't make a specific suggestion.

I don't know about any rules regarding a restriction. I think the 
general concept of CF is to set the minimum rules. Additional rules 
applied by another group on top of CF is allowed. For example my 
organization uses additional attributes not defined in CF. I see 
quality_flag as a narrowing of the rules of status_flag not replace it. 
status_flag can still have a mixture of state and quality if the data 
provider prefers to do it that way. quality_flag can only have quality 
information. The determination of what is quality information is 
actually up to the data provider to decide.

Ken



On 2019-7-23 13:33, Andrew Barna wrote:
> Ken,
>
> Ok I see how this can be useful. Two more questions:
> * How would you deal with "legacy" flag schemes which mix "status" and
> "quality" already? I'm thinking of WOCE CTD as an example where "7"
> means Despiked (a status) and "3" means Questionable measurement (a
> quality). The way my seagoing group have dealt with both is by having
> the "quality" override "status" if the quality is anything other than
> "good", e.g. a questionable measurement which has been despiked gets
> flag 3.
>
> * Are there rules in CF regarding restricting an existing definition?
> I imagine there are many datasets already using the "status_flag" name
> as either a stand alone standard name or a standard name modifier.
> This change seems to be "breaking" in that previously compliant
> datasets would now have quality information in a purely status field.
>
> Thanks
> -Barna
>
> On Tue, Jul 23, 2019 at 10:08 AM Kehoe, Kenneth E.  wrote:
>> Martin,
>>
>> Thanks for your reply. I would prefer to keep the proposal simple. My 
>> example of a weighted mean was just one I created off the top of my head. I 
>> don't see it as something to actually look into implementing.
>>
>> I need a way to indicate a variable is a quality status field. The 
>> distinction that the status field only contains quality information is the 
>> important distinction. The variable indicated with quality_flag will need to 
>> also use flag_meanings, same as status_flag. Hence my reason for choosing 
>> quality_flag to follow a similar naming pattern.
>>
>> Barna,
>>
>> Without a distinction that the entire variable is a quality variable the 
>> user is forced to parse the flag_meanings to see if the variable applies. 
>> This would also encourage a data provider to mix quality with source or 
>> instrument state or something else in the same variable. That would be very 
>> difficult to understand.
>>
>> As Martin points out quality is more subjective than other status 
>> information. A user may need to choose what parts of the quality variable to 
>> apply. I would prefer we not conflate absolute information with subjective 
>> information. But we need a way to distinguish the variable contains absolute 
>> information vs a variable that contains more subjective information.
>>
>> To expand on Martin's example imagine a profiling instrument that has a 
>> shutter to protect the laser from rain. The laser will always send out 
>> pulses and the receiver will always be on receiving the return from laser 
>> pulse. To know when the shutter is in the open state where the instrument is 
>> profiling we would use a state variable with a simple flag_values method.
>>
>> short shutter (time)
>>shutter:long_name = "Shutter state"
>>shutter:units = '1'
>>shutter:flag_values = 0, 1
>>shutter:flag_meanings = "closed open"
>>shutter:standard_name = "status_flag"
>>
>> This variable is just indicating the position of the shutter. There is no 
>> ambiguity with it's use. If a user wants to use the data for atmospheric 
>> reasons they should filter to only use data where profiling. In fact we can 
>> implement this variable into our code by only using data where shutter is 
>> set to open.
>>
&g

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

essarily imply that dataset A is in any sense better and B.


If you want to use it in weighted means, perhaps it should be "quality_measure" 
rather than "quality_flag"? With "status_flag" the order of integer values does 
not have any meaning, but with quality perhaps it would make more sense have 
some concept of a sequence of quality settings (so that, for example "1" always 
indicates a quality between "0" and "2" within a dataset, but could have 
different meanings in different datasets). Could the quality also be expressed 
as a floating point number without any flag meanings?


Responding to a point Barna raised: it is certainly possible to have more than 
one "status_flag" variable, but I don't think it is ideal: if information needs 
to be split across multiple variables we generally like to describe the 
difference between the variables in the standard name or in other metadata. In 
this case, I think there is a good case for using a new standard name.


regards,

Martin




____
From: CF-metadata 
<mailto:cf-metadata-boun...@cgd.ucar.edu> on 
behalf of Andrew Barna <mailto:aba...@ucsd.edu>
Sent: 23 July 2019 00:23
To: Kehoe, Kenneth E.
Cc: cf-metadata@cgd.ucar.edu<mailto:cf-metadata@cgd.ucar.edu>
Subject: Re: [CF-metadata] New standard_name of quality_flag for corresponding 
quality control variables

Ken,

I guess, I don't see this proposed change as necessary since the
distinction between the terms "quality" and "status" is really done in
the "flag_meanings" attribute and is basically free form/uncontrolled.
These attributes need to be used by this new name as well.

Let me rephrase my suggestion/question:
If this proposal is not adopted, but an example of how to use a
variable, with the standard name of "status_flag", to only indicate
data quality is included in the document, would that help?

-Barna

On Mon, Jul 22, 2019 at 1:22 PM Kehoe, Kenneth E. 
<mailto:kke...@ou.edu> wrote:



Barna,

Yes an update to the CF document should follow after the new
standard_name is implemented. I think multiple examples are needed since
status_flag covers many different types of state variables.

Ken



On 2019-7-22 10:35, Andrew Barna wrote:


Hi Martin, Ken,

Is there anything wrong with including multiple "status_flag"
variables to capture all separate state you wish? The CF document
unfortunately only includes an example of how to encode the status of
a sensor, but the actual meanings of the flag values are entirely up
to you, and this will not change with this proposal. Perhaps the CF
document would benefit from additional examples (e.g. one that only
shows data quality flags).

-Barna


On Mon, Jul 22, 2019 at 9:04 AM Kehoe, Kenneth E. 
<mailto:kke...@ou.edu> wrote:


Hi Martin,

I see status encompassing multiple metadata pieces of information. For
example it could be a state of the instrument as it cycles through a
pre-programed routine (Look at calibration target, look at sky, look at
ground, look at second calibration target, repeat...). Or the sources of
the inputs for a model where the availability or some other reason could
require making a decision on what source(s) to use. For provenance this
source information is important to report on a time step basis. Or the
status could be a data providers method to provide uncertainty
information (I see this as incorrect but some people do see it this
way). Each of these are important metadata but the method of use is
different than a strictly quality variable. A quality variable provides
information indicating if the data should be used or possibly could be
used in a weighted mean method to favor high quality data over low
quality data. The way the metadata is used is different depending on the
metadata type. A state of the instrument would be used for sub-setting
calibration vs. data. There is no ambiguity in this as data from a
calibration target is not used in a weather research analysis. But
quality is more subjective and is decided by the data user. If the
quality variable has 20 different quality tests the user would need to
decided if all 20 test results should be used or only a subset. Also,
the code for applying the quality is different than the state of the
instrument view (in my example above).

It is possible to have a quality test result from the state of the
instrument, but not the other way around (typically). So I need a way to
distinguish the two for automated or semi-automated tools. Hence my
point of quality_flag essentially being a subset of status_flag

Ken



On 2019-7-22 02:57, Martin Juckes - UKRI STFC wrote:


Dear Ken,


Can you expand on the distinction between "quality" and "status"? I understand 
that they are different in principle, but, in order to support this new 
standard name I think we need a clear objective statement of how we wo

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Barna,

Yes an update to the CF document should follow after the new 
standard_name is implemented. I think multiple examples are needed since 
status_flag covers many different types of state variables.

Ken



On 2019-7-22 10:35, Andrew Barna wrote:
> Hi Martin, Ken,
>
> Is there anything wrong with including multiple "status_flag"
> variables to capture all separate state you wish? The CF document
> unfortunately only includes an example of how to encode the status of
> a sensor, but the actual meanings of the flag values are entirely up
> to you, and this will not change with this proposal. Perhaps the CF
> document would benefit from additional examples (e.g. one that only
> shows data quality flags).
>
> -Barna
>
>
> On Mon, Jul 22, 2019 at 9:04 AM Kehoe, Kenneth E.  wrote:
>> Hi Martin,
>>
>> I see status encompassing multiple metadata pieces of information. For
>> example it could be a state of the instrument as it cycles through a
>> pre-programed routine (Look at calibration target, look at sky, look at
>> ground, look at second calibration target, repeat...). Or the sources of
>> the inputs for a model where the availability or some other reason could
>> require making a decision on what source(s) to use. For provenance this
>> source information is important to report on a time step basis. Or the
>> status could be a data providers method to provide uncertainty
>> information (I see this as incorrect but some people do see it this
>> way). Each of these are important metadata but the method of use is
>> different than a strictly quality variable. A quality variable provides
>> information indicating if the data should be used or possibly could be
>> used in a weighted mean method to favor high quality data over low
>> quality data. The way the metadata is used is different depending on the
>> metadata type. A state of the instrument would be used for sub-setting
>> calibration vs. data. There is no ambiguity in this as data from a
>> calibration target is not used in a weather research analysis. But
>> quality is more subjective and is decided by the data user. If the
>> quality variable has 20 different quality tests the user would need to
>> decided if all 20 test results should be used or only a subset. Also,
>> the code for applying the quality is different than the state of the
>> instrument view (in my example above).
>>
>> It is possible to have a quality test result from the state of the
>> instrument, but not the other way around (typically). So I need a way to
>> distinguish the two for automated or semi-automated tools. Hence my
>> point of quality_flag essentially being a subset of status_flag
>>
>> Ken
>>
>>
>>
>> On 2019-7-22 02:57, Martin Juckes - UKRI STFC wrote:
>>> Dear Ken,
>>>
>>>
>>> Can you expand on the distinction between "quality" and "status"? I 
>>> understand that they are different in principle, but, in order to support 
>>> this new standard name I think we need a clear objective statement of how 
>>> we would want to distinguish between them in CF.
>>>
>>> The conventions section on flags (3.5) mixes the two up 
>>> (http://cfconventions.org/cf-conventions/cf-conventions.html#flags ), so 
>>> some re-wording of the document would also be needed,
>>>
>>> regards,
>>> Martin
>>>
>>> 
>>> From: CF-metadata  on behalf of Kehoe, 
>>> Kenneth E. 
>>> Sent: 19 July 2019 06:42
>>> To: cf-metadata@cgd.ucar.edu
>>> Subject: [CF-metadata] New standard_name of quality_flag for corresponding 
>>> quality control variables
>>>
>>> Dear CF,
>>>
>>> I am proposing a new standard name of "quality_flag" to indicate a variable 
>>> is purely a quality control variable. A quality control variable would use 
>>> flag_values or flag_masks along with flag_meanings to allow declaring 
>>> levels of quality or results from quality indicating tests of the data 
>>> variable. This variable be a subset of the more general "status_flag" 
>>> standard name. Currently the definition of "status_flag" is:
>>>
>>> - A variable with the standard name of status_flag contains an indication 
>>> of quality or other status of another data variable. The linkage between 
>>> the data variable and the variable with the standard_name of status_flag is 
>>> achieved using the ancillary_variables attribute.
>>>
>>> This defini

Re: [CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Hi Martin,

I see status encompassing multiple metadata pieces of information. For 
example it could be a state of the instrument as it cycles through a 
pre-programed routine (Look at calibration target, look at sky, look at 
ground, look at second calibration target, repeat...). Or the sources of 
the inputs for a model where the availability or some other reason could 
require making a decision on what source(s) to use. For provenance this 
source information is important to report on a time step basis. Or the 
status could be a data providers method to provide uncertainty 
information (I see this as incorrect but some people do see it this 
way). Each of these are important metadata but the method of use is 
different than a strictly quality variable. A quality variable provides 
information indicating if the data should be used or possibly could be 
used in a weighted mean method to favor high quality data over low 
quality data. The way the metadata is used is different depending on the 
metadata type. A state of the instrument would be used for sub-setting 
calibration vs. data. There is no ambiguity in this as data from a 
calibration target is not used in a weather research analysis. But 
quality is more subjective and is decided by the data user. If the 
quality variable has 20 different quality tests the user would need to 
decided if all 20 test results should be used or only a subset. Also, 
the code for applying the quality is different than the state of the 
instrument view (in my example above).

It is possible to have a quality test result from the state of the 
instrument, but not the other way around (typically). So I need a way to 
distinguish the two for automated or semi-automated tools. Hence my 
point of quality_flag essentially being a subset of status_flag

Ken



On 2019-7-22 02:57, Martin Juckes - UKRI STFC wrote:
> Dear Ken,
>
>
> Can you expand on the distinction between "quality" and "status"? I 
> understand that they are different in principle, but, in order to support 
> this new standard name I think we need a clear objective statement of how we 
> would want to distinguish between them in CF.
>
> The conventions section on flags (3.5) mixes the two up 
> (http://cfconventions.org/cf-conventions/cf-conventions.html#flags ), so some 
> re-wording of the document would also be needed,
>
> regards,
> Martin
>
> ________
> From: CF-metadata  on behalf of Kehoe, 
> Kenneth E. 
> Sent: 19 July 2019 06:42
> To: cf-metadata@cgd.ucar.edu
> Subject: [CF-metadata] New standard_name of quality_flag for corresponding 
> quality control variables
>
> Dear CF,
>
> I am proposing a new standard name of "quality_flag" to indicate a variable 
> is purely a quality control variable. A quality control variable would use 
> flag_values or flag_masks along with flag_meanings to allow declaring levels 
> of quality or results from quality indicating tests of the data variable. 
> This variable be a subset of the more general "status_flag" standard name. 
> Currently the definition of "status_flag" is:
>
> - A variable with the standard name of status_flag contains an indication of 
> quality or other status of another data variable. The linkage between the 
> data variable and the variable with the standard_name of status_flag is 
> achieved using the ancillary_variables attribute.
>
> This definition includes a variable used to define the state or other status 
> information of a variable and can not be distinguished by standard name alone 
> from a state of the instrument, processing decision, source information, 
> needed metadata about the data variable or other ancillary variable type. 
> Since there is no other way to define a purely quality control variable, the 
> use of "status_flag" is too general for strictly quality control variables. 
> By having a method to define a variable as strictly quality control the 
> results of quality control tests can be applied to the data with a software 
> tool based on requests by the user. This would not affect current datasets 
> that do use "status_flag" nor require a change to the definition outside of 
> the indication that "quality_flag" standard name is available and a better 
> use for pure quality control variables.
>
> Proposed addition:
>
> quality_flag = A variable with the standard name of quality_flag contains an 
> indication of quality information of another data variable. The linkage 
> between the data variable and the variable or variables with the 
> standard_name of quality_flag is achieved using the ancillary_variables 
> attribute.
>
> Proposed change:
>
> status_flag = A variable with the standard name of status_flag contains an 
> in

Re: [CF-metadata] How to encode "not occurring" as distinct from "missing data"

[Kehoe, Kenneth E.]

Thanks for bringing this up Lars. I had a similar question a while back 
(http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2015/018489.html) and 
the suggestion was to use flag_values as you have proposed. As Jonathan 
indicated the use of flag_values is not prohibited but does come with 
some possible issues. Particularly software that automatically converts 
values outside the valid range to NaN or a single missing value 
indicator. Some people use this methodology instead of using 
missing_value attribute. I don't know of any software that does this 
automatically, but it could be an issue.

The proposal I gave was for some data that had history and could not be 
updated so I was looking for a solution that would work without changes. 
Discussions I had with others at meetings suggested using an ancillary 
state variable. I find that a better solution since it separates the 
data from metadata and would not require looping over multiple missing 
values (I don't know if that is supported) or run into the issue of code 
automatically changing data outside the valid range.

My program uses ancillary variables to contain corresponding quality 
control information, and one of the pieces of information is that the 
data variable is set to missing value. This allows for faster quality 
control analysis by only needing to look at the ancillary variable to 
find missing data instead of looking at both quality control and data 
variable.

My suggestion is to use a single missing value indicator with the data 
variable and then indicate the greater detail of why it is missing in 
the ancillary variable using flag_values. Additional quality information 
could be provided beyond the reason for being set to missing value. You 
can also provided multiple pieces (inclusive state) of information on a 
single value using flag_masks instead of flag_values.

For example:

cloud_layer_base_height(time, layer):float
     long_name = "Cloud base height of hydrometeor layers” ;
     units = "m” ;
     missing_value = -.f ;
     ancillary_varialbes = "qc_cloud_layer_base_height"
qc_cloud_layer_base_height (time, layer): short
     long_name = "Quality information for Cloud base height of 
hydrometeor layers"
     units = "1"
     flag_values = 0, 1, 2, 3, 4
     flag_meanings = "data_available input_value_missing 
input_data_exists_but_the_computation_did_not_result_in_a_valid_numeric_value 
value_missing_because_of_birds value_was_computed_but_I_would_not_use_it"
     standard_name = "status_flag"

I'm curious to hear your thoughts and others,

Ken


On 2019-7-19 06:57, Jonathan Gregory wrote:
> Dear Lars
>
> I think that using a flag_value would be a good CF way to do this. I am not
> sure whether it's a good idea to choose a value which is outside the valid
> range. That's not a problem for CF (that is, it's not prohibited), but maybe
> it might not suit some software, which could object if it wasn't aware of CF
> flag_values.
>
> Best wishes
>
> Jonathan
>
>
> - Forwarded message from Bärring Lars  -
>
>> Date: Fri, 19 Jul 2019 10:20:35 +
>> From: Bärring Lars 
>> To: "cf-metadata@cgd.ucar.edu" 
>> Subject: [CF-metadata] How to encode "not occurring" as distinct from
>>  "missing data"
>>
>> Dear all,
>>
>> We are considering how best to store data produced by some computation where 
>> there has to be a distinction between missing input data (i.e. no input data 
>> available) and "not occurring" (i.e. input data exists but the computation 
>> did not result in a valid numeric value).
>>
>> In practice, the situation is reasonably similar to what was discussed back 
>> in 2017  (in the thread "Recording "day of year on which something happens") 
>> where Jim Biard offered a solution 
>> (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2017/019238.html).
>>
>> We have considered his solution to use flag values outside the valid_range 
>> of the data variable to indicate "no_occurrence". We have also considered to 
>> use a separate quality variable with flag values to use as as a mask 
>> (combined with _MissVal in the data variable).
>>
>> In this work the following questions surfaced,
>>
>> -- Is there any experience regarding how 'standard software' would handle 
>> either of these alternatives, is one more generally accepted?
>>
>> -- Is there any experience to guide us regarding which is better, or 
>> generally more "in line with the CF Conventions"?
>>
>> -- Is there another better approach that we have not thought of?
>>
>>
>> Many thanks,
>> Lars
>>
>> ___
>> CF-metadata mailing list
>> CF-metadata@cgd.ucar.edu
>> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
>
> - End forwarded message -
> ___
> CF-metadata mailing list
> CF-metadata@cgd.ucar.edu
> http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

-- 
Kenneth E. Kehoe
   Research Associate - University of Oklahoma
   Cooperative 

[CF-metadata] New standard_name of quality_flag for corresponding quality control variables

[Kehoe, Kenneth E.]

Dear CF,

I am proposing a new standard name of "quality_flag" to indicate a variable is 
purely a quality control variable. A quality control variable would use 
flag_values or flag_masks along with flag_meanings to allow declaring levels of 
quality or results from quality indicating tests of the data variable. This 
variable be a subset of the more general "status_flag" standard name. Currently 
the definition of "status_flag" is:

- A variable with the standard name of status_flag contains an indication of 
quality or other status of another data variable. The linkage between the data 
variable and the variable with the standard_name of status_flag is achieved 
using the ancillary_variables attribute.

This definition includes a variable used to define the state or other status 
information of a variable and can not be distinguished by standard name alone 
from a state of the instrument, processing decision, source information, needed 
metadata about the data variable or other ancillary variable type. Since there 
is no other way to define a purely quality control variable, the use of 
"status_flag" is too general for strictly quality control variables. By having 
a method to define a variable as strictly quality control the results of 
quality control tests can be applied to the data with a software tool based on 
requests by the user. This would not affect current datasets that do use 
"status_flag" nor require a change to the definition outside of the indication 
that "quality_flag" standard name is available and a better use for pure 
quality control variables.

Proposed addition:

quality_flag = A variable with the standard name of quality_flag contains an 
indication of quality information of another data variable. The linkage between 
the data variable and the variable or variables with the standard_name of 
quality_flag is achieved using the ancillary_variables attribute.

Proposed change:

status_flag = A variable with the standard name of status_flag contains an 
indication of status of another data variable. The linkage between the data 
variable and the variable with the standard_name of status_flag is achieved 
using the ancillary_variables attribute. For data quality information use 
quality_flag.

Thanks,

Ken



--
Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climate Research Facility - Data Quality Office
  e-mail: kke...@ou.edu | Office: 303-497-4754
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Adding sensor_elevation_angle to standard_name

[Kehoe, Kenneth E.]

Hi Martin,

Yeah I went round and round with this trying to reuse the current definitions, 
but they were a little confusing to me. Using the statement "Local zenith is a 
line perpendicular to the Earth's surface at a given location." in 
sensor_zenith_angle (and platform_zenith_angle) does not work for me because I 
want zenith to be defined by direction of gravity not the Earth's surface.  The 
Earth's surface is often tilted. I think that needs to be updated according to 
my understanding of the term zenith. I assume there could be a use for angle 
between local surface of Earth, but I think that would be a different term than 
zenith. I think zenith_angle is currently OK. Is it worth defining zenith in 
the table for referencing by other standard_name's? I don't really know if that 
is how the standard_name table is designed to work.

I want the reference horizontal plane to be perpendicular to gravity, not the 
other definition where horizon is where the Earth surface and sky 
meet<https://www.merriam-webster.com/dictionary/horizon>. Is it worth creating 
a horizontal_plane standard name? I guess we could add something to allow a 
user to add "A comment attribute should be added to a data variable with this 
standard name to specify the reference direction." like sensor_azimuth_angle if 
they wanted to change it? But to be honest I don't understand this instruction 
in the sensor_azimuth_angle description statement and would probably never use 
it.

You may also need to update sensor_view_angle and sensor_zenith_angle from 
"line of sight from the sensor" to "line of sight of the sensor", but I don't 
want this proposal to get bogged down with changing the other standard name 
definitions. It may just be my interpretation and the existing definitions of 
sensor_zenith_angle and sensor_view_angle are correct.


Here is an updated definition:

proposed definition = sensor_elevation_angle is the angle measured in the 
vertical plane between the line of sight of the sensor and a horizontal plane 
perpendicular to local zenith. This angle is measured starting at zero from the 
horizontal plane directly in front of sensor. The angle is positive above the 
horizontal plane, and negative below the horizontal plane. Local zenith is 
directly overhead in the direction of gravity. A standard name also exists for 
sensor_view_angle and sensor_zenith_angle.

Ken



On 2019-3-19 03:56, Martin Juckes - UKRI STFC wrote:

Hello Ken,


this looks like a straight forward extension of existing names, but I'm puzzled 
by the definition of "zenith" in sensor_zenith_angle, where it is defined as 
"Local zenith is a line perpendicular to the Earth's surface at a given 
location." For the standard name "zenith_angle" the definition is different: 
"Zenith angle is the angle to the local vertical". The latter, with vertical 
defined by the direction of gravity, appears to be the standard usage: would 
you agree with this?


Is the reference to the Earth's surface in the definition of 
sensor_zenith_angle a mistake, which should be modified to bring it into line 
with zenith_angle, or is there a real need to have angles measured relative to 
the local plane of the Earth's surface?


For the sensor_elevation_angle, do you want a specific interpretation of 
"horizontal" (e.g. perpendicular to the direction of gravity), or is it 
acceptable to use this term with other interpretations of the reference 
horizontal plane?


regards,

Martin


________
From: CF-metadata 
<mailto:cf-metadata-boun...@cgd.ucar.edu> on 
behalf of Kehoe, Kenneth E. <mailto:kke...@ou.edu>
Sent: 19 March 2019 00:27
To: CF Metadata List
Subject: [CF-metadata] Adding sensor_elevation_angle to standard_name

I would like to add sensor_elevation_angle to complement the existing 
sensor_azimuth_angle in the standard_name table.

proposed standard_name = sensor_elevation_angle

proposed canonical units = degree

proposed definition = sensor_elevation_angle is the angle measured in the 
vertical plane between the line of sight of the sensor and a horizontal plane 
perpendicular to local zenith. This angle is measured starting from the 
horizontal plane directly in front of sensor. The angle is positive above the 
horizontal plane, and negative below the horizontal plane. Local zenith is 
directly overhead. A standard name also exists for sensor_view_angle and 
sensor_zenith_angle.

Thanks,

Ken


--
Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climate Research Facility - Data Quality Office
  e-mail: 
kke...@ou.edu<mailto:kke...@ou.edu><mailto:kke...@ou.edu><mailto:kke...@ou.edu> 
| Office: 303-497-4754



--
Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Stud

[CF-metadata] Adding sensor_elevation_angle to standard_name

[Kehoe, Kenneth E.]

I would like to add sensor_elevation_angle to complement the existing 
sensor_azimuth_angle in the standard_name table.

proposed standard_name = sensor_elevation_angle

proposed canonical units = degree

proposed definition = sensor_elevation_angle is the angle measured in the 
vertical plane between the line of sight of the sensor and a horizontal plane 
perpendicular to local zenith. This angle is measured starting from the 
horizontal plane directly in front of sensor. The angle is positive above the 
horizontal plane, and negative below the horizontal plane. Local zenith is 
directly overhead. A standard name also exists for sensor_view_angle and 
sensor_zenith_angle.

Thanks,

Ken


--
Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climate Research Facility - Data Quality Office
  e-mail: kke...@ou.edu | Office: 303-497-4754
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

Re: [CF-metadata] Platform Heave

[Kehoe, Kenneth E.]

I second Jim's thanks. Long discussion with a good outcome. The reason for this 
listserv!

Ken


On 2018-10-9 08:11, Jim Biard wrote:

Alison, Roy, Nan, (and the rest of you)

Thanks for all the hard work!

Jim

On 10/9/18 10:06 AM, Alison Pamment - UKRI STFC wrote:

Dear Roy,

Many thanks for checking through all the text.

We seem to have reached full consensus in this discussion, so all the platform 
names are accepted and will be included in the 15th October standard name table 
update. Altogether this will result in the addition of 28 new names, 3 aliases 
and 17 updated definitions. For the full list, please see the following link to 
the standard names editor: 
http://cfeditor.ceda.ac.uk/proposals/1?status=active=platform=+and+display=Filter.

I agree that we can always do further work to improve names and definitions, 
but also that now is the time to draw this particular discussion to a close. 
Thank you again to all those who submitted comments - these names are very much 
the result of community effort.

Best wishes,
Alison

--
Alison Pamment Tel: +44 1235 778065
NCAS/Centre for Environmental Data ArchivalEmail: 
alison.pamm...@stfc.ac.uk
STFC Rutherford Appleton Laboratory
R25, 2.22
Harwell Oxford, Didcot, OX11 0QX, U.K.

From: Lowry, Roy K. 
Sent: 07 October 2018 17:05
To: Pamment, Alison (STFC,RAL,RALSP) 
; CF-metadata 
(cf-metadata@cgd.ucar.edu) 

Subject: Re: Platform Heave

Dear Alison,

I have read through the definitions and can see no errors.

As always when doing this, one sees possible 'improvements' but these are not 
critical and so are best kept to myself or we will never reach end game with 
this proposal.

Cheers, Roy.

I have now retired but will continue to be active through an Emeritus 
Fellowship using this e-mail address.


From: CF-metadata on behalf of Alison Pamment - UKRI STFC
Sent: 03 October 2018 18:10
To: CF-metadata (mailto:cf-metadata@cgd.ucar.edu)
Subject: [CF-metadata] Platform Heave
Dear Jim, Roy, Nan, Jonathan, et al.,

I have drawn together what I hope is the final list for the platform names.

We seem to be agreed on the need to have triplets of names to cope with 
opposite sign conventions and the case where the sign convention is unknown. 
I've followed Roy's comment 
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2018/020540.html
 regarding cross-referencing in the definitions, i.e. adding the statement to 
only choose the unsigned names if the convention is truly unknown. I plan to 
create aliases for some of the existing names as follows:
platform_yaw_angle -> platform_yaw
platform_pictch_angle -> platform_pitch
platform_roll_angle > platform_roll
to make them consistent with the new names. In addition to Jim's 12th September 
proposals 
(http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2018/020513.html),
 new names for platform_sway, platform_sway_rate, platform_surge and 
platform_surge_rate are needed to provide complete triplets of names for all 
the quantities.

The definitions of the new quantities are all based on Jim's text, with the 
'platform' description moved to the end. 'Instrument' has been added to the 
list of platforms as I think this was requested by Nan earlier in the 
discussion. (I think we can regard an 'instrument' as a way to mount one or 
more 'sensors'). The definitions of all existing platform names will be updated 
for consistency.

On a grammatical point, I noticed that the definitions of yaw/pitch/roll/surge 
rate in Jim's message all said "Yaw/pitch/roll/surge rate *might not* include 
changes to the "at rest" position of the platform ..." whereas the definitions 
of sway/heave rate said "Sway/heave rate *may not* include changes to the "at 
rest" position of the platform ...". I have changed all the definitions to say 
"might not" 

Re: [CF-metadata] positive attribute expansion of use and reserved values

[Kehoe, Kenneth E.]

o know how to relate quantities with
opposite sign conventions in their standard names. The standard names are
mostly systematically constructed, but for use by humans; they aren't designed
to be parsed by machines. If there is a need for some means to deal with this,
I would favour recording the sign convention as a machine-readable extra piece
of information in the standard name table. If we put it in the table, it must
be consistent with the standard name; you'd just have to look it up, instead of
trying to extract it from the standard name.

Best wishes

Jonathan

- Forwarded message from "Kehoe, Kenneth E." 
<kke...@ou.edu<mailto:kke...@ou.edu>> -

Date: Wed, 27 Sep 2017 22:49:20 +
From: "Kehoe, Kenneth E." <kke...@ou.edu<mailto:kke...@ou.edu>>
To: "CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>" 
<CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>>
Subject: [CF-metadata] positive attribute expansion of use and reserved
values

CF-metadata,

I would like to propose an expansion of the use and reserved values for the 
“positive” attribute (section 4.3), specifically to include values in addition 
to the two reserved values of “up” and “down” to include “towards” and “away”.

Most variables define direction in the standard_name, but with instrumentation 
a standard name is often not available or the correct definition of the name 
exists with the wrong positive direction. Also, needing to understand or 
extract direction from the standard_name can be difficult for a simple tool not 
wanting to review the standard_name definition just to see if a transformation 
is needed. Expanding the use of the “positive” attribute would reduce the 
number of standard names by not needing to include positive direction in the 
definition. This would also follow the recommendation of being consistent with 
the definition between the standard_name and positive attribute. The “positive” 
attribute is currently reserved for use with coordinate dimensions only, but 
the same logic can be used with data variables to indicate direction. For 
example rate of speed for vertical velocities could be described by indicating 
positive = “up” when vertical velocities are positive when moving away from the 
surface.

Instruments are also often not installed perpendicular to the surface, and the 
coordinate system is better described as towards or away from the instrument. 
Specifically for radial instruments. Errors in misunderstanding direction with 
radial velocities or accelerations are comment when not specifically defined. 
There is no vendor standard.

I’ll leave my suggestion at towards and away, but this could also be expanded 
to include cardinal direction for East-West/North-South directions.

Thanks,

Ken




Kenneth E. Kehoe
 Research Associate - University of Oklahoma
 Cooperative Institute for Mesoscale Meteorological Studies
 ARM Climate Research Facility - Data Quality Office
 e-mail: kke...@ou.edu<mailto:kke...@ou.edu><mailto:kke...@ou.edu> | Office: 
303-497-4754 | Cell: 405-826-0299

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

- End forwarded message -
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
/formerly NOAA’s National Climatic Data Center/
151 Patton Ave, Asheville, NC 28801
e: jbi...@cicsnc.org<mailto:jbi...@cicsnc.org> <mailto:jbi...@cicsnc.org>
o: +1 828 271 4900

/Connect with us on Facebook for climate
<https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics
<https://www.facebook.com/NOAANCEIoceangeo> information, and follow
us on Twitter at @NOAANCEIclimate
<https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo
<https://twitter.com/NOAANCEIocngeo>. /



___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata


- End forwarded message -
___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu<mailto:CF-metadata@cgd.ucar.edu>
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata



Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climat

[CF-metadata] positive attribute expansion of use and reserved values

[Kehoe, Kenneth E.]

CF-metadata,

I would like to propose an expansion of the use and reserved values for the 
“positive” attribute (section 4.3), specifically to include values in addition 
to the two reserved values of “up” and “down” to include “towards” and “away”.

Most variables define direction in the standard_name, but with instrumentation 
a standard name is often not available or the correct definition of the name 
exists with the wrong positive direction. Also, needing to understand or 
extract direction from the standard_name can be difficult for a simple tool not 
wanting to review the standard_name definition just to see if a transformation 
is needed. Expanding the use of the “positive” attribute would reduce the 
number of standard names by not needing to include positive direction in the 
definition. This would also follow the recommendation of being consistent with 
the definition between the standard_name and positive attribute. The “positive” 
attribute is currently reserved for use with coordinate dimensions only, but 
the same logic can be used with data variables to indicate direction. For 
example rate of speed for vertical velocities could be described by indicating 
positive = “up” when vertical velocities are positive when moving away from the 
surface.

Instruments are also often not installed perpendicular to the surface, and the 
coordinate system is better described as towards or away from the instrument. 
Specifically for radial instruments. Errors in misunderstanding direction with 
radial velocities or accelerations are comment when not specifically defined. 
There is no vendor standard.

I’ll leave my suggestion at towards and away, but this could also be expanded 
to include cardinal direction for East-West/North-South directions.

Thanks,

Ken




Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climate Research Facility - Data Quality Office
  e-mail: kke...@ou.edu | Office: 303-497-4754 | Cell: 
405-826-0299

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Using the same time grid but different cell averaging periods

[Kehoe, Kenneth E.]

CF metadata,

I’m currently working on a dataset that has a suite of variables with two 
different averaging periods. Going through the archives I can’t find any 
discussions of this type. For example one variable measures average temperature 
each minute while a second variable measures average pressure over the 
preceding 5 minutes (essentially a running box car mean with overlapping 
samples.) The time step is the same for both variables. Is there a way to list 
two different time bounds variables associated with the same time step?

I was thinking of something like this:


dimensions:
  time = UNLIMITED; // (1440 currently)
  nv = 2;

variables:
  float temperature(time);
temperature:long_name = “Atmospheric temperature";
temperature:units = “degC";
temperature:cell_methods = "time: mean";

  float pressure(time);
pressure:long_name = “Atmospheric pressure over preceding 5 minutes";
pressure:units = “hPa";
pressure:cell_methods = "time: mean”;
pressure:bounds = "time_bnds2";

  double time(time);
time:long_name = "time";
time:units = “minutes since 1998-04-19 06:00:00 0:00";
time:bounds = "time_bnds1";
  double time_bnds1(time,nv);
  double time_bnds2(time,nv);

data:
  time = 0., 1., 2., 3., 4. ...;
  time_bnds1 = -1,0, 0,1, 1,2, 2,3, 3,4  ...;
  time_bnds2 = -5,0, -4,1, -3,2, -2,3, -1,4  ...;


Thanks,

Ken


Kenneth E. Kehoe
  Research Associate - University of Oklahoma
  Cooperative Institute for Mesoscale Meteorological Studies
  ARM Climate Research Facility Data Quality Office
e-mail: kke...@ou.edu
Office: 303-497-4754

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] New standard_name: downward_air_velocity

[Kehoe, Kenneth E.]

CF,

Can we add downward_air_velocity to be the counter to the existing 
upward_air_velocity.

Definition = A velocity is a vector quantity. “Downward" indicates a vector 
component which is positive when directed downward (negative upward). Downward 
air velocity is the vertical component of the 3D air velocity vector.

Thanks,

Ken



Kenneth E. Kehoe
Research Associate
ARM Climate Research Facility Data Quality Office

Office: 303-497-4754
Cell: 405-826-0299

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] Mixing data and flag values in one variable

[Kehoe, Kenneth E.]

I have a question about mixing data values and flag values. I know the correct 
method is to separate the data information from the flag information, but my 
colleague would prefer to keep the two in the same variable to simplify 
processing and reduce file size. The data files are already quite large. I 
looked through the discussion archive but could not find any discussion about 
this.

For example:

I have a data vector that occasionally has missing value indicators. They want 
to use multiple missing value indicators to indicate why the data is missing. A 
value of - indicates no data at all, a value of -2 indicates possible clear 
sky, -1 indicates it is clear sky, and anything greater than or equal to 0 
represents a data value. We will use valid_range to indicate that any value 
below 0 is not valid data and will be excluded. (I can’t use valid_min for 
historical reasons. We are sticking with missing_value for historical reasons.)

cloud_layer_base_height(time, layer):float
long_name = "Cloud base height of hydrometeor layers” ;
units = "m” ;
missing_value = -.f ;
valid_range = 0.f, 10.f ;
flag_values = -2.f , -1.f ;
flag_meanings = "possible_clear_sky clear_sky” ;
comment = "-2. Possible clear sky (No MPL observations available, 
Ceilometer obscured, but no cloud detected), -1. Clear sky, >= 0. Valid cloud 
base height” ;

Would this be acceptable?

Ken

Kenneth E. Kehoe
Research Associate
ARM Climate Research Facility Data Quality Office

Office: 303-497-4754
Cell: 405-826-0299

___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata

[CF-metadata] downward_air_velocity

[Kehoe, Kenneth E.]

Dear CF-metadata,

I would like to propose a new standard_name to complement upward_air_velocity. 
upward_air_velocity uses positive values to indicate upward direction, but we 
have many instrument that use a convention of positive values to indicate 
downward direction. Can we clone upward_air_velocity to downward_air_velocity 
with the sign convention reversed?

Thanks,

Ken



Kenneth E. Kehoe
Research Associate
ARM Climate Research Facility Data Quality Office

Office: 303-497-4754
Cell: 405-826-0299




___
CF-metadata mailing list
CF-metadata@cgd.ucar.edu
http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata