[weewx-user] Re: Trouble viewing new data column(s)

[James Runser]

Incremental progress. As described above but with the addition of a restart 
I have data in the statistics section but still N/A for Water Temp in 
"Current Conditions". Further, the data in statistics is formatted as 
"00.00" instead of the hoped-for "00.00  °F". I've walked through the 
template and inc files but still can't find the right building blocks. I 
even tried changing the waterTemp column from a double to a double(4,2) but 
am still getting all the extra zeros.

On Monday, January 16, 2023 at 2:16:12 PM UTC-5 James Runser wrote:

> I'm trying to integrate a home-built tide and temperature monitor into 
> weewx (configured and working without issue with an Acurite Atlas). I'm 
> using a MySQL DB and tried simply inserting my data (waterTemp, tideHeight) 
> into archive. I use current timestamp, usUnits and 10 for the required 
> interval values and let the remaining columns default to null ( except - 
> duh - waterTemp and tideHeight). 
>
> The data are written without issue but I'm never seeing the results in the 
> Seasons report - I chose waterTemp as the learning value since the 
> skin.conf already had a reference to water temp.
>
> My thinking is that my data is not returned by the report engine - based, 
> perhaps - on timestamp? Is there a better way to hook net new data into the 
> flow? I do in fact see the waterTemp tag in the current conditions box that 
> was enabled via current.inc. - the issue is that the value is always N/A. I 
> haven't dug deeply enough to figure out where the actual native Weewx 
> insert is done but am wary of the effort required to piggy-back on that 
> process.
>
> Advice or suggestions appreciated but a solid answer would not be kicked 
> out of bed for eating crackers ;)
>  
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/862a42c4-b38d-46a5-933c-aaf93a48bc43n%40googlegroups.com.

[weewx-user] Re: Trouble viewing new data column(s)

[gjr80]

If you add a new observation to the database you need to tell WeeWX what 
unit group it belongs to in order to be able to effectively use the 
formatting and unit conversion capabilities of the WeeWX tag system in 
reports. For example, if you add a field that is a pressure once WeeWX 
knows the new field is a pressure WeeWX knows what default formatting to 
apply, the correct unit label and you can use the formatting and unit 
conversion aspects of the tag system to change the format/convert units in 
a WeeWX report template. If WeeWX does not know what unit group the 
observation belongs to you just get the raw value from the database (which 
is typically a float with many decimal places - ie what you are seeing now).

There are a number of ways to assign an observation to a unit group. In 
your case the easiest approach is to add a few lines of code to 
/home/weewx/bin/user/extensions.py (or /usr/share/weewx/user/extensions.py 
if you installed WeeWX as a package). Try adding the following to the 
bottom of extensions.py:

import weewx.units
weewx.units.obs_group_dict['waterTemp'] = 'group_temperature'
weewx.units.obs_group_dict['tideHeight'] = 'group_length'

(group_temperature was the obvious choice for waterTemp; tideHeight could 
be group_length, group_altitude, group_rain or group_distance. The deciding 
factor here is the available units for each group - if you look at the Units 
 appendix to the Customisation 
Guide you will see what I mean)

You will need to restart WeeWX for the changes to take effect. The 
extensions.py code is run at WeeWX startup and the above lines will make 
the appropriate unit group assignments for your new fields. You should now 
be able to use tags based on waterTemp and tideHeight in your reports. You 
will find some further info on assigning unit groups in the Customising 
units and unit groups 
 section of the 
Customisation Guide. You will also find information on the WeeWX tag system 
in the Tags  section of the 
Customisation Guide.

A couple of notes about units.

For the WeeWX tag system to correctly and consistently display observation 
values and units you need to ensure the data you are adding to the database 
is in the correct units. How you do this depends on how you are inserting 
data into the database. If you are using a WeeWX service to augment loop 
packets (the preferred approach) then your service needs to take cognisance 
of the unit system (ie WeeWX field usUnits) of the loop packet being 
augmented and add waterTemp and tideHeight to the loop packet using the 
appropriate units for the unit group each observation belongs to for the 
unit system used by the loop packet. For example, if the loop packet uses 
US customary units (ie usUnits = 0) then waterTemp would need to be in 
Fahrenheit and tideHeight in inches. Once you do this WeeWX takes care of 
everything else. You will find information on the unit systems, unit groups 
and units used by WeeWX in the Units 
 appendix to the Customisation 
Guide.

If you are directly inserting data in the WeeWX database (not the preferred 
approach) then it is up to you to ensure that the values are inserted using 
units applicable to the database unit system and the unit group used for 
each observation.

Apologies for the long response but the WeeWX tag/unit/formatting system 
has a lot of moving parts and you need to have them all correctly 
configured or you will have problems.

Gary
On Tuesday, 17 January 2023 at 07:19:24 UTC+10 ja...@runser.net wrote:

> Incremental progress. As described above but with the addition of a 
> restart I have data in the statistics section but still N/A for Water Temp 
> in "Current Conditions". Further, the data in statistics is formatted as 
> "00.00" instead of the hoped-for "00.00  °F". I've walked through the 
> template and inc files but still can't find the right building blocks. I 
> even tried changing the waterTemp column from a double to a double(4,2) but 
> am still getting all the extra zeros.
>
> On Monday, January 16, 2023 at 2:16:12 PM UTC-5 James Runser wrote:
>
>> I'm trying to integrate a home-built tide and temperature monitor into 
>> weewx (configured and working without issue with an Acurite Atlas). I'm 
>> using a MySQL DB and tried simply inserting my data (waterTemp, tideHeight) 
>> into archive. I use current timestamp, usUnits and 10 for the required 
>> interval values and let the remaining columns default to null ( except - 
>> duh - waterTemp and tideHeight). 
>>
>> The data are written without issue but I'm never seeing the results in 
>> the Seasons report - I chose waterTemp as the learning value since the 
>> skin.conf already had a reference to water temp.
>>
>> My thinking is that my data is not returned by the report engine - based, 
>> perhaps

Re: [weewx-user] Re: Trouble viewing new data column(s)

['Ian Millard' via weewx-user]

Gary,Follow up question from me in a similar vein. What is the correct way to add a completely new measurement group to WeeWX, for example part_per_million_per_cubic_meter?IanSent from my iPad.On 17 Jan 2023, at 05:20, gjr80  wrote:If you add a new observation to the database you need to tell WeeWX what unit group it belongs to in order to be able to effectively use the formatting and unit conversion capabilities of the WeeWX tag system in reports. For example, if you add a field that is a pressure once WeeWX knows the new field is a pressure WeeWX knows what default formatting to apply, the correct unit label and you can use the formatting and unit conversion aspects of the tag system to change the format/convert units in a WeeWX report template. If WeeWX does not know what unit group the observation belongs to you just get the raw value from the database (which is typically a float with many decimal places - ie what you are seeing now).There are a number of ways to assign an observation to a unit group. In your case the easiest approach is to add a few lines of code to /home/weewx/bin/user/extensions.py (or /usr/share/weewx/user/extensions.py if you installed WeeWX as a package). Try adding the following to the bottom of extensions.py:import weewx.unitsweewx.units.obs_group_dict['waterTemp'] = 'group_temperature'weewx.units.obs_group_dict['tideHeight'] = 'group_length'(group_temperature was the obvious choice for waterTemp; tideHeight could be group_length, group_altitude, group_rain or group_distance. The deciding factor here is the available units for each group - if you look at the Units appendix to the Customisation Guide you will see what I mean)You will need to restart WeeWX for the changes to take effect. The extensions.py code is run at WeeWX startup and the above lines will make the appropriate unit group assignments for your new fields. You should now be able to use tags based on waterTemp and tideHeight in your reports. You will find some further info on assigning unit groups in the Customising units and unit groups section of the Customisation Guide. You will also find information on the WeeWX tag system in the Tags section of the Customisation Guide.A couple of notes about units.For the WeeWX tag system to correctly and consistently display observation values and units you need to ensure the data you are adding to the database is in the correct units. How you do this depends on how you are inserting data into the database. If you are using a WeeWX service to augment loop packets (the preferred approach) then your service needs to take cognisance of the unit system (ie WeeWX field usUnits) of the loop packet being augmented and add waterTemp and tideHeight to the loop packet using the appropriate units for the unit group each observation belongs to for the unit system used by the loop packet. For example, if the loop packet uses US customary units (ie usUnits = 0) then waterTemp would need to be in Fahrenheit and tideHeight in inches. Once you do this WeeWX takes care of everything else. You will find information on the unit systems, unit groups and units used by WeeWX in the Units appendix to the Customisation Guide.If you are directly inserting data in the WeeWX database (not the preferred approach) then it is up to you to ensure that the values are inserted using units applicable to the database unit system and the unit group used for each observation.Apologies for the long response but the WeeWX tag/unit/formatting system has a lot of moving parts and you need to have them all correctly configured or you will have problems.GaryOn Tuesday, 17 January 2023 at 07:19:24 UTC+10 ja...@runser.net wrote:Incremental progress. As described above but with the addition of a restart I have data in the statistics section but still N/A for Water Temp in "Current Conditions". Further, the data in statistics is formatted as "00.00" instead of the hoped-for "00.00 

°F". I've walked through the template and inc files but still can't find the right building blocks. I even tried changing the waterTemp column from a double to a double(4,2) but am still getting all the extra zeros.On Monday, January 16, 2023 at 2:16:12 PM UTC-5 James Runser wrote:I'm trying to integrate a home-built tide and temperature monitor into weewx (configured and working without issue with an Acurite Atlas). I'm using a MySQL DB and tried simply inserting my data (waterTemp, tideHeight) into archive. I use current timestamp, usUnits and 10 for the required interval values and let the remaining columns default to null ( except - duh - waterTemp and tideHeight). The data are written without issue but I'm never seeing the results in the Seasons report - I chose waterTemp as the learning value since the skin.conf already had a reference to water temp.My thinking is that my data is not returned by the report engine - based, perhaps - on timestamp? Is there a better way to hook net new data into the flow? I do in fact see the waterTemp 

Re: [weewx-user] Re: Trouble viewing new data column(s)

[gjr80]

Ian,

When you say measurement group I presume you mean unit - 
part_per_million_per_cubic_meter is a unit whereas what you call a 
'measurement group' sounds to me to be more like a unit group, eg 
group_temperature or group_pressure which are groups that contain details 
of all of the units used for temperature and pressure respectively. Not 
trying to be pedantic, but it's important to know what part of the WeeWX 
unit system we are dealing with - there are lots of moving parts. I'll 
assume your question is about adding a new unit.

Same basic process but instead of manipulating weewx.units.obs_group_dict 
you might need to manipulate weewx.units.USUnits, weewx.units.MetricUnits 
and weewx.units.MetricWXUnits unit systems if your new unit will be the 
default unit used for one or more of the three unit systems. If it is not 
to be a default but rather another unit in an existing unit group then you 
would need to add it to the weewx.units.conversionDict so that WeeWX knows 
how to convert to and from your new unit in it's unit group. If your new 
unit is not a default unit in a unit system or is not a unit to be 
converted to/from then you essentially won't be using your new unit. You 
may also need to define default unit formatting and unit labels for your 
new unit in units.default_unit_format_dict and units.default_unit_label_dict 
respectively.

Ignoring the existing WeeWX group_concentration, let's say you want a new 
unit group, 'group_new_concentration' and it will be the only unit in that 
unit group (ie no unit conversions required). Default unit formatting will 
be 1 decimal place and the default unit label will be 'ppm/m3'. So putting 
that altogether you might end up with something like:

import weewx.units
# set the unit system units
weewx.units.USUnits['group_new_concentration'] = 
'part_per_million_per_cubic_meter'
weewx.units.MetricUnits['group_new_concentration'] = 
'part_per_million_per_cubic_meter'
weewx.units.MetricWXUnits['group_new_concentration'] = 
'part_per_million_per_cubic_meter'
# define default formatting
weewx.units.default_unit_format_dict['part_per_million_per_cubic_meter'] = 
'%.1f'
# define default unit label
weewx.units.default_unit_label_dict['part_per_million_per_cubic_meter'] = ' 
ppm/m3'

Let's say you want to add part_per_million_per_cubic_meter to the existing 
group_concentration unit group. part_per_million_per_cubic_meter will not 
be the default group_concentration unit in any of the unit systems. Same 
default formatting and unit label. For the sake of argument lets say to 
convert from microgram_per_meter_cubed to part_per_million_per_cubic_meter 
you divide by 10, ie 10 microgram_per_meter_cubed = 1 
part_per_million_per_cubic_meter (this is a nonsense of course, it is just 
to illustrate a point). In this case you might use:

 import weewx.units
# define conversion functions for group_concentration
weewx.units.conversionDict['microgram_per_meter_cubed'] = 
{'part_per_million_per_cubic_meter': lambda x: x / 10.0}
weewx.units.conversionDict['part_per_million_per_cubic_meter'] = 
{'microgram_per_meter_cubed': lambda x: x * 10.0}
# define default formatting
weewx.units.default_unit_format_dict['part_per_million_per_cubic_meter'] = 
'%.1f'
# define default unit label
weewx.units.default_unit_label_dict['part_per_million_per_cubic_meter'] = ' 
ppm/m3'

If you wanted to set part_per_million_per_meter_cubed as the the default US 
customary unit then you would add in the following line:

weewx.units.USUnits['group_concentration'] = 
'part_per_million_per_cubic_meter'

Finally, where to put the code. Of course you could put it in extensions.py 
as per my previous post. If you were using this as part of a driver or 
service you have written (and that is the only place it will be used, ie 
someone who does not have your driver will unlikely use these settings) 
then you can put it in your driver or service file with zero indent, that 
way the code will be read and executed when your driver or service is 
loaded. My preference is the latter if it is applicable.

Gary


On Wednesday, 18 January 2023 at 19:31:52 UTC+10 steep...@btinternet.com 
wrote:

> Gary,
> Follow up question from me in a similar vein. What is the correct way to 
> add a completely new measurement group to WeeWX, for example 
> part_per_million_per_cubic_meter?
> Ian
>
> Sent from my iPad.
>
> On 17 Jan 2023, at 05:20, gjr80  wrote:
>
> If you add a new observation to the database you need to tell WeeWX what 
> unit group it belongs to in order to be able to effectively use the 
> formatting and unit conversion capabilities of the WeeWX tag system in 
> reports. For example, if you add a field that is a pressure once WeeWX 
> knows the new field is a pressure WeeWX knows what default formatting to 
> apply, the correct unit label and you can use the formatting and unit 
> conversion aspects of the tag system to change the format/convert units in 
> a WeeWX report template. If WeeWX does not know w

Re: [weewx-user] Re: Trouble viewing new data column(s)

['Cameron D' via weewx-user]

I hope this is not a real example you are suggesting -  part_per_million  
would be a concentration, which already has an implied volume component 
according to whether it is in air, water etc.   A unit of  
 part_per_million_per_cubic_meter does not make any sense to me. 

On Wednesday, 18 January 2023 at 7:31:52 pm UTC+10 steep...@btinternet.com 
wrote:

> Gary,
> Follow up question from me in a similar vein. What is the correct way to 
> add a completely new measurement group to WeeWX, for example 
> part_per_million_per_cubic_meter?
> Ian
>
> Sent from my iPad.
>
> On 17 Jan 2023, at 05:20, gjr80  wrote:
>
> If you add a new observation to the database you need to tell WeeWX what 
> unit group it belongs to in order to be able to effectively use the 
> formatting and unit conversion capabilities of the WeeWX tag system in 
> reports. For example, if you add a field that is a pressure once WeeWX 
> knows the new field is a pressure WeeWX knows what default formatting to 
> apply, the correct unit label and you can use the formatting and unit 
> conversion aspects of the tag system to change the format/convert units in 
> a WeeWX report template. If WeeWX does not know what unit group the 
> observation belongs to you just get the raw value from the database (which 
> is typically a float with many decimal places - ie what you are seeing now).
>
>
> There are a number of ways to assign an observation to a unit group. In 
> your case the easiest approach is to add a few lines of code to 
> /home/weewx/bin/user/extensions.py (or /usr/share/weewx/user/extensions.py 
> if you installed WeeWX as a package). Try adding the following to the 
> bottom of extensions.py:
>
> import weewx.units
> weewx.units.obs_group_dict['waterTemp'] = 'group_temperature'
> weewx.units.obs_group_dict['tideHeight'] = 'group_length'
>
> (group_temperature was the obvious choice for waterTemp; tideHeight could 
> be group_length, group_altitude, group_rain or group_distance. The 
> deciding factor here is the available units for each group - if you look at 
> the Units  appendix to the 
> Customisation Guide you will see what I mean)
>
> You will need to restart WeeWX for the changes to take effect. The 
> extensions.py code is run at WeeWX startup and the above lines will make 
> the appropriate unit group assignments for your new fields. You should now 
> be able to use tags based on waterTemp and tideHeight in your reports. 
> You will find some further info on assigning unit groups in the Customising 
> units and unit groups 
>  section of the 
> Customisation Guide. You will also find information on the WeeWX tag system 
> in the Tags  section of the 
> Customisation Guide.
>
> A couple of notes about units.
>
> For the WeeWX tag system to correctly and consistently display observation 
> values and units you need to ensure the data you are adding to the database 
> is in the correct units. How you do this depends on how you are inserting 
> data into the database. If you are using a WeeWX service to augment loop 
> packets (the preferred approach) then your service needs to take cognisance 
> of the unit system (ie WeeWX field usUnits) of the loop packet being 
> augmented and add waterTemp and tideHeight to the loop packet using the 
> appropriate units for the unit group each observation belongs to for the 
> unit system used by the loop packet. For example, if the loop packet uses 
> US customary units (ie usUnits = 0) then waterTemp would need to be in 
> Fahrenheit and tideHeight in inches. Once you do this WeeWX takes care of 
> everything else. You will find information on the unit systems, unit groups 
> and units used by WeeWX in the Units 
>  appendix to the 
> Customisation Guide.
>
> If you are directly inserting data in the WeeWX database (not the 
> preferred approach) then it is up to you to ensure that the values are 
> inserted using units applicable to the database unit system and the unit 
> group used for each observation.
>
> Apologies for the long response but the WeeWX tag/unit/formatting system 
> has a lot of moving parts and you need to have them all correctly 
> configured or you will have problems.
>
> Gary
> On Tuesday, 17 January 2023 at 07:19:24 UTC+10 ja...@runser.net wrote:
>
>> Incremental progress. As described above but with the addition of a 
>> restart I have data in the statistics section but still N/A for Water Temp 
>> in "Current Conditions". Further, the data in statistics is formatted as 
>> "00.00" instead of the hoped-for "00.00  °F". I've walked through the 
>> template and inc files but still can't find the right building blocks. I 
>> even tried changing the waterTemp column from a double to a double(4,2) but 
>> am still getting all the extra zeros.
>>
>> On Monday, January 16, 2023 at

Re: [weewx-user] Re: Trouble viewing new data column(s)

['Steeple Ian' via weewx-user]

Gary,
Thank you for your very extensive and detailed reply which confirmed my 
thoughts that it is not a simple process. However it has given a greater 
understanding of what is required.
Ian

On Thursday, January 19, 2023 at 12:27:15 AM UTC gjr80 wrote:

> Ian,
>
> When you say measurement group I presume you mean unit - 
> part_per_million_per_cubic_meter is a unit whereas what you call a 
> 'measurement group' sounds to me to be more like a unit group, eg 
> group_temperature or group_pressure which are groups that contain details 
> of all of the units used for temperature and pressure respectively. Not 
> trying to be pedantic, but it's important to know what part of the WeeWX 
> unit system we are dealing with - there are lots of moving parts. I'll 
> assume your question is about adding a new unit.
>
> Same basic process but instead of manipulating weewx.units.obs_group_dict 
> you might need to manipulate weewx.units.USUnits, weewx.units.MetricUnits 
> and weewx.units.MetricWXUnits unit systems if your new unit will be the 
> default unit used for one or more of the three unit systems. If it is not 
> to be a default but rather another unit in an existing unit group then you 
> would need to add it to the weewx.units.conversionDict so that WeeWX 
> knows how to convert to and from your new unit in it's unit group. If your 
> new unit is not a default unit in a unit system or is not a unit to be 
> converted to/from then you essentially won't be using your new unit. You 
> may also need to define default unit formatting and unit labels for your 
> new unit in units.default_unit_format_dict and 
> units.default_unit_label_dict respectively.
>
> Ignoring the existing WeeWX group_concentration, let's say you want a new 
> unit group, 'group_new_concentration' and it will be the only unit in that 
> unit group (ie no unit conversions required). Default unit formatting will 
> be 1 decimal place and the default unit label will be 'ppm/m3'. So putting 
> that altogether you might end up with something like:
>
> import weewx.units
> # set the unit system units
> weewx.units.USUnits['group_new_concentration'] = 
> 'part_per_million_per_cubic_meter'
> weewx.units.MetricUnits['group_new_concentration'] = 
> 'part_per_million_per_cubic_meter'
> weewx.units.MetricWXUnits['group_new_concentration'] = 
> 'part_per_million_per_cubic_meter'
> # define default formatting
> weewx.units.default_unit_format_dict['part_per_million_per_cubic_meter'] = 
> '%.1f'
> # define default unit label
> weewx.units.default_unit_label_dict['part_per_million_per_cubic_meter'] = 
> ' ppm/m3'
>
> Let's say you want to add part_per_million_per_cubic_meter to the 
> existing group_concentration unit group. part_per_million_per_cubic_meter 
> will not be the default group_concentration unit in any of the unit 
> systems. Same default formatting and unit label. For the sake of argument 
> lets say to convert from microgram_per_meter_cubed to 
> part_per_million_per_cubic_meter you divide by 10, ie 10 
> microgram_per_meter_cubed = 1 part_per_million_per_cubic_meter (this is a 
> nonsense of course, it is just to illustrate a point). In this case you 
> might use:
>
>  import weewx.units
> # define conversion functions for group_concentration
> weewx.units.conversionDict['microgram_per_meter_cubed'] = 
> {'part_per_million_per_cubic_meter': lambda x: x / 10.0}
> weewx.units.conversionDict['part_per_million_per_cubic_meter'] = 
> {'microgram_per_meter_cubed': lambda x: x * 10.0}
> # define default formatting
> weewx.units.default_unit_format_dict['part_per_million_per_cubic_meter'] = 
> '%.1f'
> # define default unit label
> weewx.units.default_unit_label_dict['part_per_million_per_cubic_meter'] = 
> ' ppm/m3'
>
> If you wanted to set part_per_million_per_meter_cubed as the the default 
> US customary unit then you would add in the following line:
>
> weewx.units.USUnits['group_concentration'] = 
> 'part_per_million_per_cubic_meter'
>
> Finally, where to put the code. Of course you could put it in 
> extensions.py as per my previous post. If you were using this as part of 
> a driver or service you have written (and that is the only place it will be 
> used, ie someone who does not have your driver will unlikely use these 
> settings) then you can put it in your driver or service file with zero 
> indent, that way the code will be read and executed when your driver or 
> service is loaded. My preference is the latter if it is applicable.
>
> Gary
>
>
> On Wednesday, 18 January 2023 at 19:31:52 UTC+10 steep...@btinternet.com 
> wrote:
>
>> Gary,
>> Follow up question from me in a similar vein. What is the correct way to 
>> add a completely new measurement group to WeeWX, for example 
>> part_per_million_per_cubic_meter?
>> Ian
>>
>> Sent from my iPad.
>>
>> On 17 Jan 2023, at 05:20, gjr80  wrote:
>>
>> If you add a new observation to the database you need to tell WeeWX what 
>> unit group it belongs to in order to be able to effectively use