I encountered the same problem trying to calculate the all time humidex in
the index.html.tmpl template. This was taking about 3 minutes on a Pi 4.
Rather than add humidex to the database (probably the best solution), I
calculated all-time humidex once a day using a small program and invoking
it with cron. I saved it in a text file that I could then use the Cheetah
include statement to read it into index.the html.tmpl template. This runs
in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
appTemp, a similar approach might work. I was not comfortable with
modifying the database because that's not my area of expertise:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Program: WeeWX Humidex calculator
Author: Paul M Dunphy
Date: 12 December 2023
Version: 1.0.0
Description: This script sweeps through the Weewx database and
calculates the
derived parameter humidex for each record. It then returns
the
maximum humidex and the date/time when it occured. This
takes
about 1.5 seconds on a 900,000 record database as opposed
to 3.5
minutes by the WeeWX system if the recommended aggregate
values
are implemented in the Standard/index.html.tmpl file:
$alltime.humidex.max ($alltime.humidex.max.degree_F)
$alltime.humidex.maxtime.format($ALLTIMEFMT)
It's not clear why the weeWX system takes so long. Possibly
because it's doing many other things at the same time.
This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
"""
# Required Libraries
import sqlite3
import math
from datetime import datetime
import pytz
from pytz import timezone
def calculate_humidex(temperature, dewpoint):
# Constants for the formula
k = 273.15 # Conversion factor for Celsius to Kelvin
# Convert temperatures to Kelvin
temperature + k
dewpoint_k = dewpoint + k
# Calculate vapor pressure in hPa
vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
dewpoint_k)))
# Humidex calculation
humidex = temperature + ((vapor_pressure - 10) * 0.5555)
return humidex
def get_max_humidex(database_path):
# Connect to the WeeWX database
conn = sqlite3.connect(database_path)
cursor = conn.cursor()
# Query for temperature, dewpoint, and dateTime for all records
query = "SELECT outTemp, dewpoint, dateTime FROM archive"
cursor.execute(query)
# Variables to track the maximum humidex and its date
maximum_humidex = None
maximum_humidex_date = None
for row in cursor:
temperature, dewpoint, date_time = row
if temperature is not None and dewpoint is not None:
humidex = calculate_humidex(temperature, dewpoint)
if maximum_humidex is None or humidex > maximum_humidex:
maximum_humidex = humidex
maximum_humidex_date = date_time
conn.close()
return maximum_humidex, maximum_humidex_date
def convert_utc_to_ast_adt(utc_datetime):
"""
Convert a UTC datetime to AST or ADT depending on the date.
Parameters:
utc_datetime (datetime): UTC datetime object.
Returns:
datetime: Datetime object in AST or ADT.
"""
# Define the time zone for AST/ADT
ast_tz = timezone('America/Halifax')
# Localize the UTC datetime to the AST/ADT timezone
local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
return local_datetime
def format_date(unix_timestamp):
"""
Format a UNIX timestamp into a human-readable string in AST or ADT.
Parameters:
unix_timestamp (int): UNIX timestamp.
Returns:
str: Formatted date string.
"""
# Create a UTC datetime from the UNIX timestamp
utc_datetime =
datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
# Convert UTC to AST or ADT
local_datetime = convert_utc_to_ast_adt(utc_datetime)
# Format the local datetime
return local_datetime.strftime("%I:%M %p on %d %b %Y")
db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
max_humidex, max_humidex_date = get_max_humidex(db_path)
# Convert Humidex to Fahrenheit
max_humidex_f = max_humidex * 9 / 5 + 32
# Format and print the result
print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
{format_date(max_humidex_date)}")
# Format the result
output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
{format_date(max_humidex_date)}"
# Write the result to a file
output_file_path =
'/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
with open(output_file_path, 'w') as file:
file.write(output)
print(f"Output written to {output_file_path}")
On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:
> Using Blaine's database, I was able to isolate the performance problems.
>
> It's in the template records/index.html.tmpl
>
> Specifically, apparent temperature (appTemp). It does not appear in the
> database, so a simple tag such as $alltime.appTemp.max requires searching
> the engine database, calculating apparent temperature for each record, then
> finding the max value.
>
> In V4, this would not have even been attempted.
>
> The solution is to add appTemp to the database.
>
> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>
>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive
>> period.
>> The historical files dating back to 2014 generated fine.
>>
>> Unrelated but interesting.....
>>
>> - this db shows the Belchertown taking too long issue
>> - running htop I see it peg two cpus and StdReport aborts due to time
>> vs. the Simulator 300 sec on a pi3+
>> - Belchertown 'always' takes longer than 300 secs and weewx always
>> aborts due to StdReport not completing in 300 secs
>> - yet the same installation with 'my' db works fine with no issues,
>> so it 'has' to be content of the db being used doesn't it ?
>> - I have been unable to get Belchertown to succeed even at 600 secs
>> archive period.
>>
>> Question - why would the db be 'locked' if a skin is reading it in order
>> to process its contents ??????
>>
>> last restart/failure...
>>
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>> version 5.0.0
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is
>> -0.21 seconds (positive is fast)
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>> 'wx_binding' to database 'weewx.sdb'
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>> backfill of daily summaries
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily summaries
>> up to date
>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>> packet loop.
>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>> Generated 8 files for report SeasonsReport in 3.96 seconds
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>> Generated 30 images for report SeasonsReport in 4.80 seconds
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5
>> files to /home/pi/weewx-data/public_html
>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version 1.3.1
>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>> exiting. Shutting engine down.
>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>> StdReport thread
>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut
>> down StdReport thread
>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>> OperationalError exception: database is locked
>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>> Waiting 2 minutes then retrying...
>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>> Generated 12 files for report Belchertown in 617.64 seconds
>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40
>> files to /home/pi/weewx-data/public_html/belchertown
>>
>>
>> weewx related open files:
>>
>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>> python3 1633 pi mem REG 179,2
>> 217360 272588
>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>> python3 1633 pi mem REG 179,2
>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>> python3 1633 pi mem REG 179,2
>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imaging.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>> libastro.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingft.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>
>> When its cpus are pegged trying to process Belchertown:
>>
>> python3 1633 pi cwd DIR 179,2
>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>> python3 1633 pi mem REG 179,2
>> 217360 272588
>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>> python3 1633 pi mem REG 179,2
>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>> python3 1633 pi mem REG 179,2
>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imaging.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>> libastro.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingft.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi mem REG 179,2
>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>> namemapper.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 pi 4uw REG 179,2
>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>> python3 1633 pi 5uw REG 179,2
>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>> python3 1633 pi 6u REG 179,2
>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>> python3 1633 1853 python3 pi cwd DIR 179,2
>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>> python3 1633 1853 python3 pi mem REG 179,2
>> 217360 272588
>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>> python3 1633 1853 python3 pi mem REG 179,2
>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>> python3 1633 1853 python3 pi mem REG 179,2
>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imaging.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 1853 python3 pi mem REG 179,2
>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>> libastro.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 1853 python3 pi mem REG 179,2
>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 1853 python3 pi mem REG 179,2
>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>> imagingft.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 1853 python3 pi mem REG 179,2
>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>> namemapper.cpython-311-arm-linux-gnueabihf.so
>> python3 1633 1853 python3 pi 4uw REG 179,2
>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>> python3 1633 1853 python3 pi 5uw REG 179,2
>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>> python3 1633 1853 python3 pi 6u REG 179,2
>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>
>>
>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>
>>> Thank you for the reply! Yes, I have tried this many times. It continues
>>> to regenerate them with the missing data.
>>>
>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>
>>> Have you tried completely deleting all NOAA reports and letting weewx
>>> regenerate them?
>>>
>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>
>>> Way back in 2020 my NOAA reports stopped populating their daily values.
>>> The average values for the month continue to be populated at the bottom of
>>> the table. Since 2020 every year or so I have spent hours banging my head
>>> against the wall following any and all instructions on the wiki, github,
>>> google and this user group to attempt to repair this issue. None of the
>>> published instructions seem to help. I have never been successful.
>>>
>>> If this is a problem that can be solved I would be more than happy to
>>> pay someone to help fix this issue knowing full well there are no
>>> guarantees recovery is possible. This data is important to me and having
>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-user+...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com.
