No need to be gentle, Tom. My approach was probably the most complicated
(worst?) solution possible. Without going into detail, when I was working,
I messed up a mission-critical Oracle database by trying to change
something I didn't understand. I can program fairly well, but I'm still
paranoid about databases! So, I need to get my head around adding fields
to the weeWX database and not devise these complex ways to avoid touching
it. :-)
On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
> I mean this in the gentlest way, but this is basically what the SQLite
> database is doing for you: saving various values in a file where they can
> be retrieved via a general query statement.
>
> On Fri, Jan 19, 2024 at 6:16 AM WindnFog <foc...@gmail.com> wrote:
>
>> I encountered the same problem trying to calculate the all time humidex
>> in the index.html.tmpl template. This was taking about 3 minutes on a Pi
>> 4. Rather than add humidex to the database (probably the best solution), I
>> calculated all-time humidex once a day using a small program and invoking
>> it with cron. I saved it in a text file that I could then use the Cheetah
>> include statement to read it into index.the html.tmpl template. This runs
>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
>> appTemp, a similar approach might work. I was not comfortable with
>> modifying the database because that's not my area of expertise:
>>
>> #!/usr/bin/env python3
>> # -*- coding: utf-8 -*-
>>
>> """
>> Program: WeeWX Humidex calculator
>> Author: Paul M Dunphy
>> Date: 12 December 2023
>> Version: 1.0.0
>> Description: This script sweeps through the Weewx database and
>> calculates the
>> derived parameter humidex for each record. It then
>> returns the
>> maximum humidex and the date/time when it occured. This
>> takes
>> about 1.5 seconds on a 900,000 record database as opposed
>> to 3.5
>> minutes by the WeeWX system if the recommended aggregate
>> values
>> are implemented in the Standard/index.html.tmpl file:
>>
>> $alltime.humidex.max ($alltime.humidex.max.degree_F)
>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>
>> It's not clear why the weeWX system takes so long.
>> Possibly
>> because it's doing many other things at the same time.
>>
>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>> """
>>
>> # Required Libraries
>>
>> import sqlite3
>> import math
>> from datetime import datetime
>> import pytz
>> from pytz import timezone
>>
>>
>> def calculate_humidex(temperature, dewpoint):
>> # Constants for the formula
>> k = 273.15 # Conversion factor for Celsius to Kelvin
>>
>> # Convert temperatures to Kelvin
>> temperature + k
>> dewpoint_k = dewpoint + k
>>
>> # Calculate vapor pressure in hPa
>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
>> dewpoint_k)))
>>
>> # Humidex calculation
>> humidex = temperature + ((vapor_pressure - 10) * 0.5555)
>> return humidex
>>
>>
>> def get_max_humidex(database_path):
>> # Connect to the WeeWX database
>> conn = sqlite3.connect(database_path)
>> cursor = conn.cursor()
>>
>> # Query for temperature, dewpoint, and dateTime for all records
>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>> cursor.execute(query)
>>
>> # Variables to track the maximum humidex and its date
>> maximum_humidex = None
>> maximum_humidex_date = None
>>
>> for row in cursor:
>> temperature, dewpoint, date_time = row
>> if temperature is not None and dewpoint is not None:
>> humidex = calculate_humidex(temperature, dewpoint)
>> if maximum_humidex is None or humidex > maximum_humidex:
>> maximum_humidex = humidex
>> maximum_humidex_date = date_time
>>
>> conn.close()
>> return maximum_humidex, maximum_humidex_date
>>
>>
>> def convert_utc_to_ast_adt(utc_datetime):
>> """
>> Convert a UTC datetime to AST or ADT depending on the date.
>>
>> Parameters:
>> utc_datetime (datetime): UTC datetime object.
>>
>> Returns:
>> datetime: Datetime object in AST or ADT.
>> """
>> # Define the time zone for AST/ADT
>> ast_tz = timezone('America/Halifax')
>>
>> # Localize the UTC datetime to the AST/ADT timezone
>> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>>
>> return local_datetime
>>
>>
>> def format_date(unix_timestamp):
>> """
>> Format a UNIX timestamp into a human-readable string in AST or ADT.
>>
>> Parameters:
>> unix_timestamp (int): UNIX timestamp.
>>
>> Returns:
>> str: Formatted date string.
>> """
>> # Create a UTC datetime from the UNIX timestamp
>> utc_datetime =
>> datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
>>
>> # Convert UTC to AST or ADT
>> local_datetime = convert_utc_to_ast_adt(utc_datetime)
>>
>> # Format the local datetime
>> return local_datetime.strftime("%I:%M %p on %d %b %Y")
>>
>>
>> db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
>> max_humidex, max_humidex_date = get_max_humidex(db_path)
>>
>> # Convert Humidex to Fahrenheit
>> max_humidex_f = max_humidex * 9 / 5 + 32
>>
>> # Format and print the result
>> print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>> {format_date(max_humidex_date)}")
>>
>> # Format the result
>> output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>> {format_date(max_humidex_date)}"
>>
>> # Write the result to a file
>> output_file_path =
>> '/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
>> with open(output_file_path, 'w') as file:
>> file.write(output)
>>
>> print(f"Output written to {output_file_path}")
>>
>>
>>
>> On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:
>>
>>> Using Blaine's database, I was able to isolate the performance problems.
>>>
>>> It's in the template records/index.html.tmpl
>>>
>>> Specifically, apparent temperature (appTemp). It does not appear in the
>>> database, so a simple tag such as $alltime.appTemp.max requires searching
>>> the engine database, calculating apparent temperature for each record, then
>>> finding the max value.
>>>
>>> In V4, this would not have even been attempted.
>>>
>>> The solution is to add appTemp to the database.
>>>
>>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>>>
>>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>>>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive
>>>> period.
>>>> The historical files dating back to 2014 generated fine.
>>>>
>>>> Unrelated but interesting.....
>>>>
>>>> - this db shows the Belchertown taking too long issue
>>>> - running htop I see it peg two cpus and StdReport aborts due to
>>>> time vs. the Simulator 300 sec on a pi3+
>>>> - Belchertown 'always' takes longer than 300 secs and weewx always
>>>> aborts due to StdReport not completing in 300 secs
>>>> - yet the same installation with 'my' db works fine with no issues,
>>>> so it 'has' to be content of the db being used doesn't it ?
>>>> - I have been unable to get Belchertown to succeed even at 600 secs
>>>> archive period.
>>>>
>>>> Question - why would the db be 'locked' if a skin is reading it in
>>>> order to process its contents ??????
>>>>
>>>> last restart/failure...
>>>>
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>>>> version 5.0.0
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is
>>>> -0.21 seconds (positive is fast)
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>>>> 'wx_binding' to database 'weewx.sdb'
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>>>> backfill of daily summaries
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily
>>>> summaries up to date
>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>>>> packet loop.
>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>> Generated 8 files for report SeasonsReport in 3.96 seconds
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>>>> Generated 30 images for report SeasonsReport in 4.80 seconds
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5
>>>> files to /home/pi/weewx-data/public_html
>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version
>>>> 1.3.1
>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>>>> exiting. Shutting engine down.
>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>>>> StdReport thread
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to
>>>> shut down StdReport thread
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>>>> OperationalError exception: database is locked
>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>>>> Waiting 2 minutes then retrying...
>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>> Generated 12 files for report Belchertown in 617.64 seconds
>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>> 40 files to /home/pi/weewx-data/public_html/belchertown
>>>>
>>>>
>>>> weewx related open files:
>>>>
>>>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>>>> python3 1633 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>
>>>> When its cpus are pegged trying to process Belchertown:
>>>>
>>>> python3 1633 pi cwd DIR 179,2
>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>> python3 1633 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 pi 4uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 pi 5uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 pi 6u REG 179,2
>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>> python3 1633 1853 python3 pi cwd DIR 179,2
>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 217360 272588
>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>> python3 1633 1853 python3 pi 4uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 1853 python3 pi 5uw REG 179,2
>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>> python3 1633 1853 python3 pi 6u REG 179,2
>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>
>>>>
>>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>>>
>>>>> Thank you for the reply! Yes, I have tried this many times. It
>>>>> continues to regenerate them with the missing data.
>>>>>
>>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>>>
>>>>> Have you tried completely deleting all NOAA reports and letting weewx
>>>>> regenerate them?
>>>>>
>>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>>>
>>>>> Way back in 2020 my NOAA reports stopped populating their daily
>>>>> values. The average values for the month continue to be populated at the
>>>>> bottom of the table. Since 2020 every year or so I have spent hours
>>>>> banging
>>>>> my head against the wall following any and all instructions on the wiki,
>>>>> github, google and this user group to attempt to repair this issue. None
>>>>> of
>>>>> the published instructions seem to help. I have never been successful.
>>>>>
>>>>> If this is a problem that can be solved I would be more than happy to
>>>>> pay someone to help fix this issue knowing full well there are no
>>>>> guarantees recovery is possible. This data is important to me and having
>>>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to weewx-user+...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx-user+...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-user+...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/19ebcada-d53b-4727-94d0-dff127a2b1f2n%40googlegroups.com.
