I mean this in the gentlest way, but this is basically what the SQLite
database is doing for you: saving various values in a file where they can
be retrieved via a general query statement.
On Fri, Jan 19, 2024 at 6:16 AM WindnFog <foc1...@gmail.com> wrote:
> I encountered the same problem trying to calculate the all time humidex in
> the index.html.tmpl template. This was taking about 3 minutes on a Pi 4.
> Rather than add humidex to the database (probably the best solution), I
> calculated all-time humidex once a day using a small program and invoking
> it with cron. I saved it in a text file that I could then use the Cheetah
> include statement to read it into index.the html.tmpl template. This runs
> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
> appTemp, a similar approach might work. I was not comfortable with
> modifying the database because that's not my area of expertise:
>
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
>
> """
> Program: WeeWX Humidex calculator
> Author: Paul M Dunphy
> Date: 12 December 2023
> Version: 1.0.0
> Description: This script sweeps through the Weewx database and
> calculates the
> derived parameter humidex for each record. It then
> returns the
> maximum humidex and the date/time when it occured. This
> takes
> about 1.5 seconds on a 900,000 record database as opposed
> to 3.5
> minutes by the WeeWX system if the recommended aggregate
> values
> are implemented in the Standard/index.html.tmpl file:
>
> $alltime.humidex.max ($alltime.humidex.max.degree_F)
> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>
> It's not clear why the weeWX system takes so long.
> Possibly
> because it's doing many other things at the same time.
>
> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
> """
>
> # Required Libraries
>
> import sqlite3
> import math
> from datetime import datetime
> import pytz
> from pytz import timezone
>
>
> def calculate_humidex(temperature, dewpoint):
> # Constants for the formula
> k = 273.15 # Conversion factor for Celsius to Kelvin
>
> # Convert temperatures to Kelvin
> temperature + k
> dewpoint_k = dewpoint + k
>
> # Calculate vapor pressure in hPa
> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
> dewpoint_k)))
>
> # Humidex calculation
> humidex = temperature + ((vapor_pressure - 10) * 0.5555)
> return humidex
>
>
> def get_max_humidex(database_path):
> # Connect to the WeeWX database
> conn = sqlite3.connect(database_path)
> cursor = conn.cursor()
>
> # Query for temperature, dewpoint, and dateTime for all records
> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
> cursor.execute(query)
>
> # Variables to track the maximum humidex and its date
> maximum_humidex = None
> maximum_humidex_date = None
>
> for row in cursor:
> temperature, dewpoint, date_time = row
> if temperature is not None and dewpoint is not None:
> humidex = calculate_humidex(temperature, dewpoint)
> if maximum_humidex is None or humidex > maximum_humidex:
> maximum_humidex = humidex
> maximum_humidex_date = date_time
>
> conn.close()
> return maximum_humidex, maximum_humidex_date
>
>
> def convert_utc_to_ast_adt(utc_datetime):
> """
> Convert a UTC datetime to AST or ADT depending on the date.
>
> Parameters:
> utc_datetime (datetime): UTC datetime object.
>
> Returns:
> datetime: Datetime object in AST or ADT.
> """
> # Define the time zone for AST/ADT
> ast_tz = timezone('America/Halifax')
>
> # Localize the UTC datetime to the AST/ADT timezone
> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>
> return local_datetime
>
>
> def format_date(unix_timestamp):
> """
> Format a UNIX timestamp into a human-readable string in AST or ADT.
>
> Parameters:
> unix_timestamp (int): UNIX timestamp.
>
> Returns:
> str: Formatted date string.
> """
> # Create a UTC datetime from the UNIX timestamp
> utc_datetime =
> datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
>
> # Convert UTC to AST or ADT
> local_datetime = convert_utc_to_ast_adt(utc_datetime)
>
> # Format the local datetime
> return local_datetime.strftime("%I:%M %p on %d %b %Y")
>
>
> db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
> max_humidex, max_humidex_date = get_max_humidex(db_path)
>
> # Convert Humidex to Fahrenheit
> max_humidex_f = max_humidex * 9 / 5 + 32
>
> # Format and print the result
> print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
> {format_date(max_humidex_date)}")
>
> # Format the result
> output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
> {format_date(max_humidex_date)}"
>
> # Write the result to a file
> output_file_path =
> '/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
> with open(output_file_path, 'w') as file:
> file.write(output)
>
> print(f"Output written to {output_file_path}")
>
>
>
> On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:
>
>> Using Blaine's database, I was able to isolate the performance problems.
>>
>> It's in the template records/index.html.tmpl
>>
>> Specifically, apparent temperature (appTemp). It does not appear in the
>> database, so a simple tag such as $alltime.appTemp.max requires searching
>> the engine database, calculating apparent temperature for each record, then
>> finding the max value.
>>
>> In V4, this would not have even been attempted.
>>
>> The solution is to add appTemp to the database.
>>
>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>>
>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive
>>> period.
>>> The historical files dating back to 2014 generated fine.
>>>
>>> Unrelated but interesting.....
>>>
>>> - this db shows the Belchertown taking too long issue
>>> - running htop I see it peg two cpus and StdReport aborts due to
>>> time vs. the Simulator 300 sec on a pi3+
>>> - Belchertown 'always' takes longer than 300 secs and weewx always
>>> aborts due to StdReport not completing in 300 secs
>>> - yet the same installation with 'my' db works fine with no issues,
>>> so it 'has' to be content of the db being used doesn't it ?
>>> - I have been unable to get Belchertown to succeed even at 600 secs
>>> archive period.
>>>
>>> Question - why would the db be 'locked' if a skin is reading it in order
>>> to process its contents ??????
>>>
>>> last restart/failure...
>>>
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>>> version 5.0.0
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is
>>> -0.21 seconds (positive is fast)
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>>> 'wx_binding' to database 'weewx.sdb'
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>>> backfill of daily summaries
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily
>>> summaries up to date
>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>>> packet loop.
>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>> Generated 8 files for report SeasonsReport in 3.96 seconds
>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>>> Generated 30 images for report SeasonsReport in 4.80 seconds
>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5
>>> files to /home/pi/weewx-data/public_html
>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version
>>> 1.3.1
>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>>> exiting. Shutting engine down.
>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>>> StdReport thread
>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut
>>> down StdReport thread
>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>>> OperationalError exception: database is locked
>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>>> Waiting 2 minutes then retrying...
>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>> Generated 12 files for report Belchertown in 617.64 seconds
>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40
>>> files to /home/pi/weewx-data/public_html/belchertown
>>>
>>>
>>> weewx related open files:
>>>
>>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>>> python3 1633 pi mem REG 179,2
>>> 217360 272588
>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>> python3 1633 pi mem REG 179,2
>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>> python3 1633 pi mem REG 179,2
>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>
>>> When its cpus are pegged trying to process Belchertown:
>>>
>>> python3 1633 pi cwd DIR 179,2
>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>> python3 1633 pi mem REG 179,2
>>> 217360 272588
>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>> python3 1633 pi mem REG 179,2
>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>> python3 1633 pi mem REG 179,2
>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi mem REG 179,2
>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 pi 4uw REG 179,2
>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>> python3 1633 pi 5uw REG 179,2
>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>> python3 1633 pi 6u REG 179,2
>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>> python3 1633 1853 python3 pi cwd DIR 179,2
>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 217360 272588
>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 1853 python3 pi mem REG 179,2
>>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>> python3 1633 1853 python3 pi 4uw REG 179,2
>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>> python3 1633 1853 python3 pi 5uw REG 179,2
>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>> python3 1633 1853 python3 pi 6u REG 179,2
>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>
>>>
>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>>
>>>> Thank you for the reply! Yes, I have tried this many times. It
>>>> continues to regenerate them with the missing data.
>>>>
>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>>
>>>> Have you tried completely deleting all NOAA reports and letting weewx
>>>> regenerate them?
>>>>
>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>>
>>>> Way back in 2020 my NOAA reports stopped populating their daily values.
>>>> The average values for the month continue to be populated at the bottom of
>>>> the table. Since 2020 every year or so I have spent hours banging my head
>>>> against the wall following any and all instructions on the wiki, github,
>>>> google and this user group to attempt to repair this issue. None of the
>>>> published instructions seem to help. I have never been successful.
>>>>
>>>> If this is a problem that can be solved I would be more than happy to
>>>> pay someone to help fix this issue knowing full well there are no
>>>> guarantees recovery is possible. This data is important to me and having
>>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx-user+...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/CAPq0zECUbntXm5c%3D8NQWQvjGpS%2BUDq5Y_cKyDgxz8_z%2BbB9%2B7Q%40mail.gmail.com.
