Sorry for this late question:
I went through this performance issue also and to have an apples to apples
comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and had it
running with a large db (280MB).
For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on a
well-equipped PC, on my Raspi-type system it is >100s).
Following the explanation from Tom Keffer ("... In V4, this would not have
even been attempted." ...) I would expect that in the output for v4.10.2
some data should be missing / empty or different compared to the 5.0.2
output. But this I do not see: Can somebody point me to the data I should
compare for this?
WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:
> Rewording Tom's post to align with one of my professors decades ago, "That
> will work, but let's do it the right way." Using 'weectl database'
> (followed by the proper parameters) is the right way. Humidex is now stored
> in the database, and '$alltime.humidex.max' takes a split second to return
> the correct result. My Rube Goldberg software machine has been dismantled.
>
> - Paul VE1DX
>
> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>
>> I mean this in the gentlest way, but this is basically what the SQLite
>> database is doing for you: saving various values in a file where they can
>> be retrieved via a general query statement.
>>
>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog <foc...@gmail.com> wrote:
>>
>>> I encountered the same problem trying to calculate the all time humidex
>>> in the index.html.tmpl template. This was taking about 3 minutes on a Pi
>>> 4. Rather than add humidex to the database (probably the best solution), I
>>> calculated all-time humidex once a day using a small program and invoking
>>> it with cron. I saved it in a text file that I could then use the Cheetah
>>> include statement to read it into index.the html.tmpl template. This runs
>>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
>>> appTemp, a similar approach might work. I was not comfortable with
>>> modifying the database because that's not my area of expertise:
>>>
>>> #!/usr/bin/env python3
>>> # -*- coding: utf-8 -*-
>>>
>>> """
>>> Program: WeeWX Humidex calculator
>>> Author: Paul M Dunphy
>>> Date: 12 December 2023
>>> Version: 1.0.0
>>> Description: This script sweeps through the Weewx database and
>>> calculates the
>>> derived parameter humidex for each record. It then
>>> returns the
>>> maximum humidex and the date/time when it occured. This
>>> takes
>>> about 1.5 seconds on a 900,000 record database as
>>> opposed to 3.5
>>> minutes by the WeeWX system if the recommended aggregate
>>> values
>>> are implemented in the Standard/index.html.tmpl file:
>>>
>>> $alltime.humidex.max ($alltime.humidex.max.degree_F)
>>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>>
>>> It's not clear why the weeWX system takes so long.
>>> Possibly
>>> because it's doing many other things at the same time.
>>>
>>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>>> """
>>>
>>> # Required Libraries
>>>
>>> import sqlite3
>>> import math
>>> from datetime import datetime
>>> import pytz
>>> from pytz import timezone
>>>
>>>
>>> def calculate_humidex(temperature, dewpoint):
>>> # Constants for the formula
>>> k = 273.15 # Conversion factor for Celsius to Kelvin
>>>
>>> # Convert temperatures to Kelvin
>>> temperature + k
>>> dewpoint_k = dewpoint + k
>>>
>>> # Calculate vapor pressure in hPa
>>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
>>> dewpoint_k)))
>>>
>>> # Humidex calculation
>>> humidex = temperature + ((vapor_pressure - 10) * 0.5555)
>>> return humidex
>>>
>>>
>>> def get_max_humidex(database_path):
>>> # Connect to the WeeWX database
>>> conn = sqlite3.connect(database_path)
>>> cursor = conn.cursor()
>>>
>>> # Query for temperature, dewpoint, and dateTime for all records
>>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>>> cursor.execute(query)
>>>
>>> # Variables to track the maximum humidex and its date
>>> maximum_humidex = None
>>> maximum_humidex_date = None
>>>
>>> for row in cursor:
>>> temperature, dewpoint, date_time = row
>>> if temperature is not None and dewpoint is not None:
>>> humidex = calculate_humidex(temperature, dewpoint)
>>> if maximum_humidex is None or humidex > maximum_humidex:
>>> maximum_humidex = humidex
>>> maximum_humidex_date = date_time
>>>
>>> conn.close()
>>> return maximum_humidex, maximum_humidex_date
>>>
>>>
>>> def convert_utc_to_ast_adt(utc_datetime):
>>> """
>>> Convert a UTC datetime to AST or ADT depending on the date.
>>>
>>> Parameters:
>>> utc_datetime (datetime): UTC datetime object.
>>>
>>> Returns:
>>> datetime: Datetime object in AST or ADT.
>>> """
>>> # Define the time zone for AST/ADT
>>> ast_tz = timezone('America/Halifax')
>>>
>>> # Localize the UTC datetime to the AST/ADT timezone
>>> local_datetime = ast_tz.normalize(utc_datetime.astimezone(ast_tz))
>>>
>>> return local_datetime
>>>
>>>
>>> def format_date(unix_timestamp):
>>> """
>>> Format a UNIX timestamp into a human-readable string in AST or ADT.
>>>
>>> Parameters:
>>> unix_timestamp (int): UNIX timestamp.
>>>
>>> Returns:
>>> str: Formatted date string.
>>> """
>>> # Create a UTC datetime from the UNIX timestamp
>>> utc_datetime =
>>> datetime.utcfromtimestamp(unix_timestamp).replace(tzinfo=pytz.utc)
>>>
>>> # Convert UTC to AST or ADT
>>> local_datetime = convert_utc_to_ast_adt(utc_datetime)
>>>
>>> # Format the local datetime
>>> return local_datetime.strftime("%I:%M %p on %d %b %Y")
>>>
>>>
>>> db_path = '/home/pdunphy/weewx-data/archive/weewx.sdb'
>>> max_humidex, max_humidex_date = get_max_humidex(db_path)
>>>
>>> # Convert Humidex to Fahrenheit
>>> max_humidex_f = max_humidex * 9 / 5 + 32
>>>
>>> # Format and print the result
>>> print(f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>>> {format_date(max_humidex_date)}")
>>>
>>> # Format the result
>>> output = f"{max_humidex:.1f} C ({max_humidex_f:.1f} F) at
>>> {format_date(max_humidex_date)}"
>>>
>>> # Write the result to a file
>>> output_file_path =
>>> '/home/pdunphy/weewx-data/skins/Standard/Standard_all_time_humidex.txt'
>>> with open(output_file_path, 'w') as file:
>>> file.write(output)
>>>
>>> print(f"Output written to {output_file_path}")
>>>
>>>
>>>
>>> On Thursday, January 18, 2024 at 7:12:14 PM UTC-4 Tom Keffer wrote:
>>>
>>>> Using Blaine's database, I was able to isolate the performance problems.
>>>>
>>>> It's in the template records/index.html.tmpl
>>>>
>>>> Specifically, apparent temperature (appTemp). It does not appear in the
>>>> database, so a simple tag such as $alltime.appTemp.max requires searching
>>>> the engine database, calculating apparent temperature for each record,
>>>> then
>>>> finding the max value.
>>>>
>>>> In V4, this would not have even been attempted.
>>>>
>>>> The solution is to add appTemp to the database.
>>>>
>>>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>>>>
>>>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>>>>> The Seasons skin NOAA file for 2024-01 'is' being updated every
>>>>> archive period.
>>>>> The historical files dating back to 2014 generated fine.
>>>>>
>>>>> Unrelated but interesting.....
>>>>>
>>>>> - this db shows the Belchertown taking too long issue
>>>>> - running htop I see it peg two cpus and StdReport aborts due to
>>>>> time vs. the Simulator 300 sec on a pi3+
>>>>> - Belchertown 'always' takes longer than 300 secs and weewx always
>>>>> aborts due to StdReport not completing in 300 secs
>>>>> - yet the same installation with 'my' db works fine with no
>>>>> issues, so it 'has' to be content of the db being used doesn't it ?
>>>>> - I have been unable to get Belchertown to succeed even at 600
>>>>> secs archive period.
>>>>>
>>>>> Question - why would the db be 'locked' if a skin is reading it in
>>>>> order to process its contents ??????
>>>>>
>>>>> last restart/failure...
>>>>>
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>>>>> version 5.0.0
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error
>>>>> is -0.21 seconds (positive is fast)
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>>>>> 'wx_binding' to database 'weewx.sdb'
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>>>>> backfill of daily summaries
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily
>>>>> summaries up to date
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>>>>> packet loop.
>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>>>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>>> Generated 8 files for report SeasonsReport in 3.96 seconds
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>>>>> Generated 30 images for report SeasonsReport in 4.80 seconds
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>>> 5 files to /home/pi/weewx-data/public_html
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version
>>>>> 1.3.1
>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>>>>> exiting. Shutting engine down.
>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>>>>> StdReport thread
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to
>>>>> shut down StdReport thread
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>>>>> OperationalError exception: database is locked
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>>>>> Waiting 2 minutes then retrying...
>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>>> Generated 12 files for report Belchertown in 617.64 seconds
>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>>> 40 files to /home/pi/weewx-data/public_html/belchertown
>>>>>
>>>>>
>>>>> weewx related open files:
>>>>>
>>>>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>>>>> python3 1633 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>>
>>>>> When its cpus are pegged trying to process Belchertown:
>>>>>
>>>>> python3 1633 pi cwd DIR 179,2
>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>>> python3 1633 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi 4uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 pi 5uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 pi 6u REG 179,2
>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>> python3 1633 1853 python3 pi cwd DIR 179,2
>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi 4uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 1853 python3 pi 5uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 1853 python3 pi 6u REG 179,2
>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>>
>>>>>
>>>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>>>>
>>>>>> Thank you for the reply! Yes, I have tried this many times. It
>>>>>> continues to regenerate them with the missing data.
>>>>>>
>>>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>>>>
>>>>>> Have you tried completely deleting all NOAA reports and letting weewx
>>>>>> regenerate them?
>>>>>>
>>>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>>>>
>>>>>> Way back in 2020 my NOAA reports stopped populating their daily
>>>>>> values. The average values for the month continue to be populated at the
>>>>>> bottom of the table. Since 2020 every year or so I have spent hours
>>>>>> banging
>>>>>> my head against the wall following any and all instructions on the wiki,
>>>>>> github, google and this user group to attempt to repair this issue. None
>>>>>> of
>>>>>> the published instructions seem to help. I have never been successful.
>>>>>>
>>>>>> If this is a problem that can be solved I would be more than happy to
>>>>>> pay someone to help fix this issue knowing full well there are no
>>>>>> guarantees recovery is possible. This data is important to me and having
>>>>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx-user+...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to weewx-user+...@googlegroups.com.
>>>>>
>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/44a584f4-b98d-4180-9aa8-4d0f5bdc67f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/eba9f50a-852a-4103-a11e-b3fc0268397dn%40googlegroups.com.
