Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?
I tried to follow the proposal of an extension of the DB schema, but as I
am using a quite different setup (similar to / derived from the Brultech
monitor) I failed in the step of updating.
My attempt was to add the until now calculated value "heating_COP" to the
database.
Adding the column worked:
root@localhost:/home/weewx2# ./bin/weectl database add-column heating_COP
--config=wpm.conf --binding=wpm_binding
Using configuration file wpm.conf
Add new column 'heating_COP' of type 'REAL' to database 'wpm.sdb' (y/n)? y
New column heating_COP of type REAL added to database.
But updating did not work, it insists in using the default binding that I
do not use:
root@localhost:/home/weewx2# ./bin/weectl database calc-missing
--config=wpm.conf --binding=wpm_binding
Using configuration file wpm.conf
Missing derived observations will be calculated for all records.
Proceed (y/n)? y
Calculating missing derived observations...
Error: 'Unknown data binding ''wx_binding'''
Perhaps StdWXCalculate is using a different binding. Check configuration
file [StdWXCalculate] stanza
Nothing done. Aborting.
Any ideas?
vince schrieb am Sonntag, 25. Februar 2024 um 21:01:22 UTC+1:
> This has come up a lot of times vs. the Seasons skin if that's what you're
> referring to. v5 calculates things it expects are available in the db as
> needed, which is slow. In most cases adding the columns to the schema
> makes the problem go away. This seems to be coming up for folks starting
> in v3 weewx who are using the original wview-compatible schema rather than
> the wview_extended bigger schema that first appeared in v4 weewx.
>
> You can check to see if you're already running the extended schema in your
> db already by just counting the number of fields:
> echo "SELECT count() FROM PRAGMA_TABLE_INFO('archive');" |
> sqlite3 weewx.sdb
>
> The extended schema will return a count of 114. The original schema had
> around 53 or so if I recall correctly. If you're weewx 3.x you are
> probably going to see 53 as your answer.
>
> If you are running the small original schema you can either figure out
> which columns are missing and add just those, or update your db to the
> newer/bigger schema and not worry it. Given you have a fast PC to do this
> on, personally I'd upgrade the db to the big one. Doesn't take that long
> if you throw some compute at it.
>
> On Sunday, February 25, 2024 at 11:30:14 AM UTC-8 loeriver wrote:
>
>> Sorry for this late question:
>>
>> I went through this performance issue also and to have an apples to
>> apples comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and
>> had it running with a large db (280MB).
>>
>> For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on
>> a well-equipped PC, on my Raspi-type system it is >100s).
>>
>> Following the explanation from Tom Keffer ("... In V4, this would not
>> have even been attempted." ...) I would expect that in the output for
>> v4.10.2 some data should be missing / empty or different compared to the
>> 5.0.2 output. But this I do not see: Can somebody point me to the data I
>> should compare for this?
>>
>> WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:
>>
>>> Rewording Tom's post to align with one of my professors decades ago,
>>> "That will work, but let's do it the right way." Using 'weectl database'
>>> (followed by the proper parameters) is the right way. Humidex is now stored
>>> in the database, and '$alltime.humidex.max' takes a split second to return
>>> the correct result. My Rube Goldberg software machine has been dismantled.
>>>
>>> - Paul VE1DX
>>>
>>> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>>>
>>>> I mean this in the gentlest way, but this is basically what the SQLite
>>>> database is doing for you: saving various values in a file where they can
>>>> be retrieved via a general query statement.
>>>>
>>>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog wrote:
>>>>
>>>>> I encountered the same problem trying to calculate the all time
>>>>> humidex in the index.html.tmpl template. This was taking about 3 minutes
>>>>> on a Pi 4. Rather than add humidex to the database (probably the best
>>>>> solution), I calculated all-time humidex once a day using a small program
>>>>> and invoking it with cron. I saved it in a text file that I could then
>>>>> use
>>>>> the Cheetah include statement to read it into index.the html.tmpl
>>>>> templa
Re: [weewx-user] I'm desperate to fix my NOAA reports. Can I please pay someone to help fix my database and recover this data?
Sorry for this late question:
I went through this performance issue also and to have an apples to apples
comparison I set up weewx 4.10.2 and 5.0.2 on the same computer and had it
running with a large db (280MB).
For 4.10.2 file generation takes around 0.3s, for 5.0.2 it takes 13s (on a
well-equipped PC, on my Raspi-type system it is >100s).
Following the explanation from Tom Keffer ("... In V4, this would not have
even been attempted." ...) I would expect that in the output for v4.10.2
some data should be missing / empty or different compared to the 5.0.2
output. But this I do not see: Can somebody point me to the data I should
compare for this?
WindnFog schrieb am Samstag, 20. Januar 2024 um 14:14:08 UTC+1:
> Rewording Tom's post to align with one of my professors decades ago, "That
> will work, but let's do it the right way." Using 'weectl database'
> (followed by the proper parameters) is the right way. Humidex is now stored
> in the database, and '$alltime.humidex.max' takes a split second to return
> the correct result. My Rube Goldberg software machine has been dismantled.
>
> - Paul VE1DX
>
> On Friday, January 19, 2024 at 10:55:50 AM UTC-4 Tom Keffer wrote:
>
>> I mean this in the gentlest way, but this is basically what the SQLite
>> database is doing for you: saving various values in a file where they can
>> be retrieved via a general query statement.
>>
>> On Fri, Jan 19, 2024 at 6:16 AM WindnFog wrote:
>>
>>> I encountered the same problem trying to calculate the all time humidex
>>> in the index.html.tmpl template. This was taking about 3 minutes on a Pi
>>> 4. Rather than add humidex to the database (probably the best solution), I
>>> calculated all-time humidex once a day using a small program and invoking
>>> it with cron. I saved it in a text file that I could then use the Cheetah
>>> include statement to read it into index.the html.tmpl template. This runs
>>> in 2 seconds on a Pi 4 and 1 second on a pi 5; if you have an algorithm for
>>> appTemp, a similar approach might work. I was not comfortable with
>>> modifying the database because that's not my area of expertise:
>>>
>>> #!/usr/bin/env python3
>>> # -*- coding: utf-8 -*-
>>>
>>> """
>>> Program:WeeWX Humidex calculator
>>> Author: Paul M Dunphy
>>> Date: 12 December 2023
>>> Version:1.0.0
>>> Description:This script sweeps through the Weewx database and
>>> calculates the
>>> derived parameter humidex for each record. It then
>>> returns the
>>> maximum humidex and the date/time when it occured. This
>>> takes
>>> about 1.5 seconds on a 900,000 record database as
>>> opposed to 3.5
>>> minutes by the WeeWX system if the recommended aggregate
>>> values
>>> are implemented in the Standard/index.html.tmpl file:
>>>
>>> $alltime.humidex.max ($alltime.humidex.max.degree_F)
>>> $alltime.humidex.maxtime.format($ALLTIMEFMT)
>>>
>>> It's not clear why the weeWX system takes so long.
>>> Possibly
>>> because it's doing many other things at the same time.
>>>
>>> This runs on a Raspberry Pi 4B (4 GB Ram) using Debian 12
>>> """
>>>
>>> # Required Libraries
>>>
>>> import sqlite3
>>> import math
>>> from datetime import datetime
>>> import pytz
>>> from pytz import timezone
>>>
>>>
>>> def calculate_humidex(temperature, dewpoint):
>>> # Constants for the formula
>>> k = 273.15 # Conversion factor for Celsius to Kelvin
>>>
>>> # Convert temperatures to Kelvin
>>> temperature + k
>>> dewpoint_k = dewpoint + k
>>>
>>> # Calculate vapor pressure in hPa
>>> vapor_pressure = 6.112 * math.exp(5417.7530 * ((1 / 273.16) - (1 /
>>> dewpoint_k)))
>>>
>>> # Humidex calculation
>>> humidex = temperature + ((vapor_pressure - 10) * 0.)
>>> return humidex
>>>
>>>
>>> def get_max_humidex(database_path):
>>> # Connect to the WeeWX database
>>> conn = sqlite3.connect(database_path)
>>> cursor = conn.cursor()
>>>
>>> # Query for temperature, dewpoint, and dateTime for all records
>>> query = "SELECT outTemp, dewpoint, dateTime FROM archive"
>>> cursor.execute(query)
>>>
>>> # Variables to track the maximum humidex and its date
>>> maximum_humidex = None
>>> maximum_humidex_date = None
>>>
>>> for row in cursor:
>>> temperature, dewpoint, date_time = row
>>> if temperature is not None and dewpoint is not None:
>>> humidex = calculate_humidex(temperature, dewpoint)
>>> if maximum_humidex is None or humidex > maximum_humidex:
>>> maximum_humidex = humidex
>>> maximum_humidex_date = date_time
>>>
>>> conn.close()
>>> return maximum_humidex, maximum_humidex_date
>>>
>>>
>>> def convert_utc_to_ast_adt(utc_datetime):
>>> """
>>> Convert a
