[weewx-user] Re: rsync: host key verification failed - problem

[Ben W.]

Thanks, Vince! 
I'm pretty sure I saw that link in my results, but the Ubuntu reference 
resulted in my continue scrolling. I will try your suggestion when I get 
back home!

On Saturday, June 1, 2024 at 7:59:44 PM UTC-5 vince wrote:

> If you google your exact error "*Host key verification failed."* and it 
> will return what's going on
>
>
> https://askubuntu.com/questions/45679/ssh-connection-problem-with-host-key-verification-failed-error
>
> The weewx-related answer is that you're using old notes that are still 
> expecting the 'root' user on the weewx system to be the local user, which 
> is no longer accurate.  In v5 it is the 'weewx' user (upgrade guide link) 
> . 
>  So it is very likely the host key of the remote computer is not known in 
> the /home/weewx/.ssh/known_hosts file.  Simplest way around this would be 
> do append whatever is in your legacy /root/.ssh/known_hosts file to your 
> /home/weewx/.ssh/known_hosts file.
>
> On Saturday, June 1, 2024 at 5:34:36 PM UTC-7 Ben W. wrote:
>
>> Greetings!
>> Thank you for logging these steps last year:
>>
>> 1. I logged as root ('sudo -i' from 'pi' account)
>> 2. generated SSH keys ('ssh-keygen')
>> 3. copied them to external server ('ssh-copy-id 
>> ace...@external.domain.com  -p 222')
>> 4. copied /home/pi/.ssh/config to /root/.ssh/config
>> 5. changed owner of 'config' ('chown root:root /root/.ssh/config')
>> 6. waited for next synchronization
>> 7. smiled because everything worked as expected :)
>>
>> I'm stuck between 6 and 7 and unsure what I'm doing wrong. My logs are 
>> below. I've copied the public key from the 'root' user and verified on my 
>> paid web hosting site that they have the same key even with 'root'. I can 
>> SSH in, but the host is still asking for a password - only if I enter the 
>> password am I able to authenticate. It's as-if either side isn't seeing the 
>> key. I'm assuming that's my problem with rsync. The logs:
>>
>> ___
>>
>> Jun 01 19:18:18 rpi systemd[1]: Started weewx.service - WeeWX.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Initializing weewxd 
>> version 5.0.2
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Command line: 
>> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Using Python 3.11.2 
>> (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Located at 
>> /usr/bin/python3
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Platform 
>> Linux-6.6.20+rpt-rpi-2712-aarch64-with-glibc2.36
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Locale: 'en_GB.UTF-8'
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Entry path: 
>> /usr/share/weewx/weewxd.py
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: WEEWX_ROOT: /etc/weewx
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Configuration file: 
>> /etc/weewx/weewx.conf
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: User module: 
>> /etc/weewx/bin/user
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station 
>> type GW1000 (user.gw1000)
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
>> version is 0.6.1
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
>> is 192.168.7.206:45000
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
>> is 20 seconds
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
>> version is 0.6.1
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
>> is 192.168.7.206:45000
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
>> is 20 seconds
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
>> unit is 0x1
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.wxservices: StdWXCalculate 
>> will use data binding wx_binding
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Archive will use 
>> data binding wx_binding
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Record generation 
>> will be attempted in 'software'
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Using archive 
>> interval of 300 seconds (software record generation)
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: StationRegistry: 
>> Station will be registered.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: Wunderground: 
>> Posting not enabled.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: PWSWeather: Data 
>> for station PROOF0FHUMBOLDT will be posted
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: CWOP: Posting not 
>> enabled.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: WOW: Posting not 
>> enabled.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: AWEKAS: Posting not 
>> enabled.
>> Jun 01 19:18

[weewx-user] Re: rsync: host key verification failed - problem

[Ben W.]

Hi Gary,

Thank you for this information. To be frank, I'm unsure how I did that, but 
I'll research how to resolve it. I have my weather station plus indoor 
sensors all feeding through the Ecowitt GW hub. My intent was to have WeeWX 
capture everything from the GW hub data.

Thank you!

On Saturday, June 1, 2024 at 8:44:39 PM UTC-5 gjr80 wrote:

> Nothing to do with your rsync issue, but I notice in your log extract that 
> you are running the Ecowitt gateway driver as both a driver and service in 
> the same WeeWX instance against the same Ecowitt gateway device:
>
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station 
> type GW1000 (user.gw1000)
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
> unit is 0x1
>
> I am hard pressed to think of a use case needing this configuration, it 
> was certainly not something I ever planned for when writing the driver. The 
> recommended use of the Ecowitt gateway driver in a given WeeWX instance is 
> to run it as a driver unless you have multiple gateway devices or another 
> device type that must be run with its own driver. Running as both a driver 
> and service against the same device on the same WeeWX instance will impact 
> performance (the gateway device is polled twice as often and the entire 
> sensor data and metadata is decoded twice irrespective of and field 
> filtering that may be applied).
>
> At best you are just increasing the system (and WeeWX) load, at worst you 
> are increasing the system load and may experience unknown 2nd order effects 
> on the data emitted by WeeWX.
>
> I would suggest dropping back to running the Ecowitt gateway driver as a 
> driver only.
>
> Gary
> On Sunday 2 June 2024 at 10:34:36 UTC+10 Ben W. wrote:
>
>> Greetings!
>> Thank you for logging these steps last year:
>>
>> 1. I logged as root ('sudo -i' from 'pi' account)
>> 2. generated SSH keys ('ssh-keygen')
>> 3. copied them to external server ('ssh-copy-id 
>> ace...@external.domain.com  -p 222')
>> 4. copied /home/pi/.ssh/config to /root/.ssh/config
>> 5. changed owner of 'config' ('chown root:root /root/.ssh/config')
>> 6. waited for next synchronization
>> 7. smiled because everything worked as expected :)
>>
>> I'm stuck between 6 and 7 and unsure what I'm doing wrong. My logs are 
>> below. I've copied the public key from the 'root' user and verified on my 
>> paid web hosting site that they have the same key even with 'root'. I can 
>> SSH in, but the host is still asking for a password - only if I enter the 
>> password am I able to authenticate. It's as-if either side isn't seeing the 
>> key. I'm assuming that's my problem with rsync. The logs:
>>
>> ___
>>
>> Jun 01 19:18:18 rpi systemd[1]: Started weewx.service - WeeWX.
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Initializing weewxd 
>> version 5.0.2
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Command line: 
>> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Using Python 3.11.2 
>> (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Located at 
>> /usr/bin/python3
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Platform 
>> Linux-6.6.20+rpt-rpi-2712-aarch64-with-glibc2.36
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Locale: 'en_GB.UTF-8'
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Entry path: 
>> /usr/share/weewx/weewxd.py
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: WEEWX_ROOT: /etc/weewx
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Configuration file: 
>> /etc/weewx/weewx.conf
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: User module: 
>> /etc/weewx/bin/user
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station 
>> type GW1000 (user.gw1000)
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
>> version is 0.6.1
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
>> is 192.168.7.206:45000
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
>> is 20 seconds
>> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
>> version is 0.6.1
>> Jun 01 19:18:18 rpi weewxd[200894]: IN

[weewx-user] Re: Belchertown and "database is locked" error

[Rob Cranfill]

Thanks, Vince. I read those other threads but didn't realize my particular 
error was from the same issue. I will look into it; meanwhile I've just 
turned Belchertown off.

On Saturday, June 1, 2024 at 6:06:07 PM UTC-7 vince wrote:

> Your computer is too slow, or your SD card is too slow, or your archive 
> period is too frequent (60 secs is very fast), or some combination of the 
> above.   For Belchertown specifically, many folks have run into issues with 
> v5 calculating items referenced in skins that are not elements in their db. 
>  Check the earlier threads from this week for the handful that might be 
> missing in your db.
>
> At a minimum run 'weectl debug' and show us which DB elements are defined. 
>  That's 114 elements in the default v4 and above schema.  If you see a 
> number more like 51 or so, you're running the old schema and you'll need to 
> add some fields (per the multiple other threads in the last week or so)
>
> Archive info
>   Database name:vp2.sdb
>   Table name:   archive
>   Version   4.0
>   Unit system:  1 (US)
>   First good timestamp: 2006-11-29 19:24:00 PST (1164857040)
>   Last good timestamp:  2024-06-01 09:10:00 PDT (1717258200)
>   Number of records:1713267
>   weewx (weewx.conf) is set to use an archive interval of 300 seconds.
>   The station hardware was not interrogated to determine the archive 
> interval.
>
> Supported SQL keys
>   dateTime  usUnits   interval
>   altimeter appTemp   appTemp1
>   barometer batteryStatus1batteryStatus2
>   batteryStatus3batteryStatus4batteryStatus5
>   batteryStatus6batteryStatus7batteryStatus8
>   cloudbase coco2
>   consBatteryVoltagedewpoint  dewpoint1
>   ETextraHumid1   extraHumid2
>   extraHumid3   extraHumid4   extraHumid5
>   extraHumid6   extraHumid7   extraHumid8
>   extraTemp1extraTemp2extraTemp3
>   extraTemp4extraTemp5extraTemp6
>   extraTemp7extraTemp8forecast
>   hail  hailBatteryStatus hailRate
>   heatindex heatindex1heatingTemp
>   heatingVoltagehumidex   humidex1
>   inDewpointinHumidityinTemp
>   inTempBatteryStatus   leafTemp1 leafTemp2
>   leafWet1  leafWet2  lightning_distance
>   lightning_disturber_count lightning_energy  lightning_noise_count
>   lightning_strike_countluminositymaxSolarRad
>   nh3   no2   noise
>   o3outHumidity   outTemp
>   outTempBatteryStatus  pbpm10_0
>   pm1_0 pm2_5 pressure
>   radiation rain  rainBatteryStatus
>   rainRate  referenceVoltage  rxCheckPercent
>   signal1   signal2   signal3
>   signal4   signal5   signal6
>   signal7   signal8   snow
>   snowBatteryStatus snowDepth snowMoisture
>   snowRate  so2   soilMoist1
>   soilMoist2soilMoist3soilMoist4
>   soilTemp1 soilTemp2 soilTemp3
>   soilTemp4 supplyVoltage txBatteryStatus
>   UVuvBatteryStatus   windBatteryStatus
>   windchill windDir   windGust
>   windGustDir   windrun   windSpeed
>
> On Saturday, June 1, 2024 at 5:58:01 PM UTC-7 Rob Cranfill wrote:
>
>> Upgraded to 4.10 to 5.0.2, thanks to help here. But now my Belchertown is 
>> dying and restarting the whole shebang with a "database is locked" error. 
>>
>> Ideas?
>>
>>
>>
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO __main__: retrying...
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Loading station 
>> type Vantage (weewx.drivers.vantage)
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: StdConvert target 
>> unit is 0x1
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.wxservices: 
>> StdWXCalculate will use data binding wx_binding
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Archive will use 
>> data binding wx_binding
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Record generation 
>> will be attempted in 'hardware'
>> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: The archive 
>> interval in the configuration file (300) does not match the station 
>> hard

[weewx-user] Re: rsync: host key verification failed - problem

[gjr80]

Nothing to do with your rsync issue, but I notice in your log extract that 
you are running the Ecowitt gateway driver as both a driver and service in 
the same WeeWX instance against the same Ecowitt gateway device:

Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station type 
GW1000 (user.gw1000)
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
version is 0.6.1
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
is 192.168.7.206:45000
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval is 
20 seconds
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
version is 0.6.1
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
is 192.168.7.206:45000
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval is 
20 seconds
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
unit is 0x1

I am hard pressed to think of a use case needing this configuration, it was 
certainly not something I ever planned for when writing the driver. The 
recommended use of the Ecowitt gateway driver in a given WeeWX instance is 
to run it as a driver unless you have multiple gateway devices or another 
device type that must be run with its own driver. Running as both a driver 
and service against the same device on the same WeeWX instance will impact 
performance (the gateway device is polled twice as often and the entire 
sensor data and metadata is decoded twice irrespective of and field 
filtering that may be applied).

At best you are just increasing the system (and WeeWX) load, at worst you 
are increasing the system load and may experience unknown 2nd order effects 
on the data emitted by WeeWX.

I would suggest dropping back to running the Ecowitt gateway driver as a 
driver only.

Gary
On Sunday 2 June 2024 at 10:34:36 UTC+10 Ben W. wrote:

> Greetings!
> Thank you for logging these steps last year:
>
> 1. I logged as root ('sudo -i' from 'pi' account)
> 2. generated SSH keys ('ssh-keygen')
> 3. copied them to external server ('ssh-copy-id ace...@external.domain.com 
>  -p 222')
> 4. copied /home/pi/.ssh/config to /root/.ssh/config
> 5. changed owner of 'config' ('chown root:root /root/.ssh/config')
> 6. waited for next synchronization
> 7. smiled because everything worked as expected :)
>
> I'm stuck between 6 and 7 and unsure what I'm doing wrong. My logs are 
> below. I've copied the public key from the 'root' user and verified on my 
> paid web hosting site that they have the same key even with 'root'. I can 
> SSH in, but the host is still asking for a password - only if I enter the 
> password am I able to authenticate. It's as-if either side isn't seeing the 
> key. I'm assuming that's my problem with rsync. The logs:
>
> ___
>
> Jun 01 19:18:18 rpi systemd[1]: Started weewx.service - WeeWX.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Initializing weewxd 
> version 5.0.2
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Command line: 
> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Using Python 3.11.2 
> (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Located at 
> /usr/bin/python3
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Platform 
> Linux-6.6.20+rpt-rpi-2712-aarch64-with-glibc2.36
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Locale: 'en_GB.UTF-8'
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Entry path: 
> /usr/share/weewx/weewxd.py
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: WEEWX_ROOT: /etc/weewx
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Configuration file: 
> /etc/weewx/weewx.conf
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: User module: 
> /etc/weewx/bin/user
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station 
> type GW1000 (user.gw1000)
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
> unit is 0x1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.wxservices: StdWXCalculate 
> will use data binding wx_binding
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Archive will use 
> data binding wx_binding
> Jun 01 19:18:18 rpi w

[weewx-user] Re: Belchertown and "database is locked" error

[vince]

Your computer is too slow, or your SD card is too slow, or your archive 
period is too frequent (60 secs is very fast), or some combination of the 
above.   For Belchertown specifically, many folks have run into issues with 
v5 calculating items referenced in skins that are not elements in their db. 
 Check the earlier threads from this week for the handful that might be 
missing in your db.

At a minimum run 'weectl debug' and show us which DB elements are defined. 
 That's 114 elements in the default v4 and above schema.  If you see a 
number more like 51 or so, you're running the old schema and you'll need to 
add some fields (per the multiple other threads in the last week or so)

Archive info
  Database name:vp2.sdb
  Table name:   archive
  Version   4.0
  Unit system:  1 (US)
  First good timestamp: 2006-11-29 19:24:00 PST (1164857040)
  Last good timestamp:  2024-06-01 09:10:00 PDT (1717258200)
  Number of records:1713267
  weewx (weewx.conf) is set to use an archive interval of 300 seconds.
  The station hardware was not interrogated to determine the archive 
interval.

Supported SQL keys
  dateTime  usUnits   interval
  altimeter appTemp   appTemp1
  barometer batteryStatus1batteryStatus2
  batteryStatus3batteryStatus4batteryStatus5
  batteryStatus6batteryStatus7batteryStatus8
  cloudbase coco2
  consBatteryVoltagedewpoint  dewpoint1
  ETextraHumid1   extraHumid2
  extraHumid3   extraHumid4   extraHumid5
  extraHumid6   extraHumid7   extraHumid8
  extraTemp1extraTemp2extraTemp3
  extraTemp4extraTemp5extraTemp6
  extraTemp7extraTemp8forecast
  hail  hailBatteryStatus hailRate
  heatindex heatindex1heatingTemp
  heatingVoltagehumidex   humidex1
  inDewpointinHumidityinTemp
  inTempBatteryStatus   leafTemp1 leafTemp2
  leafWet1  leafWet2  lightning_distance
  lightning_disturber_count lightning_energy  lightning_noise_count
  lightning_strike_countluminositymaxSolarRad
  nh3   no2   noise
  o3outHumidity   outTemp
  outTempBatteryStatus  pbpm10_0
  pm1_0 pm2_5 pressure
  radiation rain  rainBatteryStatus
  rainRate  referenceVoltage  rxCheckPercent
  signal1   signal2   signal3
  signal4   signal5   signal6
  signal7   signal8   snow
  snowBatteryStatus snowDepth snowMoisture
  snowRate  so2   soilMoist1
  soilMoist2soilMoist3soilMoist4
  soilTemp1 soilTemp2 soilTemp3
  soilTemp4 supplyVoltage txBatteryStatus
  UVuvBatteryStatus   windBatteryStatus
  windchill windDir   windGust
  windGustDir   windrun   windSpeed

On Saturday, June 1, 2024 at 5:58:01 PM UTC-7 Rob Cranfill wrote:

> Upgraded to 4.10 to 5.0.2, thanks to help here. But now my Belchertown is 
> dying and restarting the whole shebang with a "database is locked" error. 
>
> Ideas?
>
>
>
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO __main__: retrying...
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Loading station 
> type Vantage (weewx.drivers.vantage)
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: StdConvert target 
> unit is 0x1
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.wxservices: StdWXCalculate 
> will use data binding wx_binding
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Archive will use 
> data binding wx_binding
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Record generation 
> will be attempted in 'hardware'
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: The archive 
> interval in the configuration file (300) does not match the station 
> hardware interval (60).
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Using archive 
> interval of 60 seconds (specified by hardware)
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: StationRegistry: 
> Station will be registered.
> Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: Wunderground-PWS: 
> Data for station KWASEATT418 will be posted
> Jun 01 17:4

[weewx-user] Re: rsync: host key verification failed - problem

[vince]

If you google your exact error "*Host key verification failed."* and it 
will return what's going on

https://askubuntu.com/questions/45679/ssh-connection-problem-with-host-key-verification-failed-error

The weewx-related answer is that you're using old notes that are still 
expecting the 'root' user on the weewx system to be the local user, which 
is no longer accurate.  In v5 it is the 'weewx' user (upgrade guide link) 
.  So 
it is very likely the host key of the remote computer is not known in the 
/home/weewx/.ssh/known_hosts file.  Simplest way around this would be do 
append whatever is in your legacy /root/.ssh/known_hosts file to your 
/home/weewx/.ssh/known_hosts file.

On Saturday, June 1, 2024 at 5:34:36 PM UTC-7 Ben W. wrote:

> Greetings!
> Thank you for logging these steps last year:
>
> 1. I logged as root ('sudo -i' from 'pi' account)
> 2. generated SSH keys ('ssh-keygen')
> 3. copied them to external server ('ssh-copy-id ace...@external.domain.com 
>  -p 222')
> 4. copied /home/pi/.ssh/config to /root/.ssh/config
> 5. changed owner of 'config' ('chown root:root /root/.ssh/config')
> 6. waited for next synchronization
> 7. smiled because everything worked as expected :)
>
> I'm stuck between 6 and 7 and unsure what I'm doing wrong. My logs are 
> below. I've copied the public key from the 'root' user and verified on my 
> paid web hosting site that they have the same key even with 'root'. I can 
> SSH in, but the host is still asking for a password - only if I enter the 
> password am I able to authenticate. It's as-if either side isn't seeing the 
> key. I'm assuming that's my problem with rsync. The logs:
>
> ___
>
> Jun 01 19:18:18 rpi systemd[1]: Started weewx.service - WeeWX.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Initializing weewxd 
> version 5.0.2
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Command line: 
> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Using Python 3.11.2 
> (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Located at 
> /usr/bin/python3
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Platform 
> Linux-6.6.20+rpt-rpi-2712-aarch64-with-glibc2.36
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Locale: 'en_GB.UTF-8'
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Entry path: 
> /usr/share/weewx/weewxd.py
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: WEEWX_ROOT: /etc/weewx
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Configuration file: 
> /etc/weewx/weewx.conf
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: User module: 
> /etc/weewx/bin/user
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station 
> type GW1000 (user.gw1000)
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
> version is 0.6.1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
> is 192.168.7.206:45000
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval 
> is 20 seconds
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
> unit is 0x1
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.wxservices: StdWXCalculate 
> will use data binding wx_binding
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Archive will use 
> data binding wx_binding
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Record generation 
> will be attempted in 'software'
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Using archive 
> interval of 300 seconds (software record generation)
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: StationRegistry: 
> Station will be registered.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: Wunderground: 
> Posting not enabled.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: PWSWeather: Data for 
> station PROOF0FHUMBOLDT will be posted
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: CWOP: Posting not 
> enabled.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: WOW: Posting not 
> enabled.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: AWEKAS: Posting not 
> enabled.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO user.mqtt: service version is 0.24
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: MQTT: No config 
> info. Skipped.
> Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: 'pyephem' detected, 
> extended almanac data is available
> Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Starting up weewx 
> 

[weewx-user] Belchertown and "database is locked" error

[Rob Cranfill]

Upgraded to 4.10 to 5.0.2, thanks to help here. But now my Belchertown is 
dying and restarting the whole shebang with a "database is locked" error. 

Ideas?



Jun 01 17:45:41 pi4 weewxd[3438608]: INFO __main__: retrying...
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Loading station 
type Vantage (weewx.drivers.vantage)
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: StdConvert target 
unit is 0x1
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.wxservices: StdWXCalculate 
will use data binding wx_binding
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Archive will use 
data binding wx_binding
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Record generation 
will be attempted in 'hardware'
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: The archive 
interval in the configuration file (300) does not match the station 
hardware interval (60).
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: Using archive 
interval of 60 seconds (specified by hardware)
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: StationRegistry: 
Station will be registered.
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: Wunderground-PWS: 
Data for station KWASEATT418 will be posted
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: PWSWeather: Data for 
station SEAWALL01 will be posted
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: CWOP: Posting not 
enabled.
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: WOW: Posting not 
enabled.
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.restx: AWEKAS: Posting not 
enabled.
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO weewx.engine: 'pyephem' detected, 
extended almanac data is available
Jun 01 17:45:41 pi4 weewxd[3438608]: INFO __main__: Starting up weewx 
version 5.0.2
Jun 01 17:45:42 pi4 weewxd[3438608]: INFO weewx.engine: Clock error is 
-0.23 seconds (positive is fast)
Jun 01 17:45:42 pi4 weewxd[3438608]: INFO weewx.engine: Using binding 
'wx_binding' to database 'weewx.sdb'
Jun 01 17:45:42 pi4 weewxd[3438608]: INFO weewx.manager: Starting backfill 
of daily summaries
Jun 01 17:45:42 pi4 weewxd[3438608]: INFO weewx.manager: Daily summaries up 
to date
Jun 01 17:45:43 pi4 weewxd[3438608]: INFO weewx.engine: Starting main 
packet loop.
Jun 01 17:45:46 pi4 weewxd[3438608]: ERROR weewx.restx: StationRegistry: 
Failed to publish record 2024-06-01 17:43:00 PDT (1717288980): HTTP Error 
429: TOO MANY REQUESTS
Jun 01 17:47:20 pi4 weewxd[3438608]: INFO weewx.engine: Main loop exiting. 
Shutting engine down.
Jun 01 17:47:20 pi4 weewxd[3438608]: INFO weewx.engine: Shutting down 
StdReport thread
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator: 
Evaluation of template 
/etc/weewx/skins/Belchertown/json/weewx_data.json.tmpl failed with 
exception ''
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
Ignoring template /etc/weewx/skins/Belchertown/json/weewx_data.json.tmpl
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
Reason: database is locked
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 Traceback (most recent call last):
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:    
 File "/usr/share/weewx/weedb/sqlite.py", line 38, in guarded_fn
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 return fn(*args, **kwargs)
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
^^^
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:    
 File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 return sqlite3.Cursor.execute(self, *args, **kwargs)
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
^
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 sqlite3.OperationalError: database is locked
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator: 
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 During handling of the above exception, another exception occurred:
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator: 
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 Traceback (most recent call last):
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:    
 File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
 unicode_string = compiled_template.respond()
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:  
  ^^^
Jun 01 17:47:32 pi4 weewxd[3438608]: ERROR weewx.cheetahgenerator:    
 File "_etc_weewx_skins_Belchertown_json_weewx_data_json_tmpl.py", line 
3368, in respond
Jun 01 17:47:32 pi4 

[weewx-user] Re: rsync: host key verification failed - problem

[Ben W.]

Greetings!
Thank you for logging these steps last year:

1. I logged as root ('sudo -i' from 'pi' account)
2. generated SSH keys ('ssh-keygen')
3. copied them to external server ('ssh-copy-id ace...@external.domain.com 
 -p 222')
4. copied /home/pi/.ssh/config to /root/.ssh/config
5. changed owner of 'config' ('chown root:root /root/.ssh/config')
6. waited for next synchronization
7. smiled because everything worked as expected :)

I'm stuck between 6 and 7 and unsure what I'm doing wrong. My logs are 
below. I've copied the public key from the 'root' user and verified on my 
paid web hosting site that they have the same key even with 'root'. I can 
SSH in, but the host is still asking for a password - only if I enter the 
password am I able to authenticate. It's as-if either side isn't seeing the 
key. I'm assuming that's my problem with rsync. The logs:

___

Jun 01 19:18:18 rpi systemd[1]: Started weewx.service - WeeWX.
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Initializing weewxd 
version 5.0.2
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Command line: 
/usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Using Python 3.11.2 
(main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Located at 
/usr/bin/python3
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Platform 
Linux-6.6.20+rpt-rpi-2712-aarch64-with-glibc2.36
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Locale: 'en_GB.UTF-8'
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Entry path: 
/usr/share/weewx/weewxd.py
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: WEEWX_ROOT: /etc/weewx
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Configuration file: 
/etc/weewx/weewx.conf
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: User module: 
/etc/weewx/bin/user
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Debug: 0
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Loading station type 
GW1000 (user.gw1000)
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayDriver: 
version is 0.6.1
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
is 192.168.7.206:45000
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval is 
20 seconds
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: GatewayService: 
version is 0.6.1
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  device address 
is 192.168.7.206:45000
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000:  poll interval is 
20 seconds
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: StdConvert target 
unit is 0x1
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.wxservices: StdWXCalculate 
will use data binding wx_binding
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Archive will use 
data binding wx_binding
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Record generation 
will be attempted in 'software'
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Using archive 
interval of 300 seconds (software record generation)
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: StationRegistry: 
Station will be registered.
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: Wunderground: Posting 
not enabled.
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: PWSWeather: Data for 
station PROOF0FHUMBOLDT will be posted
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: CWOP: Posting not 
enabled.
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: WOW: Posting not 
enabled.
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: AWEKAS: Posting not 
enabled.
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.mqtt: service version is 0.24
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.restx: MQTT: No config info. 
Skipped.
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: 'pyephem' detected, 
extended almanac data is available
Jun 01 19:18:18 rpi weewxd[200894]: INFO __main__: Starting up weewx 
version 5.0.2
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Using binding 
'wx_binding' to database 'weewx.sdb'
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.manager: Starting backfill 
of daily summaries
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.manager: Daily summaries up 
to date
Jun 01 19:18:18 rpi weewxd[200894]: INFO weewx.engine: Starting main packet 
loop.
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: Using 't_rainyear' 
for rain total
Jun 01 19:18:18 rpi weewxd[200894]: INFO user.gw1000: skipping rain 
measurement of 282.6: no last rain
Jun 01 19:18:19 rpi weewxd[200894]: INFO user.gw1000: Using 't_rainyear' 
for rain total
Jun 01 19:18:19 rpi weewxd[200894]: INFO user.gw1000: skipping rain 
measurement of 282.6: no last rain
Jun 01 19:20:19 rpi weewxd[200894]: INFO weewx.manager: Added record 
2024-06-01 19:20:00 CDT (1717287600) to database 'weewx.sdb'
Jun 01 19:20:19 rpi weewxd[200894]: INFO weewx.manager: Added record 
2024-06-01 19:20:0

Re: [weewx-user] HTML perms issue for 5.0

[Rob Cranfill]

That seems to have done it! Thanks :)

One odd thing: I had to copy the old seasons.css and seasons.js files over, 
by hand. Whatev.


On Saturday, June 1, 2024 at 1:31:47 PM UTC-7 vince wrote:

> I would "sudo chown -R weewx:weewx /var/www/html/weather" so the 
> directory and its contents are owned by the weewx user that the weewx 
> process runs as. 
>
> On Saturday, June 1, 2024 at 1:18:32 PM UTC-7 Rob Cranfill wrote:
>
>> Thanks for the idea, but that didn't fix it.
>>
>> On Saturday, June 1, 2024 at 12:42:47 PM UTC-7 Warren Gill wrote:
>>
>>> There might be files left over from pre-upgrade that are still owned by 
>>> www-data (daybarometer.png, for example). I'd just go into the 
>>> /var/www/html/weather/ directory and remove everything and let weewx build 
>>> new ones. Or, chown weewx:weewx * in the weather directory
>>>
>>>
>>> On Sat, Jun 1, 2024 at 2:22 PM Rob Cranfill  wrote:
>>>
 Hey, all!

 Just upgraded to 5.02 from 4.10.something, and now WeeWX can't output 
 to my web server directory. I'm sure it's some dumb permissions thing I'm 
 missing, but I am not seeing it.

 I get a bazillion errors like this in the log:

 Jun 01 12:16:15 pi4 weewxd[3424852]: ERROR weewx.imagegenerator: Unable 
 to save to file '/var/www/html/weather/daybarometer.png' [Errno 13] 
 Permission denied: '/var/www/html/weather/daybarometer.png':

 My new install is running as user weewx, as intended:
 rob@pi4: $ ps aux | grep wee
 weewx3424852  4.9  2.8 358576 53528 ?Ssl  11:38   1:42 
 python3 /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf

 I added that user to some groups (for my USB interface and the HTML 
 directory)
 rob@pi4: $ groups weewx
 weewx : weewx dialout www-data

 And here are the permissions along that directory path. See anything 
 untowards?
 drwxr-xr-x 12 root root 4.0K Dec  7 20:36 /var
 drwxr-xr-x 3 root root 4.0K Dec  7 20:36 /var/www/
 drwxr-xr-x 4 www-data www-data 4.0K Jun  1 10:58 /var/www/html/
 drwxrwxr-x 6 www-data www-data 4.0K Jun  1 11:25 /var/www/html/weather/

 Thanks for any suggestions. I can post more info if that will help. The 
 logs show nothing else interesting that I can see.

  /rob
  

 -- 
 You received this message because you are subscribed to the Google 
 Groups "weewx-user" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to weewx-user+...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/weewx-user/08ee001f-caa9-4982-9f13-1c2be9d3bdaan%40googlegroups.com
  
 
 .

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/e116cca9-0778-451b-aa77-e23b321b971en%40googlegroups.com.

Re: [weewx-user] Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

[Tom Keffer]

Some ideas:

1. Check the slow query log. Anything stand out?
2. Check the regular query log. Any queries that are being run repeatedly?
Or, require full database searches?
3. Divide and conquer. Disable the Cheetah generator (go to the bottom of
Season's skin.conf and remove weewx.cheetahgenerator.CheetahGenerator from
the option generator_list). Now start removing plots under
[ImageGenerator]. Is there one plot that is causing the problem?

In short, you're going to have to isolate the problem.

-tk

On Sat, Jun 1, 2024 at 12:49 PM Bartosz Francman 
wrote:

> Hi,
>
> The skin has not been modified in any way.
> In my long description of the problem ;) I forgot to write that after
> automatic upgrade from 4.10.2 to 5.02 and this method did not work, I
> uninstalled the entire weewx (sudo apt purge) and installed it fresh.
> Later, I just copied the entries from the old configuration file to the new
> one. And not replacing this file, just individual entries.
> No files were edited or changed except the configuration file.
> I realize that changing from weewx to root shouldn't increase CPU usage. I
> only marked it to describe the problem with port 80, on which the raspberry
> must listen for data from the weather station.
>
>
> Br
> Bartosz
>
>
> sobota, 1 czerwca 2024 o 18:21:13 UTC+2 vince napisał(a):
>
>> oops - thank you.  I missed that link in the long wording of the problem
>> description.   Like the new weectl debug output too :-)
>>
>> What I see is he's using the old original wview-compatible schema, so
>> that certainly leans toward the db schema issue.  Again, I'd suggest
>> temporarily switching to the old hard-coded Standard or Smartphone or
>> Mobile skin (as a test) to see if the problem goes away.
>>
>> Is there a way to instrument a test version of something under the hood
>> in order to log which (if any) xtypes are being calculated due to not being
>> found in the db ?   This issue seems to be hitting quite a few legacy
>> users.  Would some test instrumentation perhaps help folks identify what's
>> missing in their setup ?
>>
>> On Saturday, June 1, 2024 at 9:02:14 AM UTC-7 Tom Keffer wrote:
>>
>>> Vince, he posted the output of weectl debug, which includes the schema
>>> and weewx.conf.
>>>
>>> -tk
>>>
>>>
>>> On Sat, Jun 1, 2024 at 8:26 AM vince  wrote:
>>>
 If you have 6 years of data you probably are using the old
 wview-compatible schema, so this 'sounds' like you are missing db elements
 referenced in your skins.  You are also running on a pi 3b (slow) and
 mentioned you have a 2 minute archive period (a little fast) making the pi
 work harder than usual.

 As a quick test, switch to the simplest available skin (Mobile or
 Smartphone or Standard) and see if the problem goes away.  If so it is
 almost certainly missing elements in your database.

 Can you post your Seasons skin.conf, weewx.conf (with no
 usernames/passwords in it), and your db schema ?

 On Saturday, June 1, 2024 at 7:52:50 AM UTC-7 Tom Keffer wrote:

> Permission problems tend to be "all or nothing", so I doubt running as
> 'root' (as opposed to 'weewx') has anything to do with the excessive use 
> of
> the CPU.
>
> The problem is almost surely caused by the need to calculate an
> aggregate of an xtype that is not in the database (as described in this
> wiki article ).
> Because this involves many small queries to the database, using a remote
> database can really exacerbate the problem.
>
> I see that you are using the Seasons skin. Have you modified it at
> all? To check, you may want to use the stock Seasons skin and compare
> times.
>
>
>
> On Sat, Jun 1, 2024 at 6:25 AM Bartosz Francman 
> wrote:
>
>> Hi,
>> For several months I have been postponing the upgrade from version
>> 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that 
>> it
>> was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade
>> (sudo apt update & upgrade).
>> When asked about saving the configuration file, I left the old
>> version, as everything worked so far.
>> My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station
>> and a database (MariaDB 10) located on the NAS so as not to write data to
>> the memory card. This system has been operating for 6 years (the first
>> entries in the database are from 2018). So far I haven't had any 
>> problems.
>> Everything worked as it should.
>> As the weather station only works via WiFi, I had to use the
>> Interceptor controller to read data from the station. And unfortunately,
>> only port 80 could be used. It is not possible to configure the station,
>> apart from turning it on and setting access data to 3 weather services.
>> Now I will briefly describe the problems with the upg

Re: [weewx-user] HTML perms issue for 5.0

[vince]

I would "sudo chown -R weewx:weewx /var/www/html/weather" so the directory 
and its contents are owned by the weewx user that the weewx process runs 
as. 

On Saturday, June 1, 2024 at 1:18:32 PM UTC-7 Rob Cranfill wrote:

> Thanks for the idea, but that didn't fix it.
>
> On Saturday, June 1, 2024 at 12:42:47 PM UTC-7 Warren Gill wrote:
>
>> There might be files left over from pre-upgrade that are still owned by 
>> www-data (daybarometer.png, for example). I'd just go into the 
>> /var/www/html/weather/ directory and remove everything and let weewx build 
>> new ones. Or, chown weewx:weewx * in the weather directory
>>
>>
>> On Sat, Jun 1, 2024 at 2:22 PM Rob Cranfill  wrote:
>>
>>> Hey, all!
>>>
>>> Just upgraded to 5.02 from 4.10.something, and now WeeWX can't output to 
>>> my web server directory. I'm sure it's some dumb permissions thing I'm 
>>> missing, but I am not seeing it.
>>>
>>> I get a bazillion errors like this in the log:
>>>
>>> Jun 01 12:16:15 pi4 weewxd[3424852]: ERROR weewx.imagegenerator: Unable 
>>> to save to file '/var/www/html/weather/daybarometer.png' [Errno 13] 
>>> Permission denied: '/var/www/html/weather/daybarometer.png':
>>>
>>> My new install is running as user weewx, as intended:
>>> rob@pi4: $ ps aux | grep wee
>>> weewx3424852  4.9  2.8 358576 53528 ?Ssl  11:38   1:42 
>>> python3 /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
>>>
>>> I added that user to some groups (for my USB interface and the HTML 
>>> directory)
>>> rob@pi4: $ groups weewx
>>> weewx : weewx dialout www-data
>>>
>>> And here are the permissions along that directory path. See anything 
>>> untowards?
>>> drwxr-xr-x 12 root root 4.0K Dec  7 20:36 /var
>>> drwxr-xr-x 3 root root 4.0K Dec  7 20:36 /var/www/
>>> drwxr-xr-x 4 www-data www-data 4.0K Jun  1 10:58 /var/www/html/
>>> drwxrwxr-x 6 www-data www-data 4.0K Jun  1 11:25 /var/www/html/weather/
>>>
>>> Thanks for any suggestions. I can post more info if that will help. The 
>>> logs show nothing else interesting that I can see.
>>>
>>>  /rob
>>>  
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to weewx-user+...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/08ee001f-caa9-4982-9f13-1c2be9d3bdaan%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/84d2493d-eddb-43f6-8687-9bb02fdc5683n%40googlegroups.com.

Re: [weewx-user] HTML perms issue for 5.0

[Rob Cranfill]

Thanks for the idea, but that didn't fix it.

On Saturday, June 1, 2024 at 12:42:47 PM UTC-7 Warren Gill wrote:

> There might be files left over from pre-upgrade that are still owned by 
> www-data (daybarometer.png, for example). I'd just go into the 
> /var/www/html/weather/ directory and remove everything and let weewx build 
> new ones. Or, chown weewx:weewx * in the weather directory
>
>
> On Sat, Jun 1, 2024 at 2:22 PM Rob Cranfill  wrote:
>
>> Hey, all!
>>
>> Just upgraded to 5.02 from 4.10.something, and now WeeWX can't output to 
>> my web server directory. I'm sure it's some dumb permissions thing I'm 
>> missing, but I am not seeing it.
>>
>> I get a bazillion errors like this in the log:
>>
>> Jun 01 12:16:15 pi4 weewxd[3424852]: ERROR weewx.imagegenerator: Unable 
>> to save to file '/var/www/html/weather/daybarometer.png' [Errno 13] 
>> Permission denied: '/var/www/html/weather/daybarometer.png':
>>
>> My new install is running as user weewx, as intended:
>> rob@pi4: $ ps aux | grep wee
>> weewx3424852  4.9  2.8 358576 53528 ?Ssl  11:38   1:42 python3 
>> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
>>
>> I added that user to some groups (for my USB interface and the HTML 
>> directory)
>> rob@pi4: $ groups weewx
>> weewx : weewx dialout www-data
>>
>> And here are the permissions along that directory path. See anything 
>> untowards?
>> drwxr-xr-x 12 root root 4.0K Dec  7 20:36 /var
>> drwxr-xr-x 3 root root 4.0K Dec  7 20:36 /var/www/
>> drwxr-xr-x 4 www-data www-data 4.0K Jun  1 10:58 /var/www/html/
>> drwxrwxr-x 6 www-data www-data 4.0K Jun  1 11:25 /var/www/html/weather/
>>
>> Thanks for any suggestions. I can post more info if that will help. The 
>> logs show nothing else interesting that I can see.
>>
>>  /rob
>>  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/08ee001f-caa9-4982-9f13-1c2be9d3bdaan%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/c4811dfd-2187-44e7-afa3-8294cc93c109n%40googlegroups.com.

Re: [weewx-user] Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

[Bartosz Francman]

Hi,

The skin has not been modified in any way.
In my long description of the problem ;) I forgot to write that after 
automatic upgrade from 4.10.2 to 5.02 and this method did not work, I 
uninstalled the entire weewx (sudo apt purge) and installed it fresh. 
Later, I just copied the entries from the old configuration file to the new 
one. And not replacing this file, just individual entries.
No files were edited or changed except the configuration file.
I realize that changing from weewx to root shouldn't increase CPU usage. I 
only marked it to describe the problem with port 80, on which the raspberry 
must listen for data from the weather station.


Br
Bartosz


sobota, 1 czerwca 2024 o 18:21:13 UTC+2 vince napisał(a):

> oops - thank you.  I missed that link in the long wording of the problem 
> description.   Like the new weectl debug output too :-)
>
> What I see is he's using the old original wview-compatible schema, so that 
> certainly leans toward the db schema issue.  Again, I'd suggest temporarily 
> switching to the old hard-coded Standard or Smartphone or Mobile skin (as a 
> test) to see if the problem goes away.
>
> Is there a way to instrument a test version of something under the hood in 
> order to log which (if any) xtypes are being calculated due to not being 
> found in the db ?   This issue seems to be hitting quite a few legacy 
> users.  Would some test instrumentation perhaps help folks identify what's 
> missing in their setup ?
>
> On Saturday, June 1, 2024 at 9:02:14 AM UTC-7 Tom Keffer wrote:
>
>> Vince, he posted the output of weectl debug, which includes the schema 
>> and weewx.conf. 
>>
>> -tk
>>
>>
>> On Sat, Jun 1, 2024 at 8:26 AM vince  wrote:
>>
>>> If you have 6 years of data you probably are using the old 
>>> wview-compatible schema, so this 'sounds' like you are missing db elements 
>>> referenced in your skins.  You are also running on a pi 3b (slow) and 
>>> mentioned you have a 2 minute archive period (a little fast) making the pi 
>>> work harder than usual.
>>>
>>> As a quick test, switch to the simplest available skin (Mobile or 
>>> Smartphone or Standard) and see if the problem goes away.  If so it is 
>>> almost certainly missing elements in your database.
>>>
>>> Can you post your Seasons skin.conf, weewx.conf (with no 
>>> usernames/passwords in it), and your db schema ?
>>>
>>> On Saturday, June 1, 2024 at 7:52:50 AM UTC-7 Tom Keffer wrote:
>>>
 Permission problems tend to be "all or nothing", so I doubt running as 
 'root' (as opposed to 'weewx') has anything to do with the excessive use 
 of 
 the CPU.

 The problem is almost surely caused by the need to calculate an 
 aggregate of an xtype that is not in the database (as described in this 
 wiki article ). 
 Because this involves many small queries to the database, using a remote 
 database can really exacerbate the problem. 

 I see that you are using the Seasons skin. Have you modified it at all? 
 To check, you may want to use the stock Seasons skin and compare times. 



 On Sat, Jun 1, 2024 at 6:25 AM Bartosz Francman  
 wrote:

> Hi,
> For several months I have been postponing the upgrade from version 
> 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that 
> it 
> was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade 
> (sudo apt update & upgrade).
> When asked about saving the configuration file, I left the old 
> version, as everything worked so far.
> My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station 
> and a database (MariaDB 10) located on the NAS so as not to write data to 
> the memory card. This system has been operating for 6 years (the first 
> entries in the database are from 2018). So far I haven't had any 
> problems. 
> Everything worked as it should.
> As the weather station only works via WiFi, I had to use the 
> Interceptor controller to read data from the station. And unfortunately, 
> only port 80 could be used. It is not possible to configure the station, 
> apart from turning it on and setting access data to 3 weather services.
> Now I will briefly describe the problems with the upgrade.
> Port 80 cannot be used by any user other than root, so I had to change 
> the way weewx runs from the weewx user to the root user. Data began to 
> appear and the script did not end with the lack of access to port 80. I 
> thought that it was not so bad.
> However, I saw that the processes related to weewx were using 100% of 
> the load of two raspberry processors. I found information on this group 
> that it may be a problem with the lack of appropriate columns in the 
> archive table (
> https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/S0Ek9ZVaBwAJ) 
> I did everything accordi

Re: [weewx-user] HTML perms issue for 5.0

[Warren Gill]

There might be files left over from pre-upgrade that are still owned by
www-data (daybarometer.png, for example). I'd just go into the
/var/www/html/weather/ directory and remove everything and let weewx build
new ones. Or, chown weewx:weewx * in the weather directory


On Sat, Jun 1, 2024 at 2:22 PM Rob Cranfill  wrote:

> Hey, all!
>
> Just upgraded to 5.02 from 4.10.something, and now WeeWX can't output to
> my web server directory. I'm sure it's some dumb permissions thing I'm
> missing, but I am not seeing it.
>
> I get a bazillion errors like this in the log:
>
> Jun 01 12:16:15 pi4 weewxd[3424852]: ERROR weewx.imagegenerator: Unable to
> save to file '/var/www/html/weather/daybarometer.png' [Errno 13] Permission
> denied: '/var/www/html/weather/daybarometer.png':
>
> My new install is running as user weewx, as intended:
> rob@pi4: $ ps aux | grep wee
> weewx3424852  4.9  2.8 358576 53528 ?Ssl  11:38   1:42 python3
> /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
>
> I added that user to some groups (for my USB interface and the HTML
> directory)
> rob@pi4: $ groups weewx
> weewx : weewx dialout www-data
>
> And here are the permissions along that directory path. See anything
> untowards?
> drwxr-xr-x 12 root root 4.0K Dec  7 20:36 /var
> drwxr-xr-x 3 root root 4.0K Dec  7 20:36 /var/www/
> drwxr-xr-x 4 www-data www-data 4.0K Jun  1 10:58 /var/www/html/
> drwxrwxr-x 6 www-data www-data 4.0K Jun  1 11:25 /var/www/html/weather/
>
> Thanks for any suggestions. I can post more info if that will help. The
> logs show nothing else interesting that I can see.
>
>  /rob
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/08ee001f-caa9-4982-9f13-1c2be9d3bdaan%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CA%2Bz%2BvD4-O-%3DtFv%2BQ5Qu%3DT4zP-j5iAKkLcd6B-0ass3VT4-noWA%40mail.gmail.com.

[weewx-user] HTML perms issue for 5.0

[Rob Cranfill]

Hey, all!

Just upgraded to 5.02 from 4.10.something, and now WeeWX can't output to my 
web server directory. I'm sure it's some dumb permissions thing I'm 
missing, but I am not seeing it.

I get a bazillion errors like this in the log:

Jun 01 12:16:15 pi4 weewxd[3424852]: ERROR weewx.imagegenerator: Unable to 
save to file '/var/www/html/weather/daybarometer.png' [Errno 13] Permission 
denied: '/var/www/html/weather/daybarometer.png':

My new install is running as user weewx, as intended:
rob@pi4: $ ps aux | grep wee
weewx3424852  4.9  2.8 358576 53528 ?Ssl  11:38   1:42 python3 
/usr/share/weewx/weewxd.py /etc/weewx/weewx.conf

I added that user to some groups (for my USB interface and the HTML 
directory)
rob@pi4: $ groups weewx
weewx : weewx dialout www-data

And here are the permissions along that directory path. See anything 
untowards?
drwxr-xr-x 12 root root 4.0K Dec  7 20:36 /var
drwxr-xr-x 3 root root 4.0K Dec  7 20:36 /var/www/
drwxr-xr-x 4 www-data www-data 4.0K Jun  1 10:58 /var/www/html/
drwxrwxr-x 6 www-data www-data 4.0K Jun  1 11:25 /var/www/html/weather/

Thanks for any suggestions. I can post more info if that will help. The 
logs show nothing else interesting that I can see.

 /rob
 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/08ee001f-caa9-4982-9f13-1c2be9d3bdaan%40googlegroups.com.

Re: [weewx-user] Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

[vince]

oops - thank you.  I missed that link in the long wording of the problem 
description.   Like the new weectl debug output too :-)

What I see is he's using the old original wview-compatible schema, so that 
certainly leans toward the db schema issue.  Again, I'd suggest temporarily 
switching to the old hard-coded Standard or Smartphone or Mobile skin (as a 
test) to see if the problem goes away.

Is there a way to instrument a test version of something under the hood in 
order to log which (if any) xtypes are being calculated due to not being 
found in the db ?   This issue seems to be hitting quite a few legacy 
users.  Would some test instrumentation perhaps help folks identify what's 
missing in their setup ?

On Saturday, June 1, 2024 at 9:02:14 AM UTC-7 Tom Keffer wrote:

> Vince, he posted the output of weectl debug, which includes the schema and 
> weewx.conf. 
>
> -tk
>
>
> On Sat, Jun 1, 2024 at 8:26 AM vince  wrote:
>
>> If you have 6 years of data you probably are using the old 
>> wview-compatible schema, so this 'sounds' like you are missing db elements 
>> referenced in your skins.  You are also running on a pi 3b (slow) and 
>> mentioned you have a 2 minute archive period (a little fast) making the pi 
>> work harder than usual.
>>
>> As a quick test, switch to the simplest available skin (Mobile or 
>> Smartphone or Standard) and see if the problem goes away.  If so it is 
>> almost certainly missing elements in your database.
>>
>> Can you post your Seasons skin.conf, weewx.conf (with no 
>> usernames/passwords in it), and your db schema ?
>>
>> On Saturday, June 1, 2024 at 7:52:50 AM UTC-7 Tom Keffer wrote:
>>
>>> Permission problems tend to be "all or nothing", so I doubt running as 
>>> 'root' (as opposed to 'weewx') has anything to do with the excessive use of 
>>> the CPU.
>>>
>>> The problem is almost surely caused by the need to calculate an 
>>> aggregate of an xtype that is not in the database (as described in this 
>>> wiki article ). 
>>> Because this involves many small queries to the database, using a remote 
>>> database can really exacerbate the problem. 
>>>
>>> I see that you are using the Seasons skin. Have you modified it at all? 
>>> To check, you may want to use the stock Seasons skin and compare times. 
>>>
>>>
>>>
>>> On Sat, Jun 1, 2024 at 6:25 AM Bartosz Francman  
>>> wrote:
>>>
 Hi,
 For several months I have been postponing the upgrade from version 
 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that 
 it 
 was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade 
 (sudo apt update & upgrade).
 When asked about saving the configuration file, I left the old version, 
 as everything worked so far.
 My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station 
 and a database (MariaDB 10) located on the NAS so as not to write data to 
 the memory card. This system has been operating for 6 years (the first 
 entries in the database are from 2018). So far I haven't had any problems. 
 Everything worked as it should.
 As the weather station only works via WiFi, I had to use the 
 Interceptor controller to read data from the station. And unfortunately, 
 only port 80 could be used. It is not possible to configure the station, 
 apart from turning it on and setting access data to 3 weather services.
 Now I will briefly describe the problems with the upgrade.
 Port 80 cannot be used by any user other than root, so I had to change 
 the way weewx runs from the weewx user to the root user. Data began to 
 appear and the script did not end with the lack of access to port 80. I 
 thought that it was not so bad.
 However, I saw that the processes related to weewx were using 100% of 
 the load of two raspberry processors. I found information on this group 
 that it may be a problem with the lack of appropriate columns in the 
 archive table (
 https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/S0Ek9ZVaBwAJ) I 
 did everything according to this post, plus from 
 https://weewx.com/docs/5.0/utilities/weectl-database/#update-a-database 
 update the database and check it. All this took about 2 hours due to the 
 large amount of data I have in this database. By the way, apart from the 
 high CPU load, I noticed that every half a minute there is a very large 
 data read from the database. And it's related to weewx. I thought that 
 rebuilding the database would reduce the CPU load and eliminate the 
 frequent polling, but unfortunately it didn't help. I don't have any very 
 advanced visualizations, just a basic diagram.
 I left it because there is nothing else to do with this raspberry and 
 two out of four processors can be fully loaded. But unfortunately, about 
 two hours after starting the system, I received information

Re: [weewx-user] Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

[Tom Keffer]

Vince, he posted the output of weectl debug, which includes the schema and
weewx.conf.

-tk


On Sat, Jun 1, 2024 at 8:26 AM vince  wrote:

> If you have 6 years of data you probably are using the old
> wview-compatible schema, so this 'sounds' like you are missing db elements
> referenced in your skins.  You are also running on a pi 3b (slow) and
> mentioned you have a 2 minute archive period (a little fast) making the pi
> work harder than usual.
>
> As a quick test, switch to the simplest available skin (Mobile or
> Smartphone or Standard) and see if the problem goes away.  If so it is
> almost certainly missing elements in your database.
>
> Can you post your Seasons skin.conf, weewx.conf (with no
> usernames/passwords in it), and your db schema ?
>
> On Saturday, June 1, 2024 at 7:52:50 AM UTC-7 Tom Keffer wrote:
>
>> Permission problems tend to be "all or nothing", so I doubt running as
>> 'root' (as opposed to 'weewx') has anything to do with the excessive use of
>> the CPU.
>>
>> The problem is almost surely caused by the need to calculate an aggregate
>> of an xtype that is not in the database (as described in this wiki
>> article ).
>> Because this involves many small queries to the database, using a remote
>> database can really exacerbate the problem.
>>
>> I see that you are using the Seasons skin. Have you modified it at all?
>> To check, you may want to use the stock Seasons skin and compare times.
>>
>>
>>
>> On Sat, Jun 1, 2024 at 6:25 AM Bartosz Francman 
>> wrote:
>>
>>> Hi,
>>> For several months I have been postponing the upgrade from version
>>> 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that it
>>> was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade
>>> (sudo apt update & upgrade).
>>> When asked about saving the configuration file, I left the old version,
>>> as everything worked so far.
>>> My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station
>>> and a database (MariaDB 10) located on the NAS so as not to write data to
>>> the memory card. This system has been operating for 6 years (the first
>>> entries in the database are from 2018). So far I haven't had any problems.
>>> Everything worked as it should.
>>> As the weather station only works via WiFi, I had to use the Interceptor
>>> controller to read data from the station. And unfortunately, only port 80
>>> could be used. It is not possible to configure the station, apart from
>>> turning it on and setting access data to 3 weather services.
>>> Now I will briefly describe the problems with the upgrade.
>>> Port 80 cannot be used by any user other than root, so I had to change
>>> the way weewx runs from the weewx user to the root user. Data began to
>>> appear and the script did not end with the lack of access to port 80. I
>>> thought that it was not so bad.
>>> However, I saw that the processes related to weewx were using 100% of
>>> the load of two raspberry processors. I found information on this group
>>> that it may be a problem with the lack of appropriate columns in the
>>> archive table (
>>> https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/S0Ek9ZVaBwAJ) I
>>> did everything according to this post, plus from
>>> https://weewx.com/docs/5.0/utilities/weectl-database/#update-a-database
>>> update the database and check it. All this took about 2 hours due to the
>>> large amount of data I have in this database. By the way, apart from the
>>> high CPU load, I noticed that every half a minute there is a very large
>>> data read from the database. And it's related to weewx. I thought that
>>> rebuilding the database would reduce the CPU load and eliminate the
>>> frequent polling, but unfortunately it didn't help. I don't have any very
>>> advanced visualizations, just a basic diagram.
>>> I left it because there is nothing else to do with this raspberry and
>>> two out of four processors can be fully loaded. But unfortunately, about
>>> two hours after starting the system, I received information from one
>>> weather service that I was not sending data to them. I tried to access this
>>> raspberry via ssh, but it was impossible. Connection timed out. Although
>>> the raspberry was running.
>>> Only today I sat down to work on it and managed to connect to the
>>> raspberry. The last log entries after issuing the sudo systemctl status
>>> weewx command are provided below:
>>>
>>> pi@raspberry-pi:~ $ sudo systemctl status weewx
>>> × weewx.service - WeeWX
>>>  Loaded: loaded (/lib/systemd/system/weewx.service; enabled; preset:
>>> enabled)
>>>  Active: failed (Result: signal) since Sat 2024-06-01 10:36:12 CEST;
>>> 1min 2s ago
>>>Duration: 19h 10min 2.367s
>>>Docs: https://weewx.com/docs
>>> Process: 720 ExecStart=weewxd /etc/weewx/weewx.conf (code=killed,
>>> signal=KILL)
>>>Main PID: 720 (code=killed, signal=KILL)
>>> CPU: 1h 59min 21.575s
>>>
>>> maj 31 17:51:09 raspb

Re: [weewx-user] Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

[vince]

If you have 6 years of data you probably are using the old wview-compatible 
schema, so this 'sounds' like you are missing db elements referenced in 
your skins.  You are also running on a pi 3b (slow) and mentioned you have 
a 2 minute archive period (a little fast) making the pi work harder than 
usual.

As a quick test, switch to the simplest available skin (Mobile or 
Smartphone or Standard) and see if the problem goes away.  If so it is 
almost certainly missing elements in your database.

Can you post your Seasons skin.conf, weewx.conf (with no 
usernames/passwords in it), and your db schema ?

On Saturday, June 1, 2024 at 7:52:50 AM UTC-7 Tom Keffer wrote:

> Permission problems tend to be "all or nothing", so I doubt running as 
> 'root' (as opposed to 'weewx') has anything to do with the excessive use of 
> the CPU.
>
> The problem is almost surely caused by the need to calculate an aggregate 
> of an xtype that is not in the database (as described in this wiki article 
> ). Because this 
> involves many small queries to the database, using a remote database can 
> really exacerbate the problem. 
>
> I see that you are using the Seasons skin. Have you modified it at all? To 
> check, you may want to use the stock Seasons skin and compare times. 
>
>
>
> On Sat, Jun 1, 2024 at 6:25 AM Bartosz Francman  wrote:
>
>> Hi,
>> For several months I have been postponing the upgrade from version 
>> 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that it 
>> was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade 
>> (sudo apt update & upgrade).
>> When asked about saving the configuration file, I left the old version, 
>> as everything worked so far.
>> My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station and 
>> a database (MariaDB 10) located on the NAS so as not to write data to the 
>> memory card. This system has been operating for 6 years (the first entries 
>> in the database are from 2018). So far I haven't had any problems. 
>> Everything worked as it should.
>> As the weather station only works via WiFi, I had to use the Interceptor 
>> controller to read data from the station. And unfortunately, only port 80 
>> could be used. It is not possible to configure the station, apart from 
>> turning it on and setting access data to 3 weather services.
>> Now I will briefly describe the problems with the upgrade.
>> Port 80 cannot be used by any user other than root, so I had to change 
>> the way weewx runs from the weewx user to the root user. Data began to 
>> appear and the script did not end with the lack of access to port 80. I 
>> thought that it was not so bad.
>> However, I saw that the processes related to weewx were using 100% of the 
>> load of two raspberry processors. I found information on this group that it 
>> may be a problem with the lack of appropriate columns in the archive table (
>> https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/S0Ek9ZVaBwAJ) I 
>> did everything according to this post, plus from 
>> https://weewx.com/docs/5.0/utilities/weectl-database/#update-a-database 
>> update the database and check it. All this took about 2 hours due to the 
>> large amount of data I have in this database. By the way, apart from the 
>> high CPU load, I noticed that every half a minute there is a very large 
>> data read from the database. And it's related to weewx. I thought that 
>> rebuilding the database would reduce the CPU load and eliminate the 
>> frequent polling, but unfortunately it didn't help. I don't have any very 
>> advanced visualizations, just a basic diagram.
>> I left it because there is nothing else to do with this raspberry and two 
>> out of four processors can be fully loaded. But unfortunately, about two 
>> hours after starting the system, I received information from one weather 
>> service that I was not sending data to them. I tried to access this 
>> raspberry via ssh, but it was impossible. Connection timed out. Although 
>> the raspberry was running.
>> Only today I sat down to work on it and managed to connect to the 
>> raspberry. The last log entries after issuing the sudo systemctl status 
>> weewx command are provided below:
>>
>> pi@raspberry-pi:~ $ sudo systemctl status weewx
>> × weewx.service - WeeWX
>>  Loaded: loaded (/lib/systemd/system/weewx.service; enabled; preset: 
>> enabled)
>>  Active: failed (Result: signal) since Sat 2024-06-01 10:36:12 CEST; 
>> 1min 2s ago
>>Duration: 19h 10min 2.367s
>>Docs: https://weewx.com/docs
>> Process: 720 ExecStart=weewxd /etc/weewx/weewx.conf (code=killed, 
>> signal=KILL)
>>Main PID: 720 (code=killed, signal=KILL)
>> CPU: 1h 59min 21.575s
>>
>> maj 31 17:51:09 raspberry-pi weewxd[720]: INFO weewx.restx: PWSWeather: 
>> Published record 2024-05-31 17:48:00 CEST (1717170480)
>> maj 31 17:51:22 raspberry-pi weewxd[720]: INFO weewx.restx: 
>> Wunderground-RF: Pub

[weewx-user] Re: Upgrade from 4.10.2-1 to 5.0.2 - high CPU and external database load.

['michael.k...@gmx.at' via weewx-user]

Belchertown? It is known Belchertown consumes a lot of CPU time when 
certain derived values are not in the database, after a 4 => 5 upgrade. 
Check the group for solutions.

I had maybe something like that a couple weeks ago (without Belchertown), 
it might have startet after I did an apt upgrade (but no weewx upgrade), I 
had to restart the RPi4 every day, after 24h I couldn't even ssh into the 
machine any more. In the logs were messages like "database locked" and 
those "report thread still running" messages. I didn't care too much 
because I upgraded my server hardware and installed everything from scratch 
and don't have any issues any more. I had four weewxd and a lot of other 
stuff running, I never found out, what caused the issues.

Bartosz Francman schrieb am Samstag, 1. Juni 2024 um 15:25:36 UTC+2:

> Hi,
> For several months I have been postponing the upgrade from version 
> 4.10.2-1 to version 5.02. To tell you the truth, I didn't even know that it 
> was a jump from 4.x to 5.x and a few days ago I did an automatic upgrade 
> (sudo apt update & upgrade).
> When asked about saving the configuration file, I left the old version, as 
> everything worked so far.
> My system consists of a RaspberryPi 3B, a Eurochron EFWS 2900 station and 
> a database (MariaDB 10) located on the NAS so as not to write data to the 
> memory card. This system has been operating for 6 years (the first entries 
> in the database are from 2018). So far I haven't had any problems. 
> Everything worked as it should.
> As the weather station only works via WiFi, I had to use the Interceptor 
> controller to read data from the station. And unfortunately, only port 80 
> could be used. It is not possible to configure the station, apart from 
> turning it on and setting access data to 3 weather services.
> Now I will briefly describe the problems with the upgrade.
> Port 80 cannot be used by any user other than root, so I had to change the 
> way weewx runs from the weewx user to the root user. Data began to appear 
> and the script did not end with the lack of access to port 80. I thought 
> that it was not so bad.
> However, I saw that the processes related to weewx were using 100% of the 
> load of two raspberry processors. I found information on this group that it 
> may be a problem with the lack of appropriate columns in the archive table (
> https://groups.google.com/g/weewx-user/c/6rl2FIbqVp4/m/S0Ek9ZVaBwAJ) I 
> did everything according to this post, plus from 
> https://weewx.com/docs/5.0/utilities/weectl-database/#update-a-database 
> update the database and check it. All this took about 2 hours due to the 
> large amount of data I have in this database. By the way, apart from the 
> high CPU load, I noticed that every half a minute there is a very large 
> data read from the database. And it's related to weewx. I thought that 
> rebuilding the database would reduce the CPU load and eliminate the 
> frequent polling, but unfortunately it didn't help. I don't have any very 
> advanced visualizations, just a basic diagram.
> I left it because there is nothing else to do with this raspberry and two 
> out of four processors can be fully loaded. But unfortunately, about two 
> hours after starting the system, I received information from one weather 
> service that I was not sending data to them. I tried to access this 
> raspberry via ssh, but it was impossible. Connection timed out. Although 
> the raspberry was running.
> Only today I sat down to work on it and managed to connect to the 
> raspberry. The last log entries after issuing the sudo systemctl status 
> weewx command are provided below:
>
> pi@raspberry-pi:~ $ sudo systemctl status weewx
> × weewx.service - WeeWX
>  Loaded: loaded (/lib/systemd/system/weewx.service; enabled; preset: 
> enabled)
>  Active: failed (Result: signal) since Sat 2024-06-01 10:36:12 CEST; 
> 1min 2s ago
>Duration: 19h 10min 2.367s
>Docs: https://weewx.com/docs
> Process: 720 ExecStart=weewxd /etc/weewx/weewx.conf (code=killed, 
> signal=KILL)
>Main PID: 720 (code=killed, signal=KILL)
> CPU: 1h 59min 21.575s
>
> maj 31 17:51:09 raspberry-pi weewxd[720]: INFO weewx.restx: PWSWeather: 
> Published record 2024-05-31 17:48:00 CEST (1717170480)
> maj 31 17:51:22 raspberry-pi weewxd[720]: INFO weewx.restx: 
> Wunderground-RF: Published record 2024-05-31 17:48:47 CEST (1717170527)
> maj 31 17:51:28 raspberry-pi weewxd[720]: ERROR weewx.restx: WOW: Failed 
> to publish record 2024-05-31 17:48:00 CEST (1717170480): Failed upload 
> after 3 tries
> maj 31 17:51:28 raspberry-pi weewxd[720]: INFO weewx.restx: 
> Wunderground-RF: Published record 2024-05-31 17:49:19 CEST (1717170559)
> maj 31 17:51:53 raspberry-pi weewxd[720]: INFO weewx.restx: 
> Wunderground-RF: Published record 2024-05-31 17:49:51 CEST (1717170591)
> maj 31 17:52:13 raspberry-pi weewxd[720]: INFO weewx.restx: 
> Wunderground-RF: Published record 2024-05-31 17:50:23 CEST (1717170623)
> maj