Re: [weewx-user] How to remove spikes from data?

I have only cleaned the database in post-processing and then replaced it. On Tue, Oct 12, 2021, 5:04 PM HRM Resident wrote: > Regardless of whether it’s a median filter, a Kalman filter or > anything else, integrating it into weewx doesn’t seem trivial to me. This > is something Tom and th

Re: [weewx-user] How to remove spikes from data?

We've sort of been around this buoy before. Issue #3 and PR #228 refer, with the last activity seemingly closure of PR #228 back in 2018. I suspect that if someone wants to take on the work there's probably no r

Re: [weewx-user] How to remove spikes from data?

There's been an example alarm.py in weewx for over a decade. Search for "weewx alarm" for some old threads. On Tuesday, October 12, 2021 at 5:04:20 PM UTC-7 WindnFog wrote: > Lastly, I wonder if there’s already a way to trigger an alarm. That’s a > feature that I haven’t researched, if it do

Re: [weewx-user] How to remove spikes from data?

Regardless of whether it’s a median filter, a Kalman filter or anything else, integrating it into weewx doesn’t seem trivial to me. This is something Tom and the other developers would need to consider and decide the best way to proceed. Also, how many users would take advantage of it. T

Re: [weewx-user] How to remove spikes from data?

It does seem like an ideal use-case for a Kalman filter. https://medium.com/@jaems33/understanding-kalman-filters-with-python-2310e87b8f48 On Wed, 13 Oct 2021 at 09:54, Richard Horobin wrote: > I accept that you want to remove spikes. > > Do you think this process could be used to detect "warni

Re: [weewx-user] How to remove spikes from data?

I accept that you want to remove spikes. Do you think this process could be used to detect "warning" levels? eg tomato plants are not frost-hardy at all, so we need a warning when the temperature goes to 5C. Many other processes require warnings at plus or minus 2.5 standard deviations, as use

Re: [weewx-user] How to remove spikes from data?

FWIW, here's how I'm doing it in C on an Arduino to de-spike/smooth temperature and humidity: #define buff_size 7  // Must be an odd number.  Should be greater than 5.  7 works well.   >snip< void bubble_sort(float sort_array[], int n)   {   int i, j;   float  temp;   for (i = 0 ; i <

Re: [weewx-user] How to remove spikes from data?

This would be out of my league to develop in Python. In my Fortran and Pascal programming days of data acquisition, we routinely used 5,7, or 9 point median filters to smooth and de-spike oceanographic data. I would think this might be a valuable addition to weewx. It's not a lot of code (bub

Re: [weewx-user] How to remove spikes from data?

Here is a description of my adventures cleaning up the database http://www.adelieresources.com//2019/01/conversion-from-wview-to-weewx-weather-station-software/ On Monday, October 4, 2021 at 6:30:16 AM UTC-7 wfs...@gmail.com wrote: > Thanks Tom! I love Weewx and get a kick out of playing with th

Re: [weewx-user] How to remove spikes from data?

Thanks Tom! I love Weewx and get a kick out of playing with the data. My biggest variance here in Springfield Illinois was 4.4 degrees. On 4/21/21 at 12:15pm i went from 49.1 to 56.3 then fell back to 54.7. No rain that day, just clouds and sun I guess. Breezy. Walt On Sunday, October 3, 20

Re: [weewx-user] How to remove spikes from data?

Wow! Some serious SQL fu! Very nice. On Sun, Oct 3, 2021 at 12:20 PM wfs...@gmail.com wrote: > Here's an updated query that does better checking and it can delete the > rows or update the temperatures. Test it out on a copy of your database > first. This works for sqlite. Don't know about ot

Re: [weewx-user] How to remove spikes from data?

Here's an updated query that does better checking and it can delete the rows or update the temperatures. Test it out on a copy of your database first. This works for sqlite. Don't know about others. Walt On Sunday, October 3, 2021 at 6:48:23 AM UTC-5 anc...@gmail.com wrote: > Exactly. Is th

Re: [weewx-user] How to remove spikes from data?

Exactly. Is there a way to consider only cases such as +0, -10, +0 (i.e. spikes) and not those like +0, -6. -8 ? Meanwhile I thank you for your precious help. Andrea Il giorno giovedì 30 settembre 2021 alle 23:03:35 UTC+2 wfs...@gmail.com ha scritto: > Just a reminder, this delete query will

Re: [weewx-user] How to remove spikes from data?

Just a reminder, this delete query will delete all situations where the criteria is met. Some of these situations are legit, temps can change rapidly. You want to make sure it's going to delete only the records you want before running it. There are probably some enhancements that can be made

Re: [weewx-user] How to remove spikes from data?

Well, make sure you back up your database before you try to delete anything in case this goes wrong. I would 1. Stop weewx 2. Make a copy of the database 3. Execute the delete SQL 4. Run the original query again to see if things are OK 5. Rebuild weewx dailies 6. Start weewx Hopefully I'm not f

Re: [weewx-user] How to remove spikes from data?

Thank you Walt, this script works fine! in fact it highlighted that spike in my graph: DT datetimetemp temp_prior temp_next varianced_back d_forw --- -- -

Re: [weewx-user] How to remove spikes from data?

Here's a query using the "join a table to itself" method if you don't have the LAG function. It prints observations that are outside the average of the surrounding observations by 2+ degrees. I get about 30 observations this year and I think it's legit. Although I don't get spikes in my data,

Re: [weewx-user] How to remove spikes from data?

Yes, it's possible, but it's not a simple SELECT statement. For example, this query will return all rows where the temperature difference between adjacent rows is greater than 1 degree: *SELECT g.* FROM (SELECT dateTime, datetime(dateTime,'unixepoch','localtime'), LAG(outTemp) OVER (ORDER BY dat

[weewx-user] How to remove spikes from data?

Hi there, I need to remove some temperature spikes from my weewx.sdb database. I know this has been already asked many times, but my case is different. I need to delete spikes in which the values are climatically acceptable: so, it is not sufficient to give (temperature is intended in °C) echo