Re: [weewx-user] Handling scarce external data

[Pila]

This possibly returns me to my first try. Why the line_gap_fraction has no 
effect? It describes the problem perfectly. I have a gap in the data. But 
neither data points are connected nor even shown when they are in the gap..

If there is a time gap in the data, the option line_gap_fraction controls 
how line plots will be drawn.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3159b06e-7d5a-4c50-b162-fc6e85030ab7%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Pila]

Precisely what I found out: there are no (soilTemp2 = '' ) records, records 
are either: (soilTemp2 IS NULL) or (soilTemp2 > 1). There is no third 
option.

But, what stil bugs me: Plot does not acknowledge in any way (dot, segment 
nor anything) sporadic data in soilTemp2 field. If data is in every 5 
minutes spaced records - all is well. As soon as one record has an empty 
soilTemp2 field, plot breaks. It does not display dot alone or segment 
alone. Nothing. Plot resumes only when both neighbour records have data in 
the field soilTemp2.

But OK, I can try this variants now :) Thanks.



-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8dd45854-b84c-494e-8294-e9374cd50519%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Tom Keffer]

Sorry. My fault for not thinking this through. It will still end up being a
null value in the database, so when it's read back in, it takes on the
value of None.

I think you're back to either a separate database, or using large
aggregation intervals and using aggregation type ".last".

-tk


On Wed, May 20, 2020 at 3:48 AM Pila  wrote:

> If a line has an embedded value of None, it is considered to be two
> separate line segments, regardless of the setting of gap_fraction. So, I
> would suggest leaving the data value out entirely, rather than setting it
> to None.
>>
>>
>> Does that make sense?
>>
>>
> I believe I got it! I think my problem is with using the pond.py to read
> the data and how it reacts when the field is empty. If I understand well, a
> command:
>
> event.record['leafTemp1'] = float(value[0])
>
> will add the data to the database even if nothing is there to be add.
> Instead of simply skipping the add operation, it will add empty data
> represented by None. None breaks line in plot. If the above line would not
> execute, nothing would be placed in the database and the line would not be
> broken.
>
> Now I save something in each pass: real data or "Hey, this is empty!".
> Instead, I want to save the data only once per hour and nothing in between!
>
> I do not know Python but am a programmer. I expected this in pond.py
> should fix the issue:
> if value[0]:
>event.record['leafTemp1'] = float(value[0])
>
> I tried this but it seems to not work. But, if I am on the right track, I
> may have been to quick. To test, I can clear the fields between
> measurements.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/1e3e3177-4e1c-4e62-a51c-4ba746930079%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAO7YppjED_1iznvO4Re7K13fuOhbF2XVyU8Q2w6sW3OQ%40mail.gmail.com.

Re: [weewx-user] Handling scarce external data

[Pila]

If a line has an embedded value of None, it is considered to be two 
separate line segments, regardless of the setting of gap_fraction. So, I 
would suggest leaving the data value out entirely, rather than setting it 
to None.
>
>
> Does that make sense?
>
>
I believe I got it! I think my problem is with using the pond.py to read 
the data and how it reacts when the field is empty. If I understand well, a 
command:

event.record['leafTemp1'] = float(value[0])

will add the data to the database even if nothing is there to be add. 
Instead of simply skipping the add operation, it will add empty data 
represented by None. None breaks line in plot. If the above line would not 
execute, nothing would be placed in the database and the line would not be 
broken.

Now I save something in each pass: real data or "Hey, this is empty!". 
Instead, I want to save the data only once per hour and nothing in between!

I do not know Python but am a programmer. I expected this in pond.py should 
fix the issue:
if value[0]: 
   event.record['leafTemp1'] = float(value[0])

I tried this but it seems to not work. But, if I am on the right track, I 
may have been to quick. To test, I can clear the fields between 
measurements.

  

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/1e3e3177-4e1c-4e62-a51c-4ba746930079%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Tom Keffer]

If a line has an embedded value of None, it is considered to be two
separate line segments, regardless of the setting of gap_fraction. So, I
would suggest leaving the data value out entirely, rather than setting it
to None.

Does that make sense?

-tk

On Tue, May 19, 2020 at 2:48 PM Pila  wrote:

> Plot will not draw line when data points are one hour apart. Only when
> data points are 5 minutes apart, line will be plotted.
>
> But, since I am unable to fix python breaking when a field is missing, I
> can not continue.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/5b63ac31-f321-430c-8799-4b122c136b2a%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEBZ%3DHJ%3DWO6tr1%3DtfS0wrNwT8JJ4ELU_2xqZNOEsXS0QPA%40mail.gmail.com.

Re: [weewx-user] Handling scarce external data

[Pila]

Plot will not draw line when data points are one hour apart. Only when data 
points are 5 minutes apart, line will be plotted.

But, since I am unable to fix python breaking when a field is missing, I 
can not continue.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/5b63ac31-f321-430c-8799-4b122c136b2a%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Tom Keffer]

I'm not following, but in the hope that we get lucky with a random answer,
in WeeWX, if a data value is not available, maybe because the sensor is
offline, or can't produce a value, you set the value to None. If the data
value just plain doesn't exist, maybe because you don't own that sensor,
you leave it out of the record completely.

-tk

On Tue, May 19, 2020 at 12:22 PM Pila  wrote:

> Not working. I have two problems.
>
> 1) First location has this sporadic (hourly) temp as the last temp it is
> reading from pond.txt. When the field is empty, it logs "pond: cannot read
> value: could not convert string to float:." as defined in pond.py. When
> there is a temp, it is properly added to the database. If it matters, new
> sporadic data is saved at :15 minutes each hour.
>
> The plot is ignoring all the data saved at one hour interval. Plotting
> stopped with the last datappint saved at 5 min interval. Last plotted
> datapoint was at 13:15. All further hourly datapoints starting from 14:15
> to 20:15 inclusive are not ploted. After I reverted to the old way, from
> 20:30 data is plotted again from that point on.
>
> That particular plot displays 3 different temperatures: other 2 are being
> saved at 5 min intervals.
>
> 2) Second location apparently breaks after the empty field, when I save
> the hourly data into the second position at the pond.txt. like ",24.1"
> without quotes. Pond.py never loads the number after the empty field and
> the comma to the database. As soon as I reverted to saving both temps, both
> fields are being read correctly.
>
> What to save as an empty field so that pond.py does not break further
> reading after the empty field? There is no data, logically, I shouldn't
> save anything there. Seems like Python problem? I do not know Python.
> Should I read data differently?
>
> I am saving data into pond.txt like this:
> Location 1:
> 23.4,21.8,22.4,23.3,18.0,17.4
> Location 2:
> 18,24.1
>
> Reading it with code modified to read multiple comma separated fields:
>
> with open(self.filename) as f:
>line = f.readline()
>value = line.split(',')
>syslog.syslog(syslog.LOG_DEBUG, "pond: found value of %s" % value)
> event.record['leafTemp1'] = float(value[0])
> event.record['extraTemp2'] = float(value[1])
> ...
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/ac9eabf1-8170-43de-9958-090d8f537c1a%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLkJisKh4ZeT%2Bt6D6%2BS_nQqfNsmh3n-gJ8YLSvzEK1fg%40mail.gmail.com.

Re: [weewx-user] Handling scarce external data

[Pila]

Not working. I have two problems.

1) First location has this sporadic (hourly) temp as the last temp it is 
reading from pond.txt. When the field is empty, it logs "pond: cannot read 
value: could not convert string to float:." as defined in pond.py. When 
there is a temp, it is properly added to the database. If it matters, new 
sporadic data is saved at :15 minutes each hour. 

The plot is ignoring all the data saved at one hour interval. Plotting 
stopped with the last datappint saved at 5 min interval. Last plotted 
datapoint was at 13:15. All further hourly datapoints starting from 14:15 
to 20:15 inclusive are not ploted. After I reverted to the old way, from 
20:30 data is plotted again from that point on.

That particular plot displays 3 different temperatures: other 2 are being 
saved at 5 min intervals.

2) Second location apparently breaks after the empty field, when I save the 
hourly data into the second position at the pond.txt. like ",24.1" without 
quotes. Pond.py never loads the number after the empty field and the comma 
to the database. As soon as I reverted to saving both temps, both fields 
are being read correctly.

What to save as an empty field so that pond.py does not break further 
reading after the empty field? There is no data, logically, I shouldn't 
save anything there. Seems like Python problem? I do not know Python. 
Should I read data differently?

I am saving data into pond.txt like this:
Location 1:
23.4,21.8,22.4,23.3,18.0,17.4
Location 2:
18,24.1

Reading it with code modified to read multiple comma separated fields:

with open(self.filename) as f:
   line = f.readline()
   value = line.split(',')
   syslog.syslog(syslog.LOG_DEBUG, "pond: found value of %s" % value)
event.record['leafTemp1'] = float(value[0])
event.record['extraTemp2'] = float(value[1])
...



-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/ac9eabf1-8170-43de-9958-090d8f537c1a%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Pila]

I misspoke :( I was pulling the data every hour but it was kept available 
to WeeWx to read it for the entire hour, not just once per observation. I 
left it that way as I could not fix the plot gaps.

I adjusted for both sporadic observations to be available to WeeWx only 
once. WeeWx saves all the other data every 5 minutes. But now: graphs do 
not even show this lone data points. One is the temperature observation 
every hour (may be several hours) and the second is the sea temperature 
with 3 observations per day. Two different locations, separate computers 
and installations, no new data plots for these any of these observations. 
Everything else works as normal. Graphs are updated regularly.

My program places the data when it is available to pond.txt (at say 13th 
min). Leaves it to be read by WeeWx at 15th min and then my program removes 
the data at 18th min. The index page shows correctly the last existing 
observations. Hence, I believe data is correctly added into the database. 

As for pot: I am using line_gap_fraction. Docs are very brief. it is 
written:

... No line_gap_fraction specified ... to connect data points. The only 
thing I can do is to add:

line_gap_fraction = 0

and it seems to be ignored. No data points and nothing connected since I 
changed availability of the data from permanent to single observation only 
when the new one exists.

Another related question arose. WeeWx reads my data using pond.py. How, 
that I have started serving it empty fields, it logs a line:

pond: cannot read value: could not convert string to float

Data is prepared for pond.py and saved to pond.txt like this:

18,24.4 (both temps available)

or

,24.4   (no sea, only air)

or just a , (only comma in the pond.txt) when no data is to be read by the 
WeeWx.

Is that a problem? They are not logged as errors, but observation of a 
fact: no data to read.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/e84214d2-f8da-476d-89f3-e677ac07d644%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Pila]

2. Use aggregate type 'last'. It will return the last non-null value. For 
example,
>
>
> The last measured sea temperature is $day.seaTemperature.last at 
> $day.seaTemperature.lasttime
>
> You don't want to use $latest.seaTemperature because that would give you 
> seaTemperature in the last record of the database, which is likely to be 
> null.
> pond.txt for reading 60 minutes and remove it at 9:18. No data is 
> available for reading until new data is read
>
> Thanks! I am glad to confirm .last does perfectly what I mistakenly tried 
to achieve with latest. 

I have adjusted my program to provide the sea temp only when new 
measurement is available, so by tomorrow, I should have few separate rare 
data points to play with.

Doing that, I recalled I am already adding hourly temperature at this other 
location, I added it to the plot. Plot connects hourly temp points well. I 
just do not like square form of hourly data so I am off looking how to make 
lines connecting  data points round :)

Progress is made :) Many thanks for the nudge in the right direction!

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/30e8365b-8fcd-4fc6-8263-835807b496e1%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Pila]

To make things shorter: OK, thanks, I will try but this is a slow thing to 
test :))

This sea temp is not important and I would avoid building anything separate 
for it. It is being displayed together with external temp. As I am pulling 
it now fir days, I can see there are 3 values every day: 8:00, 14:00 and 
15:00 every day.

I am pulling this data from our national Weather service and extract 
temperatures. Then save them to be read by the pond.py. Interval for saving 
data is 5 minutes for everything.

Last night, I started another instance of the same problem. I am trying to 
find a better position for my external weather sensors. So, I keep one at 
its usual place were it is affected by the building (too high if sunny). 
Second sensor I moved to potentially better place. For a reference, I am 
pulling official temperature as I dowith the sea above, but I get it only 
once per hour with 13 minute delay from the full hour. 

So, If I repeat saving this same temp for entire hour (as I do now) and 
other 2 sensors I get and save new data every 5 minutes, this one plot line 
gets quite jagged. It would be better if I can save data as I get it - once 
per hour) and connect points with smooth line. But, since I did not manage 
to do it with the sea temp, I did not even try it with this hourly temp.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3d3f4e0b-863a-46a0-9fa4-76ad3a7bff05%40googlegroups.com.

Re: [weewx-user] Handling scarce external data

[Tom Keffer]

Interesting problem.

1. You should archive only what you observed. That is, when a datum is
observed, it should be put in the database. Then you're done with it ---
the value should not persist in the database.

That means your strategy 1a is best --- you put in a value for temperature
when it was observed. Otherwise, it's null.

You didn't say what software setup you have, but I can see a couple of
options. One would be include a field for sea-temperature (call is '
seaTemperature') in your otherwise weather-oriented database schema. The
archive interval would be 5 minutes or so, and the archive record, and
database, would be mostly filled with weather data. Most values of
seaTemperature would be null, but an occasional row would have a non-null
value. In this case, seaTemperature would come from a WeeWX service.

Another option is a weewx instance dedicated to sea temperature. No weather
data. Its "LOOP" packets would arrive only rarely,  once every few hours,
so most of the time this instance would be blocking on its driver, waiting
for that rare packet. Its archive interval would be pretty long too, maybe
3 to 6 hours. The resultant database would have very few entries, but most
would be non-null. In this case, you'd write a driver for seaTemperature.

Either approach would work.

2. Use aggregate type 'last'. It will return the last non-null value. For
example,

The last measured sea temperature is $day.seaTemperature.last at
$day.seaTemperature.lasttime

You don't want to use $latest.seaTemperature because that would give you
seaTemperature in the last record of the database, which is likely to be
null.

-tk


On Sun, May 10, 2020 at 2:33 AM Pila  wrote:

> I have added one measurement (sea temperature) to WeeWx, which I get
> probably only 3 times a day. How to handle such scarce data source? As
> there are multiple variables which should be balanced here, I would
> appreciate pointers. Small amount of data points makes for annoyingly slow
> testing.
>
> Theoretical maximum is 6 measurements from 7:00 to 17:00 but the frequency
> depends on the location. Temperature will vary mostly 1 deg C between
> measurements, rarely 2 degs. The number of measurements may vary. I get new
> data when it is available and my system knows know this is a new
> measurement. Currently, measurements are made at 8:00, 14:00 and 15:00. I
> have two questions.
>
> 1) I would like normal day/week/month graphs with all points connected.
> Data from my weather station is saved once in 5 minutes. I use example set
> by pond.py to read data as extraTemp1. I see 3 ways to supply the data to
> pond.py.
>
> a) Save the data and have it available for only one reading. E.g. save it
> at 8:13, it is read by WeeWx at 8:15 and removed from pond.txt at 8:18.
> Until new data arrives, nothing is to be read by WeeWx into this field. I
> just turned this mode on to test it again.
>
> b) I could have one reading available for one hour (or some other
> duration). 8:00 reading is being received by my system at 8:18. I can leave
> it available at pond.txt for reading 60 minutes and remove it at 9:18. No
> data is available for reading until new data is read
>
> c) Let the data be available until replaced with the new data. This I
> tried last day and it makes for a very jagged and wrong plot. It keeps the
> last temp from yesterday (the highest) overnight and it drops suddenly at
> 8:00 the next morning to the lowest temperature.
>
> Logically, I should collect data using a) and have it plotted with line
> connected. I tried it briefly with line_gap_fraction = 0 to connect my
> scarce data points but I still had breaks.
>
> 2) on the index page where the current values are, I should always see the
> last standing value. Longest interval with no data is between 15:00 and
> 8:00 the next day. This is related to above methods a) and b). In the
> skin's index.html.tmpl I tried replacing $current.extraTemp1 with
> $latest.extraTemp1 but I was getting N/A.
>
> I limit yscale to something like 15,25,1 to see the trends better.
>
> Maybe I was to quick in testing a) and something did not refresh or
> regenerate. Since multiple features need to be balanced, I may have failed
> to see the desired result using method a) which I am retesting again.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/a0cb5a23-f0dd-494f-99cf-6f0a69bdaddc%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to