Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

It turned out that this thread was only a step on the way for me. I ended 
up writing an issue (#939)  after 
all.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/ec559475-7e09-46f9-a786-ec50e619f1dbn%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[bell...@gmail.com]

Well I am pretty sure that at some point in time I needed the ‘is_null’ 
method. I’ve decided to keep my implementation because I can implement a 
more efficient/specific one. I’ve started to prepend my XType so its 
implementation is used instead of XTypeTable.  (As an aside, it feels like 
XTypeTable should always be the last XType in the list.)

On Friday 23 February 2024 at 18:00:07 UTC-5 Tom Keffer wrote:

> You should not have to implement either.
>
> The implementation of has_data runs a not_null query over the aggregation 
> period. In turn not_null either consults the database or, if it's an xtype, 
> runs get_scalar() over the aggregation period. The first non-null value 
> seen causes it to return True.
>
> Of course, that's a fairly compute intensive way of checking for non-null 
> xtype values, but it will work for any xtype. One could optimize the 
> calculation with a specialized version for your type, but so far I haven't 
> heard of a case where it's needed.
>
> -tk
>
>
> On Fri, Feb 23, 2024 at 7:48 AM Karen K  wrote:
>
>> Do I have to implement both the not_null and has_data aggregation in an 
>> XType extension?
>>
>>
>> Tom Keffer schrieb am Freitag, 23. Februar 2024 um 14:01:45 UTC+1:
>>
>> The aggregation not_null does not check first to see if the type exists. 
>> If you know the type exists, it's slightly faster. 
>>
>> In practice, I don't think it's very useful. I may remove it from the 
>> documentation.
>>
>> On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:
>>
>> What is the difference between the aggregation types has_data and 
>> not_null according to the documentation in 
>> http://weewx.com/docs/5.0/reference/aggtypes/.
>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/8abfecfb-7fe5-4e6c-aace-540ed933ea8en%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/a672e629-cd8c-4d29-b159-4be45961f8efn%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Tom Keffer]

I think you're right that the function can be simplified. Commit de738ca

.


On Sat, Feb 24, 2024 at 12:30 PM Karen K  wrote:

> I guess PR #929 is an entirely different topic.
>
> It seams to me, that the if condition in
> weewx.tags.ObservationBinder.has_data is useless, because what happens
> afterwards is almost the same in both cases.
>
> May be, the intension for that function was like that really:
>
> @property
> def has_data(self):
> db_manager = self.db_lookup(self.data_binding)
> # First see if the type exists in the database.
> if db_manager.exists(self.obs_type):
> # Yes. Is it non-null?
> val = bool(db_manager.has_data(self.obs_type, self.timespan))
> else:
> # Nope. Try the xtypes system.
> val = weewx.xtypes.has_data(self.obs_type, self.timespan,
> db_manager)
> return val
>
> See the changed line with yellow background. There could be a connection
> to the performance issues some people experience I guess. Unfortunately I
> still have no 5.0 installation working to do tests.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/0cddb249-fefb-4179-84c3-d9fff8e70bc6n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAXCtAgB48Fs4_yVPdMRXzzBQH2%3Dy5objVA6qeqoMe18A%40mail.gmail.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

I guess PR #929 is an entirely different topic.

It seams to me, that the if condition in 
weewx.tags.ObservationBinder.has_data is useless, because what happens 
afterwards is almost the same in both cases. 

May be, the intension for that function was like that really:

@property
def has_data(self):
db_manager = self.db_lookup(self.data_binding)
# First see if the type exists in the database.
if db_manager.exists(self.obs_type):
# Yes. Is it non-null?
val = bool(db_manager.has_data(self.obs_type, self.timespan))
else:
# Nope. Try the xtypes system.
val = weewx.xtypes.has_data(self.obs_type, self.timespan, 
db_manager)
return val

See the changed line with yellow background. There could be a connection to 
the performance issues some people experience I guess. Unfortunately I 
still have no 5.0 installation working to do tests. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/0cddb249-fefb-4179-84c3-d9fff8e70bc6n%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Tom Keffer]

Rich Bell brought this up in PR #929
. Go there for the discussion. Try
the "master" branch of the repository and see if it works better for you.

On Fri, Feb 23, 2024 at 11:56 PM Karen K  wrote:

> Ah.
>
> Tom Keffer schrieb am Samstag, 24. Februar 2024 um 00:00:07 UTC+1:
>
> You should not have to implement either.
>
>
> For WeeWX 5.0 people experienced that the "GTS", "yearGDD", and "seasonGDD"
> observation types (XTypes calculated out of "outTemp") were not graphed
> any more. After an explicit implementation of "not_null" the graph
> appeared again.
>
>- https://github.com/roe-dl/weewx-GTS/issues/26
>- https://github.com/roe-dl/weewx-GTS/blob/master/bin/user/GTS.py
>
> tags.ObservationBinder.has_data has changed from WeeWX 4.X to 5.0. The
> old version only looked into the database by calling 
> self.db_lookup(self.data_binding).has_data(self.obs_type,
> self.timespan). The new version is:
>
> @property
> def has_data(self):
> db_manager = self.db_lookup(self.data_binding)
> # First see if the type exists in the database.
> if db_manager.exists(self.obs_type):
> # Yes. Is it non-null?
> val = bool(weewx.xtypes.get_aggregate(self.obs_type,
> self.timespan,
>   'not_null',
> db_manager)[0])
> else:
> # Nope. Try the xtypes system.
> val = weewx.xtypes.has_data(self.obs_type, self.timespan,
> db_manager)
> return val
>
> If you look into weewx.xtypes.get_aggregate(...,'not_null',...) and
> weewx.xtypes.has_data() they do almost the same. There is a loop over
> xtypes, and for each xtype get_aggreate() is called. If I understand the
> code right, the result of the has_data property in WeeWX 5.0 is
> independent of the value of the condition of the if statement. And the
> database lookup, that is used in WeeWX 4.X, has gone at all.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/e5010568-d6cd-4297-a608-89d1dee91bf0n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEBW7OZQkXwiQPqdTGG6nPa0TXcO9%2B5Uhnw9pBHG%3Dg4t5A%40mail.gmail.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

Ah.

Tom Keffer schrieb am Samstag, 24. Februar 2024 um 00:00:07 UTC+1:

You should not have to implement either.


For WeeWX 5.0 people experienced that the "GTS", "yearGDD", and "seasonGDD" 
observation types (XTypes calculated out of "outTemp") were not graphed any 
more. After an explicit implementation of "not_null" the graph appeared 
again.

   - https://github.com/roe-dl/weewx-GTS/issues/26
   - https://github.com/roe-dl/weewx-GTS/blob/master/bin/user/GTS.py
   
tags.ObservationBinder.has_data has changed from WeeWX 4.X to 5.0. The old 
version only looked into the database by calling 
self.db_lookup(self.data_binding).has_data(self.obs_type, 
self.timespan). The new version is:

@property
def has_data(self):
db_manager = self.db_lookup(self.data_binding)
# First see if the type exists in the database.
if db_manager.exists(self.obs_type):
# Yes. Is it non-null?
val = bool(weewx.xtypes.get_aggregate(self.obs_type, 
self.timespan,
  'not_null', 
db_manager)[0])
else:
# Nope. Try the xtypes system.
val = weewx.xtypes.has_data(self.obs_type, self.timespan, 
db_manager)
return val

If you look into weewx.xtypes.get_aggregate(...,'not_null',...) and 
weewx.xtypes.has_data() they do almost the same. There is a loop over xtypes, 
and for each xtype get_aggreate() is called. If I understand the code 
right, the result of the has_data property in WeeWX 5.0 is independent of 
the value of the condition of the if statement. And the database lookup, 
that is used in WeeWX 4.X, has gone at all.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/e5010568-d6cd-4297-a608-89d1dee91bf0n%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

Ah.

Tom Keffer schrieb am Samstag, 24. Februar 2024 um 00:00:07 UTC+1:

You should not have to implement either.


For WeeWX 5.0 people experienced that the "GTS", "yearGDD", and "seasonGDD" 
observation types (XTypes calculated out of "outTemp")  were not graphed 
any more. After an explicit implementation of "not_null" the graph appeared 
again. 

https://github.com/roe-dl/weewx-GTS/issues/26

https://github.com/roe-dl/weewx-GTS/blob/master/bin/user/GTS.py




-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/270bcad8-5fce-4356-a921-33ac77d9dc42n%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Tom Keffer]

You should not have to implement either.

The implementation of has_data runs a not_null query over the aggregation
period. In turn not_null either consults the database or, if it's an xtype,
runs get_scalar() over the aggregation period. The first non-null value
seen causes it to return True.

Of course, that's a fairly compute intensive way of checking for non-null
xtype values, but it will work for any xtype. One could optimize the
calculation with a specialized version for your type, but so far I haven't
heard of a case where it's needed.

-tk


On Fri, Feb 23, 2024 at 7:48 AM Karen K  wrote:

> Do I have to implement both the not_null and has_data aggregation in an
> XType extension?
>
>
> Tom Keffer schrieb am Freitag, 23. Februar 2024 um 14:01:45 UTC+1:
>
> The aggregation not_null does not check first to see if the type exists.
> If you know the type exists, it's slightly faster.
>
> In practice, I don't think it's very useful. I may remove it from the
> documentation.
>
> On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:
>
> What is the difference between the aggregation types has_data and not_null
> according to the documentation in
> http://weewx.com/docs/5.0/reference/aggtypes/.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/8abfecfb-7fe5-4e6c-aace-540ed933ea8en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAGi1sLuP_23WvoXMc3F1eusA1LK45OMVCtgxCRq39rPQ%40mail.gmail.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[bell...@gmail.com]

I found that you only have to implement ‘not_null’. I believe this is 
because the ‘has_data’ tag (tags.py line 389) enters the xtype system with 
an aggregate of ‘has_data’. This then looks for the aggregate of ‘not_null’ 
(xtypes.py line 156).
Looking forward to hearing if my conclusion was correct.

On Friday 23 February 2024 at 10:48:51 UTC-5 Karen K wrote:

> Do I have to implement both the not_null and has_data aggregation in an 
> XType extension?
>
>
> Tom Keffer schrieb am Freitag, 23. Februar 2024 um 14:01:45 UTC+1:
>
> The aggregation not_null does not check first to see if the type exists. 
> If you know the type exists, it's slightly faster. 
>
> In practice, I don't think it's very useful. I may remove it from the 
> documentation.
>
> On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:
>
> What is the difference between the aggregation types has_data and not_null 
> according to the documentation in 
> http://weewx.com/docs/5.0/reference/aggtypes/.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/da719dec-cf71-4ffb-8d0e-a7cb1bdd7d26n%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

Do I have to implement both the not_null and has_data aggregation in an 
XType extension?


Tom Keffer schrieb am Freitag, 23. Februar 2024 um 14:01:45 UTC+1:

The aggregation not_null does not check first to see if the type exists. If 
you know the type exists, it's slightly faster. 

In practice, I don't think it's very useful. I may remove it from the 
documentation.

On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:

What is the difference between the aggregation types has_data and not_null 
according to the documentation in 
http://weewx.com/docs/5.0/reference/aggtypes/.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8abfecfb-7fe5-4e6c-aace-540ed933ea8en%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

Thank you.

Tom Keffer schrieb am Freitag, 23. Februar 2024 um 14:01:45 UTC+1:

> The aggregation not_null does not check first to see if the type exists. 
> If you know the type exists, it's slightly faster. 
>
> In practice, I don't think it's very useful. I may remove it from the 
> documentation.
>
> On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:
>
>> What is the difference between the aggregation types has_data and 
>> not_null according to the documentation in 
>> http://weewx.com/docs/5.0/reference/aggtypes/. 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/3eef09c7-c2fe-49a2-9e89-f1957c731a54n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/92bba5f3-93ed-4d16-963e-06e520c606c1n%40googlegroups.com.

Re: [weewx-user] Difference between aggregation types has_data and not_null

[Tom Keffer]

The aggregation not_null does not check first to see if the type exists. If
you know the type exists, it's slightly faster.

In practice, I don't think it's very useful. I may remove it from the
documentation.

On Fri, Feb 23, 2024 at 3:48 AM Karen K  wrote:

> What is the difference between the aggregation types has_data and not_null
> according to the documentation in
> http://weewx.com/docs/5.0/reference/aggtypes/.
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/3eef09c7-c2fe-49a2-9e89-f1957c731a54n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zECeyc90n6f6O1KbmwYTExvOr0NaDvZcNYRtLe-9m7qL4Q%40mail.gmail.com.

[weewx-user] Difference between aggregation types has_data and not_null

[Karen K]

What is the difference between the aggregation types has_data and not_null 
according to the documentation 
in http://weewx.com/docs/5.0/reference/aggtypes/.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3eef09c7-c2fe-49a2-9e89-f1957c731a54n%40googlegroups.com.