[weewx-user] Re: fileparse polling interval

[jf42]

Hi Bill,

in fact I have such solution. The computer doing the quality check dumps 
the checked data in highest resolution to the disk
and additionally publishes it to a MQTT server, where other clients can 
access it. 
I have a Vantage Pro2 station running with weewx at home and the idea was, 
to use weewx for nice looking visualization 
of the weather data, current as well as historical, for the lidar station 
too. 
Anyway, if time allows I will upgrade to weewx v4 and see how fast it will 
be.

Thanks a lot,
Jens

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/ca6cbd8f-a2f3-40cc-8e9f-14a2642ddd26%40googlegroups.com.

Re: [weewx-user] Re: fileparse polling interval

[Bill Burton]

Hello,

On Tue, Apr 21, 2020 at 8:53 AM jf42  wrote:

> My station is a german "MWS 55V", which is read out by a Python program,
> writing the data to a tcp port. Another computer
> collects the data, does a quality check, many other stuff, and writes a
> file. Weewx uses the file parser to get the data.
> The station provides the data every 2 secs, the file resolution is about 5
> secs. We use the data for a lidar instrument
> (light radar) for weather monitoring during atmospheric measurements.
> Thus: the faster the better ;-)
> That's for the background of my question.
>

So it sounds like you are using the WeeWX report generation or archive
records creation to send data to the lidar?

Normally, the archive interval is configured to match reporting
requirements that will work for a day/week/month/year which is why the
default is 300 seconds/5 minutes. Also as some weather stations send data
less frequently, this gives time to collect most if not all the data of
each type. The archive interval is a different requirement than how fast a
WeeWX driver receives data.

Since you are trying to get data to the lidar as fast as possible, have you
considered using MQTT to transfer data? The message based publish-subscribe
model may be more suitable for event based data transfer than polling for
files. For instance:

   - The program that writes the data file after quality control could
   instead publish it to an MQTT topic
   
   .
   - WeeWX would be configured to use the MQTT Subscribe Driver
    and receive these
   messages as loop packets. At the appropriate time, WeeWX would write the
   archive record and generate the reports.
   - You would implement another (Python) program that subscribes to the
   same MQTT topic
   
   and sends that information to the lidar as fast as data is received.

With this approach, the lidar would receive the data almost immediately
after the QC machine publishes it without going through the WeeWX archival
and report generation process. This way, the archive interval could be
configured at the right amount to meet your reporting requirements would
likely be much less frequent then how often data is received. However, if
your requirement is that all data sent to the lidar must be logged, then
you either have to log it elsewhere or use a short archive interval.
However, the MQTT driver can be configured to publish archive data which
should make it available more quickly than other methods that provide the
archive data.
The page, How to setup your own MQTT Broker

 is a nice
overview on setting up MQTT to work with WeeWX even if you don't use the
Belchertown skin. However, the MQTT extension for WeeWX mentioned in that
post is used to *publish* loop packets after WeeWX receives them, it cannot
act as a driver to *subscribe* to an MQTT topic for receiving data.

Hope this helps,
-Bill

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAKKfUm9PjshLTTbB_BCwo2m0Vm5cHfkgxLHwPUf1StcbG8ch1w%40mail.gmail.com.

Re: [weewx-user] Re: fileparse polling interval

[Thomas Keffer]

Sounds like an interesting project.

On Tue, Apr 21, 2020 at 5:53 AM jf42  wrote:

> My station is a german "MWS 55V", which is read out by a Python program,
> writing the data to a tcp port. Another computer
> collects the data, does a quality check, many other stuff, and writes a
> file. Weewx uses the file parser to get the data.
> The station provides the data every 2 secs, the file resolution is about 5
> secs. We use the data for a lidar instrument
> (light radar) for weather monitoring during atmospheric measurements.
> Thus: the faster the better ;-)
> That's for the background of my question.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/beae49e6-f7d2-472a-8a6b-efcab5d9fc0d%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAs1wn17cVp3ZSM4ctqytjHg5Lzp-hhg%2BXepS%3DkPWuRcw%40mail.gmail.com.

[weewx-user] Re: fileparse polling interval

[jf42]

My station is a german "MWS 55V", which is read out by a Python program, 
writing the data to a tcp port. Another computer
collects the data, does a quality check, many other stuff, and writes a 
file. Weewx uses the file parser to get the data.
The station provides the data every 2 secs, the file resolution is about 5 
secs. We use the data for a lidar instrument
(light radar) for weather monitoring during atmospheric measurements. Thus: 
the faster the better ;-)
That's for the background of my question.


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/beae49e6-f7d2-472a-8a6b-efcab5d9fc0d%40googlegroups.com.

Re: [weewx-user] Re: fileparse polling interval

[Thomas Keffer]

In theory, yes, you will be able to use an archive interval of less than a
minute. But, some weather stations emit data only every 30-60 seconds or so
and, even then, not all data.

On Tue, Apr 21, 2020 at 5:01 AM jf42  wrote:

> Thanks for the info.
>
> Seems that with Version 4 archive intervals less than 60 secs would be
> possible.
> I guess, my computer is fast enough to handle the neccessary tasks ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/dcc3a091-dcd0-4934-9250-6e3f62cfa7ba%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAeD_y7sas8w4Sk9Dd-HB%3DsHkMjPPZGKWZDspxW_c0Vjw%40mail.gmail.com.

[weewx-user] Re: fileparse polling interval

[jf42]

Thanks for the info.

Seems that with Version 4 archive intervals less than 60 secs would be 
possible.
I guess, my computer is fast enough to handle the neccessary tasks ...

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/dcc3a091-dcd0-4934-9250-6e3f62cfa7ba%40googlegroups.com.

Re: [weewx-user] Re: fileparse polling interval

[Thomas Keffer]

In V3.x, the option archive_interval must divide evenly into an hour.
Version 4.X will remove that restriction. See Issue #469
.

-tk

On Mon, Apr 20, 2020 at 6:58 AM jf42  wrote:

> Addition:
>
> I found my mistake: archive_interval was 60 in weewx.conf ... But:
>
> Values below 60 sec cause a crash of the software.
>
> Any ideas for that?
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/6c8e638d-7aa5-4ae9-9a82-bcad43dcd6e7%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEAbvqteK2nHOb85w7nZmKQhdpJPonDKJDurmt7Dvfz%2BMA%40mail.gmail.com.

[weewx-user] Re: fileparse polling interval

[Andrew Milner]

If you have set archive interval too small there is probably too little 
time to 
a) store archive record
b) generate HTML pages
c) generate NOAA reports
d) do any uploads

etc etc

Normally the loop data (obtained at small time intervals) is NOT written to 
the dstabase, or used to trigger reports etc - the loop data is just 
accumulated until the archive interval and then the archive record is 
written to the database.

So as to what are you doing wrong, my response would be to suggest you are 
attempting to do something with weewx which it is not designed to do ,



On Monday, 20 April 2020 16:58:02 UTC+3, jf42 wrote:
>
> Addition:
>
> I found my mistake: archive_interval was 60 in weewx.conf ... But:
>
> Values below 60 sec cause a crash of the software.
>
> Any ideas for that?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/024faf8e-8de1-458e-973e-af5f9e71fa8f%40googlegroups.com.

[weewx-user] Re: fileparse polling interval

[jf42]

Addition:

I found my mistake: archive_interval was 60 in weewx.conf ... But:

Values below 60 sec cause a crash of the software.

Any ideas for that?

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/6c8e638d-7aa5-4ae9-9a82-bcad43dcd6e7%40googlegroups.com.

Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Initializing weewx version 3.9.2
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Using Python 2.7.17 (default, Oct 
24 2019, 12:43:41) [GCC]
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Platform 
Linux-5.3.8-1-default-x86_64-with-glibc2.2.5
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Locale is 'en_US.UTF-8'
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Using configuration file 
/home/weewx/weewx.conf
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Loading station type FileParse 
(user.fileparse)
Apr 20 13:49:43 rmr-srv weewx[27136]: fileparse: data file is 
/var/tmp/weewx-input.dat
Apr 20 13:49:43 rmr-srv weewx[27136]: fileparse: polling interval is 5.0
Apr 20 13:49:43 rmr-srv weewx[27136]: fileparse: label map is {}
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: StdConvert target unit is 0x1
Apr 20 13:49:43 rmr-srv weewx[27136]: wxcalculate: The following values will be 
calculated: barometer=prefer_hardware, windchill=prefer_hardware, 
dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, 
windrun=prefer_
hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, 
humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, 
ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
Apr 20 13:49:43 rmr-srv weewx[27136]: wxcalculate: The following algorithms 
will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Archive will use data binding 
wx_binding
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Record generation will be 
attempted in 'software'
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Using archive interval of 59 
seconds (software record generation)
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Using binding 'wx_binding' to 
database 'weewx.sdb'
Apr 20 13:49:43 rmr-srv weewx[27136]: manager: Starting backfill of daily 
summaries
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: StationRegistry: Registration not 
requested.
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: Wunderground: Posting not enabled.
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: PWSweather: Posting not enabled.
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: CWOP: Posting not enabled.
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: WOW: Posting not enabled.
Apr 20 13:49:43 rmr-srv weewx[27136]: restx: AWEKAS: Posting not enabled.
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Starting up weewx version 3.9.2
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Starting main packet loop.
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Main loop exiting. Shutting 
engine down.
Apr 20 13:49:43 rmr-srv weewx[27136]: engine: Caught unrecoverable exception in 
engine:
Apr 20 13:49:43 rmr-srv weewx[27136]:   integer division or modulo by 
zero
Apr 20 13:49:43 rmr-srv weewx[27136]:   Traceback (most recent call 
last):
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weewx/engine.py", line 894, in main
Apr 20 13:49:43 rmr-srv weewx[27136]:   engine.run()
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weewx/engine.py", line 191, in run
Apr 20 13:49:43 rmr-srv weewx[27136]:   
self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weewx/engine.py", line 224, in dispatchEvent
Apr 20 13:49:43 rmr-srv weewx[27136]:   callback(event)
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weewx/engine.py", line 540, in new_loop_packet
Apr 20 13:49:43 rmr-srv weewx[27136]:   self.accumulator = 
self._new_accumulator(event.packet['dateTime'])
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weewx/engine.py", line 650, in _new_accumulator
Apr 20 13:49:43 rmr-srv weewx[27136]:   self.archive_interval)
Apr 20 13:49:43 rmr-srv weewx[27136]: File 
"/home/weewx/bin/weeutil/weeutil.py", line 197, in startOfInterval
Apr 20 13:49:43 rmr-srv weewx[27136]:   m = int(time_tt.tm_min // 
interval_m * interval_m)
Apr 20 13:49:43 rmr-srv weewx[27136]:   ZeroDivisionError: integer 
division or modulo by zero
Apr 20 13:49:43 rmr-srv weewx[27136]:   Exiting.