Re: [weewx-user] Simplest Way to Read Current Data

[Pogz Sy Ortile]

@mwall: the diagrams are awesome. what I had in mind is approach 3 or 4.
ill look for some uploader code and see how its implemented.

@thomas: the template might be the best approach. create a dead simple
template and just put the values separated by comma.. then have the
'aggregator' to pull the data from there and other non-weewx sensors and
send via http.

this project of mine is gonna get turned over to a bunch of people. thats
why im avoiding over complicating things.. i had to explain to them what an
http requests is, so i dont want to burden myself of explaining what mqtt
is. lol.

On Mon, Jan 8, 2018 at 1:14 AM, mwall  wrote:

> alan,
>
> maybe this wiki page will help clarify the options (now with yummy
> diagrams!):
>
> https://github.com/weewx/weewx/wiki/dashboards
>
> just to be clear about the MQTT and HTTP options:
>
> 3) HTTP as transport.  write your own uploading service that sends data to
> your servers on each LOOP packet or archive record.  this would require
> some python coding.  the code is really simple, and there are many existing
> uploaders that you could adapt or from which you could derive.
>
> 4) MQTT as transport.  use the weewx-mqtt extension on each weewx instance
> to feed data to a single MQTT broker.  optionally connect an influx server
> to the broker so that you get centralized data retention.  then your server
> subscribes to the broker to get the data from all of the stations and/or
> the influx server for historical data.  or you can skip the MQTT transport
> and go right to influx.  either way, you'll have to write some server-side
> code, but that can be any language you want with any framework you want.
>
> the MQTT/influx option is not as complicated as you might think.  the
> result is far more robust than most methods, and much easier to
> manage/maintain.  if you're aggregating data from a bunch of
> sensors/stations, don't reinvent the wheel!  using MQTT/influx lets you
> focus on the value you provide with your servers/systems instead of mucking
> around with "do i need a timestamp?" or "should i use GET or PUT or POST?"
> or "how do i handle three different types of weather stations?"
>
> btw, the reliability of the 'wee_device --current' approach is going to
> vary a lot, depending mostly on the quality of the hardware and how it is
> physically wired up.  for example, it will be pretty much rubbish for fine
> offset stations connected directly to a raspberry pi, but it should be
> almost bulletproof for a rainwise or vantage.  but if you want continuous
> operation, just use weewxd.
>
> m
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/weewx-user/P8L4qHt5e_o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[vince]

On Sunday, January 7, 2018 at 9:14:04 AM UTC-8, mwall wrote:
>
> the MQTT/influx option is not as complicated as you might think.  the 
> result is far more robust than most methods, and much easier to 
> manage/maintain.  if you're aggregating data from a bunch of 
> sensors/stations, don't reinvent the wheel!  using MQTT/influx lets you 
> focus on the value you provide with your servers/systems instead of mucking 
> around with "do i need a timestamp?" or "should i use GET or PUT or POST?" 
> or "how do i handle three different types of weather stations?"
>
>
One of the benefits of aggregating 'into' a weewx instance is the ease of 
getting plots and reports 'out' of the aggregate db using weewx's skins 
magic.

For the MQTT/influxdb scenario-5 do you have a recommendation for how to 
generate nice-looking skins (so to speak) that would reasonably equivalent 
to weewx's ease of use ?   Would that be jumping into grafana, or writing 
something with express ala weeRT's approach, or some other mechanism ? 

(thanks for the diagrams incidentally - helps lots !)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[mwall]

alan,

maybe this wiki page will help clarify the options (now with yummy 
diagrams!):

https://github.com/weewx/weewx/wiki/dashboards

just to be clear about the MQTT and HTTP options:

3) HTTP as transport.  write your own uploading service that sends data to 
your servers on each LOOP packet or archive record.  this would require 
some python coding.  the code is really simple, and there are many existing 
uploaders that you could adapt or from which you could derive.

4) MQTT as transport.  use the weewx-mqtt extension on each weewx instance 
to feed data to a single MQTT broker.  optionally connect an influx server 
to the broker so that you get centralized data retention.  then your server 
subscribes to the broker to get the data from all of the stations and/or 
the influx server for historical data.  or you can skip the MQTT transport 
and go right to influx.  either way, you'll have to write some server-side 
code, but that can be any language you want with any framework you want.

the MQTT/influx option is not as complicated as you might think.  the 
result is far more robust than most methods, and much easier to 
manage/maintain.  if you're aggregating data from a bunch of 
sensors/stations, don't reinvent the wheel!  using MQTT/influx lets you 
focus on the value you provide with your servers/systems instead of mucking 
around with "do i need a timestamp?" or "should i use GET or PUT or POST?" 
or "how do i handle three different types of weather stations?"

btw, the reliability of the 'wee_device --current' approach is going to 
vary a lot, depending mostly on the quality of the hardware and how it is 
physically wired up.  for example, it will be pretty much rubbish for fine 
offset stations connected directly to a raspberry pi, but it should be 
almost bulletproof for a rainwise or vantage.  but if you want continuous 
operation, just use weewxd.

m

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[Thomas Keffer]

You did not say whether you need the "raw" LOOP data, or the archive data.
Because you played with the RSS feed, I'm going to assume the latter.
There are lots of ways to accomplish this, but here are two easy ways:

1. Access the sqlite database directly. It is not encrypted. You do not
need the WeeWX python library to access it --- just use SQL. That's what
it's for. You either had the wrong file location, or you lacked privileges.
Its exact location depends on the install method, but is usually
/var/lib/weewx/weewx.sdb or /home/weewx/archive/weewx.sdb. Did you look to
see if the file existed??

2. Create a template with exactly the information you need. Something like

$current.dateTime.raw $current.outTemp $current.barometer


That's it. Put it in the Standard skin directory along with all the other
templates. Add its name to the list of all the other templates to be
processed in skin.conf (section [CheetahGenerator]
). See the
Customizing Guide for details. The template will be processed at the end of
an archive period and a file created. Then FTP (or rsync) the file to your
aggregator.

Honestly, this is not hard. You're not giving yourself enough time to think
through what you want and how to do it.

-tk

On Sun, Jan 7, 2018 at 5:34 AM, Pogz Sy Ortile 
wrote:

> ill check mqtt but i think it might be overkill.. did i understand that
> its gonna be like
>
> weewx -> mqtt -> my app to read stuff from mqtt -> send to my server via
> http request in my own format
>
>
> On Sun, Jan 7, 2018 at 9:32 PM, jmltech  wrote:
>
>> Look at the MQTT under “uploaders” on the weewx wiki.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/weewx-user/P8L4qHt5e_o/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> weewx-user+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[Pogz Sy Ortile]

what i had in mind is that the server would just wait for http requests
from all the raspberry pis i have from all over the city. the raspberry pi
would have a small python script which would act as an aggregator and would
assemble all the sensor readings along with the time stamp and run it as a
cron job every minute..

i thought of implementing it that way so that all sensors (bought off the
shelf or not) would all funnel into that aggregator and send all the
readings as 1 whole request.

my ideal setup would be just to reuse the weewx python library to get the
current data, format it to conform with my http request format and send it.
clean. simple.

my plan b was to read the latest database entry

plan c was to parse the rss

but then again nothing is going my way currently. lol

On Sun, Jan 7, 2018 at 9:54 PM, jmltech  wrote:

> That looks right.
>
> I assume that your “server” is on an external host outside of your local
> network?  If so, couldn’t you write your owm specific uploader using one of
> the existing uploaders as your guide? Or maybe create a new report skin
> with the exact info that you need, and have the FTP service upload it to
> your server. The skin could even generate the exact html that your looking
> for, so you wouldn’t need to parse it. Not sure what your looking to do on
> the “server” so hard to offer specific suggestions. Also, I’m no expert
> myself.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/weewx-user/P8L4qHt5e_o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[jmltech]

That looks right. 

I assume that your “server” is on an external host outside of your local 
network?  If so, couldn’t you write your owm specific uploader using one of the 
existing uploaders as your guide? Or maybe create a new report skin with the 
exact info that you need, and have the FTP service upload it to your server. 
The skin could even generate the exact html that your looking for, so you 
wouldn’t need to parse it. Not sure what your looking to do on the “server” so 
hard to offer specific suggestions. Also, I’m no expert myself. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Simplest Way to Read Current Data

[Pogz Sy Ortile]

ill check mqtt but i think it might be overkill.. did i understand that its
gonna be like

weewx -> mqtt -> my app to read stuff from mqtt -> send to my server via
http request in my own format


On Sun, Jan 7, 2018 at 9:32 PM, jmltech  wrote:

> Look at the MQTT under “uploaders” on the weewx wiki.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/weewx-user/P8L4qHt5e_o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Simplest Way to Read Current Data

[jmltech]

Look at the MQTT under “uploaders” on the weewx wiki. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Simplest Way to Read Current Data

[jmltech]

I have found the simplistic way to get weewx data into external programs is to 
add the MQTT driver into weewx. Then any external python or PHP program can 
subscribe to the stream. Including apps on your phone if you make the stream 
accessable through your router. 

Or, look at the weewx-wd skin. If you add that skin, it will generate external 
reports that a program can parse as often as you need. It may be overkill for 
what you need, but it does have a lot of data available. 
Joe

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Simplest Way to Read Current Data

[Allan Ortile]

Hi, I've been looking through a bunch of documentation and group threads on 
how to get the current data as simple as possible. Right now im thinking of 
these approaches: 

1.) Using the RSS feed of Weewx, read the current data which is enclosed in 
CDATA. 

I tried this one and throught to myself, how hard can it be? Then I tried 
beautifulsoup to read/parse the rss, tried some other libraries but ended 
up wasting an hour not getting anywhere. 

2.) Maybe I can just run wee_device --current and just parse the data there 

It works for a time, til I get the annoying "weewx.WeeWxIOError: [Errno 16] 
Resource busy" errors. Dont get me wrong, the device is still polling and 
is still updating the HTTP website with fresh data its just that wee_device 
is so unreliable that i'd even consider this as an option

3.) Directly pull the latest data from the SQLite database 

Well, i tried the command sqlite /var/lib/weewx/weewx.sdb and got the error 
Unable to open database "/var/lib/weewx/weewx.sdb": file is encrypted or is 
not a database 

Man this gets weirder and weirder. 

4.) Use the weewx python library 

I saw some guys use the weewx python via import, but theres no reference 
libraries i can turn to. 

5.) Curl an endpoint? 

Is there an endpoint i can curl to just get raw data from the sensor? 



I honestly just want the current values from weewx. Is it too much to ask? 
:( 

My end goal would be to write a simple python aggregator to get my sensor 
data from weewx + some other sensor data directly connected to the pi 
(soil, uv, etc) then send it via an http request to my server which stores 
it on a database and processes everything collectively (this wont be just 
one weather station, we will deploy many). 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.