Re: [Dorset] YAWMT Question
Hi Terry, On Thursday, 22 June 2017, at 13:23:35 BST, Ralph Corderoy wrote: > That's sounding as if you'd have a database server sitting on > the network with all the slave and the master being its > clients, sending SQL to modify tables. And if the slaves are > writing to the tables, and the master is trailing behind, > reading them, then you're (mis-)using the database as message > queues. Perhaps a message queue would be a better fit. > As Tim said, it sounds like overkill, and adds another major > piece to understand, configure, monitoring, debug, ... A > message queue, like RabbitMQ, or zero.mq, also has those > problems given your simple needs. I think Ralph has hit the nail on the head with these two paragraphs. My thoughts were much the same, but not clear enough in my mind to be able to articulate them this well! Patrick -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
On Thursday, 22 June 2017 13:23:35 BST Ralph Corderoy wrote: > I'm assuming that minimising power consumption won't be an issue either > way. Obviously we won't want to install a heavier sub-station for the WMT, but no; we have no known issues with power ATM :-) > I'm guessing slave Pis will only be in the dozens, and readings and > all-hands-to-the-pump commands in the order of 1 Hz or slower? Essentially correct, although the initial installation won't include dozens of slaves (apart from the volunteers :-) ). > That's sounding as if you'd have a database server sitting on the > network with all the slave and the master being its clients, sending SQL > to modify tables. And if the slaves are writing to the tables, and the > master is trailing behind, reading them, then you're (mis-)using the > database as message queues. Perhaps a message queue would be a better > fit. OK. I've vaguely heard of message queues. > As Tim said, it sounds like overkill, and adds another major piece to > understand, configure, monitoring, debug, ... A message queue, like > RabbitMQ, or zero.mq, also has those problems given your simple needs. I'll look into those. > You're sticking with Python? And you've been dabbling with > http://flask.pocoo.org/, the web micro-framework? I'd try and stick > with those. Python has been adopted as the preferred language for WMT software by consensus. (Despite the fact that I had to fall back on C for the lighting software due to lack of Library support for hardware PWM in piCore.) I haven't done much with the flask stuff yet, due to 'one damn thing after another' on the home front. The web updater for the Audio Guide / Quiz was always a 'Desirable' rather than 'Essential' feature, so I'm focusing on fixing the blocked drain, broken tumble drier, central heating always on issues at home and trying to keep up with the River System development during odd bits of spare time. > How about having the master poll the slaves regularly, with a timeout. > That way it learns of slaves not responding. The slaves present a web > server that sends back the last-obtained readings when asked; GET > /readings. The same web server can provide a path for writing by the > master; POST /pump-speed. In the slave, you'd have flask manage > calling your bits of code to format the variables holding the readings > into the text reply, and parsing the new speed out of the text request. > Something like the APScheduler you're already familiar with can > regularly call your routine that reads the Pi's I/O. To safely stash > those readings from variables local to the function to those shared with > the web-serving side, look at > https://docs.python.org/2.7/library/threading.html#lock-objects to > provide the https://en.wikipedia.org/wiki/Critical_section Tim > mentioned. Thanks. I'll look into that. We want a webserver in the system anyway to provide a control panel for the WMT staff and an information panel for the visitors. Maybe I'll understand enough about flask by the time we've finished to get the web updater done :-) > (IIRC storing a value in a Python global doesn't need a lock, so you > could have a global holding all of the readings, and switch them > wholesale for a new lot with a single atomic STORE_GLOBAL opcode, but > it's simpler to tell you to use a lock. :-) I'm sure that when I've read up on all this, I might understand what you're telling me ;-( (It's the documentation that I have a problem with and that's mainly due to my lack of knowledge of the terminology used in the CS world.) -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
Hi Terry, > The control system will be a distributed network of Raspberry Pis, > interconnected by Ethernet cable (we are trialling Power over Ethernet > to see if we can power the remote devices without needing mains local > to them). I'm assuming that minimising power consumption won't be an issue either way. > Each device will be responsible for taking one or more measurements of > water depth, flow, etc and, in certain locations, for controlling > pumps. My question is about distributing data (eg measurements and > speed demands). There will be a master controller orchestrating > everything, so we need to get measurement data from the Remote Pis to > the Master Pi and possibly pump speed demands the other way. I'm guessing slave Pis will only be in the dozens, and readings and all-hands-to-the-pump commands in the order of 1 Hz or slower? > As a systems engineer my instinct is to use a central database which > each of the Pis can write to or read from. So for example, a level > measurement value gets written to a record and the Master Pi reads > that record to use when determining if a pump speed needs to be > changed. The Master Pi then writes the speed demand to another > record, which the Remote Pi reads and acts upon. That's sounding as if you'd have a database server sitting on the network with all the slave and the master being its clients, sending SQL to modify tables. And if the slaves are writing to the tables, and the master is trailing behind, reading them, then you're (mis-)using the database as message queues. Perhaps a message queue would be a better fit. > My reasoning for selecting a database is that there won't be problems > if a Remote Pi is trying to write to a record at the same time as the > master is tring to read it, plus, I believe that it should be possible > to do all these transactions over the Ethernet link. As Tim said, it sounds like overkill, and adds another major piece to understand, configure, monitoring, debug, ... A message queue, like RabbitMQ, or zero.mq, also has those problems given your simple needs. You're sticking with Python? And you've been dabbling with http://flask.pocoo.org/, the web micro-framework? I'd try and stick with those. How about having the master poll the slaves regularly, with a timeout. That way it learns of slaves not responding. The slaves present a web server that sends back the last-obtained readings when asked; GET /readings. The same web server can provide a path for writing by the master; POST /pump-speed. In the slave, you'd have flask manage calling your bits of code to format the variables holding the readings into the text reply, and parsing the new speed out of the text request. Something like the APScheduler you're already familiar with can regularly call your routine that reads the Pi's I/O. To safely stash those readings from variables local to the function to those shared with the web-serving side, look at https://docs.python.org/2.7/library/threading.html#lock-objects to provide the https://en.wikipedia.org/wiki/Critical_section Tim mentioned. (IIRC storing a value in a Python global doesn't need a lock, so you could have a global holding all of the readings, and switch them wholesale for a new lot with a single atomic STORE_GLOBAL opcode, but it's simpler to tell you to use a lock. :-) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
Hi Terry On 22/06/17 12:25, Terry Coles wrote: On Thursday, 22 June 2017 12:17:56 BST Terry Coles wrote: I'm not sure that I understand how this works. Taking the Remote Pi; it reads the water level and writes the value to a variable. Fine so far, but how can the Master Pi, then read that variable? That variable is a location in the Remote Pi's RAM, so is inaccessible to another device. Thinking about this a bit more; clearly the Master Pi can get access to the Remote Pi's variables through SSH, but wouldn't it then have to maintain multiple SSH sessions? Sorry, from your description I'd assumed you'd got the data in the master. To get it there you would need a connection to each slave for sure. At the lowest level you can use sockets but it's not an area I'm familiar with in the Linux world. Tim -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
On Thursday, 22 June 2017 12:17:56 BST Terry Coles wrote: > I'm not sure that I understand how this works. Taking the Remote Pi; it > reads the water level and writes the value to a variable. Fine so far, but > how can the Master Pi, then read that variable? That variable is a > location in the Remote Pi's RAM, so is inaccessible to another device. Thinking about this a bit more; clearly the Master Pi can get access to the Remote Pi's variables through SSH, but wouldn't it then have to maintain multiple SSH sessions? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
On Thursday, 22 June 2017 12:12:29 BST t...@ls83.eclipse.co.uk wrote: > Using a database to deal with concurrent reads and writes is a bit of > overkill. These are only an issue if you have multiple threads running > on the master AND reads and writes are not atomic. Using variables equal > to or less than the word width of the processor (32 or 64 bit?) is > therefore fine without worrying about concurrency and 32 bit or less > variables should be more than adequate for this application. If you did > need variables with non-atomic access from multiple threads then simply > placing read/writes to them in critical sections (interrupts briefly > disabled) protects them. All far simpler than bringing a database to bear. I'm not sure that I understand how this works. Taking the Remote Pi; it reads the water level and writes the value to a variable. Fine so far, but how can the Master Pi, then read that variable? That variable is a location in the Remote Pi's RAM, so is inaccessible to another device. Or am I missing something? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] YAWMT Question
Hi Terry On 22/06/17 11:54, Terry Coles wrote: Hi, (Yet Another Wimborne Model Town Question) :-) The system design for the WMT River System Water Sustainability Programme (that really is its name) is progressing slowly and we have tentatively agreed on a number of things. The control system will be a distributed network of Raspberry Pis, interconnected by Ethernet cable (we are trialling Power over Ethernet to see if we can power the remote devices without needing mains local to them). Each device will be responsible for taking one or more measurements of water depth, flow, etc and, in certain locations, for controlling pumps. My question is about distributing data (eg measurements and speed demands). There will be a master controller orchestrating everything, so we need to get measurement data from the Remote Pis to the Master Pi and possibly pump speed demands the other way. As a systems engineer my instinct is to use a central database which each of the Pis can write to or read from. So for example, a level measurement value gets written to a record and the Master Pi reads that record to use when determining if a pump speed needs to be changed. The Master Pi then writes the speed demand to another record, which the Remote Pi reads and acts upon. My reasoning for selecting a database is that there won't be problems if a Remote Pi is trying to write to a record at the same time as the master is tring to read it, plus, I believe that it should be possible to do all these transactions over the Ethernet link. However, my knowledge of database functionality is very much at the systems level, so before I embark on a programme of research, does anyone have any comments on this approach? If there is a better way, I'd like to hear about it. Using a database to deal with concurrent reads and writes is a bit of overkill. These are only an issue if you have multiple threads running on the master AND reads and writes are not atomic. Using variables equal to or less than the word width of the processor (32 or 64 bit?) is therefore fine without worrying about concurrency and 32 bit or less variables should be more than adequate for this application. If you did need variables with non-atomic access from multiple threads then simply placing read/writes to them in critical sections (interrupts briefly disabled) protects them. All far simpler than bringing a database to bear. Tim -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
[Dorset] YAWMT Question
Hi, (Yet Another Wimborne Model Town Question) :-) The system design for the WMT River System Water Sustainability Programme (that really is its name) is progressing slowly and we have tentatively agreed on a number of things. The control system will be a distributed network of Raspberry Pis, interconnected by Ethernet cable (we are trialling Power over Ethernet to see if we can power the remote devices without needing mains local to them). Each device will be responsible for taking one or more measurements of water depth, flow, etc and, in certain locations, for controlling pumps. My question is about distributing data (eg measurements and speed demands). There will be a master controller orchestrating everything, so we need to get measurement data from the Remote Pis to the Master Pi and possibly pump speed demands the other way. As a systems engineer my instinct is to use a central database which each of the Pis can write to or read from. So for example, a level measurement value gets written to a record and the Master Pi reads that record to use when determining if a pump speed needs to be changed. The Master Pi then writes the speed demand to another record, which the Remote Pi reads and acts upon. My reasoning for selecting a database is that there won't be problems if a Remote Pi is trying to write to a record at the same time as the master is tring to read it, plus, I believe that it should be possible to do all these transactions over the Ethernet link. However, my knowledge of database functionality is very much at the systems level, so before I embark on a programme of research, does anyone have any comments on this approach? If there is a better way, I'd like to hear about it. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2017-07-04 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
