Multithreading? How?
I need to develop a Python application that is a sort of gateway between
to networks: a "serial bus" network (think of a serial port or a USB
connection) and a MQTT connection (in my case, it's AWS IoT service).
On the bus, a proprietary protocol is implemented. From the bus, the app
knows the status of the system (think of a simple ON/OFF status).
The user can retrieve the status of the system through MQTT: it sends a
message to read/status MQTT topic and receive back a message with the
current status on read/status/reply MQTT topic. Of course, they are just
examples.
On the contrary, when the app detects a status change reported from the
serial bus (think of a transition from ON to OFF), it sends a MQTT message.
I'm thinking to split the application in three classes: Bus, App and
IoT. Bus and IoT are also threads.
The thread of Bus manages the serial protocol, while the thread of IoT
manages MQTT connection with the broker (the server).
However I don't know if it could be a good architecture. Suppone Bus
thread receives a new status from the system. In the context of
ThreadBus, the object Bus could call a method of App object:
app.set_status(new_status_from_the_bus)
In the App I have:
class App():
..
def set_status(new_status): # Could be called from ThreadBus
if new_status != self.new_status:
self.new_status = new_status
# Do some actions on status change
def get_status():# Could be called from ThreadIoT
return self.status
Of course, IoT object needs to know the current status of the system
when a message is received from MQTT. So ThreadIoT could call
app.get_status().
I think this architecture has some problems with race conditions or
threads synchronization. What happens if IoT calls get_status() exactly
when set_status() called by ThreadBus is executing? If status is a big
data structure, set_status() could be interrupted by get_status() that
could get a completely corrupted status, because it was only partly
updated by set_status().
I know I can use locks or semaphores in get_status() and set_status(),
but I don't know if this is a good approach. Consider that the system is
complex, it isn't composed by a simple single status. It has many many
parameters that are collected from the serial bus. Should I use a lock
for every [get|set]_status(), [get|set]_dimensions(),
[get|set]_battery_level(), [get|set]_mains_present(), and so on?
Another possibility is to use a Queue for Bus and a Queue for IoT. So
the set_status(new_status) called from Bus object will be transformed in
a put in the queue:
app_queue.put({"type": "new_status", "data": ...})
However how could be transformed the get_status() from IoT? How the
return value (the current status) is real retrieved?
class IoT():
..
def status_request_from_MQTT():
app_queue.put({"type": "get_status"})
# How to get the status to return?
return current_status
Should the app put the status on the same queue and should IoT waits for
a new message in the Queue?
def status_request_from_MQTT():
app_queue.put({"type": "get_status"})
try:
current_status = app_queue.get(timeout=10)
except Empty:
# What to do?
return current_status
Again another approach is to avoid multi-threading at all and create a
single "main loop" function that waits at the same time for incoming
events on the serial bus and MQTT (how?). I don't know if this could be
done in my case, because I'm using awscrt Python module and it works
through callbacks that I think is called from another thread.
Any suggestions on this architecture?
--
https://mail.python.org/mailman/listinfo/python-list
How to 'ignore' an error in Python?
I'm sure I'm missing something obvious here but I can't see an elegant way to do this. I want to create a directory, but if it exists it's not an error and the code should just continue. So, I have:- for dirname in listofdirs: try: os.mkdir(dirname) except FileExistsError: # so what can I do here that says 'carry on regardless' except: # handle any other error, which is really an error # I want code here to execute whether or not dirname exists Do I really have to use a finally: block? It feels rather clumsy. I suppose I could test if the directory exists before the os.mkdir() but again that feels a bit clumsy somehow. I suppose also I could use os.mkdirs() with exist_ok=True but again that feels vaguely wrong somehow. -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On 4/28/23 09:55, Chris Green wrote: I'm sure I'm missing something obvious here but I can't see an elegant way to do this. I want to create a directory, but if it exists it's not an error and the code should just continue. So, I have:- for dirname in listofdirs: try: os.mkdir(dirname) except FileExistsError: # so what can I do here that says 'carry on regardless' except: # handle any other error, which is really an error # I want code here to execute whether or not dirname exists Do I really have to use a finally: block? It feels rather clumsy. For this specific case, you can use os.makedirs: os.makedirs(dirname, exist_ok=True) The mkdir in pathlib also takes the exist_ok flag As to the way you asked the question, you can use "pass" or the ellipses for the "# so what can I do here" -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On 2023-04-28 16:55, Chris Green wrote: I'm sure I'm missing something obvious here but I can't see an elegant way to do this. I want to create a directory, but if it exists it's not an error and the code should just continue. So, I have:- for dirname in listofdirs: try: os.mkdir(dirname) except FileExistsError: # so what can I do here that says 'carry on regardless' except: # handle any other error, which is really an error # I want code here to execute whether or not dirname exists Do I really have to use a finally: block? It feels rather clumsy. I suppose I could test if the directory exists before the os.mkdir() but again that feels a bit clumsy somehow. I suppose also I could use os.mkdirs() with exist_ok=True but again that feels vaguely wrong somehow. I'd do this: from contextlib import suppress for dirname in listofdirs: with suppress(FileExistsError): os.mkdir(dirname) -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On 4/28/23 11:05, MRAB wrote: On 2023-04-28 16:55, Chris Green wrote: I'm sure I'm missing something obvious here but I can't see an elegant way to do this. I want to create a directory, but if it exists it's not an error and the code should just continue. So, I have:- for dirname in listofdirs: try: os.mkdir(dirname) except FileExistsError: # so what can I do here that says 'carry on regardless' except: # handle any other error, which is really an error I'd do this: from contextlib import suppress for dirname in listofdirs: with suppress(FileExistsError): os.mkdir(dirname) I'm fond of that approach too, though you can't use if it you really wanted to do the except: # handle any other error, which is really an error If you're okay letting Python just raise whatever other error it found, then great! -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On 28Apr2023 16:55, Chris Green wrote:
for dirname in listofdirs:
try:
os.mkdir(dirname)
except FileExistsError:
# so what can I do here that says 'carry on regardless'
except:
# handle any other error, which is really an error
# I want code here to execute whether or not dirname exists
Do I really have to use a finally: block? It feels rather clumsy.
You don't. Provided the "handle any other error" part reraises or does a
break/continue, so as to skip the bottom of the loop body.
ok = true
for dirname in listofdirs:
try:
os.mkdir(dirname)
except FileExistsError:
pass
except Exception as e:
warning("mkdir(%r): %s", dirname, e)
ok = False
continue
rest of the loop body ...
if not ok:
... not all directories made ...
2 notes on the above:
- catching Exception, not a bare except (which catches a rather broader
suit of things)
- reporting the other exception
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On 28Apr2023 10:39, Mats Wichmann wrote: For this specific case, you can use os.makedirs: os.makedirs(dirname, exist_ok=True) I'm not a great fan of makedirs because it will make all the missing components, not just the final one. So as an example, if you've got a NAS mounted backup area at eg: /mnt/nas/backups/the-thing/backups-subdir and your config file has this mistyped as: /mn/nas/backups/the-thing/backups-subdir and makedirs is used, then it will make the backup area on eg the root drive where there's no room. (I'm looking at you, Docker, grr). There are plenty of similar situations. Because of this I usually am prepared to make a missing final component with mkdir(), but not a potentially deep path with makedirs(). Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On Fri, Apr 28 2023 at 04:55:41 PM, Chris Green wrote: > I'm sure I'm missing something obvious here but I can't see an elegant > way to do this. I want to create a directory, but if it exists it's > not an error and the code should just continue. > > So, I have:- > > for dirname in listofdirs: > try: > os.mkdir(dirname) > except FileExistsError: > # so what can I do here that says 'carry on regardless' > except: > # handle any other error, which is really an error > > # I want code here to execute whether or not dirname exists > > > Do I really have to use a finally: block? It feels rather clumsy. > > I suppose I could test if the directory exists before the os.mkdir() > but again that feels a bit clumsy somehow. > > I suppose also I could use os.mkdirs() with exist_ok=True but again > that feels vaguely wrong somehow. > Why does exist_ok=True feel wrong to you? This is exactly what it is there for. -- regards, kushal -- https://mail.python.org/mailman/listinfo/python-list
Re: How to 'ignore' an error in Python?
On Sat, 29 Apr 2023 at 14:27, Kushal Kumaran wrote: > > On Fri, Apr 28 2023 at 04:55:41 PM, Chris Green wrote: > > I'm sure I'm missing something obvious here but I can't see an elegant > > way to do this. I want to create a directory, but if it exists it's > > not an error and the code should just continue. > > > > So, I have:- > > > > for dirname in listofdirs: > > try: > > os.mkdir(dirname) > > except FileExistsError: > > # so what can I do here that says 'carry on regardless' > > except: > > # handle any other error, which is really an error > > > > # I want code here to execute whether or not dirname exists > > > > > > Do I really have to use a finally: block? It feels rather clumsy. > > > > I suppose I could test if the directory exists before the os.mkdir() > > but again that feels a bit clumsy somehow. > > > > I suppose also I could use os.mkdirs() with exist_ok=True but again > > that feels vaguely wrong somehow. > > > > Why does exist_ok=True feel wrong to you? This is exactly what it is > there for. > Using mkdirs when you only want to make one is inviting problems of being subtly wrong, where it creates too many levels of directory. Personally, I would just do: try: os.mkdir(dirname) except FileExistsError: pass and not try to handle anything else at all. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
