[issue40844] Alternate ways of running coroutines

[Matthew Francis]

New submission from Matthew Francis <4576fran...@gmail.com>:

Currently, using await inside a coroutine will block inside the coroutine.  
This behavior would usually be fine, but for some usecases a way to 
nonblockingly run coroutines without creating a Task could be useful, because 
tasks don't allow for a callback.  I'm suggesting a method on coroutines that 
runs them without blocking, and will run a callback when it's complete.

--
components: asyncio
messages: 370614
nosy: asvetlov, matthewfrancis, yselivanov
priority: normal
severity: normal
status: open
title: Alternate ways of running coroutines
type: enhancement
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue40844>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Building a HPC data assimilation system using Python?

[Matthew Francis]

I have a prototype data assimilation code ( an ionospheric nowcast/forecast 
model driven by GPS data ) that is written in IDL (interactive data language) 
which is a horrible language choice for scaling the application up to large 
datasets as IDL is serial and slow (interpreted).

I am embarking on a project to convert this prototype into an operational 
parallel HPC code. In the past I've used C++ for this kind of project and am 
comfortable using MPI. On the other hand, I've recently started using python 
and appreciate the flexibility and speed of development using python compared 
with C++. I have read that there is a trend to use python as the high level 
'glue' for these kind of large number crunching projects, so it would seem 
appropriate to go down that path. There are a number of C++ and FORTRAN(!) 
libraries I'd need to incorporate that handle things such as the processing of 
raw GPS data and computing ionospheric models, so I'd need to be able to make 
the appropriate interface for these into python.

If anyone uses python is this way, I'd appreciate any tips, hints, things to be 
careful about and in general any war stories you can relate that you wish you'd 
heard before making some mistake.

Here are the things I have investigated that it looks like I'd probably need to 
use:

* scipy/numpy/matplotlib
* Cython (or pyrex?) for speeding up any bottlenecks that occur in python code 
(as opposed to C++/FORTRAN libraries)
* MPI for Python (mpi4py). Does this play nice with Cython?
* Something to interface python with other language libraries. ctypes, swig, 
boost? Which would be best for this application?
* Profiling. profile/cprofile are straightforward to use, but how do they cope 
with a parallel (mpi4py) code?
* If a C++ library call has its own MPI calls, does that work smoothly with 
mpi4py operating in the python part of the code?

Sorry if some of this is a little basic, I'm trying to get up to speed on this 
a quick as I can.

Thanks in advance!
-- 
http://mail.python.org/mailman/listinfo/python-list