On 11/21/18 5:54 PM, Alan Gauld via Tutor wrote: > On 21/11/2018 16:31, Avi Gross wrote:
>> An obvious speedup might be had by starting up N threads with each opening >> one file and doing what I said above into one shared process with N >> variables now available. But will it be faster? > > Trying to calculate (or guess) this kind of thing in > advance is near impossible. The best solution is to > prototype and measure, making sure to do so on typical > data volumes. > > That having been said if you know (or discover) that > you definitely need parallelism then its definitely worth > revisiting the design to ensure the data structures > and overall workflow are optimised for a parallel approach. > >> ...I will pause and simply say that I opted not to bother >> as the darn program finished in 5 or 10 seconds. > > Exactly so. > AS the famous quote says "Premature optimisation is..." > >> For heavy industrial uses, like some of the applications in the cloud >> dealing with huge problems, it may well be worth it. > > In many cases it's the only practical solution. > Almost all of my industrial programming has involved multi > processing and threading. Almost none (I think one )of my > personal programming projects has needed it. > People play all kinds of parallelism tricks with Python because Python has a certain Impedimet Which Shall Remain Nameless (except I'm certain someone will mention it). Anyway, it's one thing to try to decompose a massive problem, that's interesting on a certain level (see some of the talks companies like Google have done on scaling their services) but is really hard to replicate at home. But another use for non-linear programming, if you want to call it that, is task that just needs a different programming model. That's where a lot of the async stuff with coroutines and event loops that has been beefed up recently is quite interesting. Even very simple programs can run into cases where it may make sense, usually if there are things you have to wait for and want to be able to do other work while doing so. I actually got around to watching David Beazley's talk from a couple years ago, and it was pretty impressive - something I'd flagged as "watch later" I don't know how long ago, more than a year at least. Wish I'd watched it earlier now! I hate posting those obscure YouTube links where people don't know what they are clicking on, so search for this title if interested: David Beazley - Python Concurrency From the Ground Up (it's from the 2015 PyCon) _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
