I see, so the logic is simply wait a bit and try again for a certain
number of times. That was my question I guess (if I need to do anything
else).
I shall put that in as well.
On 17/12/13 09:43, Nathan Rusch wrote:
We get them frequently enough (using Isilon) that we have a decorator
that we attach to most functions and methods that perform file
operations. They're surprisingly frequent with distributed or
cluster-based file systems once you start scaling up the number of
users and connections.
Here's a simple example:
import errno
import time
tryCount = 0
while True:
try:
# Do some file operation
except (OSError, IOError) as e:
if e.errno == errno.ESTALE:
if tryCount >= 5:
raise
time.sleep(.5)
tryCount += 1
else:
raise
You can do fancier things like scaling up the sleep duration with the
try count, but that's the general idea. In your case, you wouldn't
necessarily need a decorator since you've only got one block to worry
about, but you can see how the same logic translates nicely into one.
-Nathan
*From:* Frank Rueter <mailto:[email protected]>
*Sent:* Monday, December 16, 2013 12:16 PM
*To:* Nuke Python discussion
<mailto:[email protected]>
*Subject:* Re: [Nuke-python] nuke localise
Hi Nathan,
just to follow up:
how often would you expect to see stale nfs handles? I see very, very
few of them and usually when I screw up somethign accidentally, so my
retry logic would be "wait for culprit to clean up there mess".
In bigger environments and virtual file systems this can of course
happen a lot more, but I'm not sure what retry logix would be
suitable?! Any tips? I'm happy to throw in whatever people deem
sensible error handling, but working frmo my home office I won't be
able to test some of those.
Cheers,
frank
On 17/12/13 07:32, Nathan Rusch wrote:
Cool stuff Frank. Thanks for sharing.
In line with what Sebastian mentioned about Windows, I would agree
that threading beyond maybe 2 to 4 workers on any platform is going
to result in a pretty severe performance drop past some "sweet spot",
which will depend on your storage type, load, etc. If you're using a
distributed file system or a storage array like Isilon with a
potentially high uncached seek overhead, parallel copying can be
quite a bit quicker, but trying to load up 12 or 16 threads is
probably a optimistic unless you've got some kind of ridiculous
storage cluster and link (in which case... why localize in the first
place?).
I would also recommend adding exception handling and retry logic for
stale NFS handles (errno.ESTALE), and you may want to consider just
using a multiprocessing.ThreadPool and adding paths individually or
in chunks of 5 or 10 to dice things up without having to go to Qt.
-Nathan
*From:* Sebastian Elsner <mailto:[email protected]>
*Sent:* Monday, December 16, 2013 1:24 AM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: AW: [Nuke-python] nuke localise
On 12/16/2013 09:29 AM, Frank Rueter wrote:
Hola everybody,
I had a quick look at this this morning and realised I would have to
re-write everything from scratch - but then couldn't resists :-D.
Could some of you test the attached file and tell me how you get on?
Just put it into your NUKE_PATH and put this into your menu.py:
import LocaliseThreaded
LocaliseThreaded.register()
This will replace the default localising behaviour with a threaded one.
The maximum threads are half of your Nuke threads (nuke.THREADS) at
the moment (I will make this a preference though). There will be
info about concurrent threads in the progress bar as it does it's thing.
It's work in progress at this stage but since my day is coming to an
end, I thought it would be good to get it out there for a test run,
so I know more in the morning.
Things I still need/want to do:
* support split file knobs for stereo projects
* support proxy knobs (should I, not sure if the default does?)
* refactor the code so that multiple threads can tackle the same
read node (at the moment one Read node is allocated one task)
* benchmark the copy function (currently shutil.copy2). Pretty
sure it's not the fastest one for large files and I might have
to roll my own to speed things up.
On windows copy2 is really slow compared to native copying. You can
use this snippet to speed things up:
import platform
if platform.system() == "Windows":
from ctypes import windll
def copy2(src, dst):
windll.kernel32.CopyFileA(src, dst, False)
else:
from shutil import copy2
Also copying in multiple threads on windows does not help at all to
speed things up. Worse, it will make things slower. But this is
mostly depending on your fileserver. I never do more than one thread
on our machines, its not worth the effort of handling the threads
dynamically. If you still want that and need an easy solution try
QThreadPool and QRunnable. Its not as flexible as handling the
threads yourself, but it is difficult to get it wrong, which is
something you cannot say about QThread.
Cheers
Sebastian
*
Let me know how it works, especially if you are on windows as I
can't test that here.
Cheers,
frank
P.S.: If somebody is an expert with python threading with a little
bit of time on their hands, get in touch, I'm pretty sure the way
I'm doing this can be optimised, especially for trying to
dynamically allocate threads to efficiently deal with outstanding tasks.
On 14/12/13 23:23, Howard Jones wrote:
I was wondering that. It was going to be my next question (honest)
Howard
On 14 Dec 2013, at 08:19, Thorsten
Kaufmannmailto:[email protected] wrote:
This also sounds like a job for "import nuke" no? ;)
Thorsten Kaufmann
Production Pipeline Architect
____________________________________
Mackevision Medien Design GmbH
Forststra?e 7
D-70174 Stuttgart
T T +49 711 93 30 48 78
F +49 711 93 30 48 90
M +49 151 19 55 55 02
[email protected]
http://www.mackevision.de
Gesch?ftsf?hrer: Armin Pohl, Joachim Lincke, Karin Suttheimer
HRB 243735 Amtsgericht Stuttgart
________________________________________
Von:[email protected]
[[email protected]] im Auftrag von Frank Rueter
[[email protected]]
Gesendet: Samstag, 14. Dezember 2013 01:11
An: Nuke Python discussion; Justin Fpc
Betreff: Re: [Nuke-python] nuke localise
I have wrote my own localising script from scratch just before this feature was
implemented. I will have a peek next week if I can quickly adapt it to use the
localising settings in the preferences and nodes. If so it will be threaded and
we should get the best of both worlds until the built in feature is more
flexible to allow background processing.
Sent with AquaMail for Android
http://www.aqua-mail.com
On 13 December 2013 9:58:43 PM Justin Fpc wrote:
Hi all,
I would be very interested if there is anyway to manage this localising in
background.
I've also tested to use the threading method and found the same problem/cause
as Frank.
Justin
2013/12/13 Howard Jones <[email protected]mailto:[email protected]>
Thanks for testing. That would have stumped me.
I contacted support.
Howard
On 12 Dec 2013, at 23:48, Frank Rueter
<[email protected]mailto:[email protected]> wrote:
I remember now:
I tried this a while ago myself and failed because doLocalise() is a wrapper
function using nuke.localiseFiles which seems to be compiled.
Since nuke.localiseFiles takes care of the progress bar (presumably juggling
it's own threads) it's not just a matter of using
thread = threading.Thread(target=doLocalise, args=(True,))
thread.start()
or
thread = threading.Thread(target=nuke.localiseFiles, args=(readKnobList,))
thread.start()
Both the above do the job, but you won't get the progress bar and the main
thread is still blocked.
There might be a way but I don't know how, other than basically writing the
localisation logic yourself.
So best to push that feature request to make nuke.localiseFiles thread-able.
Cheers,
frank
On 13/12/13 12:15, Frank Rueter wrote:
Yes, you should be able to. I have a quick peek...
On 13/12/13 11:29, Howard Jones wrote:
Ok done. Out of interest can this be run in a separate thread? My python brain
hasn't got round threading, but i can run doLocalise(0) so could I thread it
instead?
Howard
On 12 Dec 2013, at 22:02, Frank
Ruetermailto:[email protected]:[email protected] wrote:
I have asked for this in the pas as well, so please bug support to up the
priority ;)
On 11/12/13 05:23, Howard Jones wrote:
Hi
Is it possible to run localise from a shell or in the background?
H
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected]mailto:[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
--
check outwww.pointcloud9.com
Sebastian Elsner - Pipeline Technical Director - RISE
t: +49 30 [email protected]
f: +49 30 61651074www.risefx.com
RISE FX GmbH
Schlesische Strasse 28, Aufgang B, 10997 Berlin
c/o action concept, An der Hasenkaule 1-7, 50354 Hürth
Geschaeftsfuehrer: Sven Pannicke, Robert Pinnow
Handelsregister Berlin HRB 106667 B
------------------------------------------------------------------------
_______________________________________________
Nuke-python mailing list
[email protected], http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected],http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
------------------------------------------------------------------------
_______________________________________________
Nuke-python mailing list
[email protected], http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected], http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
[email protected], http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
