From: Sylvain Lebeau Sent: Tuesday, July 31, 2012 4:26 PM To: softimage@listproc.autodesk.com Subject: Re: Renderfarm options
> cool to have another chime! >When i say old blades... it doesnt mean they are old x486... ;-) they are >also Dell Poweredge Xeon servers with 6 gig ram. still very usable . > Here we mostly do beauty renders in Arnold with AOV's and a couple of puzz > matte. And renders that takes 1h on our best blades can take up to 3-4 hours > on older ones. So it's better for us to keep the power on these tasks so we > get the passes as fast as possible. It is very rare that a nuke comp takes > more then 5 minutes per frame. We also use deadline and i've never tried the > concurrent task thing. I will look into it. thanks for pointing this out.. > What's is your gain by using it in % of speed increase? well, in our setup it’s almost linear – by which I mean, when a sequence of say 100 frames with concurrent tasks at 1 might take say 100 minutes at 1min/frame, with concurrent tasks at 4 it could end up close to 25 minutes (let’s give it a margin – 30 minutes). This is true as long as: - your blades don’t run into memory limits and start to swap – at which point it will become slow as hell. with 6gb you will hit that limit quite fast – with our 24Gb machines we do 4 concurrent tasks (that’s ~6gb per task) and it almost never poses a problem (30 to 150 image readers in the nuke scripts, all HD res, half float .Exr is what we have) - your server can follow the demand – we constantly have 10 to 15 blades for comp, and can increase to 30 blades at peak times, but you will feel this on the server. That could be 120 nukes rendering, each asking a multitude of images – at some point it becomes painful for artists in their interactive sessions – and you will get longer rendertimes overall. Compositing is *much* more taxing on server and network than 3D rendering. - the comp rendering doesn’t have very high CPU usage (which happens when all or most cores are being fully used, very common for 3D rendering, not so common for comp). Look at task manager when rendering - if CPU percentage is low, and you are using just a part of available RAM, you have a good candidate for concurrent tasks. - beware of certain plugins (DOF, MB,...) because they can be very CPU intensive or not at all, thoroughly multithreaded or not – and they can mess with your carefully tweaked setup. Obviously, don’t take any of this for granted – there are so many things that influence rendering, strategies will vary for different studios. For us, concurrent tasks for Nuke renders has proven invaluable, and the Sysadmin has carefully thought about which blades this runs on (as said, these are the top machines of the farm), and the consequences it has on server and network. Doing the opposite, favoring 3D rendering, can make just as much sense – because without finished 3D renders, no need for comp renders. > as reading images from the server , As for network I/O, we have a system > called shed_latest_3d that copies the latest 3d renders to the comp server. > This way, nuke comps always source the same sequence and we dont need to > relink them and it also makes the load on the servers/switches better > balanced. sounds good – until now we had one server (HP Ibrix) for 3D and comp, holding everything for projects – assets, textures, 3D renders, comp renders, deliverables. It has served us well in the past – but we anticipate splitting to 2 or 3 servers, so we can decouple between 3D and comp – since demands for storage size and performance are quite different for both disciplines. > will try the concurrent task thing! sly -- Sylvain Lebeau // SHED V-P/Visual effects supervisor 1410, RUE STANLEY, 11E ÉTAGE MONTRÉAL (QUÉBEC) H3A 1P8 T 514 849-1555 F 514 849-5025 WWW.SHEDMTL.COM <http://WWW.SHEDMTL.COM> pete...@skynet.be Tuesday, July 31, 2012 6:16 AM Sylvain wrote: >> 35 blades total with 14 dedicated to nuke (old ones that doesnt have enough ram and power for arnold) >> recycle your old blade for comp! ... just jumping in I am surprised about this – as we do the exact opposite. we prioritize the best renderblades for Nuke, most with 24Gb ram, some more – and about any old blade will do for 3D rendering. (10-12Gb is plenty) the logic is that -during working hours- the rendered composite is what artists are actually waiting for to move their work forward – while 3D sequence rendering can usually plough on in the background. We’ve verified the 3D render on a few stills (that can even be rendered local if the artist so chooses) so we don’t really lose time waiting on the full sequence. we’re using Deadline, and something we really push with Nuke rendering is “concurrent tasks” – so each blade is rendering several frames at once – at which point having a lot of ram is crucial. This reduces total task time almost linearly – eg 4 tasks at once means the total render of a sequence will be almost 4 times faster. Another factor is that nuke rendering puts more load on the network – so we rather have a small amount of very performant blades for that, and a large amount of less performant ones for 3D rendering. Something which no doubt comes into play too, is that we are not doing beauty rendering – so we have a lot of discreet passes, none of them very slow to render – in return, our comps are more demanding, with the simplest of shots having 30 odd passes. So yeah, best blades for compositing here all the way. As for Leoung’s original question – for 3D rendering, you can never have enough cores – assuming your renderer uses them well (which mr mostly does) – so I’d be tempted by the 32 cores. That’s double the 16 cores – it’s bound to make a real difference. Only downside is that for those tasks that aren’t multithreaded you have a lot of cores idling. Wouldn’t it be cool if they designed these muticore machines with one “master” core, that would have very high clock speed, and a lot of “slave” cores that kick in for multithreading? Fast pre-render + fast render = . just my 0.02 eurocent. Sylvain Lebeau Monday, July 30, 2012 11:37 PM ok! recycle your old blade for comp! ... or for ethernet gagming!! ....Quake 2 arena is massivly comming back here.... ;-) and no i didnt compare.... i buy new shits and hope for the best upon the specs i get.. !! yeah boxx are very expensive and ... maybe overated... in my humble opinion. Maybe others could chip in to tell us... i would be very open to hear experiences with those boxxes... sly Leoung O'Young Monday, July 30, 2012 11:30 PM We render in old Mental Ray. We do have a few systems with not enough ram collecting dust . Have you compare a single intel i7 3930 (6 cores, 12 thread) with 2x Xeon X5650 I spoke to Boxx about their Renderpro 2x Xeon E5 2670 16 cores 32 thread but it comes with a price On 7/30/2012 11:01 PM, Sylvain Lebeau wrote: Sylvain Lebeau Monday, July 30, 2012 11:01 PM Hey no problem!! We only have Dell Poweredge servers in the farm. 35 blades total with 14 dedicated to nuke (old ones that doesnt have enough ram and power for arnold) ... The other ones + the artists workstations are dedicated to 3D.... Our latest machines (15 wich we named Ultras) are Xeon X5650 with 2 physical mutli-threaded processors for a total of 24 threads. Each 16 gigs of ram. They are rock solid!!!! We render everything in Arnold here and they are very performant.. Glad you've never had any problems.... i've recently read that in the Arnold or XSI list a couple weeks ago.... Maybe it is related to how your render engine is programmed too?... i dont know much about instruction sets like i've said... Are you Arnold based?. or Mental Ray? Just curious.. as you can see, my farm is not doing much right now ;-) wich is pretty temporary... Because with the load of work heading among us.... i may need 5-10 more blades soon enough.... dam it!!! sly Leoung O'Young Monday, July 30, 2012 10:37 PM Hi Sly, Thanks so much for quick responds. I had a mix of AMD Opteron and Intel Xeon previously with no problem, but that doesn't mean it won't be a problem now. Did you guys build your own computers for the farm or bought them off the shelf, are they dual processor systems? Thanks, Leoung On 7/30/2012 10:02 PM, Sylvain Lebeau wrote:
<<wlEmoticon-smile[1].png>>
<<compose-unknown-contact.jpg>>
<<image.png>>
