Re: [CinCVS] hardware question
Scott C. Frase wrote: On Sat, 2007-12-29 at 20:27 +0100, Christian Thaeter wrote: Cinelerra has some race problems between threads which let them wait on each other doing nothing, this is hard to fix unfortunally. But generally I think adding more CPU's will add some performance improvement. While in doubt, you may better opt for more ram (4 or more GB of ram). For long time future I we will fix this (with what is called cinelerra 3 now) to have no races and utilize multi cpu systems much bettter, but don't hold your breath yet for that. Christian Hi Christian, Yes, I continue to notice race conditions here and there. For example, after doing some basic editing on a HDV project with two stereo audio tracks and one video track, I wanted to move one audio track down in the timeline. When I did this, I triggered a race condition that hung a CPU at 100% for 30 minutes. Bummer. Hannes gave me some instruction last year on how to track these issues down using kdbg: http://crazedmuleproductions.blogspot.com/search/label/kdbg I will try to capture some debug information the next time it happens. Imo waste of time, I know some of the code and (see condition.C) it might be evolved due workaround problems in early linuxthread implementations but it is horribly ugly and broken by design. In a experimental branch I started to add my 'NoBug' library with a resource/deadlock checker to cinelerra which shown a few too. But finding this races is really the simplest thing. Fixing them is a really hard design issue, I won't say it is impossible but it is certainly not simple and very intrusive. I started to write more bullet proof locking classes for cin2 just to find out that going over the code and refactoring/adding that was more complicated than I imagined. Actually the locking problems in cinelerra are one of the main issues I proprosed cin3 as new rewrite. If you have the patience and want to make some efforts to fix it, that would be a major contribution to cinelerras stability and performance, but for myself I dont wan't to touch that code anymore currently. Christian ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
Imo waste of time, I know some of the code and (see condition.C) it might be evolved due workaround problems in early linuxthread implementations but it is horribly ugly and broken by design. In a experimental branch I started to add my 'NoBug' library with a resource/deadlock checker to cinelerra which shown a few too. But finding this races is really the simplest thing. Fixing them is a really hard design issue, I won't say it is impossible but it is certainly not simple and very intrusive. I started to write more bullet proof locking classes for cin2 just to find out that going over the code and refactoring/adding that was more complicated than I imagined. Actually the locking problems in cinelerra are one of the main issues I proprosed cin3 as new rewrite. If you have the patience and want to make some efforts to fix it, that would be a major contribution to cinelerras stability and performance, but for myself I dont wan't to touch that code anymore currently. Christian Christian, OK. Its understandable that you'd want to work on new, more resilient software. My C chops aren't that strong and I'm working on some other projects right now. I might continue sending some debug streams to bugs.cinelerra, just in case someone else has the inclination to work on this issue.. scott ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
Hi all, Just to carry on with my initial post regarding hardware. Considering that I just need power to render, not for editing (which seems to trigger the problem with multi cpus), I always thought that getting a rendering farm of PS3 could be smart ? I am most probably not aware of potential issues. So consider it as a naive question. cheers E ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
E Chalaron wrote: Hi all, Just to carry on with my initial post regarding hardware. Considering that I just need power to render, not for editing (which seems to trigger the problem with multi cpus), I always thought that getting a rendering farm of PS3 could be smart ? I am most probably not aware of potential issues. So consider it as a naive question. cheers Cinelerra is in no way optimized for Power PC or the SPE's in a Cell processor, at best you could get it working on the (main) PPC processor of a Cell CPU with moderate performance. Probably even that would give some serious headache (maybe easier, sony supports linux on cell). Adding support for the Cell SPE processors which give the magic performance boost would introduce a new code path for rendering, thats something which is somewhere between much much work and impossible in current cinelerra, I doubt anyone would waste massive developers resources to for such a very specific hardware. Even for cin3 we won't support that, it is just to much work for now. The design of cin3 should be open enough to add such things but the work for implementing such extra code paths would be massive and likely not be done anytime soon. I recognize that the advantages would be enormous but if we want some (far) future support for special hardware we should at least choose something which has an open API and will be available/compatible for the years coming. I suggest to delay this decision for the time being and then see whats there (GPU cards for general purpose computing become available now, maybe Cell blades / expansion cards, future will tell). Supporting a game console might be nice because of the low price tag, but product cycles are likely faster than we can write software for it :P Christian ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
On Thu, 2007-12-27 at 11:45 -0600, Timothy Baldridge wrote: Your disk performance may be holding you back as well. On the oil effect, you are CPU bound, on a faster effect (e.g. grayscale) you will be primarily IO bound. Here's an example. Only recently (in the past 3 years) has Discreet moved to x86. Before that they were selling their highest end product (for editing 4K film) on a quad CPU 1Ghz. But they smoked the competition when it came to performance. Why? Most of the time these systems had a 10-30 fibre channel RAID behind them. This allowed the Tezro and Onyx3 systems to edit 5 streams of real-time 4k video on a system that had basically no number crunching power. 9 times out of 10 you will be limited by your disk performance, not the CPU. I would suggest running a bonnie++ benchmark on your disks, from there you can calculate an approximate max fps processing rate for the disks. Timothy Hi Timothy, Thanks for the thought. I have been monitoring wait i/o via mpstat and it doesn't seem to be the issue. Here's recent output of mpstat using a bunch of effects on a HDV video render, including oil painting. The sixth column is iowait: 01:15:45 PM CPU %user %nice%sys %iowait%irq %soft %steal %idleintr/s 01:15:55 PM0 87.300.000.600.000.000.300.00 11.80150.20 01:15:55 PM1 87.400.000.700.000.000.000.00 11.90127.90 01:15:55 PM2 86.700.000.400.200.000.300.00 12.40150.20 01:15:55 PM3 87.710.000.300.000.000.000.00 11.99128.00 01:15:55 PM4 87.400.000.400.000.000.000.00 12.20134.30 01:15:55 PM5 87.100.000.600.000.000.000.00 12.30128.00 01:15:55 PM6 86.700.000.100.000.000.000.00 13.20134.10 01:15:55 PM7 86.800.000.200.000.000.000.00 13.00128.50 Notice that iowait is zero. I do have a couple of RAID 0 filesystems, one based on software RAID and the other hardware. scott ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
Scott C. Frase wrote: On Thu, 2007-12-27 at 11:45 -0600, Timothy Baldridge wrote: Your disk performance may be holding you back as well. On the oil effect, you are CPU bound, on a faster effect (e.g. grayscale) you will be primarily IO bound. Here's an example. Only recently (in the past 3 years) has Discreet moved to x86. Before that they were selling their highest end product (for editing 4K film) on a quad CPU 1Ghz. But they smoked the competition when it came to performance. Why? Most of the time these systems had a 10-30 fibre channel RAID behind them. This allowed the Tezro and Onyx3 systems to edit 5 streams of real-time 4k video on a system that had basically no number crunching power. 9 times out of 10 you will be limited by your disk performance, not the CPU. I would suggest running a bonnie++ benchmark on your disks, from there you can calculate an approximate max fps processing rate for the disks. Timothy Hi Timothy, Thanks for the thought. I have been monitoring wait i/o via mpstat and it doesn't seem to be the issue. Here's recent output of mpstat using a bunch of effects on a HDV video render, including oil painting. The sixth column is iowait: 01:15:45 PM CPU %user %nice%sys %iowait%irq %soft %steal %idleintr/s 01:15:55 PM0 87.300.000.600.000.000.300.00 11.80150.20 01:15:55 PM1 87.400.000.700.000.000.000.00 11.90127.90 01:15:55 PM2 86.700.000.400.200.000.300.00 12.40150.20 01:15:55 PM3 87.710.000.300.000.000.000.00 11.99128.00 01:15:55 PM4 87.400.000.400.000.000.000.00 12.20134.30 01:15:55 PM5 87.100.000.600.000.000.000.00 12.30128.00 01:15:55 PM6 86.700.000.100.000.000.000.00 13.20134.10 01:15:55 PM7 86.800.000.200.000.000.000.00 13.00128.50 Notice that iowait is zero. I do have a couple of RAID 0 filesystems, one based on software RAID and the other hardware. scott Cinelerra has some race problems between threads which let them wait on each other doing nothing, this is hard to fix unfortunally. But generally I think adding more CPU's will add some performance improvement. While in doubt, you may better opt for more ram (4 or more GB of ram). For long time future I we will fix this (with what is called cinelerra 3 now) to have no races and utilize multi cpu systems much bettter, but don't hold your breath yet for that. Christian ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
On Sat, 2007-12-29 at 20:27 +0100, Christian Thaeter wrote: Cinelerra has some race problems between threads which let them wait on each other doing nothing, this is hard to fix unfortunally. But generally I think adding more CPU's will add some performance improvement. While in doubt, you may better opt for more ram (4 or more GB of ram). For long time future I we will fix this (with what is called cinelerra 3 now) to have no races and utilize multi cpu systems much bettter, but don't hold your breath yet for that. Christian Hi Christian, Yes, I continue to notice race conditions here and there. For example, after doing some basic editing on a HDV project with two stereo audio tracks and one video track, I wanted to move one audio track down in the timeline. When I did this, I triggered a race condition that hung a CPU at 100% for 30 minutes. Bummer. Hannes gave me some instruction last year on how to track these issues down using kdbg: http://crazedmuleproductions.blogspot.com/search/label/kdbg I will try to capture some debug information the next time it happens. scott ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
[CinCVS] hardware question
Hello Just wondering about a config for a new machine. With the quad cores coming up I am wondering if I should invest in an average quad core or a good duo, being tied to a budget. Basically my question is would a FBS 1333 with a duo core 3Ghz be faster than a 2.4 Ghz quad core 1066 FBS, all other thing being equal? Most of my needs are : unsharp, histogram correction, saturation on progresive frames of 1400x1040. Cheers and happy new year to all. E. ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
Hi E, My opinions are based upon only twenty-four hours of real editing time, so I haven't had the chance to test all of Cinelerra's effects, but I'll give you a quick summary of my impressions. I bought a Dell SC1430, 64-bit Intel dual quad core running at 1.6Ghz. Since Cinelerra is optimized for multiple CPUs (cores), I thought the more the better, right? Well..the reality is a little different. For specific video effects that are really CPU intensive, like the Oil Painting effect, all processors are utilized at 100%. This is great. However, I've noticed that for most editing and project rendering tasks (using a project with simple transitions/fades/keyframes/effects like histogram), Cinelerra only utilizes a small portion of each CPU, roughly 30% at peak. This bugged me, because I had hoped that all CPUs should be utilized 100% all the time during any task. But my opinion was based on a naive view of how Cinelerra utilizes system resources. One nice thing about having lots of cores is to use them for background rendering, a feature I never used on my single core box because it took up too much CPU. Background rendering allows you to see the effects you've applied to the timeline, rendered in realtime. The realtime part of this doesn't really happen for me; even with eight cores, there is lag time before I can playback the pre-rendered hdv video. I believe the bottleneck in this case is slow CPU speed of each core: 1.6Ghz in the case of my Dell. My thought so far is that if I had to do it again, I'd want a dual, dual core running at 3.2Ghz. That way, I get the benefits of multicores, plus the benefits of fast CPUs. Also, I'd probably go for an AMD Opteron multicore, because the AMD hyper-transport technology has much better throughput than the Intel Front Side Bus technology. This article in Linux Journal was very instructive: http://64.233.169.104/search?q=cache:R95w9nhv680J:www.linux-mag.com/microsites.php%3Fsite%3Dbusiness-class-hpc%26sid%3Dmain%26p%3D4183+site:linux-mag.com+opteron+quad+core+intel+clovertonhl=enct=clnkcd=2gl=uslr=lang_enclient=firefox-a Unlike a single FSB, hyper-transport allows more total memory bandwidth throughout the system and there is less contention for memory. I suspect that Cinelerra would perform much better on an AMD system with less memory bandwidth constraints than the Intel Clovertons I use. In sum, I think the most important factors to you should be the CPU speed of each core and the memory throughput of the server architecture. To answer your question specifically, my observations lead me to believe that perhaps an FSB 1333 with duo core 3Ghz would better utilize each CPU than a slower CPU FSB quad core. But look into the AMDs. I'm willing to bet that they'd give you much better performance. scott On Thu, 2007-12-27 at 21:01 +1300, E Chalaron wrote: Hello Just wondering about a config for a new machine. With the quad cores coming up I am wondering if I should invest in an average quad core or a good duo, being tied to a budget. Basically my question is would a FBS 1333 with a duo core 3Ghz be faster than a 2.4 Ghz quad core 1066 FBS, all other thing being equal? Most of my needs are : unsharp, histogram correction, saturation on progresive frames of 1400x1040. Cheers and happy new year to all. E. ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
Re: [CinCVS] hardware question
Your disk performance may be holding you back as well. On the oil effect, you are CPU bound, on a faster effect (e.g. grayscale) you will be primarily IO bound. Here's an example. Only recently (in the past 3 years) has Discreet moved to x86. Before that they were selling their highest end product (for editing 4K film) on a quad CPU 1Ghz. But they smoked the competition when it came to performance. Why? Most of the time these systems had a 10-30 fibre channel RAID behind them. This allowed the Tezro and Onyx3 systems to edit 5 streams of real-time 4k video on a system that had basically no number crunching power. 9 times out of 10 you will be limited by your disk performance, not the CPU. I would suggest running a bonnie++ benchmark on your disks, from there you can calculate an approximate max fps processing rate for the disks. Timothy On Dec 27, 2007 10:55 AM, Scott C. Frase [EMAIL PROTECTED] wrote: Hi E, My opinions are based upon only twenty-four hours of real editing time, so I haven't had the chance to test all of Cinelerra's effects, but I'll give you a quick summary of my impressions. I bought a Dell SC1430, 64-bit Intel dual quad core running at 1.6Ghz. Since Cinelerra is optimized for multiple CPUs (cores), I thought the more the better, right? Well..the reality is a little different. For specific video effects that are really CPU intensive, like the Oil Painting effect, all processors are utilized at 100%. This is great. However, I've noticed that for most editing and project rendering tasks (using a project with simple transitions/fades/keyframes/effects like histogram), Cinelerra only utilizes a small portion of each CPU, roughly 30% at peak. This bugged me, because I had hoped that all CPUs should be utilized 100% all the time during any task. But my opinion was based on a naive view of how Cinelerra utilizes system resources. One nice thing about having lots of cores is to use them for background rendering, a feature I never used on my single core box because it took up too much CPU. Background rendering allows you to see the effects you've applied to the timeline, rendered in realtime. The realtime part of this doesn't really happen for me; even with eight cores, there is lag time before I can playback the pre-rendered hdv video. I believe the bottleneck in this case is slow CPU speed of each core: 1.6Ghz in the case of my Dell. My thought so far is that if I had to do it again, I'd want a dual, dual core running at 3.2Ghz. That way, I get the benefits of multicores, plus the benefits of fast CPUs. Also, I'd probably go for an AMD Opteron multicore, because the AMD hyper-transport technology has much better throughput than the Intel Front Side Bus technology. This article in Linux Journal was very instructive: http://64.233.169.104/search?q=cache:R95w9nhv680J:www.linux-mag.com/microsites.php%3Fsite%3Dbusiness-class-hpc%26sid%3Dmain%26p%3D4183+site:linux-mag.com+opteron+quad+core+intel+clovertonhl=enct=clnkcd=2gl=uslr=lang_enclient=firefox-a Unlike a single FSB, hyper-transport allows more total memory bandwidth throughout the system and there is less contention for memory. I suspect that Cinelerra would perform much better on an AMD system with less memory bandwidth constraints than the Intel Clovertons I use. In sum, I think the most important factors to you should be the CPU speed of each core and the memory throughput of the server architecture. To answer your question specifically, my observations lead me to believe that perhaps an FSB 1333 with duo core 3Ghz would better utilize each CPU than a slower CPU FSB quad core. But look into the AMDs. I'm willing to bet that they'd give you much better performance. scott On Thu, 2007-12-27 at 21:01 +1300, E Chalaron wrote: Hello Just wondering about a config for a new machine. With the quad cores coming up I am wondering if I should invest in an average quad core or a good duo, being tied to a budget. Basically my question is would a FBS 1333 with a duo core 3Ghz be faster than a 2.4 Ghz quad core 1066 FBS, all other thing being equal? Most of my needs are : unsharp, histogram correction, saturation on progresive frames of 1400x1040. Cheers and happy new year to all. E. ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra ___ Cinelerra mailing list Cinelerra@skolelinux.no https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra -- If you put tomfoolery into a computer, nothing comes out but tomfoolery. But this tomfoolery, having passed through a very expensive machine, is somehow ennobled and no one dares criticize it. (Pierre Gallois) ___ Cinelerra mailing list Cinelerra@skolelinux.no
