Re: [Mjpeg-users] lav2wav | sox poor performance
Hi Brian, On Mon, 21 Jun 2004, Brian J. Murrell wrote: > $ lav2wav file.eli | sox -t wav - -t wav /dev/null stat -v > but both lav2wav and sox both use up negligible CPU in doing this job I think people (in the past) have said that adding 'buffer' in the middle could significantly improve this. It's (like the name says) a buffering utility. Ronald --- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND ___ Mjpeg-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/mjpeg-users
Re: [Mjpeg-users] lav2wav | sox poor performance
On Mon, Jun 21, 2004 at 09:42:52AM -0400, Brian J. Murrell wrote: > ... > So my command line is: > > $ lav2wav file.eli | sox -t wav - -t wav /dev/null stat -v > > but both lav2wav and sox both use up negligible CPU in doing this > job, and it takes far too long. lav2wav's CPU usage is about 2% > and sox's CPU usage is about 4.5%. Seems like there is a > bottleneck somewhere but where I am not sure. I have tried putting > a "buffer" between them but no help. Check your disk I/O read bandwidth. Lav2wav is heavily read I/O bandwidth bound. It sounds like your disk read bandwidth is way low for some reason, and therefore lav2wav is not creating PCM data very quickly, which means there is very little data for sox to work with. Both result in very low CPU utilization. --- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND ___ Mjpeg-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/mjpeg-users
Re: [Mjpeg-users] lav2wav | sox poor performance
On Mon, 2004-06-21 at 10:28 -0400, Richard Ellis wrote: > Check your disk I/O read bandwidth. Lav2wav is heavily read I/O > bandwidth bound. It sounds like your disk read bandwidth is way low > for some reason, That was one of the first things I checked. This is a UDMA5 capable drive although the IDE interface is only UDMA4 capable, so it's running UDMA4 right now. But hdparm's performance test looks good: /dev/hda: Timing buffer-cache reads: 636 MB in 2.00 seconds = 318.00 MB/sec Timing buffered disk reads: 124 MB in 3.02 seconds = 41.06 MB/sec > and therefore lav2wav is not creating PCM data very > quickly, which means there is very little data for sox to work with. > Both result in very low CPU utilization. Right. I don't think this is the situation in this case. The wav data file created from a lav2wav is 317184228 bytes. At 40MB/sec that is only (317184228 / 1024 / 1024 / 40) 7.5 seconds. The actual: $ time lav2wav file.eli > file.wav INFO: [lav2wav] WAV done 0.58user 12.01system 1:49.86elapsed 11%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (204major+938minor)pagefaults 0swaps It's worth noting that lav2rec does not really read/write that much data per read() syscall. I wonder if this is the cause. ... read(4, "9\6\372\v:\6\340\v\272\6\343\v\234\7\314\v\217\10\246\v"..., 3676) = 3676 read(4, "M\0\275\372\241\377\371\372\367\376D\373\207\376\271\373"..., 2212) = 2212 write(1, "9\6\372\v:\6\340\v\272\6\343\v\234\7\314\v\217\10\246\v"..., 5888) = 5888 read(4, "~\376\344\372\t\376\347\372o\375\371\372\365\374\'\373"..., 1884) = 1884 read(4, "\256\373\235\375\342\372-\375y\372\272\374\213\372?\374"..., 4000) = 4000 write(1, "~\376\344\372\t\376\347\372o\375\371\372\365\374\'\373"..., 5884) = 5884 read(4, "?\7\366\7\245\7S\10\252\7\244\10\302\7\367\10\377\7N\t"..., 96) = 96 read(4, "A\n\247\0046\n\216\4-\n\235\4-\n\305\4\372\t\356\4\243"..., 4096) = 4096 read(4, "\332\16}\r\35\17\17\23>\16?\24\230\ft\16~\v\230\10\243"..., 1692) = 1692 write(1, "?\7\366\7\245\7S\10\252\7\244\10\302\7\367\10\377\7N\t"..., 5884) = 5884 read(4, "m\4\v\n\'\4G\t\364\3\204\t}\3\25\t\312\2\262\6\257\1\270"..., 2404) = 2404 read(4, "\341\3\v\f2\5\251\r\362\5r\16\f\6\341\16\246\5\317\16-"..., 3484) = 3484 write(1, "m\4\v\n\'\4G\t\364\3\204\t}\3\25\t\312\2\262\6\257\1\270"..., 5888) = 5888 read(4, "6\6\213\16K\6$\17M\6a\16a\6\326\r\"\6\366\r0\6\24\16\345"..., 612) = 612 read(4, "\201\370\315\374\326\371u\376\252\373\'\376\0\375\271\376"..., 4096) = 4096 read(4, "\35\2<\372\31\1\t\371\345\377.\367s\375#\366\t\373\234"..., 1176) = 1176 write(1, "6\6\213\16K\6$\17M\6a\16a\6\326\r\"\6\366\r0\6\24\16\345"..., 5884) = 5884 read(4, "\30\7\335\r4\7\2\16l\7a\rs\7C\f\250\6;\v\32\5\311\n\260"..., 2920) = 2920 read(4, "V\374i\375\204\375\340\376\261\376\247\375R\376\204\372"..., 2968) = 2968 write(1, "\30\7\335\r4\7\2\16l\7a\rs\7C\f\250\6;\v\32\5\311\n\260"..., 5888) = 5888 read(4, "\216\375\220\375\360\374\356\377\21\376\344\1\242\377\240"..., 1128) = 1128 read(4, "\21\2\247\3r\1T\3\r\1\253\3B\1O\3\335\1\234\1\25\2\6\0"..., 4096) = 4096 read(4, "\302\372\2\365\1\375\342\367g\376j\373h\376\"\376\337\373"..., 660) = 660 write(1, "\216\375\220\375\360\374\356\377\21\376\344\1\242\377\240"..., 5884) = 5884 read(4, "[EMAIL PROTECTED]"..., 3436) = 3436 read(4, "\177\370\317\361-\371\344\360T\372\10\362\202\373\21\363"..., 2452) = 2452 write(1, "[EMAIL PROTECTED]"..., 5888) = 5888 read(4, "\356\3704\375:\370\261\373\206\367\307\371\34\366B\367"..., 1644) = 1644 read(4, "\312\16[\364\353\16\321\371\252\16\n\376M\16\24\0\215\16"..., 4096) = 4096 read(4, "\356\373\253\365w\372\343\363}\371b\364\313\370=\363\265"..., 144) = 144 write(1, "\356\3704\375:\370\261\373\206\367\307\371\34\366B\367"..., 5884) = 5884 ... Thots? b. signature.asc Description: This is a digitally signed message part
Re: [Mjpeg-users] lav2wav | sox poor performance
On Mon, Jun 21, 2004 at 11:11:09AM -0400, Brian J. Murrell wrote: > On Mon, 2004-06-21 at 10:28 -0400, Richard Ellis wrote: > > > Check your disk I/O read bandwidth. Lav2wav is heavily read I/O > > bandwidth bound. ... > > /dev/hda: > Timing buffer-cache reads: 636 MB in 2.00 seconds = 318.00 MB/sec > Timing buffered disk reads: 124 MB in 3.02 seconds = 41.06 MB/sec That should be more than enough, as long as you've not got something in the background consuming 40.06MB/sec of read bandwidth, or something else writing loads of data to disk at the same time. > Thots? Your original command line was writing the sox output to /dev/null. That shouldn't be a time consumer, but as a quick check, switch to writing to a read file on disk instead. Does that change anything? Also, you might want to try running a buffer between the two processes, that might help as well. That would let lav2wav run ahead of sox instead of having to stop each time the pipeline buffer gets full. --- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND ___ Mjpeg-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/mjpeg-users
Re: [Mjpeg-users] lav2wav | sox poor performance
On Mon, 2004-06-21 at 11:26 -0400, Richard Ellis wrote: > > That should be more than enough, as long as you've not got something > in the background consuming 40.06MB/sec of read bandwidth, or > something else writing loads of data to disk at the same time. Right. > Your original command line was writing the sox output to /dev/null. Right but since then I have simplified the test to just lav2wav sucking. See my previous e-mail: $ time lav2wav file.eli > file.wav INFO: [lav2wav] WAV done 0.58user 12.01system 1:49.86elapsed 11%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (204major+938minor)pagefaults 0swaps > That shouldn't be a time consumer, but as a quick check, switch to > writing to a read file on disk instead. Does that change anything? As above, the lav2wav alone takes almost 2 minutes. > Also, you might want to try running a buffer between the two > processes, that might help as well. Right, once lav2wav is taking all of the CPU I can throw at it and the pipeline to sox still sucks, I will. But right now, lav2rec seems to be the bottleneck. b. signature.asc Description: This is a digitally signed message part
Re: [Mjpeg-users] lav2wav | sox poor performance
On Mon, Jun 21, 2004 at 03:25:54PM -0400, Brian J. Murrell wrote: > Right but since then I have simplified the test to just lav2wav sucking. > See my previous e-mail: > > $ time lav2wav file.eli > file.wav >INFO: [lav2wav] WAV done > 0.58user 12.01system 1:49.86elapsed 11%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (204major+938minor)pagefaults 0swaps > > > That shouldn't be a time consumer, but as a quick check, switch to > > writing to a read file on disk instead. Does that change anything? > > As above, the lav2wav alone takes almost 2 minutes. That's not at all surprising. Time how long this one would take: dd if=/your/video/file/location of=/dev/null bs=64k Run it against the full capture file, all multiple gigabytes of capture. I'd bet it takes about 2 minutes just to read the capture data off your disk. I've seen this effect all too many times in the past when encoding lavrec recordings to mpeg. Lav2wav will saturate your disk read bandwidth, but maybe only 1/10 to 1/20 the disk read bandwidth will trickle out the other end as wav data. I suspect it's an internal difficulty in lav2wav, something to do with the way it reads the wav data from the avi container. I suspect it's reading every byte of the file, demuxing it internally, and then throwing away the video data and outputting the audio, but I've never dug deep enough into it's code to divine exactly how it works. Try running a vmstat on a one or two second delay in another window while you run your lav2wav test. I'll bet you see pretty close to your 40MB/s of read bandwidth being consumed for the entire 2 minutes of time lav2wav takes to create the wav file. > > Also, you might want to try running a buffer between the two > > processes, that might help as well. > > Right, once lav2wav is taking all of the CPU I can throw at it and > the pipeline to sox still sucks, I will. But right now, lav2rec > seems to be the bottleneck. And without some code changes inside lav2wav it's likely to always be the bottleneck. I never bothered reporting it as a problem myself, figuring that 5-20 minutes to encode up an mp2 was paltry compared to several hours or more to code up the mpeg video. Often I'd run the audio encode after I'd started video coding, so with the overlap, the time loss to a slow lav2wav was essentially zero. --- This SF.Net email sponsored by Black Hat Briefings & Training. Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com ___ Mjpeg-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/mjpeg-users
