Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-15 Thread Richard A. Smith
On 05/13/2010 08:07 AM, Benjamin M. Schwartz wrote: On 05/13/2010 03:26 AM, James Cameron wrote: We discussed this briefly in team meeting this morning ... Record is writing an uncompressed stream to SD, before then compressing it in the Save step once the video recording is stopped. (I could

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-15 Thread John Watlington
Given that the main memory is one eight to one quarter the size of the storage medium, I'm surprised at the effort being invested in recording straight to the storage medium. Especially if that storage medium is arbitrarily slow! Cheers, wad On May 15, 2010, at 7:02 AM, Richard A. Smith wrote:

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-13 Thread James Cameron
On Wed, May 12, 2010 at 10:31:50AM -0400, Martin Langhoff wrote: On Wed, May 12, 2010 at 1:46 AM, James Cameron qu...@laptop.org wrote: With the 250Mb test I've been doing, this causes 8 seconds in dd, 147 seconds in sync, and a nice distribution of write latencies (largest samples 0.2s

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-13 Thread Benjamin M. Schwartz
On 05/13/2010 03:26 AM, James Cameron wrote: We discussed this briefly in team meeting this morning ... Record is writing an uncompressed stream to SD, before then compressing it in the Save step once the video recording is stopped. (I could be wrong). This means that no matter what the

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-12 Thread Martin Langhoff
On Wed, May 12, 2010 at 1:46 AM, James Cameron qu...@laptop.org wrote: With the 250Mb test I've been doing, this causes 8 seconds in dd, 147 seconds in sync, and a nice distribution of write latencies (largest samples 0.2s 0.1s 44ms 15ms 8ms, median 85us, smallest sample 80us). Great! Must say

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-12 Thread Tiago Marques
On Tue, May 11, 2010 at 12:47 AM, James Cameron qu...@laptop.org wrote: On Mon, May 10, 2010 at 05:33:07PM +0100, Tiago Marques wrote: On Mon, May 10, 2010 at 7:37 AM, James Cameron qu...@laptop.org wrote: On Sat, May 08, 2010 at 05:24:00PM +, Tiago Marques wrote: I was just

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread James Cameron
On Mon, May 10, 2010 at 11:14:33PM -0400, Martin Langhoff wrote: You're definitely onto something there. Do old images with earlier kernels show the problem? IOWs is there a reasonable starting point for a bisection? Sorry, no. This symptom has been with us always. Some writes just take a

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Daniel Drake
On 11 May 2010 04:19, James Cameron qu...@laptop.org wrote: On Mon, May 10, 2010 at 11:14:33PM -0400, Martin Langhoff wrote: You're definitely onto something there. Do old images with earlier kernels show the problem? IOWs is there a reasonable starting point for a bisection? Sorry, no.  

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Martin Langhoff
On Tue, May 11, 2010 at 7:08 AM, Daniel Drake d...@laptop.org wrote: I previously went even deeper: http://dev.laptop.org/ticket/9688 Um, ugly. Wad seems to blame the SD card there -- his scripts possibly favour cards with longer life, even if writes are slow. Does this get better with a Class

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Daniel Drake
On 11 May 2010 10:15, Martin Langhoff martin.langh...@gmail.com wrote: On Tue, May 11, 2010 at 7:08 AM, Daniel Drake d...@laptop.org wrote: I previously went even deeper: http://dev.laptop.org/ticket/9688 Um, ugly. Wad seems to blame the SD card there -- his scripts possibly favour cards

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Mikus Grinbergs
Does this get better with a Class 6 card? Yes, you then get consistent high performance. No need to run any tests, just use the system and note its smoothness. I've been running a class 6 card (by PQI) in my B2 XO-1.5 for about two weeks now. While there is an improvement in responsiveness,

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread John Gilmore
There's a classic Unix problem with distribution of disk access times that relates to how older Unixes did sync -- every 30 seconds there was an instant traffic jam at the interface to the drives. This has been studied to death; here are some assorted papers:

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread John Watlington
On May 11, 2010, at 3:55 PM, Mikus Grinbergs wrote: Does this get better with a Class 6 card? Yes, you then get consistent high performance. No need to run any tests, just use the system and note its smoothness. I've been running a class 6 card (by PQI) in my B2 XO-1.5 for about two

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread John Watlington
On May 11, 2010, at 5:23 PM, John Gilmore wrote: There's a classic Unix problem with distribution of disk access times that relates to how older Unixes did sync -- every 30 seconds there was an instant traffic jam at the interface to the drives. This has been studied to death; here are some

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread James Cameron
On Tue, May 11, 2010 at 07:14:53PM -0400, John Watlington wrote: Writes in UNIX are typically asynchronous --- get the data into a memory buffer and the program can continue along its merry way. I strongly suspect that writes through the SD system are forced to wait for the media to complete

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread John Watlington
On May 11, 2010, at 7:24 PM, James Cameron wrote: On Tue, May 11, 2010 at 07:14:53PM -0400, John Watlington wrote: Writes in UNIX are typically asynchronous --- get the data into a memory buffer and the program can continue along its merry way. I strongly suspect that writes through the SD

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread James Cameron
On Tue, May 11, 2010 at 07:30:15PM -0400, John Watlington wrote: On May 11, 2010, at 7:24 PM, James Cameron wrote: On Tue, May 11, 2010 at 07:14:53PM -0400, John Watlington wrote: Writes in UNIX are typically asynchronous --- get the data into a memory buffer and the program can continue

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Nate Theis
Anyone want to test with a vanilla kernel? A plain Fedora userspace? On Tue, May 11, 2010 at 4:24 PM, James Cameron qu...@laptop.org wrote: On Tue, May 11, 2010 at 07:14:53PM -0400, John Watlington wrote: Writes in UNIX are typically asynchronous --- get the data into a memory buffer and

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread Martin Langhoff
On Tue, May 11, 2010 at 7:56 PM, James Cameron qu...@laptop.org wrote: No.  Most writes pass in a time consistent with caching, and the latency is so small that I'm certain they aren't reaching the media (60us). It's only some writes that are delayed by 600ms to 3000ms. What you are saying is

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread James Cameron
On Tue, May 11, 2010 at 10:16:13PM -0400, Martin Langhoff wrote: What you are saying is that this is a different issue from what dsd tracked down -- entirely possible that these are 2 orthogonal issues. Yes, entirely possible. Does the frequency and/or std deviation change if you tweak the

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread John Watlington
Reading the patch, it sounds like a better set of parameters to try would be: vm.dirty_background_ratio=10 vm.dirty_bytes=90 Cheers, wad On May 11, 2010, at 10:41 PM, James Cameron wrote: On Tue, May 11, 2010 at 10:16:13PM -0400, Martin Langhoff wrote: What you are saying is that this is a

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-11 Thread James Cameron
On Tue, May 11, 2010 at 11:29:08PM -0400, John Watlington wrote: Reading the patch, it sounds like a better set of parameters to try would be: vm.dirty_background_ratio=10 vm.dirty_bytes=90 dirty_bytes of 90 is rounded down to nearest page size; in this case zero. Instead, I suspect you

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread James Cameron
On Sat, May 08, 2010 at 05:24:00PM +, Tiago Marques wrote: I was just looking into this and the issue with SD cards' random write performance intrigued me in this particular case. Isn't recording being performed sequentially, hence no lag problem? At the moment, assuming an application

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread Tiago Marques
On Mon, May 10, 2010 at 7:37 AM, James Cameron qu...@laptop.org wrote: On Sat, May 08, 2010 at 05:24:00PM +, Tiago Marques wrote: I was just looking into this and the issue with SD cards' random write performance intrigued me in this particular case. Isn't recording being performed

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread Martin Langhoff
On Mon, May 10, 2010 at 2:37 AM, James Cameron qu...@laptop.org wrote: I've no explanation for this behaviour. Do we still have some (sugar?) log files opened 'sync'? ext3/4 still 'fsync' the whole partition, which may explain the erratic performance. Linus has been railing against this (with

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread Tiago Marques
On Mon, May 10, 2010 at 5:45 PM, Martin Langhoff martin.langh...@gmail.comwrote: On Mon, May 10, 2010 at 2:37 AM, James Cameron qu...@laptop.org wrote: I've no explanation for this behaviour. Do we still have some (sugar?) log files opened 'sync'? ext3/4 still 'fsync' the whole partition,

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread James Cameron
On Mon, May 10, 2010 at 05:33:07PM +0100, Tiago Marques wrote: On Mon, May 10, 2010 at 7:37 AM, James Cameron qu...@laptop.org wrote: On Sat, May 08, 2010 at 05:24:00PM +, Tiago Marques wrote: I was just looking into this and the issue with SD cards' random write performance

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread James Cameron
On Mon, May 10, 2010 at 12:45:45PM -0400, Martin Langhoff wrote: On Mon, May 10, 2010 at 2:37 AM, James Cameron qu...@laptop.org wrote: I've no explanation for this behaviour. Do we still have some (sugar?) log files opened 'sync'? ext3/4 still 'fsync' the whole partition, which may explain

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-10 Thread Martin Langhoff
On Mon, May 10, 2010 at 8:23 PM, James Cameron qu...@laptop.org wrote: No, no Sugar log files are opened O_SYNC at the moment.  I fixed that. The test also reproduces without Sugar running, and also in single user mode.  This excludes Sugar and all user-space, suggesting the problem is

Re: #10045 HIGH 1.5-sof: XO-1.5 Record audio/video are out of sync with each other

2010-05-08 Thread Tiago Marques
Hi, I was just looking into this and the issue with SD cards' random write performance intrigued me in this particular case. Isn't recording being performed sequentially, hence no lag problem? At least, it should be somewhat mitigated, filesystem might be causing problems? Anyhow, I have been