On Sun, Jan 15, 2012 at 4:22 PM, Raghavendra D Prabhu < raghu.prabh...@gmail.com> wrote:
> Hi Zheng, > > Interesting analysis. > * On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da < > zhengda1...@gmail.com> wrote: > >> Thanks. I was reading the code of kernel 3.0. XFS starts to support >> concurrent direct IO since kernel 3.1.5. >> But concurrent direct IO write still doesn't work well in kernel 3.2. >> > From what I have heard it has supported it from sometime back. I think you > may need to ask in xfs general ML about this. I didn't know this ML. I'll ask them for help. > > I wrote a test program that accesses a 4G file randomly (read and write), >> and >> I ran it with 8 threads and the machine has 8 cores. It turns out that >> only >> 1 core is running. I'm pretty sure xfs_rw_ilock is locked >> with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write. >> >> lockstat shows me that there is a lot of wait time in ip->i_lock. It seems >> the lock is locked exclusively. >> &(&ip->i_lock)->mr_lock-W: 31568 36170 >> 0.24 20048.25 7589157.99 130154 3146848 >> 0.00 217.70 1238310.72 >> &(&ip->i_lock)->mr_lock-R: 11251 11886 >> 0.24 20043.01 2895595.18 46671 526309 >> 0.00 63.80 264097.96 >> ------------------------- >> &(&ip->i_lock)->mr_lock 36170 >> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] >> &(&ip->i_lock)->mr_lock 11886 >> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] >> ------------------------- >> &(&ip->i_lock)->mr_lock 38555 >> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] >> &(&ip->i_lock)->mr_lock 9501 >> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] >> >> Then I used systemtap to instrument xfs_ilock and find there are at least >> 3 >> functions that lock ip->i_lock exclusively during write. >> > > From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there > is unaligned IO or there are cached pages to be invalidated after shared > lock is obtained *but* it demotes that lock to SHARED just before > generic_file_direct_write. Actually, there are two locks for an inode, i_lock and i_iolock. systemtap shows me that i_iolock is already locked to SHARED, but i_lock is locked exclusively somewhere else. Even though I don't think I have found the right spot that hurts concurrency so badly. Thanks, Da
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies