Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread J. R. Okajima
Al Viro: > Different ->i_mutex; you are holding one on the parent directory already. Let me make sure. In your scenario, - processA writes something into the union, and the unioning fs operates the writable layer. After sb_start_write() succeeds, processA should not block by the reason of fsf

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread Al Viro
On Sat, Mar 23, 2013 at 11:49:11AM +0900, J. R. Okajima wrote: > > Al Viro: > > The scenario, BTW, looks so: > > process A does sb_start_write() (on your upper layer) > > process B tries to freeze said upper layer and blocks, waiting for A to > > finish > > process C grabs ->i_mutex in your upper

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread J. R. Okajima
Al Viro: > The scenario, BTW, looks so: > process A does sb_start_write() (on your upper layer) > process B tries to freeze said upper layer and blocks, waiting for A to finish > process C grabs ->i_mutex in your upper layer > process C does vfs_write(), which blocks, since there's a pending attem

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread Al Viro
On Fri, Mar 22, 2013 at 06:11:11PM +, Al Viro wrote: > On Sat, Mar 23, 2013 at 02:37:55AM +0900, J. R. Okajima wrote: > > > > David Howells: > > > Now, looking at __sb_start_write(), I'm not entirely sure how the deadlock > > > might operate, so it's possible that this is a false alarm. Maybe

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread Al Viro
On Sat, Mar 23, 2013 at 02:37:55AM +0900, J. R. Okajima wrote: > > David Howells: > > Now, looking at __sb_start_write(), I'm not entirely sure how the deadlock > > might operate, so it's possible that this is a false alarm. Maybe Jan Kara > > can > > illuminate further, so I've added him to the

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-22 Thread J. R. Okajima
David Howells: > Now, looking at __sb_start_write(), I'm not entirely sure how the deadlock > might operate, so it's possible that this is a false alarm. Maybe Jan Kara > can > illuminate further, so I've added him to the cc list. It is related to the design of UnionMount, isn't it? UnionMount

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-20 Thread Jan Kara
On Wed 20-03-13 21:48:13, Al Viro wrote: > On Wed, Mar 20, 2013 at 08:52:22PM +0100, Jan Kara wrote: > > > do_bio_filebacked(), with some ugliness between that and callsite. Note, > > > BTW, that we have a pair of possible vfs_fsync() calls in there; how do > > > those > > > interact with freeze?

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-20 Thread Al Viro
On Wed, Mar 20, 2013 at 08:52:22PM +0100, Jan Kara wrote: > > do_bio_filebacked(), with some ugliness between that and callsite. Note, > > BTW, that we have a pair of possible vfs_fsync() calls in there; how do > > those > > interact with freeze? > Freezing code takes care that all dirty data i

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-20 Thread Jan Kara
On Wed 20-03-13 02:33:08, Al Viro wrote: > On Tue, Mar 19, 2013 at 10:10:32PM +, Al Viro wrote: > > > OK, it's going to be an interesting series - aforementioned tentative patch > > was badly incomplete ;-/ > > The interesting question is how far do we want to lift that. ->aio_write() > part

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-20 Thread David Howells
Al Viro wrote: > fs/cachefiles/rdwr.c:967: ret = file->f_op->write( > > cachefiles_write_page(); no fucking idea what locks might be held by caller > and potentially that's a rather nasty source of PITA The caller of cachefiles_write_page() (ie. fscache_write_op()) holds n

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-20 Thread Miklos Szeredi
On Tue, Mar 19, 2013 at 10:24 PM, Al Viro wrote: > it still might make sense to implement > something as a layer, but some parts of that sucker may be better off as > fs primitives. Hell, we could, in theory, implement xattrs as a layer; > just look at how reiserfs had done them. We could do the

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Al Viro
On Tue, Mar 19, 2013 at 10:10:32PM +, Al Viro wrote: > OK, it's going to be an interesting series - aforementioned tentative patch > was badly incomplete ;-/ The interesting question is how far do we want to lift that. ->aio_write() part is trivial - see vfs.git#experimental; the trouble beg

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Al Viro
On Tue, Mar 19, 2013 at 09:38:31PM +, Al Viro wrote: > On Tue, Mar 19, 2013 at 09:25:43PM +0100, Jan Kara wrote: > > > BTW, having sb_start_write() buried in individual ->splice_write() is > > > asking for trouble; could you describe the rules for that? E.g. where > > > does it nest wrt filesy

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Al Viro
On Tue, Mar 19, 2013 at 09:25:43PM +0100, Jan Kara wrote: > > BTW, having sb_start_write() buried in individual ->splice_write() is > > asking for trouble; could you describe the rules for that? E.g. where > > does it nest wrt filesystem-private locks? XFS iolock, for example... > Generally, th

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Al Viro
On Tue, Mar 19, 2013 at 07:32:42PM +0100, Miklos Szeredi wrote: > > * victim is negative => create a whiteout > > * victim is a directory, parent opaque => rmdir > > * victim is a non-directory, parent opaque => unlink > > * victim is positive, parent _not_ opaque =

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Jan Kara
On Mon 18-03-13 23:01:03, Al Viro wrote: > On Mon, Mar 18, 2013 at 09:53:34PM +, Al Viro wrote: > > On Mon, Mar 18, 2013 at 04:39:36PM +0100, Jan Kara wrote: > > > IMO the deadlock is real. In freeze_super() we wait for all writers to > > > the filesystem to finish while blocking beginning of

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Jan Kara
On Mon 18-03-13 21:53:34, Al Viro wrote: > On Mon, Mar 18, 2013 at 04:39:36PM +0100, Jan Kara wrote: > > IMO the deadlock is real. In freeze_super() we wait for all writers to > > the filesystem to finish while blocking beginning of any further writes. So > > we have a deadlock scenario like: > >

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Miklos Szeredi
On Tue, Mar 19, 2013 at 6:03 PM, Al Viro wrote: > On Tue, Mar 19, 2013 at 11:29:41AM +0100, Miklos Szeredi wrote: > >> Copy up is a once-in-a-lifetime event for an object. Optimizing it is >> way down in the list of things to do. I'd drop splice in a jiffy if >> it's in the way. > > What makes y

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Al Viro
On Tue, Mar 19, 2013 at 11:29:41AM +0100, Miklos Szeredi wrote: > Copy up is a once-in-a-lifetime event for an object. Optimizing it is > way down in the list of things to do. I'd drop splice in a jiffy if > it's in the way. What makes you think that write is any better? Same deadlock there -

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Miklos Szeredi
On Tue, Mar 19, 2013 at 12:04 PM, David Howells wrote: > Miklos Szeredi wrote: > >> > BTW, I wonder what's the right locking for that sucker; overlayfs is >> > probably too heavy - we are talking about copying a file from one fs to >> > another, which can obviously take quite a while, so holding

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread David Howells
Miklos Szeredi wrote: > > BTW, I wonder what's the right locking for that sucker; overlayfs is > > probably too heavy - we are talking about copying a file from one fs to > > another, which can obviously take quite a while, so holding ->i_mutex on > > _parent_ all along is asking for very serious

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread Miklos Szeredi
On Tue, Mar 19, 2013 at 2:38 AM, Al Viro wrote: > On Mon, Mar 18, 2013 at 11:01:03PM +, Al Viro wrote: > >> I'm looking at the existing callers and I really wonder if we ought to >> push sb_start_write() from ->splice_write()/->aio_write()/etc. into the >> callers. >> >> Something like file_st

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-19 Thread J. R. Okajima
Al Viro: > BTW, I wonder what's the right locking for that sucker; overlayfs is probably > too heavy - we are talking about copying a file from one fs to another, which > can obviously take quite a while, so holding ->i_mutex on _parent_ all along > is asking for very serious contention. OTOH, th

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-18 Thread Al Viro
On Mon, Mar 18, 2013 at 11:01:03PM +, Al Viro wrote: > I'm looking at the existing callers and I really wonder if we ought to > push sb_start_write() from ->splice_write()/->aio_write()/etc. into the > callers. > > Something like file_start_write()/file_end_write(), with check for file > bein

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-18 Thread Al Viro
On Mon, Mar 18, 2013 at 09:53:34PM +, Al Viro wrote: > On Mon, Mar 18, 2013 at 04:39:36PM +0100, Jan Kara wrote: > > IMO the deadlock is real. In freeze_super() we wait for all writers to > > the filesystem to finish while blocking beginning of any further writes. So > > we have a deadlock sc

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-18 Thread Al Viro
On Mon, Mar 18, 2013 at 04:39:36PM +0100, Jan Kara wrote: > IMO the deadlock is real. In freeze_super() we wait for all writers to > the filesystem to finish while blocking beginning of any further writes. So > we have a deadlock scenario like: > > THREAD1 THREAD2

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-18 Thread Jan Kara
On Sun 17-03-13 13:06:59, David Howells wrote: > Miklos Szeredi wrote: > > > Export do_splice_direct() to modules. Needed by overlay filesystem. > > Apparently you cannot call this from any function that is holding an i_mutex > if the target of the splice uses generic_file_splice_write(). > >

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-17 Thread Dave Chinner
On Sun, Mar 17, 2013 at 01:06:59PM +, David Howells wrote: > Miklos Szeredi wrote: > > > Export do_splice_direct() to modules. Needed by overlay filesystem. > > Apparently you cannot call this from any function that is holding an i_mutex > if the target of the splice uses generic_file_splic

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-17 Thread David Howells
Miklos Szeredi wrote: > Export do_splice_direct() to modules. Needed by overlay filesystem. Apparently you cannot call this from any function that is holding an i_mutex if the target of the splice uses generic_file_splice_write(). The problem is a potential deadlock situation: We have places

Re: [PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-13 Thread Andrew Morton
On Wed, 13 Mar 2013 15:16:26 +0100 Miklos Szeredi wrote: > From: Miklos Szeredi > > Export do_splice_direct() to modules. Needed by overlay filesystem. > > Signed-off-by: Miklos Szeredi > --- > fs/splice.c |1 + > 1 file changed, 1 insertion(+) > > diff --git a/fs/splice.c b/fs/splice.

[PATCH 2/9] vfs: export do_splice_direct() to modules

2013-03-13 Thread Miklos Szeredi
From: Miklos Szeredi Export do_splice_direct() to modules. Needed by overlay filesystem. Signed-off-by: Miklos Szeredi --- fs/splice.c |1 + 1 file changed, 1 insertion(+) diff --git a/fs/splice.c b/fs/splice.c index 718bd00..0e8f44a 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -1308,6