subject:"Re\: nfs race condition"

Re: nfs race condition

2015-10-23 Thread Nicolas Parpandet


>> does anybody have a solution ?
> 
> Fix 'crontab' to close before unlinking...
> 
> Is this the Debian/Ubuntu crontab program it looks a bit like it,
> but this bug was fixed 6 years ago.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413962

I'm using debian 7, the bug still exist with 3.0pl1-124, this case can be easly 
solved I think, I'm going to try the debian 8 version.

> Or use a tmpfs for /tmp/ rather than using NFS???

good idea, but mysql create temp files also in /var/lib/mysql (yes there is a 
tmpdir parameter, but the bug is also present with /var/lib/mysql/#sql files)

> 
> NeilBrown
>

Nicolas
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: nfs race condition

2015-10-23 Thread Nicolas Parpandet


>> does anybody have a solution ?
> 
> Fix 'crontab' to close before unlinking...
> 
> Is this the Debian/Ubuntu crontab program it looks a bit like it,
> but this bug was fixed 6 years ago.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413962

I'm using debian 7, the bug still exist with 3.0pl1-124, this case can be easly 
solved I think, I'm going to try the debian 8 version.

> Or use a tmpfs for /tmp/ rather than using NFS???

good idea, but mysql create temp files also in /var/lib/mysql (yes there is a 
tmpdir parameter, but the bug is also present with /var/lib/mysql/#sql files)

> 
> NeilBrown
>

Nicolas
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: nfs race condition

2015-10-22 Thread Neil Brown

Nicolas Parpandet  writes:

> Hello,
>
> I did some more investigations, (wireshark)
>
> The unlink of /tmp/crontab.vYPoHR/crontab is tranlated into a rename
> nfs call ? (.nfs file)

That happens if the file is still open.

>
> NFS   670 V3 READDIRPLUS Reply (Call In 731) crontab .. .
> NFS   210 V3 LOOKUP Call (Reply In 734), DH: 
> 0x038abb3d/.nfs0098008a0019
> NFS   194 V3 LOOKUP Reply (Call In 733) Error: NFS3ERR_NOENT
> NFS   254 V3 RENAME Call (Reply In 736), From DH: 0x038abb3d/crontab To 
> DH: 0x038abb3d/.nfs0098008a0019
> NFS   338 V3 RENAME Reply (Call In 735)
>
> and after, it tries to remove the directory /tmp/crontab.0wuoNx :
> NFS   198 V3 RMDIR Call (Reply In 738), DH: 0x954c28ec/crontab.0wuoNx
> NFS   222 V3 RMDIR Reply (Call In 737) Error: NFS3ERR_NOTEMPTY
>
> but the file /tmp/crontab.0wuoNx/crontab was renamed instead of being removed 
> (nfs network, I cannot see .nfs000 file in fact),
> this prevent the removing of the directory  /tmp/crontab.0wuoNx (notempty)
>
> It seems a race condition, see "nfs silly rename" keyword, but I have
> the problem with mysql temp files, and crontab temp file (crontab -e),

I don't think it is a race condition exactly.  More of an ordering
problem.

Unlinking a file before closing it is rarely useful (sometimes it is,
but not here).

> am I the only one doing mysql and crontabs over NFS !!!?!?,

Probably.

> does anybody have a solution ?

Fix 'crontab' to close before unlinking...

Is this the Debian/Ubuntu crontab program it looks a bit like it,
but this bug was fixed 6 years ago.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413962

Or use a tmpfs for /tmp/ rather than using NFS???

NeilBrown

>
>
> Regards
>
> Nicolas
>
>
> - Mail original -
> De: "npa" 
> À: linux-kernel@vger.kernel.org
> Cc: "pcoustillas" 
> Envoyé: Dimanche 4 Octobre 2015 09:57:26
> Objet: nfs race condition
>
> Hello, 
>
> I'm doing some " crontab -e, exit with :x" on NFS mount point, 
> and there is a race condition whent removing temporary files : 
>
>
> 7701 write(2, "No modification made\n", 21) = 21 
> 7701 open("/tmp/crontab.vYPoHR", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 
> 4 
> 7701 getdents(4, {{d_ino=73336409, d_off=1, d_reclen=24, d_name="."} 
> {d_ino=73335455, d_off=2, d_reclen=24, d_name=".."} {d_ino=73340547, d_off=3, 
> d_reclen=32, d_name="crontab"}}, 32768) = 80 
> 7701 unlink("/tmp/crontab.vYPoHR/crontab") = 0 
> 7701 getdents(4, {}, 32768) = 0 
> 7701 close(4) = 0 
> 7701 rmdir("/tmp/crontab.vYPoHR") = -1 ENOTEMPTY (Directory not empty) 
> 7701 write(2, "/tmp/crontab.vYPoHR: Directory n"..., 41) = 41 
> 7701 socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4
>
> The unlink is really done after the rmdir,
> nfsv4 & nfsv3 over ipv6, so the rmdir fails and the directory isn't removed.
> I have same problem with mysql temporary files.
>
> If I mount with "nocto,noac,lookupcache=none" : same problem
> kernel version on server : 3.16.0-4-amd64 (debian8)
> kernel version on client : 2.6.32-39-pve (proxmox), same thing with 
> 3.13.0-63-generic (ubuntu)
>
> Please answer me in CC, I'm not subscribed to the list.
>
> Regards
>
> Nicolas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


signature.asc
Description: PGP signature

Re: nfs race condition

2015-10-22 Thread Nicolas Parpandet


Hello,

I did some more investigations, (wireshark)

The unlink of /tmp/crontab.vYPoHR/crontab is tranlated into a rename nfs call ? 
(.nfs file)

NFS 670 V3 READDIRPLUS Reply (Call In 731) crontab .. .
NFS 210 V3 LOOKUP Call (Reply In 734), DH: 
0x038abb3d/.nfs0098008a0019
NFS 194 V3 LOOKUP Reply (Call In 733) Error: NFS3ERR_NOENT
NFS 254 V3 RENAME Call (Reply In 736), From DH: 0x038abb3d/crontab To 
DH: 0x038abb3d/.nfs0098008a0019
NFS 338 V3 RENAME Reply (Call In 735)

and after, it tries to remove the directory /tmp/crontab.0wuoNx :
NFS 198 V3 RMDIR Call (Reply In 738), DH: 0x954c28ec/crontab.0wuoNx
NFS 222 V3 RMDIR Reply (Call In 737) Error: NFS3ERR_NOTEMPTY

but the file /tmp/crontab.0wuoNx/crontab was renamed instead of being removed 
(nfs network, I cannot see .nfs000 file in fact),
this prevent the removing of the directory  /tmp/crontab.0wuoNx (notempty)

It seems a race condition, see "nfs silly rename" keyword, but I have the 
problem with mysql temp files, and crontab temp file (crontab -e),
am I the only one doing mysql and crontabs over NFS !!!?!?, does anybody have a 
solution ?


Regards

Nicolas


- Mail original -
De: "npa" 
À: linux-kernel@vger.kernel.org
Cc: "pcoustillas" 
Envoyé: Dimanche 4 Octobre 2015 09:57:26
Objet: nfs race condition

Hello, 

I'm doing some " crontab -e, exit with :x" on NFS mount point, 
and there is a race condition whent removing temporary files : 


7701 write(2, "No modification made\n", 21) = 21 
7701 open("/tmp/crontab.vYPoHR", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4 
7701 getdents(4, {{d_ino=73336409, d_off=1, d_reclen=24, d_name="."} 
{d_ino=73335455, d_off=2, d_reclen=24, d_name=".."} {d_ino=73340547, d_off=3, 
d_reclen=32, d_name="crontab"}}, 32768) = 80 
7701 unlink("/tmp/crontab.vYPoHR/crontab") = 0 
7701 getdents(4, {}, 32768) = 0 
7701 close(4) = 0 
7701 rmdir("/tmp/crontab.vYPoHR") = -1 ENOTEMPTY (Directory not empty) 
7701 write(2, "/tmp/crontab.vYPoHR: Directory n"..., 41) = 41 
7701 socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4

The unlink is really done after the rmdir,
nfsv4 & nfsv3 over ipv6, so the rmdir fails and the directory isn't removed.
I have same problem with mysql temporary files.

If I mount with "nocto,noac,lookupcache=none" : same problem
kernel version on server : 3.16.0-4-amd64 (debian8)
kernel version on client : 2.6.32-39-pve (proxmox), same thing with 
3.13.0-63-generic (ubuntu)

Please answer me in CC, I'm not subscribed to the list.

Regards

Nicolas
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: nfs race condition

2015-10-22 Thread Nicolas Parpandet


Hello,

I did some more investigations, (wireshark)

The unlink of /tmp/crontab.vYPoHR/crontab is tranlated into a rename nfs call ? 
(.nfs file)

NFS 670 V3 READDIRPLUS Reply (Call In 731) crontab .. .
NFS 210 V3 LOOKUP Call (Reply In 734), DH: 
0x038abb3d/.nfs0098008a0019
NFS 194 V3 LOOKUP Reply (Call In 733) Error: NFS3ERR_NOENT
NFS 254 V3 RENAME Call (Reply In 736), From DH: 0x038abb3d/crontab To 
DH: 0x038abb3d/.nfs0098008a0019
NFS 338 V3 RENAME Reply (Call In 735)

and after, it tries to remove the directory /tmp/crontab.0wuoNx :
NFS 198 V3 RMDIR Call (Reply In 738), DH: 0x954c28ec/crontab.0wuoNx
NFS 222 V3 RMDIR Reply (Call In 737) Error: NFS3ERR_NOTEMPTY

but the file /tmp/crontab.0wuoNx/crontab was renamed instead of being removed 
(nfs network, I cannot see .nfs000 file in fact),
this prevent the removing of the directory  /tmp/crontab.0wuoNx (notempty)

It seems a race condition, see "nfs silly rename" keyword, but I have the 
problem with mysql temp files, and crontab temp file (crontab -e),
am I the only one doing mysql and crontabs over NFS !!!?!?, does anybody have a 
solution ?


Regards

Nicolas


- Mail original -
De: "npa" 
À: linux-kernel@vger.kernel.org
Cc: "pcoustillas" 
Envoyé: Dimanche 4 Octobre 2015 09:57:26
Objet: nfs race condition

Hello, 

I'm doing some " crontab -e, exit with :x" on NFS mount point, 
and there is a race condition whent removing temporary files : 


7701 write(2, "No modification made\n", 21) = 21 
7701 open("/tmp/crontab.vYPoHR", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4 
7701 getdents(4, {{d_ino=73336409, d_off=1, d_reclen=24, d_name="."} 
{d_ino=73335455, d_off=2, d_reclen=24, d_name=".."} {d_ino=73340547, d_off=3, 
d_reclen=32, d_name="crontab"}}, 32768) = 80 
7701 unlink("/tmp/crontab.vYPoHR/crontab") = 0 
7701 getdents(4, {}, 32768) = 0 
7701 close(4) = 0 
7701 rmdir("/tmp/crontab.vYPoHR") = -1 ENOTEMPTY (Directory not empty) 
7701 write(2, "/tmp/crontab.vYPoHR: Directory n"..., 41) = 41 
7701 socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4

The unlink is really done after the rmdir,
nfsv4 & nfsv3 over ipv6, so the rmdir fails and the directory isn't removed.
I have same problem with mysql temporary files.

If I mount with "nocto,noac,lookupcache=none" : same problem
kernel version on server : 3.16.0-4-amd64 (debian8)
kernel version on client : 2.6.32-39-pve (proxmox), same thing with 
3.13.0-63-generic (ubuntu)

Please answer me in CC, I'm not subscribed to the list.

Regards

Nicolas
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: nfs race condition

2015-10-22 Thread Neil Brown

Nicolas Parpandet  writes:

> Hello,
>
> I did some more investigations, (wireshark)
>
> The unlink of /tmp/crontab.vYPoHR/crontab is tranlated into a rename
> nfs call ? (.nfs file)

That happens if the file is still open.

>
> NFS   670 V3 READDIRPLUS Reply (Call In 731) crontab .. .
> NFS   210 V3 LOOKUP Call (Reply In 734), DH: 
> 0x038abb3d/.nfs0098008a0019
> NFS   194 V3 LOOKUP Reply (Call In 733) Error: NFS3ERR_NOENT
> NFS   254 V3 RENAME Call (Reply In 736), From DH: 0x038abb3d/crontab To 
> DH: 0x038abb3d/.nfs0098008a0019
> NFS   338 V3 RENAME Reply (Call In 735)
>
> and after, it tries to remove the directory /tmp/crontab.0wuoNx :
> NFS   198 V3 RMDIR Call (Reply In 738), DH: 0x954c28ec/crontab.0wuoNx
> NFS   222 V3 RMDIR Reply (Call In 737) Error: NFS3ERR_NOTEMPTY
>
> but the file /tmp/crontab.0wuoNx/crontab was renamed instead of being removed 
> (nfs network, I cannot see .nfs000 file in fact),
> this prevent the removing of the directory  /tmp/crontab.0wuoNx (notempty)
>
> It seems a race condition, see "nfs silly rename" keyword, but I have
> the problem with mysql temp files, and crontab temp file (crontab -e),

I don't think it is a race condition exactly.  More of an ordering
problem.

Unlinking a file before closing it is rarely useful (sometimes it is,
but not here).

> am I the only one doing mysql and crontabs over NFS !!!?!?,

Probably.

> does anybody have a solution ?

Fix 'crontab' to close before unlinking...

Is this the Debian/Ubuntu crontab program it looks a bit like it,
but this bug was fixed 6 years ago.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413962

Or use a tmpfs for /tmp/ rather than using NFS???

NeilBrown

>
>
> Regards
>
> Nicolas
>
>
> - Mail original -
> De: "npa" 
> À: linux-kernel@vger.kernel.org
> Cc: "pcoustillas" 
> Envoyé: Dimanche 4 Octobre 2015 09:57:26
> Objet: nfs race condition
>
> Hello, 
>
> I'm doing some " crontab -e, exit with :x" on NFS mount point, 
> and there is a race condition whent removing temporary files : 
>
>
> 7701 write(2, "No modification made\n", 21) = 21 
> 7701 open("/tmp/crontab.vYPoHR", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 
> 4 
> 7701 getdents(4, {{d_ino=73336409, d_off=1, d_reclen=24, d_name="."} 
> {d_ino=73335455, d_off=2, d_reclen=24, d_name=".."} {d_ino=73340547, d_off=3, 
> d_reclen=32, d_name="crontab"}}, 32768) = 80 
> 7701 unlink("/tmp/crontab.vYPoHR/crontab") = 0 
> 7701 getdents(4, {}, 32768) = 0 
> 7701 close(4) = 0 
> 7701 rmdir("/tmp/crontab.vYPoHR") = -1 ENOTEMPTY (Directory not empty) 
> 7701 write(2, "/tmp/crontab.vYPoHR: Directory n"..., 41) = 41 
> 7701 socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4
>
> The unlink is really done after the rmdir,
> nfsv4 & nfsv3 over ipv6, so the rmdir fails and the directory isn't removed.
> I have same problem with mysql temporary files.
>
> If I mount with "nocto,noac,lookupcache=none" : same problem
> kernel version on server : 3.16.0-4-amd64 (debian8)
> kernel version on client : 2.6.32-39-pve (proxmox), same thing with 
> 3.13.0-63-generic (ubuntu)
>
> Please answer me in CC, I'm not subscribed to the list.
>
> Regards
>
> Nicolas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


signature.asc
Description: PGP signature

Re: nfs race condition

Re: nfs race condition

Re: nfs race condition

Re: nfs race condition

Re: nfs race condition

Re: nfs race condition

6 matches

Site Navigation

Mail list logo

Footer information