Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-02 Thread tomas
On Mon, May 02, 2022 at 10:04:00AM -0500, David Wright wrote: > I'm not very familiar with files' birth as it's a relatively new > addition to filesystems, particularly how to display it even when > present. So I looked it up, and the ext4 wiki says it's the time > at which the inode is created. I

Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-02 Thread David Wright
On Mon 02 May 2022 at 01:30:08 (+0200), Vincent Lefevre wrote: > On 2022-04-30 15:19:21 +0200, Thomas Schmitt wrote: > > An explanation of the observed problem would need: > > - a mechanism which delayed the content production of the inode while it > > was already in use for open and write, > > -

Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-01 Thread tomas
On Mon, May 02, 2022 at 01:10:04AM +0200, Vincent Lefevre wrote: > On 2022-04-30 14:06:53 +0200, to...@tuxteam.de wrote: [...] > > It sure looks like a bug. But it would be a bug at a spot where one would > > expect that it should have bitten oodles of other people by now, so that > > feeling tha

Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-01 Thread Greg Wooledge
On Mon, May 02, 2022 at 01:30:08AM +0200, Vincent Lefevre wrote: > I'm wondering whether the data are transferred from the VFS to ext4 > necessarily within the same openat system call or could just be kept > in the VFS as long as they are not needed elsewhere, i.e. the VFS > behaving like a cache.

Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-01 Thread Vincent Lefevre
On 2022-04-30 15:19:21 +0200, Thomas Schmitt wrote: > An explanation of the observed problem would need: > - a mechanism which delayed the content production of the inode while it > was already in use for open and write, > - or a mechanism which caused ext4 to hide the inode to other processes >

Re: file born 30 seconds after its creation on ext4 - bug?

2022-05-01 Thread Vincent Lefevre
On 2022-04-30 14:06:53 +0200, to...@tuxteam.de wrote: > On Sat, Apr 30, 2022 at 10:33:34AM +0200, Thomas Schmitt wrote: > > I understand that Vincent Lefevre suspects these discrepancies to be a bug > > in the ext4 driver. I rather suspect that ext4 is ok and that we observe > > the effects of some

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-30 Thread Thomas Schmitt
Hi, Curt wrote: > What does the following mean, then, in that light: > Because of delayed allocation and other performance optimizations, ext4's > behavior of writing files to disk is different from ext3. In ext4, when a > program writes to the file system, it is not guaranteed to be on-disk

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-30 Thread tomas
On Sat, Apr 30, 2022 at 12:31:07PM -, Curt wrote: > On 2022-04-30, Thomas Schmitt wrote: > > > > Indeed. With normal filesystem operations there should be no need to call > > something like sync(2) in order to get a consistent representation of the > > current filesystem state. > > > > What d

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-30 Thread Curt
On 2022-04-30, Thomas Schmitt wrote: > > Indeed. With normal filesystem operations there should be no need to call > something like sync(2) in order to get a consistent representation of the > current filesystem state. > What does the following mean, then, in that light: Because of delayed allo

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-30 Thread tomas
On Sat, Apr 30, 2022 at 10:33:34AM +0200, Thomas Schmitt wrote: > Hi, > > sp...@caiway.net wrote: > > how does one force to refresh this memory with a command? > > to...@tuxteam.de wrote: > > What do you mean by "refresh"? What's in the cache is always > > the "freshest" version: > > Indeed. Wit

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-30 Thread Thomas Schmitt
Hi, sp...@caiway.net wrote: > how does one force to refresh this memory with a command? to...@tuxteam.de wrote: > What do you mean by "refresh"? What's in the cache is always > the "freshest" version: Indeed. With normal filesystem operations there should be no need to call something like sync(2

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-29 Thread tomas
On Fri, Apr 29, 2022 at 09:41:23PM +0200, sp...@caiway.net wrote: > On Fri, 29 Apr 2022 13:45:15 -0400 > > > linux caches file system pages in memory > > how does one force to refresh this memory with a command? What do you mean by "refresh"? What's in the cache is always the "freshest" version:

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-29 Thread sp...@caiway.net
On Fri, 29 Apr 2022 13:45:15 -0400 > linux caches file system pages in memory how does one force to refresh this memory with a command? Thanks!

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-29 Thread Marc Auslander
On 4/29/2022 10:20 AM, duh wrote: On 4/27/22 11:05 PM, Greg Wooledge wrote: Having skimmed over a number of the replies, and really not being qualified, may I just toss out a probably useless ideas to use the "sync" command. Looking at the 'man sync' shows at the bottom several variant

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-29 Thread duh
On 4/27/22 11:05 PM, Greg Wooledge wrote: On Wed, Apr 27, 2022 at 10:45:09PM -0400, Stefan Monnier wrote: Another option might be that your system's time was "reset". This shouldn't happen, but it can happen if your NTP was down, the machine got out-of-sync over time and you restart the NTP se

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-28 Thread Vincent Lefevre
Hi, On 2022-04-28 11:26:36 +0200, Thomas Schmitt wrote: > Vincent Lefevre wrote: > > and one with > > openat(AT_FDCWD, "….out", O_WRONLY|O_CREAT|O_APPEND, 0666 > > <... openat resumed>) = 3 > > about 30 seconds later. > > Oh. So the script was still running when the file finally appeared to

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-28 Thread Vincent Lefevre
On 2022-04-27 11:39:17 +0200, Thomas Schmitt wrote: > It is normal that data get onto the physical storage medium only quite a > long time after a program wrote them. But this is supposed to be kept > consistent by the VFS and virtual memory of the Linux kernel. [...] > I understand that the filesy

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-28 Thread Thomas Schmitt
Hi, Vincent Lefevre wrote: > and one with > openat(AT_FDCWD, "….out", O_WRONLY|O_CREAT|O_APPEND, 0666 > <... openat resumed>) = 3 > about 30 seconds later. Oh. So the script was still running when the file finally appeared to lt, tail, and ll ? Is the text snippet " <... openat resumed>" l

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-28 Thread Vincent Lefevre
On 2022-04-27 22:45:09 -0400, Stefan Monnier wrote: > Another option might be that your system's time was "reset". > This shouldn't happen, but it can happen if your NTP was down, the > machine got out-of-sync over time and you restart the NTP server at > which point it may(!) decide to jump the cl

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Greg Wooledge
On Wed, Apr 27, 2022 at 10:45:09PM -0400, Stefan Monnier wrote: > Another option might be that your system's time was "reset". > This shouldn't happen, but it can happen if your NTP was down, the > machine got out-of-sync over time and you restart the NTP server at > which point it may(!) decide to

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Vincent Lefevre
On 2022-04-27 09:26:50 -0400, Greg Wooledge wrote: > Are you able to reproduce the issue at all, even like 1% of the time? In the past 11 years, I may have ran the script (and looked at the output just after that) hundreds or thousands of times; the commands to look at the output have been in my s

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Greg Wooledge
On Wed, Apr 27, 2022 at 03:11:22PM +0200, Vincent Lefevre wrote: > So, perhaps there were no issues with openat, but when reading > the directory, the file could not be found because some internal > structures might have been incomplete. If so, this is a bug at the kernel level, perhaps in the VFS

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Vincent Lefevre
On 2022-04-27 11:39:17 +0200, Thomas Schmitt wrote: > If it has indeed to do with the CPU cache then a particular cache would have > delayed its writing to RAM for 30 seconds but would have served its own CPU > with the full results of file system driver and virtual memory activities > around the n

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Thomas Schmitt
Hi, Vincent Lefevre wrote: > the issue [...] is probably unlikely to occur again. In this case we will hardly be able to find an explanation. > However, there's also the fact that the birth time was 30 seconds > ahead of the actual file creation, while there was no lockup. Yes, your observatio

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Vincent Lefevre
Hi, On 2022-04-27 09:07:37 +0200, Thomas Schmitt wrote: > i wrote: > > > So if it got created by the script, maybe it was deleted or renamed > > > shortly afterwards and created again 30 seconds later ? > > Vincent Lefevre wrote: > > The script doesn't do that. The file is created with > > echo

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-27 Thread Thomas Schmitt
Hi, i wrote: > > So if it got created by the script, maybe it was deleted or renamed > > shortly afterwards and created again 30 seconds later ? Vincent Lefevre wrote: > The script doesn't do that. The file is created with > echo "* $fqdn ($(${1:-.}/config.guess) / ${line#PROC:})" > "$out" > [.

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Vincent Lefevre
On 2022-04-26 14:18:58 -0400, Stefan Monnier wrote: > > On an ext4 filesystem, I got a file born 30 seconds after its > > actual creation. Is this a bug? > > I doubt it. > Note that a file's atime/mtime/ctime is a property of the file itself, > whereas "appearing" is defined by when the name you'r

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Vincent Lefevre
On 2022-04-26 12:47:53 -0500, Nicholas Geovanis wrote: > On Tue, Apr 26, 2022 at 12:37 PM Nicholas Geovanis > wrote: > > > On Tue, Apr 26, 2022 at 8:45 AM Vincent Lefevre > > wrote: > > > >> On an ext4 filesystem, I got a file born 30 seconds after its > >> actual creation. Is this a bug? > >> >

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Vincent Lefevre
On 2022-04-26 19:01:26 +0200, Thomas Schmitt wrote: > It looks as if the file indeed does not exist when you inquire it. > So if it got created by the script, maybe it was deleted or renamed > shortly afterwards and created again 30 seconds later ? The script doesn't do that. The file is created w

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Nicholas Geovanis
On Tue, Apr 26, 2022 at 12:37 PM Nicholas Geovanis wrote: > On Tue, Apr 26, 2022 at 8:45 AM Vincent Lefevre > wrote: > >> On an ext4 filesystem, I got a file born 30 seconds after its >> actual creation. Is this a bug? >> > > Only experimentation can really back me up on this, but consider the >

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Nicholas Geovanis
On Tue, Apr 26, 2022 at 8:45 AM Vincent Lefevre wrote: > On an ext4 filesystem, I got a file born 30 seconds after its > actual creation. Is this a bug? > Only experimentation can really back me up on this, but consider the following: Every time you use the "|" operator or the ";" separator on

Re: file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Thomas Schmitt
Hi, Vincent Lefevre wrote: > This script creates a file mpfrtests.cventin.lip.ens-lyon.fr.out > very early. But the first attempts to look at this file failed: It looks as if the file indeed does not exist when you inquire it. So if it got created by the script, maybe it was deleted or renamed sh

file born 30 seconds after its creation on ext4 - bug?

2022-04-26 Thread Vincent Lefevre
On an ext4 filesystem, I got a file born 30 seconds after its actual creation. Is this a bug? I know that such issues can be observed with NFS, but here this is just a local ext4 filesystem. Here are the details. I started a shell script: cventin:~> ps -p 667828 -o lstart,cmd S