Re: fsync() doesn't work as advertised?
On Jan 5, 1:08 pm, Nobody nob...@nowhere.com wrote: On Mon, 04 Jan 2010 08:09:56 -0800, Brian D wrote: If I'm running a process in a loop that runs for a long time, I occasionally would like to look at a log to see how it's going. I know about the logging module, and may yet decide to use that. Still, I'm troubled by howfsync() doesn't seem to work as advertised: http://docs.python.org/library/os.html If you’re starting with a Python file object f, first do f.flush(), and then do os.fsync(f.fileno()) The .flush() method (and the C fflush() function) causes the contents of application buffers to be sent to the OS, which basically copies the data into the OS-level buffers. fsync() causes the OS-level buffers to be written to the physical drive. File operations normally use the OS-level buffers; e.g. if one process write()s to a file and another process read()s it, the latter will see what the former has written regardless of whether the data has been written to the drive. The main reason for usingfsync() is to prevent important data from being lost in the event of an unexpected reboot or power-cycle (an expected reboot via the shutdown or halt commands will flush all OS-level buffers to the drive first). Other than that,fsync() is almost invisible (I say almost, as there are mechanisms to bypass the OS-level buffers, e.g. the O_DIRECT open() flag). An excellent explanation of the process. I've seen this in other programming environments, so I could visualize something to that effect, but couldn't have verbalized it. I certainly have a better idea what's happening. Thanks for the contribution. -- http://mail.python.org/mailman/listinfo/python-list
Re: fsync() doesn't work as advertised?
On Mon, 04 Jan 2010 08:09:56 -0800, Brian D wrote: If I'm running a process in a loop that runs for a long time, I occasionally would like to look at a log to see how it's going. I know about the logging module, and may yet decide to use that. Still, I'm troubled by how fsync() doesn't seem to work as advertised: http://docs.python.org/library/os.html If you’re starting with a Python file object f, first do f.flush(), and then do os.fsync(f.fileno()) The .flush() method (and the C fflush() function) causes the contents of application buffers to be sent to the OS, which basically copies the data into the OS-level buffers. fsync() causes the OS-level buffers to be written to the physical drive. File operations normally use the OS-level buffers; e.g. if one process write()s to a file and another process read()s it, the latter will see what the former has written regardless of whether the data has been written to the drive. The main reason for using fsync() is to prevent important data from being lost in the event of an unexpected reboot or power-cycle (an expected reboot via the shutdown or halt commands will flush all OS-level buffers to the drive first). Other than that, fsync() is almost invisible (I say almost, as there are mechanisms to bypass the OS-level buffers, e.g. the O_DIRECT open() flag). -- http://mail.python.org/mailman/listinfo/python-list
Re: fsync() doesn't work as advertised?
Le Mon, 04 Jan 2010 08:09:56 -0800, Brian D a écrit : What I've seen is that flush() alone produces a complete log when the loop finishes. When I used fsync(), I lost all of the write entries except the first, along with odd error trap and the last entry. Perhaps you are writing to the file from several threads or processes at once? By the way, you shouldn't need fsync() if you merely want to look at the log files, because your OS will have an up-to-date view of the file contents anyway. fsync() is useful if you want to be sure the data has been written to the hard disk drive, rather than just kept in the operating system's filesystem cache. -- http://mail.python.org/mailman/listinfo/python-list
Re: fsync() doesn't work as advertised?
On Jan 4, 10:29 am, Antoine Pitrou solip...@pitrou.net wrote: Le Mon, 04 Jan 2010 08:09:56 -0800, Brian D a écrit : What I've seen is that flush() alone produces a complete log when the loop finishes. When I used fsync(), I lost all of the write entries except the first, along with odd error trap and the last entry. Perhaps you are writing to the file from several threads or processes at once? By the way, you shouldn't need fsync() if you merely want to look at the log files, because your OS will have an up-to-date view of the file contents anyway. fsync() is useful if you want to be sure the data has been written to the hard disk drive, rather than just kept in the operating system's filesystem cache. Sure -- I hadn't considered how threads might affect the write process. That's a good lead to perhaps fixing the problem. Thanks for your help, Antoine. -- http://mail.python.org/mailman/listinfo/python-list
