[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I did some experimenting with your test cases. My results so far are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext31 0.430.420.501 1.00% ext32 0.320.300.330 0.00% ext33 0.190.160.200 0.00% ext34 0.250.200.250 0.00% ext35 0.250.200.250 0.00% ext36 0.440.330.460 0.00% ext41 0.450.440.50100 100.00% ext42 0.330.330.33100 100.00% ext43 0.200.200.210 0.00% ext44 0.250.250.330 0.00% ext45 0.250.250.260 0.00% ext46 0.440.330.410 0.00% Ext4 will zero-length all the files in Ext 4 with test cases 1 and 2. I am working on downloading a revised kernel with the patch. This is my first time doing a recompile for a jaunty release, so this may take a while. ** Attachment added: "Benchmarks in OpenOffice" http://launchpadlibrarian.net/23871702/Ext3Ext4Benchmarks.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@CowBoyTim I agree with you. I work with real-time industrial systems, where the shop floor systems are considered unreliable. We have all the same issues as a regular desktop user, except our users have bigger hammers. The attraction of ext3 was the journalling with the ordered data mode. If power was cut, it was possible to reassemble something to a recent point in time, with only the most recent data lost. This bug in ext4, results in zero-length files, and not only in the most recent files either. All fsync() does is bypass one layer of write-back caching. This just makes the window of data loss smaller, in the specific case of infrequent fsync() calls. By itself, fsync() does nothing to guarantee data integrity. I think this is why Bogdan was complaining about defective MySQL databases. Given the benchmarks, it is likely that the file system zero-lengthed the entire database file. Specifically, fsync() guarantees the data is on the disk, it doesn't guarantee the file system knows where the file is. As such, one could call fsync(), and still not be able to get at the data after a reboot. The arguments against telling every application developer to use fsync() are: 1. Under heavy file I/O, fsync() could potentially decrease your average I/O speed by defeating the write-back caching. This could make the window of data loss larger, especially with a real-time system where the incoming data rate is fixed. 2. Repeated calls to fsync() would be very rough on laptop mode and on SSDs (Solid State Disks). 3. Repeated calls to fsync() will limit maximum file system performance for desktop applications. Eventually, the file system developers will replace fsync() with an empty function, just like Apple did. 4. If everyone will want fsync(), why don't we just modify close() function to call fsync()? 5. There is a strong correlation between user activity and system crashes. Not using the fsync() leads to much more understandable system behavior. Imagine a typical self-inflicted system crash. This can be caused either directly: "Press Save then turn off the Computer," or indirectly: "edit video game config, hit play, and then watch the video driver crash." If the write-back cache is enabled, and fsync() is not used, the program will write data to the cache, cause a bunch of disk reads, and then during idle time, the data will be written to disk. If the user generated activity results in disk reads, then the write-back cache will "protect" the old version of the file. The user will learn that crashing the machine results in him losing his most recent changes. On the other hand, if fsync() is used to disable the write back cache, then programmers will start calling fsync() and close() from background threads. This will result in a poor user experience, as the hard disk will be thrashing during program startup (when all the disk reads are happening), and anything could happen when the system crashes during the fsync(). In the case that system crashes correlate to user activity, it is really tempting from a software point of view, to try to get the fsync() to happen before the system crash occurs. Unfortunately, in practice this is really tough to do. The journaled file system with an ordered data mode is a really good compromise for many desktop and real-time type applications. Additionally, limited fsync() use preserves the effectiveness of fsync() for applications that really need it, like databases. -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 317781] Re: Ext4 data loss
@Volodymyr I finished recompiling the kernel with Theodore Ts'o patches, and reran Volodymyr's test cases with the patched kernel. The results are: File System Method Performance (Typical, Minimum, Maximum) #Lost %Lost ext4patch 1 0.440.410.501 1.00% ext4patch 2 0.320.320.400 0.00% ext4patch 3 0.200.180.200 0.00% ext4patch 4 0.250.250.250 0.00% ext4patch 5 0.260.260.330 0.00% ext4patch 6 0.410.330.420 0.00% Essentially, the patches work. Ext4, with the patch, has the same data loss as the ext3 file system. Adding fsync() to the code results in a significant decrease in loop speed. As such, for the application writers, only add fsync() to your code when you want to be really sure the data has been written to disk, like when you are writing a database. ** Attachment added: "Benchmarks in OpenOffice V2" http://launchpadlibrarian.net/23897042/Ext3Ext4BenchmarksV2.ods -- Ext4 data loss https://bugs.launchpad.net/bugs/317781 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] Re: The partition utility in the setup program reports a bogus disk percentage used.
** Attachment added: "PictureOfPercentage" http://launchpadlibrarian.net/23897297/UbuntuInstallBug.png -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 343490] [NEW] The partition utility in the setup program reports a bogus disk percentage used.
Public bug reported: In Ubuntu 9.04 (Jaunty Jackalope) Alpha 6, the disk partitioner reports a bogus percentage disk space used. It is probably a really easy bug to fix, if someone can give me a rough idea of where to find the source code. I think the disk utility is reporting the disk size as a percentage. It should either report a percentage, or the disk size, but never the disk size converted to percent. The problem is that the percentage calculation probably breaks down when only one partition is present, or in some related special circumstance. ** Affects: ubuntu Importance: Undecided Status: New -- The partition utility in the setup program reports a bogus disk percentage used. https://bugs.launchpad.net/bugs/343490 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
