OS.File and shutdown
Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic. Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer? Another question: are the write tasks queued and completed in order, or can two writeAtomic calls to the same file race each other and the 2nd call finish first (only to have the 1st call finish and write older data) Thanks, Felipe ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/10/13 4:45 PM, Felipe Gomes wrote: Hi, does OS.File guarantees that write tasks that have started will be completed if a shutdown occurs? My use case is for writeAtomic but I'm interested about the behavior of both write and writeAtomic. Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer? I'm not sure about the rest of this question. But you should not be performing any I/O after profile-before-change. See https://wiki.mozilla.org/XPCOM_Shutdown and note that after profile-before-change, we are working on immediately exiting the browser and skipping XPCOM shutdown. xpcom-shutdown is definitely too late! --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/10/13 10:45 PM, Felipe Gomes wrote: > Hi, does OS.File guarantees that write tasks that have started will be > completed if a shutdown occurs? My use case is for writeAtomic but I'm > interested about the behavior of both write and writeAtomic. > Corner case: what if I call write/writeAtomic from an xpcom-shutdown observer? In theory: - every call to OS.File queued *before* xpcom-shutdown will be completed; - every call to OS.File queued after xpcom-sthudown will throw an asynchronous exception (once bug 845190 has landed). Note, however, that this has not been thoroughly tested yet. > Another question: are the write tasks queued and completed in order, or can > two writeAtomic calls to the same file race each other and the 2nd call > finish first (only to have the 1st call finish and write older data) All tasks to OS.File are queued and completed in order. Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/13/13 10:20 PM, David Rajchenbach-Teller wrote: > On 5/10/13 10:45 PM, Felipe Gomes wrote: >> Hi, does OS.File guarantees that write tasks that have started will be >> completed if a shutdown occurs? My use case is for writeAtomic but I'm >> interested about the behavior of both write and writeAtomic. >> Corner case: what if I call write/writeAtomic from an xpcom-shutdown >> observer? > > In theory: > - every call to OS.File queued *before* xpcom-shutdown will be completed; > - every call to OS.File queued after xpcom-sthudown will throw an > asynchronous exception (once bug 845190 has landed). Actually, what I wrote was misleading. Calls to OS.File queued during xpcom-shutdown will be honoured *if the OS.File thread has already been initialized* before xpcom-shutdown – this is the case in Firefox itself, but not necessarily in xpcshell tests. Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On Friday, May 10, 2013 7:53:01 PM UTC-3, Benjamin Smedberg wrote: > I'm not sure about the rest of this question. But you should not be > > performing any I/O after profile-before-change. See > > https://wiki.mozilla.org/XPCOM_Shutdown and note that after > > profile-before-change, we are working on immediately exiting the browser > > and skipping XPCOM shutdown. xpcom-shutdown is definitely too late! Okay. The code i'm touching does that on xpcom-shutdown but since it'll be wrong in the future I'll change it while I'm there. (it finishes some pending async sqlite stmts and rolls back any ongoing transactions). What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown. Should profile-before-change then be my call to stop accepting changes to the data and call writeAtomic to flush it? I've seen some code nearby doing it at quit-application-granted. Or perhaps there's no "correct" answer and it varies case by case (or anything goes that works and is early enough..) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/14/13 11:35 AM, Felipe Gomes wrote: On Friday, May 10, 2013 7:53:01 PM UTC-3, Benjamin Smedberg wrote: I'm not sure about the rest of this question. But you should not be performing any I/O after profile-before-change. See https://wiki.mozilla.org/XPCOM_Shutdown and note that after profile-before-change, we are working on immediately exiting the browser and skipping XPCOM shutdown. xpcom-shutdown is definitely too late! Okay. The code i'm touching does that on xpcom-shutdown but since it'll be wrong in the future I'll change it while I'm there. (it finishes some pending async sqlite stmts and rolls back any ongoing transactions). What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown. Should profile-before-change then be my call to stop accepting changes to the data and call writeAtomic to flush it? I've seen some code nearby doing it at quit-application-granted. Or perhaps there's no "correct" answer and it varies case by case (or anything goes that works and is early enough..) This sounds very similar to what I ran into with Firefox Health Report. There is some discussion in https://bugzilla.mozilla.org/show_bug.cgi?id=722648 and https://bugzilla.mozilla.org/show_bug.cgi?id=850712. Generally speaking, you have a queued async operation or set of async operations that need to complete. However, profile-before-change is observed before or during those operations. There is a race between xpcom-shutdown and those async operations finishing. FHR's solution to this problem is to initiate the shutdown procedure early during shutdown. We chose quit-application. If this shutdown sequence (which consists of waiting for async operations initiated or queued before shutdown started) has not completed by the time profile-before-change (the last observer before the profile goes away) is observed, FHR creates a nested event loop during the profile-before-change handler. This delays shutdown. But it eliminates race conditions and prevents data corruption. Code at https://hg.mozilla.org/mozilla-central/file/26ab72bfa9df/services/healthreport/healthreporter.jsm#l445 Performance *really* doesn't like this solution because it can delay shutdown. However, the last I checked Telemetry, we only had 121 reports on Nightly and Aurora of this occurring in the past month. This is because we only create a nested event loop if shutdown hasn't finished by profile-before-change and unless a major FHR operation was initiated just before shutdown, there will be no work to do except close the database handle. If we were buffering data and persistently flushing, we'd likely be spinning much more since we'd be more likely to have queued I/O. I'll conclude by saying that every scenario is different. I highly recommend you connect with someone on the Perf team for a tailored recommendation. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/14/13 8:35 PM, Felipe Gomes wrote: > Should profile-before-change then be my call to stop accepting changes to the > data and call writeAtomic to flush it? I've seen some code nearby doing it at > quit-application-granted. Or perhaps there's no "correct" answer and it > varies case by case (or anything goes that works and is early enough..) profile-before-change should be good. Any OS.File call posted before xpcom-shutdown will be completed before we exit Firefox. Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
Felipe Gomes schrieb: What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown. Just make sure you do (lazily but still) also flush that data out to disk every now and then during normal operation. For one thing, people leave Firefox running for hours, days and even weeks and if they run into a crash, they shouldn't lose state over such long periods, and for the other, we want shutdown to be fast and the ideal situation is that you already have written everything when the shutdown request comes and you don't need to care. ;-) Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
