Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Wed, Nov 7, 2012 at 2:31 AM, Simon Schampijer si...@schampijer.de wrote: Thanks Martin, Sam and Manuel for your work on this. The datastore runs here fine. I pushed the changes. Will be available in 0.97.2. Great! I was going to write a longer thank-you for the review, but I ran out of space. :-) m -- mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On 11/07/2012 12:50 PM, Martin Langhoff wrote: On Wed, Nov 7, 2012 at 2:31 AM, Simon Schampijer si...@schampijer.de wrote: Thanks Martin, Sam and Manuel for your work on this. The datastore runs here fine. I pushed the changes. Will be available in 0.97.2. Great! I was going to write a longer thank-you for the review, but I ran out of space. :-) Same for me, I wanted to write a loong and fancy release announcement, but these damn space constraints... Cheers, Simon ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On 09/26/2012 06:09 PM, Martin Langhoff wrote: On Sat, Sep 22, 2012 at 10:09 PM, Martin Langhoff mar...@laptop.org wrote: On Fri, Sep 21, 2012 at 11:36 AM, Martin Langhoff martin.langh...@gmail.com wrote: Reviews are boring. Testing is more fun! rpms for ARM at Reviews are boring, but necessary. As these patches have seen some reviews, and the ml is a bit awkward for tracking those, I've pushed a 'pu' (proposed updates) branch at http://dev.laptop.org/git/users/martin/sugar-datastore/ I have posted all the patches to the mailing list, and will continue to do so. The pu branch will have the latest, freshest set of patches (so: it rewinds/rebases as needed). m Thanks Martin, Sam and Manuel for your work on this. The datastore runs here fine. I pushed the changes. Will be available in 0.97.2. Regards, Simon ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Sat, Sep 22, 2012 at 10:09 PM, Martin Langhoff mar...@laptop.org wrote: On Fri, Sep 21, 2012 at 11:36 AM, Martin Langhoff martin.langh...@gmail.com wrote: Reviews are boring. Testing is more fun! rpms for ARM at Reviews are boring, but necessary. As these patches have seen some reviews, and the ml is a bit awkward for tracking those, I've pushed a 'pu' (proposed updates) branch at http://dev.laptop.org/git/users/martin/sugar-datastore/ I have posted all the patches to the mailing list, and will continue to do so. The pu branch will have the latest, freshest set of patches (so: it rewinds/rebases as needed). m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
On Thu, Sep 20, 2012 at 11:55 PM, Martin Langhoff mar...@laptop.org wrote: In conditions of very low disk space available, or ENOSPC, the datastore misbehaves grossly: Reviews are boring. Testing is more fun! rpms for ARM at http://dev.laptop.org/~martin/ds-enospc/ How to test: - Set Sugar to verbose debugging - Make sure sugar logs are not on /, add a file in /etc/statetab.d containing /home/olpc/.sugar/default/logs - use dd if=/dev/zero of=/bigfile bs=1M count=numberofmegs to fill up your disk I usually create one large file to put the system where it has little disk space remaining, then add/remove smaller files to tighten the free space further, or to push all the way to ENOSPC. This is faster than recreating a very large file everytime. The datastore.org will be telling you whether we are using tmpfs for the index, when a rebuild is triggered, ENOSPC is hit, etc. It is also interesting to monitor the presence of two clean markers in the datastore dir, ds_clean and index_updated. You can use inotifywatch -e create -e delete /home/olpc/.sugar/default/datastore cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- Software Architect - OLPC - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] [PATCH 0/7] datastore: handle ENOSPC gracefully
In conditions of very low disk space available, or ENOSPC, the datastore misbehaves grossly: - sometimes corrupts the index, and won't rebuild it, hiding valid entries from the user - if you edit an entry metadata, it will corrupt the whole entry - attempts at deleting entries fail, so users cannot get out of ENOSPC This patchseries is an attempt at fixing these issues. The overall user experience is not perfect when we hit ENOSPC -- metadata edits are ignored, sometimes the datastore process goes away (so the Journal is not responsive until a new datastore process is spawned and ready). However, it behaves sanely: - testing this code I do not lose ds entries when editing them - I can remove entries correctly, even at ENOSPC, effectively freeing disk space - the index works even at ENOSPC, though it may take slower for the system to start in those cases where we do rebuild it - the index recovers gracefully on restart when there is free disk space Martin Langhoff (7): Add ds_clean flag to trigger index rebuilds #2095, #2317 Remove invalid/corrupt on-disk entries #2317 metadatastore: store/change files on disk defensively #2317 metadatareader: ignore .hidden files indexstore: exit on _flush() errors, work on tmpdir datastore: make delete() more reliable, log properly datastore: handle low-disk and ENOSPC conditions gracefully src/carquinyol/datastore.py | 142 +-- src/carquinyol/indexstore.py| 52 ++ src/carquinyol/metadatareader.c |7 +- src/carquinyol/metadatastore.py | 51 +- 4 files changed, 197 insertions(+), 55 deletions(-) -- 1.7.10.4 ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
