Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 2:10 AM, James Cameron qu...@laptop.org wrote:
By branching at the 0.82 point as deployed, determining from evidence
Good theoretical re-statement. You post may clarify things for other
readers -- as for me, I know very well what I am requesting. And I do
respect that it is not easy.
It is just... very important.
I've reviewed the changes just now from 0.82 to 0.84. It was certainly
an entire rewrite, so what you are in effect asking is for someone to
support the 0.82 DS to fix a problem that is not yet well defined.
The _root cause_ is not well defined, although Tomeu has pointed out
likely culprits. The results are very well known, and it is possible
recover the situation without surgery. Read:
http://lists.laptop.org/pipermail/devel/2008-May/014601.html
And here you can find steps to repro and a workaround if you want to
see it in action:
http://dev.laptop.org/ticket/7719
(Uy users probably triggering the DS problem in a different way)
As for the code that controls this?
in shell/view/Shell.py:
def _start_journal_idle(self):
# Mount the datastore in internal flash
ds_path = env.get_profile_path('datastore')
try:
datastore.mount(ds_path, [], timeout=120 * \
DBUS_PYTHON_TIMEOUT_UNITS_PER_SECOND)
except:
# Don't explode if there's corruption; move the data out of the way
# and attempt to create a store from scratch.
shutil.move(ds_path, os.path.abspath(ds_path) + str(time.time()))
datastore.mount(ds_path, [], timeout=120 * \
DBUS_PYTHON_TIMEOUT_UNITS_PER_SECOND)
The good news: if recent Sugars have retained this logic, even if we
hit issues with newer datastore code (hope not! :-) ), the same
recovery tool will help for newer Sugar.
m
--
martin.langh...@gmail.com
mar...@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 09:51:58AM +0200, Martin Langhoff wrote: And here you can find steps to repro and a workaround if you want to see it in action: http://dev.laptop.org/ticket/7719 Thanks. I shall look there for an exception. As for the code that controls this? in shell/view/Shell.py: [...] The good news: if recent Sugars have retained this logic, even if we hit issues with newer datastore code (hope not! :-) ), the same recovery tool will help for newer Sugar. I can't find any evidence that this logic has been retained ... looked in git://git.sugarlabs.org/sugar/mainline.git and git://git.sugarlabs.org/sugar-datastore/mainline.git -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 10:58 AM, James Cameron qu...@laptop.org wrote:
I can't find any evidence that this logic has been retained ... looked
in git://git.sugarlabs.org/sugar/mainline.git and
git://git.sugarlabs.org/sugar-datastore/mainline.git
Well, this sounds promising-- but I am definitely not familiar w the
codebase, so it'd be good to hear from Aleksey
in src/carquinyol/datastore.py
except Exception:
logging.exception('Failed to open index, will rebuild')
layout_manager.index_updated = False
self._index_store.remove_index()
self._index_store.open_index()
cheers,
m
--
martin.langh...@gmail.com
mar...@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 11:15:17AM +0200, Martin Langhoff wrote:
On Fri, Sep 18, 2009 at 10:58 AM, James Cameron qu...@laptop.org wrote:
I can't find any evidence that this logic has been retained ... looked
in git://git.sugarlabs.org/sugar/mainline.git and
git://git.sugarlabs.org/sugar-datastore/mainline.git
Well, this sounds promising-- but I am definitely not familiar w the
codebase, so it'd be good to hear from Aleksey
in src/carquinyol/datastore.py
except Exception:
logging.exception('Failed to open index, will rebuild')
layout_manager.index_updated = False
self._index_store.remove_index()
self._index_store.open_index()
Thats just recreating index, not DS data itself
so, we dont lose any data.
--
Aleksey
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Thu, Sep 17, 2009 at 11:37, Martin Langhoff martin.langh...@gmail.com wrote: Welcome new maintainer! I am currently doing a bit of work with LatAm deployments, so I do have some notes related to DS. You may be aware of these issues, of course. - Dataloss on startup. We are hearing of some issues in the field with dataloss -- not real dataloss but if there is anything DS doesn't like something during startup, it renames the datastore directory, and creates a new one. This preserves the data, which can be recovered/diagnosed by a DS expert (good); but from the PoV of end users in the field, the data is lost and the diskspace is lost too. A means to recover lost DS entries and diskspace would save a lot of users' data. And the DS is all about users' data :-) Unfortunately for various reasons we are having a hard time getting samples of those lost DSs. Even without them, I am fairly sure we can make a simple recovery Activity. - Better behaviour when naming/renaming files on a USB stick. See http://www.mail-archive.com/de...@lists.laptop.org/msg19738.html - More resilient behaviour when plugging arbitrary USB disks / sticks (this may be mostly resolved in recent Sugar) In the field, correct data-preserving behaviour is what matters most. Even if we had no new features, if we address dataloss scenarios (real and perceived), users will rejoice. So the above are most important. There is also some feature work that is interesting: - Integration with the Backup on the XS. Tony Anderson's recent work is a step in teh right direction, but I'd think that it should be a Journal enhancement rather than a separate activity. - Integration with Moodle activities -- so that the Journal keeps track of documents' association to a particular Moodle-side activity, and auto-uploads to the right place. The recent GSoC project around printing exercises some of these aspects. Anyway, -- that's my list, FWIW :-) -- I agree that an activity for repairing or recovering the DS would be quite beneficial, we'd need to add an exception for Rainbow though. If we find the root cause and get a fix that seems to be worth deploying, can we deploy it to machines presently with 0.82? Regards, Tomeu -- «Sugar Labs is anyone who participates in improving and using Sugar. What Sugar Labs does is determined by the participants.» - David Farning ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Thu, Sep 17, 2009 at 16:51, Martin Langhoff martin.langh...@gmail.com wrote: On Thu, Sep 17, 2009 at 4:27 PM, Gary C Martin g...@garycmartin.com wrote: FWIW, your three major points were fixed way back by Tomeu's 0.84 data-store re-write to make for robustness. Actually I'm fairly sure it was these kind'a reports that prompted Tomeu to make that fine effort. Prior to this (0.82) I had seen occasional datastore 'loss', sometime recoverable, sometimes not (while stress testing 3 XOs here). But since 0.84 I've not seen any failures. Somewhat in agreement. Clearly, there have been huge efforts -- and huge thanks go to Tomeu. That is the good side. But a rewrite is always a risky business. Got to channel jwz here and link to this oldie: http://jwz.livejournal.com/154529.html A rewrite means we have to approach the new DS as a new piece of software (or library or service or component). May fix some bugs, may add others -- new code new bugs. Hasn't seen the field. And for example, cannot be easily backported for a trial run in a deployment, and/or to be incorporated into my papercuts release. I'm not sure if Gary said that we can forget about the DS problems in 0.82 because we rewrote the DS in 0.84. I think he was noting that the feedback we got on 0.82 doesn't apply necessarily to the current code base because it changed drastically. Actually, if it was redesigned was for addressing the issues you mention. I guess we're waiting on the OLPC timeline for an official XO signed release with 0.84, before deployments would be comfortable moving forward with the solution. 0.84 is just about 6 months old now, so happy half Birthday! :-) You are right there. But I would like to note that any widely deployed software has a very long release/deploy cycle. This happens on OLPC deployments, and will happen on all large deployments. The DS is a very important component. When we see dataloss bugs, we need a patch for the deployed version, and a rescue my data utility. It is rewritten in the next release is -- in practice -- not very useful. There is no fix for the users in any reasonable timeframe. And in fact, we don't know how that rewrite will work in the field. This is not to disparage Tomeu's work. It is positive, and necessary. But when we think about users and deployments, there is an aspect that is actually more important: getting a fix and a 'repair' path out to users, in a way deployments can use. The current situation with this specific issue is understandable, 8.2.x was of course released at a time of turmoil for us all, and the DS rewrite was done shortly after. So I am strictly looking forward -- and wondering if we can help these users of 0.82 now. And how we'll handle similar issues when 0.84 is deployed and if dataloss bugs are reported. I also wonder about that ;) Any deployer wants to talk about actual problems, possible solutions, and how a possible fix could be deployed? Regards, Tomeu -- «Sugar Labs is anyone who participates in improving and using Sugar. What Sugar Labs does is determined by the participants.» - David Farning ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 11:34 AM, Tomeu Vizoso to...@sugarlabs.org wrote: I agree that an activity for repairing or recovering the DS would be quite beneficial, we'd need to add an exception for Rainbow though. So Rainbow would prevent a regular activity from reading .sugar? I am not familiar with how one goes about exceptions and such. I've recently seen activities that do out of the sandbox things (Tony's data manager, dsd's audio fixer...), don't know how... Alternatively, a python script to run on the Terminal might help (but an activity is much better, naturally) If we find the root cause and get a fix that seems to be worth deploying, can we deploy it to machines presently with 0.82? We can offer the patch to deployers, and I can integrate it into a build. But our reach is limited there. A user-deployable repair tool is a lot more effective. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 02:10, James Cameron qu...@laptop.org wrote: On Thu, Sep 17, 2009 at 04:51:47PM +0200, Martin Langhoff wrote: So I am strictly looking forward -- and wondering if we can help these users of 0.82 now. By branching at the 0.82 point as deployed, determining from evidence (those renamed DS directories you have been unable to obtain) what the likely problem is, and then making a minor change to prevent or handle it. The authors of the 0.82 DS or the 0.84 DS might also use their knowledge of the code to comment on the likely causes so that the investigation can be refined. Sure, we are here available for help. It's not impossible, just more difficult than a rewrite. Maintenance programming is like that; more difficult. True, but there's several times more developers downstream that upstream. There's also a big investment going in support personnel, and there's also the lost learning opportunities because of bugs, and also the duplicated efforts in each deployment. I think that when the Sugar ecosystem matures further, we'll reach much bigger efficiency in development and maintenance. Maintenance is also less shiny and interesting to newcomers, so I would avoid relying only on volunteers for that. If the current writers are unable to dedicate that extra time, then an upgrade becomes the only option they will encourage. git://git.sugarlabs.org/sugar-datastore/mainline.git I've reviewed the changes just now from 0.82 to 0.84. It was certainly an entire rewrite, so what you are in effect asking is for someone to support the 0.82 DS to fix a problem that is not yet well defined. Is Benjamin Saller available? ;-) Perhaps we could ask? You've said that coincident with total datastore loss you've also observed the datastore directory is renamed and journal items are no longer listed. I've looked for this code, but have not found it. Can anyone tell me where in the sugar or sugar-datastore code this rename decision is made? And how we'll handle similar issues when 0.84 is deployed and if dataloss bugs are reported. There is no evidence yet to suggest that the same bugs will be reported, and since the datastore code is entirely different I think it is unlikely that you'll get the same bugs. Also: the new DS was designed to avoid the causes of data loss that we knew about (chiefly, index corruption), and is also designed to automatically repair the index when that happens. Have there been any instances of entire datastore loss on 0.84 or 0.86? Daniel reported some cases in which the index wasn't correctly autorebuilt and those got fixed. The saved datastore and the logs will be very pertinent to the problem. So that's the first way to handle it ... gather the critical problem data. Yes, I also look forward to that. Thanks for your interest, Tomeu -- «Sugar Labs is anyone who participates in improving and using Sugar. What Sugar Labs does is determined by the participants.» - David Farning ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
Hi all, Sorry for not being in the thread, its 0.86 release time.. On Thu, Sep 17, 2009 at 11:37:04AM +0200, Martin Langhoff wrote: Welcome new maintainer! I am currently doing a bit of work with LatAm deployments, so I do have some notes related to DS. You may be aware of these issues, of course. - Dataloss on startup. We are hearing of some issues in the field with dataloss -- not real dataloss but if there is anything DS doesn't like something during startup, it renames the datastore directory, and creates a new one. This preserves the data, which can be recovered/diagnosed by a DS expert (good); but from the PoV of end users in the field, the data is lost and the diskspace is lost too. A means to recover lost DS entries and diskspace would save a lot of users' data. And the DS is all about users' data :-) Unfortunately for various reasons we are having a hard time getting samples of those lost DSs. Even without them, I am fairly sure we can make a simple recovery Activity. That should be disappeared in 0.84 after Tomeu's work. - Better behaviour when naming/renaming files on a USB stick. See http://www.mail-archive.com/de...@lists.laptop.org/msg19738.html - More resilient behaviour when plugging arbitrary USB disks / sticks (this may be mostly resolved in recent Sugar) Not sure about 0.82 but in 0.84/0.86 datastore doesn't work with non-ds sources, its a shell level task. In the field, correct data-preserving behaviour is what matters most. Even if we had no new features, if we address dataloss scenarios (real and perceived), users will rejoice. So the above are most important. Yeah, I think Tomeu's 0.84 ds is more reliable in that case. There is also some feature work that is interesting: - Integration with the Backup on the XS. Tony Anderson's recent work is a step in teh right direction, but I'd think that it should be a Journal enhancement rather than a separate activity. Could you provide any links? - Integration with Moodle activities -- so that the Journal keeps track of documents' association to a particular Moodle-side activity, and auto-uploads to the right place. The recent GSoC project around printing exercises some of these aspects. That could be very useful and not only in case of Moodle but in common, remote Journal entries from another users, ASLO activities, global sharing on library.sugarlabs.org(I hope we'll manage to do this in 0.88 cycle).. but thats, in my mind, not ds job(which could be simple if follows only local-data scheme). I'm personally going to tackle this field in 0.88 release cycle. -- Aleksey ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 11:47:59AM +0200, Martin Langhoff wrote: We can offer the patch to deployers, and I can integrate it into a build. But our reach is limited there. A user-deployable repair tool is a lot more effective. Hmm. Yes, I can see that it would be, given the deployment constraints. Have you a way to *force* the problem on 0.82, e.g. by chown or chmod of a file in the datastore, or some other method that does not involve an SD card? Has the OLPC Support Gang been consulted? They may have already tooled up. -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
And just a question on Ivan's try/except patch that is so effective in 0.82 in allowing a child to continue using the system with a fresh journal ... would it be true to say that this also hides the cause of the failure to open the journal, and so any system with a renamed datastore would not have any *other* evidence of why, other than the datastore itself? e.g. no logs. -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 12:26, James Cameron qu...@laptop.org wrote: And just a question on Ivan's try/except patch that is so effective in 0.82 in allowing a child to continue using the system with a fresh journal ... would it be true to say that this also hides the cause of the failure to open the journal, and so any system with a renamed datastore would not have any *other* evidence of why, other than the datastore itself? e.g. no logs. The logs for the 5 or so last sessions should be kept in the same logs dir. Maybe they should be moved to the old DS dir when it is moved out of the way? Regards, Tomeu -- «Sugar Labs is anyone who participates in improving and using Sugar. What Sugar Labs does is determined by the participants.» - David Farning ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 12:26 PM, James Cameron qu...@laptop.org wrote: And just a question on Ivan's try/except patch that is so effective in 0.82 in allowing a child to continue using the system with a fresh journal ... would it be true to say that this also hides the cause of the failure to open the journal, and so any system with a renamed datastore would not have any *other* evidence of why, other than the datastore itself? e.g. no logs. As Tomeu points out, if we can get hold of the homedir of the machine soon enough we have both logs and the corrupt datastore. [ Join me on olpc-sur asking for samples! ] m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Fri, Sep 18, 2009 at 09:51, Martin Langhoff
martin.langh...@gmail.com wrote:
On Fri, Sep 18, 2009 at 2:10 AM, James Cameron qu...@laptop.org wrote:
By branching at the 0.82 point as deployed, determining from evidence
Good theoretical re-statement. You post may clarify things for other
readers -- as for me, I know very well what I am requesting. And I do
respect that it is not easy.
It is just... very important.
I've reviewed the changes just now from 0.82 to 0.84. It was certainly
an entire rewrite, so what you are in effect asking is for someone to
support the 0.82 DS to fix a problem that is not yet well defined.
The _root cause_ is not well defined, although Tomeu has pointed out
likely culprits. The results are very well known, and it is possible
recover the situation without surgery. Read:
http://lists.laptop.org/pipermail/devel/2008-May/014601.html
And here you can find steps to repro and a workaround if you want to
see it in action:
http://dev.laptop.org/ticket/7719
(Uy users probably triggering the DS problem in a different way)
As for the code that controls this?
in shell/view/Shell.py:
def _start_journal_idle(self):
# Mount the datastore in internal flash
ds_path = env.get_profile_path('datastore')
try:
datastore.mount(ds_path, [], timeout=120 * \
DBUS_PYTHON_TIMEOUT_UNITS_PER_SECOND)
except:
# Don't explode if there's corruption; move the data out of the way
# and attempt to create a store from scratch.
shutil.move(ds_path, os.path.abspath(ds_path) + str(time.time()))
datastore.mount(ds_path, [], timeout=120 * \
DBUS_PYTHON_TIMEOUT_UNITS_PER_SECOND)
The good news: if recent Sugars have retained this logic, even if we
hit issues with newer datastore code (hope not! :-) ), the same
recovery tool will help for newer Sugar.
No, what we do instead is making the index discardable so it's not our
weak point any more. For the rest of the data we use files and
directories, and as for every operation we only read the files we
need, a missing or unexpected file won't impact all operations, like
did the mount() step in the old DS.
Even when files are missing, the data loss should be contained to that entry.
Regards,
Tomeu
--
«Sugar Labs is anyone who participates in improving and using Sugar.
What Sugar Labs does is determined by the participants.» - David
Farning
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Wed, Sep 16, 2009 at 03:55:07PM -0500, David Farning wrote: Great job and congrats on handing off the modules and a heartfelt thanks to alsroot and Benjamin Berg for pick them up. +1 CU Sascha -- http://sascha.silbe.org/ http://www.infra-silbe.de/ signature.asc Description: Digital signature ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
Welcome new maintainer! I am currently doing a bit of work with LatAm deployments, so I do have some notes related to DS. You may be aware of these issues, of course. - Dataloss on startup. We are hearing of some issues in the field with dataloss -- not real dataloss but if there is anything DS doesn't like something during startup, it renames the datastore directory, and creates a new one. This preserves the data, which can be recovered/diagnosed by a DS expert (good); but from the PoV of end users in the field, the data is lost and the diskspace is lost too. A means to recover lost DS entries and diskspace would save a lot of users' data. And the DS is all about users' data :-) Unfortunately for various reasons we are having a hard time getting samples of those lost DSs. Even without them, I am fairly sure we can make a simple recovery Activity. - Better behaviour when naming/renaming files on a USB stick. See http://www.mail-archive.com/de...@lists.laptop.org/msg19738.html - More resilient behaviour when plugging arbitrary USB disks / sticks (this may be mostly resolved in recent Sugar) In the field, correct data-preserving behaviour is what matters most. Even if we had no new features, if we address dataloss scenarios (real and perceived), users will rejoice. So the above are most important. There is also some feature work that is interesting: - Integration with the Backup on the XS. Tony Anderson's recent work is a step in teh right direction, but I'd think that it should be a Journal enhancement rather than a separate activity. - Integration with Moodle activities -- so that the Journal keeps track of documents' association to a particular Moodle-side activity, and auto-uploads to the right place. The recent GSoC project around printing exercises some of these aspects. Anyway, -- that's my list, FWIW :-) -- cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
Hi Martin, FWIW, your three major points were fixed way back by Tomeu's 0.84 data- store re-write to make for robustness. Actually I'm fairly sure it was these kind'a reports that prompted Tomeu to make that fine effort. Prior to this (0.82) I had seen occasional datastore 'loss', sometime recoverable, sometimes not (while stress testing 3 XOs here). But since 0.84 I've not seen any failures. I guess we're waiting on the OLPC timeline for an official XO signed release with 0.84, before deployments would be comfortable moving forward with the solution. 0.84 is just about 6 months old now, so happy half Birthday! :-) On 17 Sep 2009, at 10:37, Martin Langhoff wrote: Welcome new maintainer! I am currently doing a bit of work with LatAm deployments, so I do have some notes related to DS. You may be aware of these issues, of course. - Dataloss on startup. We are hearing of some issues in the field with dataloss -- not real dataloss but if there is anything DS doesn't like something during startup, it renames the datastore directory, and creates a new one. This preserves the data, which can be recovered/diagnosed by a DS expert (good); but from the PoV of end users in the field, the data is lost and the diskspace is lost too. Fixed in 0.84 A means to recover lost DS entries and diskspace would save a lot of users' data. And the DS is all about users' data :-) Unfortunately for various reasons we are having a hard time getting samples of those lost DSs. Even without them, I am fairly sure we can make a simple recovery Activity. - Better behaviour when naming/renaming files on a USB stick. See http://www.mail-archive.com/de...@lists.laptop.org/msg19738.html Fixed in 0.84 - More resilient behaviour when plugging arbitrary USB disks / sticks (this may be mostly resolved in recent Sugar) Fixed in 0.84 In the field, correct data-preserving behaviour is what matters most. Even if we had no new features, if we address dataloss scenarios (real and perceived), users will rejoice. So the above are most important. +1! Regards, --Gary There is also some feature work that is interesting: - Integration with the Backup on the XS. Tony Anderson's recent work is a step in teh right direction, but I'd think that it should be a Journal enhancement rather than a separate activity. - Integration with Moodle activities -- so that the Journal keeps track of documents' association to a particular Moodle-side activity, and auto-uploads to the right place. The recent GSoC project around printing exercises some of these aspects. Anyway, -- that's my list, FWIW :-) -- cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Thu, Sep 17, 2009 at 4:27 PM, Gary C Martin g...@garycmartin.com wrote: FWIW, your three major points were fixed way back by Tomeu's 0.84 data-store re-write to make for robustness. Actually I'm fairly sure it was these kind'a reports that prompted Tomeu to make that fine effort. Prior to this (0.82) I had seen occasional datastore 'loss', sometime recoverable, sometimes not (while stress testing 3 XOs here). But since 0.84 I've not seen any failures. Somewhat in agreement. Clearly, there have been huge efforts -- and huge thanks go to Tomeu. That is the good side. But a rewrite is always a risky business. Got to channel jwz here and link to this oldie: http://jwz.livejournal.com/154529.html A rewrite means we have to approach the new DS as a new piece of software (or library or service or component). May fix some bugs, may add others -- new code new bugs. Hasn't seen the field. And for example, cannot be easily backported for a trial run in a deployment, and/or to be incorporated into my papercuts release. I guess we're waiting on the OLPC timeline for an official XO signed release with 0.84, before deployments would be comfortable moving forward with the solution. 0.84 is just about 6 months old now, so happy half Birthday! :-) You are right there. But I would like to note that any widely deployed software has a very long release/deploy cycle. This happens on OLPC deployments, and will happen on all large deployments. The DS is a very important component. When we see dataloss bugs, we need a patch for the deployed version, and a rescue my data utility. It is rewritten in the next release is -- in practice -- not very useful. There is no fix for the users in any reasonable timeframe. And in fact, we don't know how that rewrite will work in the field. This is not to disparage Tomeu's work. It is positive, and necessary. But when we think about users and deployments, there is an aspect that is actually more important: getting a fix and a 'repair' path out to users, in a way deployments can use. The current situation with this specific issue is understandable, 8.2.x was of course released at a time of turmoil for us all, and the DS rewrite was done shortly after. So I am strictly looking forward -- and wondering if we can help these users of 0.82 now. And how we'll handle similar issues when 0.84 is deployed and if dataloss bugs are reported. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] new journal and datastore maintainer
On Thu, Sep 17, 2009 at 04:51:47PM +0200, Martin Langhoff wrote: So I am strictly looking forward -- and wondering if we can help these users of 0.82 now. By branching at the 0.82 point as deployed, determining from evidence (those renamed DS directories you have been unable to obtain) what the likely problem is, and then making a minor change to prevent or handle it. The authors of the 0.82 DS or the 0.84 DS might also use their knowledge of the code to comment on the likely causes so that the investigation can be refined. It's not impossible, just more difficult than a rewrite. Maintenance programming is like that; more difficult. If the current writers are unable to dedicate that extra time, then an upgrade becomes the only option they will encourage. git://git.sugarlabs.org/sugar-datastore/mainline.git I've reviewed the changes just now from 0.82 to 0.84. It was certainly an entire rewrite, so what you are in effect asking is for someone to support the 0.82 DS to fix a problem that is not yet well defined. Is Benjamin Saller available? ;-) You've said that coincident with total datastore loss you've also observed the datastore directory is renamed and journal items are no longer listed. I've looked for this code, but have not found it. Can anyone tell me where in the sugar or sugar-datastore code this rename decision is made? And how we'll handle similar issues when 0.84 is deployed and if dataloss bugs are reported. There is no evidence yet to suggest that the same bugs will be reported, and since the datastore code is entirely different I think it is unlikely that you'll get the same bugs. Have there been any instances of entire datastore loss on 0.84 or 0.86? The saved datastore and the logs will be very pertinent to the problem. So that's the first way to handle it ... gather the critical problem data. -- James Cameron http://quozl.linux.org.au/ ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
