[Server-devel] Looking for additional hands on XS
http://www.laptop.org/en/utility/people/opportunities.shtml cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Looking for additional hands on XS
El Thu, 19-08-2010 a las 15:49 -0400, Martin Langhoff escribió: http://www.laptop.org/en/utility/people/opportunities.shtml Ha! I was just about to write you about some XS requirements for Uruguay :-) -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Looking for additional hands on XS
Whoa.. .found my dreamjob.. applying... 2010/8/19 Bernie Innocenti ber...@codewiz.org El Thu, 19-08-2010 a las 15:49 -0400, Martin Langhoff escribió: http://www.laptop.org/en/utility/people/opportunities.shtml Ha! I was just about to write you about some XS requirements for Uruguay :-) -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Cloning
Hi Martin We had some results doing this , but it is actually much more time consuming than installing and configuring the servers individually. What would be really useful is (a) some way of automatic this or (b) the ability to export the XS configuration especially the Moodle settings. David Leeming -Original Message- From: Martin Langhoff [mailto:martin.langh...@gmail.com] Sent: Friday, 13 August 2010 11:46 p.m. To: David Leeming Cc: XS Devel Subject: Re: [Server-devel] Cloning Hi David, super brief - - get the machine in 'single user mode' (init 1 on the commandline will work) - plug new disk in, setup the partition table, and the filesystems - rsync / and /library to the temp mountpoints - remove hardware-specific files: /etc/udev/rules.d/70-persisntent-net.rules - fixup /etc/fstab to match the UID of the new filesystems - run grub targetting the new disk - move new disk to new machine that will _clone_ the XS so it will keep a lot of identifiers (and all the data) from the original XS. Do NOT use this technique for cloning a base install. Use kickstart/anaconda methods to automate many-machine rollouts. cheers, martin On Fri, Aug 13, 2010 at 6:22 AM, David Leeming da...@leeming-consulting.com wrote: Are there any recommended ways of cloning an XS installation? We have tried Ping and Reflect but with difficulty. David Leeming ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[Server-devel] Schoolserver development in Uruguay
I'm currently at Plan Ceibal. As you may know, Uruguay developed its own schoolserver based on Debian, running software developed in-house and managed with CFengine. Yesterday we briefly discussed their future plans for the school server. == Debian vs Fedora == First of all, there's no way they're going to reinstall 2500 schoolservers with Fedora or even a newer release of Debian. Online upgrades would be possible, though. There's some interest in repackaging in Debian the datastore backup server and other components of the OLPC XS. This work could be contributed back to you or whoever will become the next schoolserver architect. Perhaps we could get one of the Debian maintainers in our community to get these packages accepted. I could do the same for Fedora. As you said, recommending or supporting multiple schoolserver configurations in parallel doesn't make sense, but it wouldn't hurt if some of the underlying components were shared horizontally, especially for the configurations that are already widely deployed. == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. I still had no time to review Prosody, the Jabber implementation recommended by Collabora. My hacker senses are telling me that switching from Erlang to Lua is a small step in the direction of sanity and simplicity. The Sugar Labs Infrastructure Team has setup new dedicated VM for collaboration, but at this time nobody has been working on it. It's an Ubuntu Lucid machine, but we could reinstall it if needed. Tomeu and Collabora overwhelmed the collaboration stack in Sugar 0.90 and seem to have plans to further evolve it. They should be consulted prior to making any long-term decision on the server side. == Backups == This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. Loosing all data is sadly the status quo in both Uruguay and Paraguay. I worked on implementing remote backups for a subset of /library using rsync, but 2Mbit per school and 70Mbit on the backup server are insufficient for the initial sync and probably also for nightly updates. What numbers are we talking about, in terms of size? Here are some numbers from an actual school which has been operating for over one year with 530 registered laptops: 262M backup 19Gcache 3.4M games 1.7M orug 62Mpgsql-xs 67Muploads 238G users 20Kwebcontenido 17Mxs-activation 516M xs-activity-server 827M xs-rsync 2.7G zope-var The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. == Content management == Paraguay seems quite happy with Plone, but frankly I can't understand why. Teachers heavily use a really simple php tool called PAFM, which provides basic hierarchical file management with no access control or versioning. Oddly, I've not yet may anyone using Moodle. When I ask why, I always hear some vague comment about it being designed for higher education. Same goes for Schooltool. These more structured tools probably present an steeper learning curve and a bad fit for unsophisticated requirements of users who are being exposed to information systems for the first time. After they have functioning backups, Uruguay would like to provide a wiki. They have already looked at Dokuwiki, with which I'm not familiar. It seems to have a readable and easy to learn Creole-like syntax. I would personally recommend going either for the simplest possible wiki in this category, or straight to Mediawiki--the most feature-complete out there. Any mid-size solution such as MoninMoin is likely to bring the worst of both worlds. Having written my own minimalist wiki, perhaps I'm slightly biased on this topic. Just slightly, yeah :-) Seriously, the choice of wiki would depend on what other tools would complement it. If you already have Moodle or Schooltool, you probably need just a basic wiki for taking notes on the side. With Mediawiki, one would probably install a bunch of powerful extensions to build
Re: [Server-devel] Schoolserver development in Uruguay
Bernie, Guys: A few of my ideas are below: 2010/8/19 Bernie Innocenti ber...@codewiz.org I'm currently at Plan Ceibal. As you may know, Uruguay developed its own schoolserver based on Debian, running software developed in-house and managed with CFengine. Yesterday we briefly discussed their future plans for the school server. == Debian vs Fedora == First of all, there's no way they're going to reinstall 2500 schoolservers with Fedora or even a newer release of Debian. Online upgrades would be possible, though. There's some interest in repackaging in Debian the datastore backup server and other components of the OLPC XS. This work could be contributed back to you or whoever will become the next schoolserver architect. Whichever the distro... It should be easily maintainable and deployable. I've no problem in building the dpkg's of the XS components. I think its justified, since there is a 2500 servers base. Let me tell you that I've only worked with rpm before, but I've no problem in learning the Debian guidelines, and maybe, try to push the packages into debian testing. Perhaps we could get one of the Debian maintainers in our community to get these packages accepted. I could do the same for Fedora. As you said, recommending or supporting multiple schoolserver configurations in parallel doesn't make sense, but it wouldn't hurt if some of the underlying components were shared horizontally, especially for the configurations that are already widely deployed. == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. I still had no time to review Prosody, the Jabber implementation recommended by Collabora. My hacker senses are telling me that switching from Erlang to Lua is a small step in the direction of sanity and simplicity. The Sugar Labs Infrastructure Team has setup new dedicated VM for collaboration, but at this time nobody has been working on it. It's an Ubuntu Lucid machine, but we could reinstall it if needed. Tomeu and Collabora overwhelmed the collaboration stack in Sugar 0.90 and seem to have plans to further evolve it. They should be consulted prior to making any long-term decision on the server side. == Backups == This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. Loosing all data is sadly the status quo in both Uruguay and Paraguay. I worked on implementing remote backups for a subset of /library using rsync, but 2Mbit per school and 70Mbit on the backup server are insufficient for the initial sync and probably also for nightly updates. What numbers are we talking about, in terms of size? Here are some numbers from an actual school which has been operating for over one year with 530 registered laptops: 262M backup 19Gcache 3.4M games 1.7M orug 62Mpgsql-xs 67Muploads 238G users 20Kwebcontenido 17Mxs-activation 516M xs-activity-server 827M xs-rsync 2.7G zope-var The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. How about 2 500GB in RAID-1? I mean, specially in Paraguay, bandwidth is scarce. == Content management == Paraguay seems quite happy with Plone, but frankly I can't understand why. Teachers heavily use a really simple php tool called PAFM, which provides basic hierarchical file management with no access control or versioning. Oddly, I've not yet may anyone using Moodle. When I ask why, I always hear some vague comment about it being designed for higher education. Same goes for Schooltool. These more structured tools probably present an steeper learning curve and a bad fit for unsophisticated requirements of users who are being exposed to information systems for the first time. After they have functioning backups, Uruguay would like to provide a wiki. They have already looked at Dokuwiki, with which I'm not familiar. It seems to have a readable and easy to learn Creole-like syntax. I would
[Server-devel] Schoolserver development in Uruguay
Comments regarding the initial paraguayian deployment, i'm not very familiar with the current status Regarding distros, when the initial setup was made, the XS (fedora based) schoolserver was the only straightforward instalation that could have anything working with not so much tampering, and was pretty automatic, so XS was chosen, I'm a regular sysadmin, and i got the thing working given enough research time, and martin's help.. is not always like that.. we must take into account that there are not many people that could get an XS working on any given distro, and although there are many volunteers (like bernie) who go around the world doing this things, sustainability is very far away. If there is a distro, or many distros, is not the real problem, the real problem is that there needs to be a simple straightforward and automatic way to deploy a schoolserver without needing a masters in computer science, or even a deegree at all. It has to be fast and it has to be simple. In Plan Ceibal worked fine using debian, because they have specialized people that can do the develpment and can do the maintenance, not because the chose this or that distro In Paraguay, Fedora was chosen for the same reasons, it was the fastest way to get things done, and the simple way to sustain it in the long term, with XS and Fedora patches, which I don't know if were made later on. == Jabber == I don't really understand much how collaboration works, so, no comments == Backups == This numbers make sense to paraguay deployment but may not make sense to other deployments, so I'll explain the folders that I remember 262M backup Backup folder, where all data that was going to be rsynced to the datacentes was stored, it would amount to a backup of the plone, the databases, some configs, and other stuff 3.4M games This was a folder where a web based game was going to be stored, this would be published by the apache web server 1.7M orug Same as before, it was a game developed by a paraguayian legal team to help kids learn about thir rights 62M pgsql-xs I don't remembre 67M uploads The PAFM web folder discussed leter 238G users The datastores folder 20K webcontenido The apache default webpage, with specialized links for games, activities and others 17M xs-activation 516M xs-activity-server 827M xs-rsync I don't remember 2.7G zope-var Since plone works with a selfcontained filesystem for its webpage, this _single_ file was going to be backed up to the datacenter as well, i think this is the folder, but i remember that it had to go to the backup folder anyways The feasibility of remote backups varies depending on how much we care to backup. In Paraguay, it was decided that the journal backups are to be considered a valuable if we are to instill the idea in teachers that the laptop is the same of a notebook with homework on it. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. This problem is more related to the way the journal stores the files and the metadata, I remember little about it, but the main problem with backing up a laptop is no just about taking files, any single file in the datastore doesn't get you a back up, you have to take the whole datastore folder. Incremental, or differential backups could be made if the datastore treated the files differently, I'm sorry if I hurt some suceptibilities, but is the truth, there's no simple way to back up _just the data_ from the journal, you back it all or nothing, because _part of it_ is useless. I don't know if that improved in the newer version, but 0.82 (i think) is the one that was used in Paraguay, is like that Yesterday Daniel Castelo and I discussed the idea of performing cross-backups between nearby schools. This solution would probably work well in terms of bandwidth distribution, but it would bring some logistic complexity. Probably an acceptable trade-off. This an interesting idea, and is related to the sustainability part of the deployment, and XS Deploying a schoolserver should be made easy, this would help small deployments and big deployments, the faster the server gets to the school the better, we all know that, but the real advantage is when maintenance can be made from remote or with simple and fast solutions, like puppet, CFEngine, or even self conained rpm/deb packages, becuase this is the way that we get the masters in computer scince in every schoolserver we want. The faster that a schoolserver can be install, and for the matter reinstalled and restore the better, because then you would only need to send a guy (not a sysadmin) to go, insert a CD, next, next, next, voila.. even when changes need to be done.. they should be done in the way that can be applyed to the
Re: [Server-devel] [Tecnologia] Schoolserver development in Uruguay
El Thu, 19-08-2010 a las 21:57 -0400, Rodolfo D. Arce S. escribió: If there is a distro, or many distros, is not the real problem, the real problem is that there needs to be a simple straightforward and automatic way to deploy a schoolserver without needing a masters in computer science, or even a deegree at all. It has to be fast and it has to be simple. Yes, this is true. Basic IPv4 setup and swapping a broken network card currently require too much command line wizardry. This is probably due to lack of time to develop a nicer interface, not a sadistic design choice. Also, once you have more servers than you can count on the palm of your hand, you really need advanced management tools such as CFengine, Puppet and BCFG2 along with advanced monitoring tools. With such advanced tools come the need for advanced UNIX sysadmins to install and manage them. I don't see any practical way to sweeten this pill. In Plan Ceibal worked fine using debian, because they have specialized people that can do the develpment and can do the maintenance, not because the chose this or that distro I agree, one distro is more or less equivalent to another. What really matters is how much experience you have acquired. 20Kwebcontenido The apache default webpage, with specialized links for games, activities and others This does not seem to be visible in the schools. If I visit http://schoolserver , I get redirected to Plone. 17Mxs-activation 516M xs-activity-server 827M xs-rsync I don't remember I know about these... they don't need to be backed up. 2.7G zope-var Since plone works with a selfcontained filesystem for its webpage, this _single_ file was going to be backed up to the datacenter as well, i think this is the folder, but i remember that it had to go to the backup folder anyways Isn't there a way to perform garbage collection on this file? Or do you think that school is really storing 2.7GB of data in Plone? This problem is more related to the way the journal stores the files and the metadata, I remember little about it, but the main problem with backing up a laptop is no just about taking files, any single file in the datastore doesn't get you a back up, you have to take the whole datastore folder. Incremental, or differential backups could be made if the datastore treated the files differently, I'm sorry if I hurt some suceptibilities, but is the truth, there's no simple way to back up _just the data_ from the journal, you back it all or nothing, because _part of it_ is useless. Yes, but metadata consists of very tiny files compared to actual data files, which is often multimedia content. Sascha designed some kind of datastore object bundle which encapsulates data and metadata in a single archive, to be used for backups and data transfers. Even if it were already available, this new backup format wouldn't solve the fundamental problem that backups of all journals in a school amount to 260GB, though. Backing up the datastore with a simple rsync is simple and fast. Note that hardlinks are already being used to reduce the size of incremental backups as much as possible and the Sugar datastore does not modify or move around application data once it has been created. I don't know if that improved in the newer version, but 0.82 (i think) is the one that was used in Paraguay, is like that The datastore layout changed radically from 0.82 to 0.84. In Dextrose-1, we force an index rebuild after restoring a backup, so even a partial backup would be useful. But why would we not want to backup everything? [...] Paraguay in that sense did something really cool.. install the schoolserver _in house_, update it and send it to the field, but in the fild had to go somebody that could make the adjustments for the network to work, and that simple task was very difficult sometimes, and updating the leases and content when no conection is available is not fun either. After over one year, we're still sending out a sysadmin from Asuncion to Caacupe for every non-trivial operation that needs to be performed from the console. This would obviously not work with a nation-wide deployment. Tomorrow I'll try to interview the people of Plan Ceibal to figure out whether they managed to train field technicians even in the most remote areas or if they've got a sysadmin taskforce that gets parachuted on demand. The plone deployment was based on a decision to give the schools a simple CMS, not simple in deployment, but simple in using, using a plone CMS to create content is super friendly, but in time, none of the teachers or students got to used it (simple or not) The per-school Plone instances have been superseded by a centralized version which the education team seems to be using for uploading documents: http://biblioteca.paraguayeduca.org/ They used this tool instead because they learned how to use it during teachyer training, and they like it, it had
Re: [Server-devel] Schoolserver development in Uruguay
On 19 August 2010 18:25, Bernie Innocenti ber...@codewiz.org wrote: == Jabber == There are two people working on Jabber. They have been using ejabberd and, quite surprisingly, they've not seen any issues of high CPU load and database corruption. Tomorrow I'll get to work more with them. XS-0.6 and some of the package updates that come later fix a few bugs related to ejabberd CPU/DB. I guess in Paraguay they are still on 0.5. This is a black hole in all deployments I visited. Redundant storage is too expensive. One cheap 500GB hard-drive is typical. In one year, 3 of the 10 schoolservers in Caacupé developed a hard drive failure. But it's not a huge issue because the XOs also have a copy of the journal. So, if technical resources are available for a quick XS repair, disruption should be minimal. Journal backups, however, amount to a whopping 238GB of rapidly changing, mostly uncompressible and undeltable data. Quite not the ideal case for an incremental backup. With today's available resources, we could afford to backup everything *but* the journals. You're giving numbers but missing an important consideration - the XS backup system makes multiple backups. And it'll continue to do make more and more copes until it meets a certain threshold based on disk size (likely to be 238GB in your case). At this point, it will purge the oldest backups before making new ones. Saying that you've hit 238GB after a year isn't conclusive because its likely that you'll meet the threshold when you're measuring an active school over such a long time period. It's the design - use the available space. It's possible that within that space you have 10 backups of every journal. So you could possibly get away with a disk half the size, and only retain 5 copies. I'm inventing numbers (and they aren't strictly copies either), but you can provide real ones - how many backups (on average) are there of a journal in this server? What's the disk space used if you only total the space used by the most recent backup of each journal? Also, is it possible that your space-measuring script is counting a 5mb file with 2 hardlinks as 10mb of used disk space? Paraguay uses Puppet. We're very happy with it. Uruguay uses CFengine. They seem to be very happy with it as well. Both employ a flat hierarchy with one puppet master controlling all the schools, which is simple and straightforward, but requires excellent connectivity. Excellent is a bit subjective, but yes, the fact that it requires any form of connectivity is a roadblock in many cases. However, we came up with a way around this (ideas only, for now, but wouldn't be hard to implement) for puppet: - clone all the puppet repositories and the config files and put them on a USB disk (and do this periodically) - install puppet-server on all the XSs (but dont run it by default) - go to a school with said USB disk, plug it in and run puppet-server - run puppet-client, connecting to localhost - stop puppet-server, unplug USB disk, go home Daniel ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] Schoolserver development in Uruguay
El Fri, 20-08-2010 a las 00:51 -0300, Bernie Innocenti escribió: Heh, these are good questions, but answering them all would take quite some time, and it's 1AM over here :-) Meanwhile, my du run to find out the size of current backups completed: # du -sh --exclude datastore-200* /library/backup 92G/library/backup So, backing up the last versions of all journals would take just 92GB, which would take more that 4 days on a 2mbit link for the initial backup. -- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
