Re: Ubuntu System Restore
Hello Michael, Thanks for your reply . On Tue, Nov 8, 2011 at 8:02 PM, Michael Vogt m...@ubuntu.com wrote: On Sat, Oct 29, 2011 at 09:19:28PM +0530, Gaurav Saxena wrote: Hello Michael Hi Gaurav, sorry for my slow reply. On Fri, Oct 7, 2011 at 2:05 PM, Michael Vogt m...@ubuntu.com wrote: On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. [..] 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. You may want to look at the apt-clone package for this part of the work, it supports creating/restoring this meta-data. Could you suggest something to me that how can I use apt-clone in my system restore program to backup the states of system packages. I read articles regarding that like http://swik.net/apt-clone which say that I need to have a ZFS file system for managing snapshots and also its just a front-end to apt-get. There is a unfortuate name clash here, the apt-clone that uses zfs on solaris is different from the one we have in the archive. Ohh I see that's why I was confused with how did they use apt-clone to create such a system, Thanks for clearing that big doubt, Actually apt-clone search on google gives result related to their system. Can we implement a system like that using apt-clone and brtfs? Our apt-clone create a system-state file by capturing installed packages, auto-install states, sources.list etc. apt-clone --help should give a overview. Its possible to test using e.g. apt-clone restore --destination=/tmp/foo clonefile this will restore into a chroot dir. Careful otherwise, the default restore location is /. Ok. And then those packages backed up using dpkg-repack or by downloading are installed using dpkg ? Can this system fix broken upgrades ? Cheers, Michael Cheers, Michael On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list
Re: Ubuntu System Restore
On Sat, Oct 29, 2011 at 09:19:28PM +0530, Gaurav Saxena wrote: Hello Michael Hi Gaurav, sorry for my slow reply. On Fri, Oct 7, 2011 at 2:05 PM, Michael Vogt m...@ubuntu.com wrote: On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. [..] 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. You may want to look at the apt-clone package for this part of the work, it supports creating/restoring this meta-data. Could you suggest something to me that how can I use apt-clone in my system restore program to backup the states of system packages. I read articles regarding that like http://swik.net/apt-clone which say that I need to have a ZFS file system for managing snapshots and also its just a front-end to apt-get. There is a unfortuate name clash here, the apt-clone that uses zfs on solaris is different from the one we have in the archive. Our apt-clone create a system-state file by capturing installed packages, auto-install states, sources.list etc. apt-clone --help should give a overview. Its possible to test using e.g. apt-clone restore --destination=/tmp/foo clonefile this will restore into a chroot dir. Careful otherwise, the default restore location is /. Cheers, Michael Cheers, Michael On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Ubuntu System Restore
Hello Bear Giles Thanks a lot for your help. On Mon, Oct 31, 2011 at 2:10 AM, Bear Giles bgi...@coyotesong.com wrote: You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) Ok. A local repository for keeping the packages for being installed offline, also can I save dpkg packages using dpkg-repack and then restore them later using dpkg -i ? The dpkg-repack saves the changes done to the conffiles the man page says that. But I haven't tried that yet, do you have any idea about that ? It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. Ok. Yes this could be a problem , may be this can be solved using dpkg-repack for saving the packages .deb files. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. Yes I had also confusion regarding that. It would be a lot more difficult then (in case dpkg-repack also does not recognize the file or the changes done to that file) . Then I need to extract the archive and then see for the changes. Also if some files of a package are corrupted is there any way of checking that ? The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. Ok thanks for this idea. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. Could you please tell some detail about
Re: Ubuntu System Restore
Hello John, On Mon, Oct 31, 2011 at 2:27 AM, John Moser john.r.mo...@gmail.com wrote: The simple way to create a restore point system ... ... is to mount / as an overlay FS, which you periodically merge (to remove prior restore points), condense into a squashfs (to take a point backup), or wipe (to restore to backup). This of course means /home should be its own partition. I think this is the idea used by nexenta by mounting / on a ZFS partition, Can a similar approach used for ubuntu ? But that needs / to be mounted on a brtfs filesystem, Does that need a thorough reinstall of the system ? This approach along with package saved state can help to create a break free upgrade mechanism as is done in nexenta currently. On Sun, Oct 30, 2011 at 4:40 PM, Bear Giles bgi...@coyotesong.com wrote: You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. Could you please tell some
Re: Ubuntu System Restore
Going back slightly you can specify a script/program that will be run by dpkg at key points when packages are installed and removed. I never went beyond logging the parameters to a log but I remember at one point you get the location of the .deb file being installed. You could grab information from it at that time. (My idle thought was wondering how much effort it would be to have an add-on that would set the 'immutable' flag on all non-conffile files as they're written, then removing the flag before they're deleted. This is for a mixture of oops-proofing and making life a little harder for attackers. It wouldn't slow down a knowledgeable attacker.) Bear On Mon, Oct 31, 2011 at 11:32 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello John, On Mon, Oct 31, 2011 at 2:27 AM, John Moser john.r.mo...@gmail.comwrote: The simple way to create a restore point system ... ... is to mount / as an overlay FS, which you periodically merge (to remove prior restore points), condense into a squashfs (to take a point backup), or wipe (to restore to backup). This of course means /home should be its own partition. I think this is the idea used by nexenta by mounting / on a ZFS partition, Can a similar approach used for ubuntu ? But that needs / to be mounted on a brtfs filesystem, Does that need a thorough reinstall of the system ? This approach along with package saved state can help to create a break free upgrade mechanism as is done in nexenta currently. On Sun, Oct 30, 2011 at 4:40 PM, Bear Giles bgi...@coyotesong.com wrote: You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without
Re: Ubuntu System Restore
You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. Could you please tell some detail about autogenerated packages ? Like if we have a list of packages installed on the system, We need to reinstall those all softwares and remove those which are installed after the restore point? BTW Debian has a C++ interface to the package metadata. I've never used it - I find it easier to just scan the metadata directory myself. There's also hooks that will allow your application to be called at each step during a package installation or removal. You could, in theory, keep your snapshots current to the last minute that way. So whenever a package is installed or removed I will update my backup according to that ? By reading package metadata determine which files are changed by that package and update those files in the backup ? On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de
Re: Ubuntu System Restore
The simple way to create a restore point system ... ... is to mount / as an overlay FS, which you periodically merge (to remove prior restore points), condense into a squashfs (to take a point backup), or wipe (to restore to backup). This of course means /home should be its own partition. On Sun, Oct 30, 2011 at 4:40 PM, Bear Giles bgi...@coyotesong.com wrote: You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. Could you please tell some detail about autogenerated packages ? Like if we have a list of packages installed on the system, We need to reinstall those all softwares and remove those which are installed after the restore point? BTW Debian has a C++ interface to the package metadata. I've never used it - I find it easier to just scan the metadata directory myself. There's also hooks that will allow your application to be called at each step during a package installation or removal. You could, in theory, keep your snapshots current to the last minute that way. So
Re: Ubuntu System Restore
LVM lets you create a snapshot where the mounted filesystem looks normal but under the cover it's using a journal and the original logical volume, e.g., /dev/mapper/vg0/home, is untouched. You can then perform your backup and when you release the snapshot the journal is written to the original volume. Obviously this requires a backup app that runs against the raw device instead of the mounted filesystem. 'dump' does that, 'tar' does not. On Sun, Oct 30, 2011 at 2:57 PM, John Moser john.r.mo...@gmail.com wrote: The simple way to create a restore point system ... ... is to mount / as an overlay FS, which you periodically merge (to remove prior restore points), condense into a squashfs (to take a point backup), or wipe (to restore to backup). This of course means /home should be its own partition. On Sun, Oct 30, 2011 at 4:40 PM, Bear Giles bgi...@coyotesong.com wrote: You need to either have a local repository or download from the internet again. I've used 'apt-mirror' in the past to maintain a local cache but that was when I was building local systems with a minimal Debian installer. I don't even know if the standard Ubuntu installer can load off a local cache. (I guess the process is to do the install without updates, change your sources.list files, then upgrade from the cache.) It's also worth remembering that the specific versions of your packages may not be available when you need to restore your system. This is usually a good thing since more recent versions have a shot at preventing whatever caused you to lose your system in the first place (e.g., closing vulnerabilities) but some people may need to restore the system exactly. On checksums - I checked my system and almost none of the conffiles have checksums. (In fact that may be against packaging standards - I would have to check.) That's a bummer since it means that there's no easy way of seeing what's changed unless you peek into the .deb file. There are some deb tools that can do this but since I can do it programmatically I usually just did that. The 'monster diff' is just a comment on the number of files involved. What I actually did create two lists, one generated by walking the filesystem and the other generated by concatenating all of the *.files and *.md5sums metadata and then comparing them. I did this programmatically but you could also create actual files, sort them, and then run a diff on them. IIRC I typically had over 70k files. On Fri, Oct 28, 2011 at 3:33 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including
Re: Ubuntu System Restore
On 10/30/2011 05:17 PM, Bear Giles wrote: LVM lets you create a snapshot where the mounted filesystem looks normal but under the cover it's using a journal and the original logical volume, e.g., /dev/mapper/vg0/home, is untouched. You can then perform your backup and when you release the snapshot the journal is written to the original volume. Actually, the snapshot keeps copies of any blocks that have been modified in either the original or the snapshot volume, and when you delete the snapshot, they are all discarded. More recent versions of LVM now allow you to merge ( with lvconvert ) the snapshot into the original volume the next time it is remounted, effectively allowing you to revert the original volume to the snapshot state. Obviously this requires a backup app that runs against the raw device instead of the mounted filesystem. 'dump' does that, 'tar' does not. It does not require that since you can just mount the snapshot, then use tar, but in my experience, dump is much faster than tar. I've looked at trying to implement a system restore based on LVM for the last year or so and have concluded that it just isn't practical. There are a number of problems that make it so: 1) You must reserve space for the snapshot up front, and if that space runs out, the snapshot is lost. 2) Because it operates on a block level, it is rather inefficient. Specifically, you can write a new file to disk, and then delete it, but the blocks it was written to remain in the snapshot exception store, and may be somewhat larger than the actual file size, as the snapshot block size defaults to 64k iirc. These no longer used blocks also then have to be copied back to the original location during a merge, whether or not they actually contained any data in the snapshot. 3) Having a snapshot active has an appreciable write performance penalty on the original volume. This penalty grows linearly worse as you add more snapshots because each new write to the original volume results in data being copied out to each of the active snapshots first. It appears that btrfs is the way to go for system snapshots, and the apt-btrfs-snapshot package appears to be a good first step in this direction. It currently causes apt to make a btrfs snapshot before each upgrade/install and provides a command line interface to list, delete, and rollback to snapshots. It is written in python and I have been thinking of adding a little gtk gui to it. I spent this weekend patching dpkg to support a --force-no-syncs option so that apt can instruct it to avoid all of the syncs dpkg normally does as they are rendered completely pointless when you can just rollback to the snapshot if things go wrong, and have a large performance penalty, especially on btrfs. I've also been thinking of adding some support to grub to allow you to choose to roll back to a snapshot right from there, in case you can't even boot the system to ask apt-btrfs-snapshot to do it for you. -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Ubuntu System Restore
Hello Michael On Fri, Oct 7, 2011 at 2:05 PM, Michael Vogt m...@ubuntu.com wrote: On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. [..] 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. You may want to look at the apt-clone package for this part of the work, it supports creating/restoring this meta-data. Could you suggest something to me that how can I use apt-clone in my system restore program to backup the states of system packages. I read articles regarding that like http://swik.net/apt-clone which say that I need to have a ZFS file system for managing snapshots and also its just a front-end to apt-get. Cheers, Michael On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Ubuntu System Restore
Hello all On Fri, Oct 7, 2011 at 4:45 AM, Bear Giles bgi...@coyotesong.com wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. But it might be possible that the package files are not available on the system. That means for all the packages installed the .deb files need to be downloaded from the internet for restore purpose ? 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. Ok. Nice idea indeed , but is there checksum associated with the files in the package ? Or that can be calculated at the time of restore ? What you say ? 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. Could you suggest me some way of scanning the debian package metadata without actually downloading the packages? and how to this monster diff ? It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. Could you please tell some detail about autogenerated packages ? Like if we have a list of packages installed on the system, We need to reinstall those all softwares and remove those which are installed after the restore point? BTW Debian has a C++ interface to the package metadata. I've never used it - I find it easier to just scan the metadata directory myself. There's also hooks that will allow your application to be called at each step during a package installation or removal. You could, in theory, keep your snapshots current to the last minute that way. So whenever a package is installed or removed I will update my backup according to that ? By reading package metadata determine which files are changed by that package and update those files in the backup ? On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I
Re: Ubuntu System Restore
Hello Bear Giles,
Thanks for your reply
Sorry for my late reply.
On Fri, Oct 7, 2011 at 9:30 PM, Bear Giles bgi...@coyotesong.com wrote:
Heh, for some reason I thought this was on the local linux users group
instead of ubuntu-devel. Hence the more generic language.
Anyway feel free to take the other two points as possible products.
My prototypes have been shell commands followed by simple java apps that did
the same work. In both cases they directly read the files in
/var/lib/dpkg/info or, in a few cases, the naked .deb files. They were
always intended as proof-of-concept demos, not something suitable for
distribution.
Ok. We can gather information about the packages installed files using
these files.
The two specific tasks that came up a lot are:
1. List all files under { /bin, /sbin, /usr, /lib, /etc and parts of /var }
that:
a) are present but not in a package as either a regular file or a
conffile,
b) are present but have a different md5sum than the package, or
c) are in an installed package but not present.
This can return a simple list of files that can be used as input to other
applications.
So these files are the files which need to be backed up? Actually could you
please somewhat point out what needs to done when a restore point is created
or when a restore is required. I understand from your idea that we are
creating a restore point whenever user installs a new package and restore
would be done when the user needs that. The above things need to be done
when a restore is required by the user, I guess? So the system restore will
remove (a) , overwrite/ replace (b) and copy (c) from the package .deb files
, which will be either downloaded from the Internet.
Now these files will be copied to the directories by the restore program.
There is an issue involved in this if the files in /var/lib/dpkg/info are
themselves modified/damaged/corrupted which would leave the system in an
inconsistent state.
An extension is to check the permissions, e.g., by looking into the .deb
file itself and examining the data.tar.gz file. There's also always the
standby
$ find / -perm +1000 -ls
$ find / -perm +2000 -ls
to find suid/sgid files in the system. Simply knowing what files should
have these permissions would be helpful.
You can do a quick MD5 sanity check with
$ cat /var/lib/dpkg/info/*.md5sums | sort | uniq /tmp/md5sums
$ md5sum -c /tmp/md5sums
Ok. Thanks for help I will try these things, would checking the file
permissions help? in case user deliberately modified the permissions of
certain files so as to solve certain issues.
2. Create a local package containing modified conffiles.
You can get a list of conffiles from
$ cat /var/lib/dpkg/info/*.conffiles | sort | uniq /tmp/conffiles
and convert that to a list of md5 files with a join with a bit of work.
(conffiles have a leading slash on the path, md5sums do not). Then it would
be a simple matter of creating a tarball of the modified files and wrapping
it up so that you can perform all of your customization by installing a
single package.
So we will create a package which when installed will revert the system to
the state at the restore point/earlier state ? But this might cause problems
when the dpkg files are themselves corrupt and system is in an inconsistent
state and dpkg cannot be used for installing or removing packages, in that
case this method won't work. But still will work for a lot many situations.
We have to perform this step whenever user needs to create a restore point ?
So that the files and state of the system is saved.
3. Not related to the original question but I've also wondered about a
hardening package that installs a cron task that's package-aware. That is, I
(should) be regularly performing tasks like:
$ find { everything but /dev } -type b -ls
$ find { everything but /dev } -type c -ls
That is, looking for character or block devices that are anywhere but under
/dev. That's never a good thing.
$ find /home -uid -1000
That is, looking for any system files under /home. Again this is never a
good thing.
There's a dozen or so checks that can be done but in some cases there will
be expected hits, depending upon the actual packages installed. It should be
possible to customize this task so it knows what to ignore and what to flag
as a problem.
Ok. We can also add a feature to exclude/include certain directories as to
be restored , as user might be aware of these things like system files under
home.
Bear
On Fri, Oct 7, 2011 at 2:35 AM, Michael Vogt m...@ubuntu.com wrote:
On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote:
I've written a few prototypes and this comes down to four issues. Some
of
the details below are debian/ubuntu-specific but the same concepts will
apply to redhat.
[..]
2. Packages should NOT be backed up. All you need is the package name
and
version. Reinstall from .deb and .rpm if
Re: Ubuntu System Restore
Hello Michael Vogt, Yes I have had a look at your blog post related to apt-clone, I am looking at how to use that in this project. Thanks for helping. On Fri, Oct 7, 2011 at 2:05 PM, Michael Vogt m...@ubuntu.com wrote: On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. [..] 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. You may want to look at the apt-clone package for this part of the work, it supports creating/restoring this meta-data. Cheers, Michael On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Ubuntu System Restore
On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote: I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. [..] 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. You may want to look at the apt-clone package for this part of the work, it supports creating/restoring this meta-data. Cheers, Michael On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: Ubuntu System Restore
Heh, for some reason I thought this was on the local linux users group
instead of ubuntu-devel. Hence the more generic language.
Anyway feel free to take the other two points as possible products. My
prototypes have been shell commands followed by simple java apps that did
the same work. In both cases they directly read the files in
/var/lib/dpkg/info or, in a few cases, the naked .deb files. They were
always intended as proof-of-concept demos, not something suitable for
distribution.
The two specific tasks that came up a lot are:
1. List all files under { /bin, /sbin, /usr, /lib, /etc and parts of /var }
that:
a) are present but not in a package as either a regular file or a
conffile,
b) are present but have a different md5sum than the package, or
c) are in an installed package but not present.
This can return a simple list of files that can be used as input to other
applications.
An extension is to check the permissions, e.g., by looking into the .deb
file itself and examining the data.tar.gz file. There's also always the
standby
$ find / -perm +1000 -ls
$ find / -perm +2000 -ls
to find suid/sgid files in the system. Simply knowing what files should have
these permissions would be helpful.
You can do a quick MD5 sanity check with
$ cat /var/lib/dpkg/info/*.md5sums | sort | uniq /tmp/md5sums
$ md5sum -c /tmp/md5sums
2. Create a local package containing modified conffiles.
You can get a list of conffiles from
$ cat /var/lib/dpkg/info/*.conffiles | sort | uniq /tmp/conffiles
and convert that to a list of md5 files with a join with a bit of work.
(conffiles have a leading slash on the path, md5sums do not). Then it would
be a simple matter of creating a tarball of the modified files and wrapping
it up so that you can perform all of your customization by installing a
single package.
3. Not related to the original question but I've also wondered about a
hardening package that installs a cron task that's package-aware. That is, I
(should) be regularly performing tasks like:
$ find { everything but /dev } -type b -ls
$ find { everything but /dev } -type c -ls
That is, looking for character or block devices that are anywhere but under
/dev. That's never a good thing.
$ find /home -uid -1000
That is, looking for any system files under /home. Again this is never a
good thing.
There's a dozen or so checks that can be done but in some cases there will
be expected hits, depending upon the actual packages installed. It should be
possible to customize this task so it knows what to ignore and what to flag
as a problem.
Bear
On Fri, Oct 7, 2011 at 2:35 AM, Michael Vogt m...@ubuntu.com wrote:
On Thu, Oct 06, 2011 at 05:15:14PM -0600, Bear Giles wrote:
I've written a few prototypes and this comes down to four issues. Some of
the details below are debian/ubuntu-specific but the same concepts will
apply to redhat.
[..]
2. Packages should NOT be backed up. All you need is the package name and
version. Reinstall from .deb and .rpm if necessary since this way you're
sure that you never restore compromised files.
You may want to look at the apt-clone package for this part of the
work, it supports creating/restoring this meta-data.
Cheers,
Michael
On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.com
wrote:
Hello Aaron
Thanks a lot for your quick reply.
On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn
aa...@heyaaron.comwrote:
In Windows, the ability to snapshot is built into the filesystem.
In Linux, you must be running a filesystem that supports snapshots. I
know LVM supports snapshotting and I believe BRTFS has support, but
other than that I'm not sure.
Yes I read the logic behind windows system restore. But I think we can
take some other approach for this, that will be better as all users
won't be
able to spare an extra partition formatted brtfs.
Basically, your program would have to check the file system that is
used on the computer (remember Linux can have many types of file
systems mounted at the same time), then (in the case of LVM) make sure
there's enough free space to snapshot, and finally take the snapshot.
Ok. Do I have to snapshot the whole system partition / important
system
files to the brtfs partition ?
When the snapshots start filling up, you would either need to delete
them or detect the low space and resize them.
In my personal opinion, snapshotting in Linux is currently a pain in
the rear. It sounds like BTRFS could change that, but it's still a
ways off.
Ok. I will try another approach that will be better as suggested by
people here.
-A
On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com
wrote:
Hello all,
I want to write a windows system restore like program for ubuntu ,
which
will have options for creating restore points for the system and
then
Re: Ubuntu System Restore
I've written a few prototypes and this comes down to four issues. Some of the details below are debian/ubuntu-specific but the same concepts will apply to redhat. 1. User data (/home) must be backed up explicitly. (ditto server data on servers). 2. Packages should NOT be backed up. All you need is the package name and version. Reinstall from .deb and .rpm if necessary since this way you're sure that you never restore compromised files. 3. Configuration data (/etc) must be backed up explicitly. This is tricky because backing up the entire directory will cause its own problems. Worse, some applications keep their configuration files elsewhere. The best solution I've found is to scan the package metadata to identify configuration files and to only save those with a different checksum than the standard file. 4. Local files. Ideally everyone would keep these files under /usr/local and /opt but that's rarely the case. The best solution I've found is to scan the debian package metadata and do a monster diff between what's on the filesystem under /bin, /sbin, /usr and (chunks of) /var with what's in the metadata. It's worth noting that the last item isn't that hard if you have a strict security policy that everything under those directories MUST be in a package. It's deleted without a second thought if it's not. You can still do everything you could before, you just need to create a local package for it. So what do you do with this? The best solution, which I haven't implemented yet, is to handle #2 and #3 with autogenerated packages. You set up one or more local packages that will install the right software and then overwrite the configuration files. You can fit everything, including original package archive, on a single DVD. BTW Debian has a C++ interface to the package metadata. I've never used it - I find it easier to just scan the metadata directory myself. There's also hooks that will allow your application to be called at each step during a package installation or removal. You could, in theory, keep your snapshots current to the last minute that way. On Fri, Sep 30, 2011 at 12:01 AM, Gaurav Saxena grvsaxena...@gmail.comwrote: Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
RE: Ubuntu System Restore
Hello Aaron Thanks a lot for your quick reply. On Fri, Sep 30, 2011 at 10:03 AM, Aaron C. de Bruyn aa...@heyaaron.comwrote: In Windows, the ability to snapshot is built into the filesystem. In Linux, you must be running a filesystem that supports snapshots. I know LVM supports snapshotting and I believe BRTFS has support, but other than that I'm not sure. Yes I read the logic behind windows system restore. But I think we can take some other approach for this, that will be better as all users won't be able to spare an extra partition formatted brtfs. Basically, your program would have to check the file system that is used on the computer (remember Linux can have many types of file systems mounted at the same time), then (in the case of LVM) make sure there's enough free space to snapshot, and finally take the snapshot. Ok. Do I have to snapshot the whole system partition / important system files to the brtfs partition ? When the snapshots start filling up, you would either need to delete them or detect the low space and resize them. In my personal opinion, snapshotting in Linux is currently a pain in the rear. It sounds like BTRFS could change that, but it's still a ways off. Ok. I will try another approach that will be better as suggested by people here. -A On Thu, Sep 29, 2011 at 21:00, Gaurav Saxena grvsaxena...@gmail.com wrote: Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss -- Thanks and Regards , Gaurav -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Ubuntu System Restore
Hello all, I want to write a windows system restore like program for ubuntu , which will have options for creating restore points for the system and then restoring it back to that point. Also I will as an extension provide support for older version of a file as is in windows currently. I need your help to find how to start with this in ubuntu. I know that I have to snapshot the system when creating a restore point and then restore it. I need some starting pointers so that I can start doing this work. Also if this has already been done please inform me. I got this idea from https://wiki.ubuntu.com/SystemRestore. -- Thanks and Regards , Gaurav -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
