Re: Online migration of arbitrary filesystems, possible?
On Mon, 01 Apr 2013 17:50:43 -0300, Daniel Hilst said: > > Any reason you can't just 'rsync /source-fs /dest-fs'? > > > because I can't use dest-fs while rsynching Sure you can. You just have to remember to pay attention to race conditions - if you create foo/bar.dat on the dest and then rsync wants to copy over a foo/bar.tar from the source, things will go poorly. However, if you wanted to write to the dest while doing your sync, you'll have that issue no matter *what* method you use to do it. >> Read operations: >> 1. See if data is already on dest fs, >> 2. If is then read data and bright back to caller (lets call this >> cold read) >> 3. If is not, then read file from source fs, put it on page cache, >> and change the backstorage of that page.. >> 3.1 So when this page get dirty or too old, it will be writed to You may want to look for 'overlayfs' and 'unionfs', which may provide you the function you need. (Note there's several different patchsets calling themselves 'unionfs'). pgpTPJOBUFmxB.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Online migration of arbitrary filesystems, possible?
> > > Ive had to do something similar before. It is painful, but doable. * create the directory structure on the target. (find -exec...) * create a symlink for every file on the target device, obviously pointing to the origin device. (Find) File system should be useable. * progressively replace the symlinks with the actual file (find or script) * watch out for actual symlinks... Optionally replace the origin file with a symlink to the new file. -- -- Teach your kids Science, or somebody else will :/ ja...@ball.net vk2...@google.com callsign: vk2vjb ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Online migration of arbitrary filesystems, possible?
On 29-03-2013 17:28, Greg Freemyer wrote: > On Fri, Mar 29, 2013 at 4:09 PM, Daniel Hilst wrote: >> Suppose that I have two big filesystems, dest fs and source fs.. I want >> to copy whole source fs to dest fs, but I need to keep >> source filesystem online I can't mount source fs read only and copy.. >> >> The idea is, mount both filesystems "together", and make write/read >> operations go on this way >> Read operations: >> 1. See if data is already on dest fs, >> 2. If is then read data and bright back to caller (lets call this >> cold read) >> 3. If is not, then read file from source fs, put it on page cache, >> and change the backstorage of that page.. >> 3.1 So when this page get dirty or too old, it will be writed to >> dest fs >> >> The problem here is, I need to remap every data and metadata on >> step 3 (inodes and stuff), I think that read only data, isn't writed to >> disk, >> so in case of old page, this page would be freed from memory and >> not write to dest fs.. so I need read only pages to be forced dirty.. >> >> Write operations: >> 1. Write data to dest fs >> 2. Mark data as present on dest fs >> >> >> Also I need to know when copy is finished, so I need a tool to >> crosscheck both filesystems and say what was copied and what wasn't, and >> give me some percent of remaining unsynced data. When I get 100% of data >> in sync I can umount source filesystem and use its storage for other >> propose. >> >> The go here is move data online, where I can't bring data offline.. >> >> I'm a kernel newbie, I have just read a lot about page cache and VSF, >> but this seems possible, right? > > Are you looking for a pre-existing solution? Or you want to write something? > > Greg > I can do this with btrfs snapshots, I think LVM2 can do this with snapshots too... I was searching for something like that, but using a kind of VFS hack So I want to know if is possible, in theory, then I can start write something like that -- Follow the white rabbit! ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Online migration of arbitrary filesystems, possible?
On 29-03-2013 18:06, valdis.kletni...@vt.edu wrote: > On Fri, 29 Mar 2013 17:09:14 -0300, Daniel Hilst said: > >> The idea is, mount both filesystems "together", and make write/read >> operations go on this way >> Read operations: >> 1. See if data is already on dest fs, >> 2. If is then read data and bright back to caller (lets call this >> cold read) >> 3. If is not, then read file from source fs, put it on page cache, >> and change the backstorage of that page.. >> 3.1 So when this page get dirty or too old, it will be writed to > > Any reason you can't just 'rsync /source-fs /dest-fs'? > because I can't use dest-fs while rsynching -- Follow the white rabbit! ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Online migration of arbitrary filesystems, possible?
On Fri, 29 Mar 2013 17:09:14 -0300, Daniel Hilst said: > The idea is, mount both filesystems "together", and make write/read > operations go on this way > Read operations: > 1. See if data is already on dest fs, > 2. If is then read data and bright back to caller (lets call this > cold read) > 3. If is not, then read file from source fs, put it on page cache, > and change the backstorage of that page.. > 3.1 So when this page get dirty or too old, it will be writed to Any reason you can't just 'rsync /source-fs /dest-fs'? pgpCeLeHQ48_a.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Online migration of arbitrary filesystems, possible?
On Fri, Mar 29, 2013 at 4:09 PM, Daniel Hilst wrote: > Suppose that I have two big filesystems, dest fs and source fs.. I want > to copy whole source fs to dest fs, but I need to keep > source filesystem online I can't mount source fs read only and copy.. > > The idea is, mount both filesystems "together", and make write/read > operations go on this way > Read operations: > 1. See if data is already on dest fs, > 2. If is then read data and bright back to caller (lets call this > cold read) > 3. If is not, then read file from source fs, put it on page cache, > and change the backstorage of that page.. > 3.1 So when this page get dirty or too old, it will be writed to > dest fs > > The problem here is, I need to remap every data and metadata on > step 3 (inodes and stuff), I think that read only data, isn't writed to > disk, > so in case of old page, this page would be freed from memory and > not write to dest fs.. so I need read only pages to be forced dirty.. > > Write operations: > 1. Write data to dest fs > 2. Mark data as present on dest fs > > > Also I need to know when copy is finished, so I need a tool to > crosscheck both filesystems and say what was copied and what wasn't, and > give me some percent of remaining unsynced data. When I get 100% of data > in sync I can umount source filesystem and use its storage for other > propose. > > The go here is move data online, where I can't bring data offline.. > > I'm a kernel newbie, I have just read a lot about page cache and VSF, > but this seems possible, right? Are you looking for a pre-existing solution? Or you want to write something? Greg ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Online migration of arbitrary filesystems, possible?
Suppose that I have two big filesystems, dest fs and source fs.. I want to copy whole source fs to dest fs, but I need to keep source filesystem online I can't mount source fs read only and copy.. The idea is, mount both filesystems "together", and make write/read operations go on this way Read operations: 1. See if data is already on dest fs, 2. If is then read data and bright back to caller (lets call this cold read) 3. If is not, then read file from source fs, put it on page cache, and change the backstorage of that page.. 3.1 So when this page get dirty or too old, it will be writed to dest fs The problem here is, I need to remap every data and metadata on step 3 (inodes and stuff), I think that read only data, isn't writed to disk, so in case of old page, this page would be freed from memory and not write to dest fs.. so I need read only pages to be forced dirty.. Write operations: 1. Write data to dest fs 2. Mark data as present on dest fs Also I need to know when copy is finished, so I need a tool to crosscheck both filesystems and say what was copied and what wasn't, and give me some percent of remaining unsynced data. When I get 100% of data in sync I can umount source filesystem and use its storage for other propose. The go here is move data online, where I can't bring data offline.. I'm a kernel newbie, I have just read a lot about page cache and VSF, but this seems possible, right? -- Follow the white rabbit! ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
