Re: Online defragmentation and ext4migrate
Hi, In my opinion, to keep the ioctl simple and small is very important for ease of maintenance. So I would rather not support indirect block files in the ioctl. Instead, I can add the call of the migration ioctl to my defrag tool in order to defragment indirect block files. How do you think of it? That should be fine. So i will start moving the ext4migrate code as a ext4 ioctl and later will send you a patch to the defrag tool that will migrate and defrag. OK. I am looking forward to your patch. Cheers, Takashi - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
On Mon, 2007-05-21 at 12:38 +0200, Jan Kara wrote: Yes. On the other hand I believe that some people would like to use defragmentation but stay with ext3. For them conversion to extents is no-go. [...] I've written a patch that defragments non-extent files but after discussion with XFS guys I've decided that the interfaces should be made more generic, so that XFS and other filesystems can use them too... I see no reason why the ioctl to convert a file to extents and then defragment it should be different from the ioctl to defragment a non-extent file. After all, whether a file's blocks are tracked as lists of blocks or a set of extents is just bookkeeping, right? The set of data blocks that make up the file and their order is the same regardless of whether the extent flag is set in the inode. I agree that at least part of the interface should be independent on the particular representation of data references - especially because I want it to be useful for more filesystems than just ext2/3/4. Currently I think that defragmenting data blocks itself can have fs-independent interface. Of course, when you decide to defragment metadata (i.e. indirect blocks, inodes, etc.) you have to have fs-specific interfaces, probably ioctls... If the user is running the ext2/3 driver or the ext4 driver with the noextents option, just defragment the file. If the user is running ext4 without the noextents option, convert to extents and then defragment. Defragmentation ioctl definitely should not touch the way the file is represented. I.e. if the file uses indirect blocks it should use indirect blocks after defragmentation. If it uses extents, it should use extents afterwards too. It should be the userspace utility which decides whether the file should be converted or not and uses appropriate call for that... The only problem that I can think of is that defragmenting metadata (including indirect block and/or whatever the equivalent is in extent-land) presumably has performance benefits too, so maybe a defragmenter in userspace would want to have some knowledge/control over this process. Yes, it has measurable benefit (especially for indirect blocks) so eventually we should do it. Honza -- Jan Kara [EMAIL PROTECTED] SuSE CR Labs - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
On 5/19/07, Eric [EMAIL PROTECTED] wrote: On Fri, 2007-05-18 at 18:36 +0530, Aneesh Kumar K.V wrote: The reason why i am asking this is to understand the usefulness of doing a ext4migrate followed by defrag. [...] Also looking at the version 0.4 I see that defrag ioctl only work if we have EXT4_EXTENTS_FL flag set. ext4migrate is necessary because the current ext4 defrag routines will only defragment files stored as extents. AFAIK, converting a file to extents does not allow the defrag routine to defragment it better than an indirect block map inode, but converting any file to extents has performance benefits regardless of whether it is later defragmented. What are the plans for making defrag work with indirect block map inode ? I think there is a second set of patches to defragment non-extent files. I was looking at this and didn't find the changes needed to defrag the non extent files. http://www.mail-archive.com/linux-ext4@vger.kernel.org/msg01522.html An even better defragmentation routine knows how to balance the time lost to defragmentation with the performance gained from a defragmented filesystem. IMHO, this requires detailed knowledge of the layout of a file's blocks on the disk. Right now, we get this information by looping over the FIBMAP ioctl, which I understand can take quite a long time. With the takashi's code we use ext4_ext_alloc_blocks and see if the number of extents that we got is less than the number of extents that we have with the original file that we intent to defrag. I am not sure an ioctl is involved here. Well the intent of my mail was to find the advantage of doing an online migration. If we are not relocating the blocks corresponding to extent index then doing a online migration doesn't bring any specific performance bonus. But yes i agree that there is a performance impact with defrag by moving the data blocks closer. -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
Hi Aneesh san, While doing online defragmentation do we move the blocks corresponding to extent index ? The reason why i am asking this is to understand the usefulness of doing a ext4migrate followed by defrag. I understand that defragmentation in general will improve the performance. But with respect to ext4migrate we are not touching the data blocks. Instead we build the extent map and if that requires to have an extent index block then we allocate one. I am trying to understand what would be the performance impact of this and whether doing a defrag really improve the performance. I think converting a file to extents has the benefit for the performance of block searching. If we want to improve also the performance of reading file data, we have to run the defrag after that. Also looking at the version 0.4 I see that defrag ioctl only work if we have EXT4_EXTENTS_FL flag set. What are the plans for making defrag work with indirect block map inode ? Unfortunately, my defrag doesn't support an indirect block file. But we can reduce fragments in the file with the defrag just after ext4migrate. In my opinion, to keep the ioctl simple and small is very important for ease of maintenance. So I would rather not support indirect block files in the ioctl. Instead, I can add the call of the migration ioctl to my defrag tool in order to defragment indirect block files. How do you think of it? Cheers, Takashi - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
Hello, While doing online defragmentation do we move the blocks corresponding to extent index ? The reason why i am asking this is to understand the usefulness of doing a ext4migrate followed by defrag. I understand that defragmentation in general will improve the performance. But with respect to ext4migrate we are not touching the data blocks. Instead we build the extent map and if that requires to have an extent index block then we allocate one. I am trying to understand what would be the performance impact of this and whether doing a defrag really improve the performance. I think converting a file to extents has the benefit for the performance of block searching. If we want to improve also the performance of reading file data, we have to run the defrag after that. Yes. On the other hand I believe that some people would like to use defragmentation but stay with ext3. For them conversion to extents is no-go. Also looking at the version 0.4 I see that defrag ioctl only work if we have EXT4_EXTENTS_FL flag set. What are the plans for making defrag work with indirect block map inode ? Unfortunately, my defrag doesn't support an indirect block file. But we can reduce fragments in the file with the defrag just after ext4migrate. In my opinion, to keep the ioctl simple and small is very important for ease of maintenance. So I would rather not support indirect block files in the ioctl. Instead, I can add the call of the migration ioctl to my defrag tool in order to defragment indirect block files. How do you think of it? Yes that could be useful but I don't think it's a complete solution for people that don't want to migrate. Honza -- Jan Kara [EMAIL PROTECTED] SuSE CR Labs - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
Takashi Sato wrote: Hi Aneesh san, In my opinion, to keep the ioctl simple and small is very important for ease of maintenance. So I would rather not support indirect block files in the ioctl. Instead, I can add the call of the migration ioctl to my defrag tool in order to defragment indirect block files. How do you think of it? That should be fine. So i will start moving the ext4migrate code as a ext4 ioctl and later will send you a patch to the defrag tool that will migrate and defrag. -aneesh - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Online defragmentation and ext4migrate
On Mon, 2007-05-21 at 12:38 +0200, Jan Kara wrote: Yes. On the other hand I believe that some people would like to use defragmentation but stay with ext3. For them conversion to extents is no-go. [...] I've written a patch that defragments non-extent files but after discussion with XFS guys I've decided that the interfaces should be made more generic, so that XFS and other filesystems can use them too... I see no reason why the ioctl to convert a file to extents and then defragment it should be different from the ioctl to defragment a non-extent file. After all, whether a file's blocks are tracked as lists of blocks or a set of extents is just bookkeeping, right? The set of data blocks that make up the file and their order is the same regardless of whether the extent flag is set in the inode. If the user is running the ext2/3 driver or the ext4 driver with the noextents option, just defragment the file. If the user is running ext4 without the noextents option, convert to extents and then defragment. The only problem that I can think of is that defragmenting metadata (including indirect block and/or whatever the equivalent is in extent-land) presumably has performance benefits too, so maybe a defragmenter in userspace would want to have some knowledge/control over this process. Cheers, Eric signature.asc Description: This is a digitally signed message part
Re: Online defragmentation and ext4migrate
On Fri, 2007-05-18 at 18:36 +0530, Aneesh Kumar K.V wrote: The reason why i am asking this is to understand the usefulness of doing a ext4migrate followed by defrag. [...] Also looking at the version 0.4 I see that defrag ioctl only work if we have EXT4_EXTENTS_FL flag set. ext4migrate is necessary because the current ext4 defrag routines will only defragment files stored as extents. AFAIK, converting a file to extents does not allow the defrag routine to defragment it better than an indirect block map inode, but converting any file to extents has performance benefits regardless of whether it is later defragmented. What are the plans for making defrag work with indirect block map inode ? I think there is a second set of patches to defragment non-extent files. When I started investigating this topic, I would have preferred a defrag routine for indirect block map inodes since it would work with the filesystems that I and others are using right now. However, as I read more code and documentation, I'm beginning to warm up to extents. A defragmentation routine makes files contiguous on disk. A better defragmentation routine intelligently locates data structures on the disk so that files and directories are placed to minimize latency and maximize throughput now, AND so that this will continue to happen in the future. Typically this means not only making files contiguous, but also consolidating free space at the end of the volume so that the block allocator can pick contiguous blocks for new files. An even better defragmentation routine knows how to balance the time lost to defragmentation with the performance gained from a defragmented filesystem. IMHO, this requires detailed knowledge of the layout of a file's blocks on the disk. Right now, we get this information by looping over the FIBMAP ioctl, which I understand can take quite a long time. But on an extent file there is a logical, high-performance mapping between the on-disk structures that keep track of which blocks belong to which files and the data returned by the as-yet-to-be-implemented FIEMAP ioctl, which could make defragging faster and more fun. http://www.mail-archive.com/linux-ext4@vger.kernel.org/msg01434.html Cheers, Eric signature.asc Description: This is a digitally signed message part
Re: Online defragmentation and ext4migrate
On May 18, 2007 13:19 -0700, Eric wrote: A defragmentation routine makes files contiguous on disk. A better defragmentation routine intelligently locates data structures on the disk so that files and directories are placed to minimize latency and maximize throughput now, AND so that this will continue to happen in the future. Typically this means not only making files contiguous, but also consolidating free space at the end of the volume so that the block It is not necessarily best to put free space at the END of the volume (that is very FAT-centric) but it does make sense to consolidate free space within each block group. But on an extent file there is a logical, high-performance mapping between the on-disk structures that keep track of which blocks belong to which files and the data returned by the as-yet-to-be-implemented FIEMAP ioctl, which could make defragging faster and more fun. http://www.mail-archive.com/linux-ext4@vger.kernel.org/msg01434.html Yeah, I wish I had time to finish working on this. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. - To unsubscribe from this list: send the line unsubscribe linux-ext4 in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
