Re: ext4 extents: How to determine if an extent points to a hole.
Hi Peter, On Tue, Sep 15, 2009 at 7:15 AM, Peter Teoh wrote: > just to share what i have read, based on "lwn.net surfing": > > From the article that started it all: > > http://lwn.net/Articles/260795/ > > From the specs, if u used ioctl() with FIEMAP_EXTENT_HOLE it will keep > returning u the next hole. > > http://lwn.net/Articles/297696/ > http://lwn.net/Articles/260803/ > http://lwn.net/Articles/332610/ > http://lwn.net/Articles/275722/ > > a few of the articles above have userspace program that shows how to > use the primitive FIEMAP_EXTENT_HOLE to identify holes. > > have fun!! > Thanks for pointing to these docs. But just for everyones benefit, wanted to mention that most of these documents are obsolete and the implementation has changed a lot since then. The conversation I pasted yesterday on the mail was from one of the authors (Kalpak Shah) of FS_IOC_FIEMAP infrastructure. Look at the latest file: http://lxr.linux.no/#linux+v2.6.31/include/linux/fiemap.h Otherwise, the documents are really good to understand the infrastructure. -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
just to share what i have read, based on "lwn.net surfing": >From the article that started it all: http://lwn.net/Articles/260795/ >From the specs, if u used ioctl() with FIEMAP_EXTENT_HOLE it will keep returning u the next hole. http://lwn.net/Articles/297696/ http://lwn.net/Articles/260803/ http://lwn.net/Articles/332610/ http://lwn.net/Articles/275722/ a few of the articles above have userspace program that shows how to use the primitive FIEMAP_EXTENT_HOLE to identify holes. have fun!! -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
Hi Manish, On Mon, Sep 14, 2009 at 3:12 PM, Manish Katiyar wrote: > On Mon, Sep 14, 2009 at 3:09 PM, SandeepKsinha > wrote: >> Hi Manish, >> >> On Mon, Sep 14, 2009 at 2:57 PM, Manish Katiyar wrote: >>> On Mon, Sep 14, 2009 at 11:59 AM, SandeepKsinha >>> wrote: Hi Greg, Thanks a lot for such nice explanation. But I still have the same query lingering... On Mon, Sep 14, 2009 at 9:36 AM, Greg Freemyer wrote: > On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha > wrote: >> On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar >> wrote: >>> On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha >>> wrote: Hi, On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: > On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha > wrote: >> Hi all, >> >> Looking at the flags in the extent info, Is there any specifc flags >> which indicates an extent to be a HOLE?? > > I am not sure if I understand the question correctly .. why would > you need that ? Can you give an example where it should be used ?? > Look at e4defrag.c, it checks the file size and allocates the same number of blocks for donor inode. Which will eventually make a holey file into a normal one. Any tool/application should make sure that they leave a sparse file as sparse. I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get such information. >>> >>> Yes, but I think bmap would be costly if the file is large and is not >>> holey :-( but that would be probably same calling fiemap if the >>> file is completely fragmented such that each extent size is 1. >>> >> >> Since, ext2/ext3 did not have mutli block allocation thats why this is >> the only way that we might have. >> But generally most of the new features work on with extent based files >> on ext4. >> >> I am still wondering that how to we represent a hole using extents in >> a extent based file. >> Just like we had a convention of having the block number 0 in case of >> holes. >> >> Similarly, what do we look at to figure out if its a hole or not. BMAP >> is one way. But since, in a extent based file, we have only extents, >> there should be some flag to indicate the same. > > Sandeep, > > If you look at e4defrag, it first gets a list of all the extents. I'm > pretty sure extents only exist for allocated extents. Holes do not > have any associated extents. >>> >>> I did a extent dump of a holey file. >>> >>> Inode: 12 Type: regular Mode: 0644 Flags: 0x8 >>> Generation: 4284390079 Version: 0x0001 >>> User: 5572 Group: 5573 Size: 20877312000 >>> File ACL: 0 Directory ACL: 0 >>> Links: 1 Blockcount: 16000 >>> Fragment: Address: 0 Number: 0 Size: 0 >>> ctime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 >>> atime: 0x4aad34e9 -- Sun Sep 13 23:37:37 2009 >>> mtime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 >>> EXTENTS: >>> (0-3999): 10241-14240, (20384000-20387999): 14337-18336 >>> >>> Level Entries Logical Physical Length Flags >>> 0/ 0 1/ 2 0 - 3999 10241 - 14240 4000 >>> 0/ 0 2/ 2 20384000 - 20387999 14337 - 18336 4000 >>> >>> I am looking at the source, but as you can see it is not printing the >>> extents for the hole part. >>> >> >> I suspect the debugfs code masking that out for the output. > > yes... and that is the flag/method that we are looking for ... isn't it :-) > ??? > read the above conversation, this answers all the question. me: How does ext4 represent holes ? Do we have a extent (unitialized for holes) ?? Kalpak: physical block no = 0 me: But we keep an extent for the same. If I do a FIEMAP Kalpak: yup me: I will also get an extent for the holes Kalpak: FIEMAP wont return hole extents me: ok Kalpak: but u can find out that if there was hole by FIEMAP output if last block != this block there was a hole in between Hope this helps everyone. Also, now we do not bother much about holes. >> If you remember, for a normal block based file also, it does something >> similar, it does not prints the values for holes. >> >> It can be a case here as well. >>> > This is what my actual question is, think of this situation... |-|-|-|--| 0 500 700 1200 3000 [Logical block numbers for an inode] In this situation you will have four extents for sure. ext_1 -> 0 -- 500 ext_2 -> 501 --700 [ This will be an initialized extent] ext_3 -> 701 -- 1200 ext_4 -> 1200 -- 3000 After looking at the sources and some comments in the ext4 source code,
Re: ext4 extents: How to determine if an extent points to a hole.
On Mon, Sep 14, 2009 at 3:09 PM, SandeepKsinha wrote: > Hi Manish, > > On Mon, Sep 14, 2009 at 2:57 PM, Manish Katiyar wrote: >> On Mon, Sep 14, 2009 at 11:59 AM, SandeepKsinha >> wrote: >>> Hi Greg, >>> >>> Thanks a lot for such nice explanation. But I still have the same >>> query lingering... >>> >>> >>> On Mon, Sep 14, 2009 at 9:36 AM, Greg Freemyer >>> wrote: On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha wrote: > On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar > wrote: >> On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha >> wrote: >>> Hi, >>> >>> >>> On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar >>> wrote: On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha wrote: > Hi all, > > Looking at the flags in the extent info, Is there any specifc flags > which indicates an extent to be a HOLE?? I am not sure if I understand the question correctly .. why would you need that ? Can you give an example where it should be used ?? >>> >>> Look at e4defrag.c, >>> it checks the file size and allocates the same number of blocks for >>> donor inode. Which will eventually make a holey file into a normal >>> one. >>> Any tool/application should make sure that they leave a sparse file as >>> sparse. >>> >>> I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get >>> such information. >> >> Yes, but I think bmap would be costly if the file is large and is not >> holey :-( but that would be probably same calling fiemap if the >> file is completely fragmented such that each extent size is 1. >> > > Since, ext2/ext3 did not have mutli block allocation thats why this is > the only way that we might have. > But generally most of the new features work on with extent based files on > ext4. > > I am still wondering that how to we represent a hole using extents in > a extent based file. > Just like we had a convention of having the block number 0 in case of > holes. > > Similarly, what do we look at to figure out if its a hole or not. BMAP > is one way. But since, in a extent based file, we have only extents, > there should be some flag to indicate the same. Sandeep, If you look at e4defrag, it first gets a list of all the extents. I'm pretty sure extents only exist for allocated extents. Holes do not have any associated extents. >> >> I did a extent dump of a holey file. >> >> Inode: 12 Type: regular Mode: 0644 Flags: 0x8 >> Generation: 4284390079 Version: 0x0001 >> User: 5572 Group: 5573 Size: 20877312000 >> File ACL: 0 Directory ACL: 0 >> Links: 1 Blockcount: 16000 >> Fragment: Address: 0 Number: 0 Size: 0 >> ctime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 >> atime: 0x4aad34e9 -- Sun Sep 13 23:37:37 2009 >> mtime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 >> EXTENTS: >> (0-3999): 10241-14240, (20384000-20387999): 14337-18336 >> >> Level Entries Logical Physical Length Flags >> 0/ 0 1/ 2 0 - 3999 10241 - 14240 4000 >> 0/ 0 2/ 2 20384000 - 20387999 14337 - 18336 4000 >> >> I am looking at the source, but as you can see it is not printing the >> extents for the hole part. >> > > I suspect the debugfs code masking that out for the output. yes... and that is the flag/method that we are looking for ... isn't it :-) ??? > If you remember, for a normal block based file also, it does something > similar, it does not prints the values for holes. > > It can be a case here as well. >> >>> >>> This is what my actual question is, think of this situation... >>> >>> |-|-|-|--| >>> 0 500 700 1200 3000 >>> [Logical block numbers for an inode] >>> >>> >>> In this situation you will have four extents for sure. >>> ext_1 -> 0 -- 500 >>> ext_2 -> 501 --700 [ This will be an initialized extent] >>> ext_3 -> 701 -- 1200 >>> ext_4 -> 1200 -- 3000 >>> >>> After looking at the sources and some comments in the ext4 source >>> code, I could figure out that holes would be having an initialized >>> extent. >>> Reference: http://lxr.linux.no/#linux+v2.6.31/fs/ext4/extents.c#L2843 >>> >>> I think we cannot have a mixture of both a BMAP and an EMAP, it will be >>> either. >>> Then it calls merge_extents to create extent groups. In e4defrag terminology, an extent group is a collection of all the logically contiguous extents. I don't know if the kernel uses that terminology or not. >>> >>> Hope they are not merging together any initialized and uninitialized >>> extents together, since they can be logically contiguous. Or rather >>> they will be. >>> >>> In other words in e4defrag terminology a sparse file is a series of: exten
Re: ext4 extents: How to determine if an extent points to a hole.
Hi Manish, On Mon, Sep 14, 2009 at 2:57 PM, Manish Katiyar wrote: > On Mon, Sep 14, 2009 at 11:59 AM, SandeepKsinha > wrote: >> Hi Greg, >> >> Thanks a lot for such nice explanation. But I still have the same >> query lingering... >> >> >> On Mon, Sep 14, 2009 at 9:36 AM, Greg Freemyer >> wrote: >>> On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha >>> wrote: On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar wrote: > On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha > wrote: >> Hi, >> >> >> On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar >> wrote: >>> On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha >>> wrote: Hi all, Looking at the flags in the extent info, Is there any specifc flags which indicates an extent to be a HOLE?? >>> >>> I am not sure if I understand the question correctly .. why would >>> you need that ? Can you give an example where it should be used ?? >>> >> >> Look at e4defrag.c, >> it checks the file size and allocates the same number of blocks for >> donor inode. Which will eventually make a holey file into a normal >> one. >> Any tool/application should make sure that they leave a sparse file as >> sparse. >> >> I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get >> such information. > > Yes, but I think bmap would be costly if the file is large and is not > holey :-( but that would be probably same calling fiemap if the > file is completely fragmented such that each extent size is 1. > Since, ext2/ext3 did not have mutli block allocation thats why this is the only way that we might have. But generally most of the new features work on with extent based files on ext4. I am still wondering that how to we represent a hole using extents in a extent based file. Just like we had a convention of having the block number 0 in case of holes. Similarly, what do we look at to figure out if its a hole or not. BMAP is one way. But since, in a extent based file, we have only extents, there should be some flag to indicate the same. >>> >>> Sandeep, >>> >>> If you look at e4defrag, it first gets a list of all the extents. I'm >>> pretty sure extents only exist for allocated extents. Holes do not >>> have any associated extents. > > I did a extent dump of a holey file. > > Inode: 12 Type: regular Mode: 0644 Flags: 0x8 > Generation: 4284390079 Version: 0x0001 > User: 5572 Group: 5573 Size: 20877312000 > File ACL: 0 Directory ACL: 0 > Links: 1 Blockcount: 16000 > Fragment: Address: 0 Number: 0 Size: 0 > ctime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 > atime: 0x4aad34e9 -- Sun Sep 13 23:37:37 2009 > mtime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 > EXTENTS: > (0-3999): 10241-14240, (20384000-20387999): 14337-18336 > > Level Entries Logical Physical Length Flags > 0/ 0 1/ 2 0 - 3999 10241 - 14240 4000 > 0/ 0 2/ 2 20384000 - 20387999 14337 - 18336 4000 > > I am looking at the source, but as you can see it is not printing the > extents for the hole part. > I suspect the debugfs code masking that out for the output. If you remember, for a normal block based file also, it does something similar, it does not prints the values for holes. It can be a case here as well. > >>> >> >> This is what my actual question is, think of this situation... >> >> |-|-|-|--| >> 0 500 700 1200 3000 >> [Logical block numbers for an inode] >> >> >> In this situation you will have four extents for sure. >> ext_1 -> 0 -- 500 >> ext_2 -> 501 --700 [ This will be an initialized extent] >> ext_3 -> 701 -- 1200 >> ext_4 -> 1200 -- 3000 >> >> After looking at the sources and some comments in the ext4 source >> code, I could figure out that holes would be having an initialized >> extent. >> Reference: http://lxr.linux.no/#linux+v2.6.31/fs/ext4/extents.c#L2843 >> >> I think we cannot have a mixture of both a BMAP and an EMAP, it will be >> either. >> >>> Then it calls merge_extents to create extent groups. In e4defrag >>> terminology, an extent group is a collection of all the logically >>> contiguous extents. I don't know if the kernel uses that terminology >>> or not. >>> >> >> Hope they are not merging together any initialized and uninitialized >> extents together, since they can be logically contiguous. Or rather >> they will be. >> >> >>> In other words in e4defrag terminology a sparse file is a series of: >>> >>> extent group - hole - extent group - hole - extent group - etc. >>> >>> Then e4defrag creates a donor file with exactly the same allocated >>> block areas by calling fallocate on the donor file for each extent >>> group with the same starting offset and length as the extent group. >
Re: ext4 extents: How to determine if an extent points to a hole.
On Mon, Sep 14, 2009 at 11:59 AM, SandeepKsinha wrote: > Hi Greg, > > Thanks a lot for such nice explanation. But I still have the same > query lingering... > > > On Mon, Sep 14, 2009 at 9:36 AM, Greg Freemyer > wrote: >> On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha >> wrote: >>> On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar wrote: On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha wrote: > Hi, > > > On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar > wrote: >> On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha >> wrote: >>> Hi all, >>> >>> Looking at the flags in the extent info, Is there any specifc flags >>> which indicates an extent to be a HOLE?? >> >> I am not sure if I understand the question correctly .. why would >> you need that ? Can you give an example where it should be used ?? >> > > Look at e4defrag.c, > it checks the file size and allocates the same number of blocks for > donor inode. Which will eventually make a holey file into a normal > one. > Any tool/application should make sure that they leave a sparse file as > sparse. > > I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get > such information. Yes, but I think bmap would be costly if the file is large and is not holey :-( but that would be probably same calling fiemap if the file is completely fragmented such that each extent size is 1. >>> >>> Since, ext2/ext3 did not have mutli block allocation thats why this is >>> the only way that we might have. >>> But generally most of the new features work on with extent based files on >>> ext4. >>> >>> I am still wondering that how to we represent a hole using extents in >>> a extent based file. >>> Just like we had a convention of having the block number 0 in case of holes. >>> >>> Similarly, what do we look at to figure out if its a hole or not. BMAP >>> is one way. But since, in a extent based file, we have only extents, >>> there should be some flag to indicate the same. >> >> Sandeep, >> >> If you look at e4defrag, it first gets a list of all the extents. I'm >> pretty sure extents only exist for allocated extents. Holes do not >> have any associated extents. I did a extent dump of a holey file. Inode: 12 Type: regularMode: 0644 Flags: 0x8 Generation: 4284390079Version: 0x0001 User: 5572 Group: 5573 Size: 20877312000 File ACL: 0Directory ACL: 0 Links: 1 Blockcount: 16000 Fragment: Address: 0Number: 0Size: 0 ctime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 atime: 0x4aad34e9 -- Sun Sep 13 23:37:37 2009 mtime: 0x4aae0975 -- Mon Sep 14 14:44:29 2009 EXTENTS: (0-3999): 10241-14240, (20384000-20387999): 14337-18336 Level Entries LogicalPhysical Length Flags 0/ 0 1/ 20 - 3999 10241 - 14240 4000 0/ 0 2/ 2 20384000 - 20387999 14337 - 18336 4000 I am looking at the source, but as you can see it is not printing the extents for the hole part. >> > > This is what my actual question is, think of this situation... > > |-|-|-|--| > 0 500 700 1200 3000 > [Logical block numbers for an inode] > > > In this situation you will have four extents for sure. > ext_1 -> 0 -- 500 > ext_2 -> 501 --700 [ This will be an initialized extent] > ext_3 -> 701 -- 1200 > ext_4 -> 1200 -- 3000 > > After looking at the sources and some comments in the ext4 source > code, I could figure out that holes would be having an initialized > extent. > Reference: http://lxr.linux.no/#linux+v2.6.31/fs/ext4/extents.c#L2843 > > I think we cannot have a mixture of both a BMAP and an EMAP, it will be > either. > >> Then it calls merge_extents to create extent groups. In e4defrag >> terminology, an extent group is a collection of all the logically >> contiguous extents. I don't know if the kernel uses that terminology >> or not. >> > > Hope they are not merging together any initialized and uninitialized > extents together, since they can be logically contiguous. Or rather > they will be. > > >> In other words in e4defrag terminology a sparse file is a series of: >> >> extent group - hole - extent group - hole - extent group - etc. >> >> Then e4defrag creates a donor file with exactly the same allocated >> block areas by calling fallocate on the donor file for each extent >> group with the same starting offset and length as the extent group. >> > > This is true and should be applicable to initialized extents as well. > I fear if they are > > >> Thus the donor file ends up have exactly the same holes as the >> original file. Then the donor blocks are used to defrag the original >> file by calling move_extent. In the kernel, the move_extent logic >> looks for holes and only replaces blocks that are allocated in the >> original file. >> > > This is true. I am sure of the kernel
Re: ext4 extents: How to determine if an extent points to a hole.
Hi Greg, Thanks a lot for such nice explanation. But I still have the same query lingering... On Mon, Sep 14, 2009 at 9:36 AM, Greg Freemyer wrote: > On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha > wrote: >> On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar wrote: >>> On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha >>> wrote: Hi, On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: > On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha > wrote: >> Hi all, >> >> Looking at the flags in the extent info, Is there any specifc flags >> which indicates an extent to be a HOLE?? > > I am not sure if I understand the question correctly .. why would > you need that ? Can you give an example where it should be used ?? > Look at e4defrag.c, it checks the file size and allocates the same number of blocks for donor inode. Which will eventually make a holey file into a normal one. Any tool/application should make sure that they leave a sparse file as sparse. I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get such information. >>> >>> Yes, but I think bmap would be costly if the file is large and is not >>> holey :-( but that would be probably same calling fiemap if the >>> file is completely fragmented such that each extent size is 1. >>> >> >> Since, ext2/ext3 did not have mutli block allocation thats why this is >> the only way that we might have. >> But generally most of the new features work on with extent based files on >> ext4. >> >> I am still wondering that how to we represent a hole using extents in >> a extent based file. >> Just like we had a convention of having the block number 0 in case of holes. >> >> Similarly, what do we look at to figure out if its a hole or not. BMAP >> is one way. But since, in a extent based file, we have only extents, >> there should be some flag to indicate the same. > > Sandeep, > > If you look at e4defrag, it first gets a list of all the extents. I'm > pretty sure extents only exist for allocated extents. Holes do not > have any associated extents. > This is what my actual question is, think of this situation... |-|-|-|--| 0 500 7001200 3000 [Logical block numbers for an inode] In this situation you will have four extents for sure. ext_1 -> 0 -- 500 ext_2 -> 501 --700 [ This will be an initialized extent] ext_3 -> 701 -- 1200 ext_4 -> 1200 -- 3000 After looking at the sources and some comments in the ext4 source code, I could figure out that holes would be having an initialized extent. Reference: http://lxr.linux.no/#linux+v2.6.31/fs/ext4/extents.c#L2843 I think we cannot have a mixture of both a BMAP and an EMAP, it will be either. > Then it calls merge_extents to create extent groups. In e4defrag > terminology, an extent group is a collection of all the logically > contiguous extents. I don't know if the kernel uses that terminology > or not. > Hope they are not merging together any initialized and uninitialized extents together, since they can be logically contiguous. Or rather they will be. > In other words in e4defrag terminology a sparse file is a series of: > > extent group - hole - extent group - hole - extent group - etc. > > Then e4defrag creates a donor file with exactly the same allocated > block areas by calling fallocate on the donor file for each extent > group with the same starting offset and length as the extent group. > This is true and should be applicable to initialized extents as well. I fear if they are > Thus the donor file ends up have exactly the same holes as the > original file. Then the donor blocks are used to defrag the original > file by calling move_extent. In the kernel, the move_extent logic > looks for holes and only replaces blocks that are allocated in the > original file. > This is true. I am sure of the kernel logic. > Greg > -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
On Sun, Sep 13, 2009 at 1:49 PM, SandeepKsinha wrote: > On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar wrote: >> On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha >> wrote: >>> Hi, >>> >>> >>> On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha wrote: > Hi all, > > Looking at the flags in the extent info, Is there any specifc flags > which indicates an extent to be a HOLE?? I am not sure if I understand the question correctly .. why would you need that ? Can you give an example where it should be used ?? >>> >>> Look at e4defrag.c, >>> it checks the file size and allocates the same number of blocks for >>> donor inode. Which will eventually make a holey file into a normal >>> one. >>> Any tool/application should make sure that they leave a sparse file as >>> sparse. >>> >>> I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get >>> such information. >> >> Yes, but I think bmap would be costly if the file is large and is not >> holey :-( but that would be probably same calling fiemap if the >> file is completely fragmented such that each extent size is 1. >> > > Since, ext2/ext3 did not have mutli block allocation thats why this is > the only way that we might have. > But generally most of the new features work on with extent based files on > ext4. > > I am still wondering that how to we represent a hole using extents in > a extent based file. > Just like we had a convention of having the block number 0 in case of holes. > > Similarly, what do we look at to figure out if its a hole or not. BMAP > is one way. But since, in a extent based file, we have only extents, > there should be some flag to indicate the same. Sandeep, If you look at e4defrag, it first gets a list of all the extents. I'm pretty sure extents only exist for allocated extents. Holes do not have any associated extents. Then it calls merge_extents to create extent groups. In e4defrag terminology, an extent group is a collection of all the logically contiguous extents. I don't know if the kernel uses that terminology or not. In other words in e4defrag terminology a sparse file is a series of: extent group - hole - extent group - hole - extent group - etc. Then e4defrag creates a donor file with exactly the same allocated block areas by calling fallocate on the donor file for each extent group with the same starting offset and length as the extent group. Thus the donor file ends up have exactly the same holes as the original file. Then the donor blocks are used to defrag the original file by calling move_extent. In the kernel, the move_extent logic looks for holes and only replaces blocks that are allocated in the original file. Greg -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha wrote: > Hi, > > > On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: >> On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha >> wrote: >>> Hi all, >>> >>> Looking at the flags in the extent info, Is there any specifc flags >>> which indicates an extent to be a HOLE?? >> >> I am not sure if I understand the question correctly .. why would >> you need that ? Can you give an example where it should be used ?? >> > > Look at e4defrag.c, > it checks the file size and allocates the same number of blocks for > donor inode. Which will eventually make a holey file into a normal > one. > Any tool/application should make sure that they leave a sparse file as sparse. > > I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get > such information. Yes, but I think bmap would be costly if the file is large and is not holey :-( but that would be probably same calling fiemap if the file is completely fragmented such that each extent size is 1. > > >>> >>> I could only find the following ones... >>> >>> 39#define FIEMAP_MAX_OFFSET (~0ULL) >>> 40 >>> 41#define FIEMAP_FLAG_SYNC 0x0001 /* sync file data before map >>> */ >>> 42#define FIEMAP_FLAG_XATTR 0x0002 /* map extended attribute >>> tree */ >>> 43 >>> 44#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) >>> 45 >>> 46#define FIEMAP_EXTENT_LAST 0x0001 /* Last extent >>> in file. */ >>> 47#define FIEMAP_EXTENT_UNKNOWN 0x0002 /* Data >>> location unknown. */ >>> 48#define FIEMAP_EXTENT_DELALLOC 0x0004 /* Location >>> still pending. >>> 49 * Sets >>> EXTENT_UNKNOWN. */ >>> 50#define FIEMAP_EXTENT_ENCODED 0x0008 /* Data can not be >>> read >>> 51 * while fs is >>> unmounted */ >>> 52#define FIEMAP_EXTENT_DATA_ENCRYPTED 0x0080 /* Data is >>> encrypted by fs. >>> 53 * Sets >>> EXTENT_NO_BYPASS. */ >>> 54#define FIEMAP_EXTENT_NOT_ALIGNED 0x0100 /* Extent >>> offsets may not be >>> 55 * block aligned. */ >>> 56#define FIEMAP_EXTENT_DATA_INLINE 0x0200 /* Data mixed >>> with metadata. >>> 57 * Sets >>> EXTENT_NOT_ALIGNED.*/ >>> 58#define FIEMAP_EXTENT_DATA_TAIL 0x0400 /* Multiple >>> files in block. >>> 59 * Sets >>> EXTENT_NOT_ALIGNED.*/ >>> 60#define FIEMAP_EXTENT_UNWRITTEN 0x0800 /* Space allocated, >>> but >>> 61 * no data >>> (i.e. zero). */ >>> 62#define FIEMAP_EXTENT_MERGED 0x1000 /* File does not >>> natively >>> 63 * support >>> extents. Result >>> 64 * merged for >>> efficiency. */ >>> >>> >>> -- >>> Regards, >>> Sandeep. >>> >>> >>> >>> >>> >>> >>> “To learn is to change. Education is a process that changes the learner.” >>> >>> -- >>> To unsubscribe from this list: send an email with >>> "unsubscribe kernelnewbies" to ecar...@nl.linux.org >>> Please read the FAQ at http://kernelnewbies.org/FAQ >>> >>> >> >> >> >> -- >> Thanks - >> Manish >> > > > > -- > Regards, > Sandeep. > > > > > > > “To learn is to change. Education is a process that changes the learner.” > -- Thanks - Manish -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
On Sun, Sep 13, 2009 at 11:08 PM, Manish Katiyar wrote: > On Sun, Sep 13, 2009 at 10:12 PM, SandeepKsinha > wrote: >> Hi, >> >> >> On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: >>> On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha >>> wrote: Hi all, Looking at the flags in the extent info, Is there any specifc flags which indicates an extent to be a HOLE?? >>> >>> I am not sure if I understand the question correctly .. why would >>> you need that ? Can you give an example where it should be used ?? >>> >> >> Look at e4defrag.c, >> it checks the file size and allocates the same number of blocks for >> donor inode. Which will eventually make a holey file into a normal >> one. >> Any tool/application should make sure that they leave a sparse file as >> sparse. >> >> I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get >> such information. > > Yes, but I think bmap would be costly if the file is large and is not > holey :-( but that would be probably same calling fiemap if the > file is completely fragmented such that each extent size is 1. > Since, ext2/ext3 did not have mutli block allocation thats why this is the only way that we might have. But generally most of the new features work on with extent based files on ext4. I am still wondering that how to we represent a hole using extents in a extent based file. Just like we had a convention of having the block number 0 in case of holes. Similarly, what do we look at to figure out if its a hole or not. BMAP is one way. But since, in a extent based file, we have only extents, there should be some flag to indicate the same. > >> >> I could only find the following ones... 39#define FIEMAP_MAX_OFFSET (~0ULL) 40 41#define FIEMAP_FLAG_SYNC 0x0001 /* sync file data before map */ 42#define FIEMAP_FLAG_XATTR 0x0002 /* map extended attribute tree */ 43 44#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) 45 46#define FIEMAP_EXTENT_LAST 0x0001 /* Last extent in file. */ 47#define FIEMAP_EXTENT_UNKNOWN 0x0002 /* Data location unknown. */ 48#define FIEMAP_EXTENT_DELALLOC 0x0004 /* Location still pending. 49 * Sets EXTENT_UNKNOWN. */ 50#define FIEMAP_EXTENT_ENCODED 0x0008 /* Data can not be read 51 * while fs is unmounted */ 52#define FIEMAP_EXTENT_DATA_ENCRYPTED 0x0080 /* Data is encrypted by fs. 53 * Sets EXTENT_NO_BYPASS. */ 54#define FIEMAP_EXTENT_NOT_ALIGNED 0x0100 /* Extent offsets may not be 55 * block aligned. */ 56#define FIEMAP_EXTENT_DATA_INLINE 0x0200 /* Data mixed with metadata. 57 * Sets EXTENT_NOT_ALIGNED.*/ 58#define FIEMAP_EXTENT_DATA_TAIL 0x0400 /* Multiple files in block. 59 * Sets EXTENT_NOT_ALIGNED.*/ 60#define FIEMAP_EXTENT_UNWRITTEN 0x0800 /* Space allocated, but 61 * no data (i.e. zero). */ 62#define FIEMAP_EXTENT_MERGED 0x1000 /* File does not natively 63 * support extents. Result 64 * merged for efficiency. */ -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ >>> >>> >>> >>> -- >>> Thanks - >>> Manish >>> >> >> >> >> -- >> Regards, >> Sandeep. >> >> >> >> >> >> >> “To learn is to change. Education is a process that changes the learner.” >> > > > > -- > Thanks - > Manish > -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha wrote: > Hi all, > > Looking at the flags in the extent info, Is there any specifc flags > which indicates an extent to be a HOLE?? I am not sure if I understand the question correctly .. why would you need that ? Can you give an example where it should be used ?? > > I could only find the following ones... > > 39#define FIEMAP_MAX_OFFSET (~0ULL) > 40 > 41#define FIEMAP_FLAG_SYNC 0x0001 /* sync file data before map */ > 42#define FIEMAP_FLAG_XATTR 0x0002 /* map extended attribute tree > */ > 43 > 44#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) > 45 > 46#define FIEMAP_EXTENT_LAST 0x0001 /* Last extent > in file. */ > 47#define FIEMAP_EXTENT_UNKNOWN 0x0002 /* Data > location unknown. */ > 48#define FIEMAP_EXTENT_DELALLOC 0x0004 /* Location > still pending. > 49 * Sets > EXTENT_UNKNOWN. */ > 50#define FIEMAP_EXTENT_ENCODED 0x0008 /* Data can not be read > 51 * while fs is > unmounted */ > 52#define FIEMAP_EXTENT_DATA_ENCRYPTED 0x0080 /* Data is > encrypted by fs. > 53 * Sets > EXTENT_NO_BYPASS. */ > 54#define FIEMAP_EXTENT_NOT_ALIGNED 0x0100 /* Extent > offsets may not be > 55 * block aligned. */ > 56#define FIEMAP_EXTENT_DATA_INLINE 0x0200 /* Data mixed > with metadata. > 57 * Sets > EXTENT_NOT_ALIGNED.*/ > 58#define FIEMAP_EXTENT_DATA_TAIL 0x0400 /* Multiple > files in block. > 59 * Sets > EXTENT_NOT_ALIGNED.*/ > 60#define FIEMAP_EXTENT_UNWRITTEN 0x0800 /* Space allocated, but > 61 * no data > (i.e. zero). */ > 62#define FIEMAP_EXTENT_MERGED 0x1000 /* File does not > natively > 63 * support > extents. Result > 64 * merged for > efficiency. */ > > > -- > Regards, > Sandeep. > > > > > > > “To learn is to change. Education is a process that changes the learner.” > > -- > To unsubscribe from this list: send an email with > "unsubscribe kernelnewbies" to ecar...@nl.linux.org > Please read the FAQ at http://kernelnewbies.org/FAQ > > -- Thanks - Manish -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
Re: ext4 extents: How to determine if an extent points to a hole.
Hi, On Sun, Sep 13, 2009 at 10:07 PM, Manish Katiyar wrote: > On Sun, Sep 13, 2009 at 5:44 PM, SandeepKsinha > wrote: >> Hi all, >> >> Looking at the flags in the extent info, Is there any specifc flags >> which indicates an extent to be a HOLE?? > > I am not sure if I understand the question correctly .. why would > you need that ? Can you give an example where it should be used ?? > Look at e4defrag.c, it checks the file size and allocates the same number of blocks for donor inode. Which will eventually make a holey file into a normal one. Any tool/application should make sure that they leave a sparse file as sparse. I think, as suggested by Greg Freemyer, we can use BMAP ioctl to get such information. >> >> I could only find the following ones... >> >> 39#define FIEMAP_MAX_OFFSET (~0ULL) >> 40 >> 41#define FIEMAP_FLAG_SYNC 0x0001 /* sync file data before map */ >> 42#define FIEMAP_FLAG_XATTR 0x0002 /* map extended attribute tree >> */ >> 43 >> 44#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) >> 45 >> 46#define FIEMAP_EXTENT_LAST 0x0001 /* Last extent >> in file. */ >> 47#define FIEMAP_EXTENT_UNKNOWN 0x0002 /* Data >> location unknown. */ >> 48#define FIEMAP_EXTENT_DELALLOC 0x0004 /* Location >> still pending. >> 49 * Sets >> EXTENT_UNKNOWN. */ >> 50#define FIEMAP_EXTENT_ENCODED 0x0008 /* Data can not be read >> 51 * while fs is >> unmounted */ >> 52#define FIEMAP_EXTENT_DATA_ENCRYPTED 0x0080 /* Data is >> encrypted by fs. >> 53 * Sets >> EXTENT_NO_BYPASS. */ >> 54#define FIEMAP_EXTENT_NOT_ALIGNED 0x0100 /* Extent >> offsets may not be >> 55 * block aligned. */ >> 56#define FIEMAP_EXTENT_DATA_INLINE 0x0200 /* Data mixed >> with metadata. >> 57 * Sets >> EXTENT_NOT_ALIGNED.*/ >> 58#define FIEMAP_EXTENT_DATA_TAIL 0x0400 /* Multiple >> files in block. >> 59 * Sets >> EXTENT_NOT_ALIGNED.*/ >> 60#define FIEMAP_EXTENT_UNWRITTEN 0x0800 /* Space allocated, but >> 61 * no data >> (i.e. zero). */ >> 62#define FIEMAP_EXTENT_MERGED 0x1000 /* File does not >> natively >> 63 * support >> extents. Result >> 64 * merged for >> efficiency. */ >> >> >> -- >> Regards, >> Sandeep. >> >> >> >> >> >> >> “To learn is to change. Education is a process that changes the learner.” >> >> -- >> To unsubscribe from this list: send an email with >> "unsubscribe kernelnewbies" to ecar...@nl.linux.org >> Please read the FAQ at http://kernelnewbies.org/FAQ >> >> > > > > -- > Thanks - > Manish > -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
ext4 extents: How to determine if an extent points to a hole.
Hi all, Looking at the flags in the extent info, Is there any specifc flags which indicates an extent to be a HOLE?? I could only find the following ones... 39#define FIEMAP_MAX_OFFSET (~0ULL) 40 41#define FIEMAP_FLAG_SYNC0x0001 /* sync file data before map */ 42#define FIEMAP_FLAG_XATTR 0x0002 /* map extended attribute tree */ 43 44#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) 45 46#define FIEMAP_EXTENT_LAST 0x0001 /* Last extent in file. */ 47#define FIEMAP_EXTENT_UNKNOWN 0x0002 /* Data location unknown. */ 48#define FIEMAP_EXTENT_DELALLOC 0x0004 /* Location still pending. 49* Sets EXTENT_UNKNOWN. */ 50#define FIEMAP_EXTENT_ENCODED 0x0008 /* Data can not be read 51* while fs is unmounted */ 52#define FIEMAP_EXTENT_DATA_ENCRYPTED0x0080 /* Data is encrypted by fs. 53* Sets EXTENT_NO_BYPASS. */ 54#define FIEMAP_EXTENT_NOT_ALIGNED 0x0100 /* Extent offsets may not be 55* block aligned. */ 56#define FIEMAP_EXTENT_DATA_INLINE 0x0200 /* Data mixed with metadata. 57* Sets EXTENT_NOT_ALIGNED.*/ 58#define FIEMAP_EXTENT_DATA_TAIL 0x0400 /* Multiple files in block. 59* Sets EXTENT_NOT_ALIGNED.*/ 60#define FIEMAP_EXTENT_UNWRITTEN 0x0800 /* Space allocated, but 61* no data (i.e. zero). */ 62#define FIEMAP_EXTENT_MERGED0x1000 /* File does not natively 63* support extents. Result 64* merged for efficiency. */ -- Regards, Sandeep. “To learn is to change. Education is a process that changes the learner.” -- To unsubscribe from this list: send an email with "unsubscribe kernelnewbies" to ecar...@nl.linux.org Please read the FAQ at http://kernelnewbies.org/FAQ
