Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
At 00:59 26/05/2001, Jeff Garzik wrote: >Anton Altaparmakov wrote: > > NTFS 1.1.15 - ChangeLog > > === > > - Support for more than 128kiB sized runlists (using vmalloc_32() instead > > of kmalloc()). > >If you are running into kmalloc size limit, please consider some >alternative method of allocation. Why? I was told that's what vmalloc is for. And anyway, Windows NT/2k operate on a 256kB sized memory allocation basis and NTFS has been designed with this in mind. Breaking away from that is a PITA. I was very happy to learn about vmalloc as it solved a lot of my headaches with regards to structures being >128kb. btw. Maybe you, or someone else, can explain me whether vmalloc() will always work or whether vmalloc_32() is required? I am thinking of highmem pages which vmalloc could return. Would I have to make sure the pages were accessible by calling some function or is this done transparently? >Can you map it into the page cache, as Al Viro has done in recent patches? If someone would document how the page cache actually works from a kernel point of view, perhaps... Sorry, but I haven't had the time to go through all the related source to understand it properly. I have an idea of how it works conceptually (well, kind of...) but not a deep enough understanding to start implementing anything that goes anywhere near the page cache. I would love to move inode metadata to the page cache. That would simplify so much in the NTFS driver and would result in an incredible speed boost due to getting rid of all the silly copying of data back and forth inside the driver as we could just pass pointers around instead (to locked pages or other form of internal locking). >Can you break your allocation into an array of pages, obtained via >get_free_page? No thank you. I would rather have it contiguous or at least logically contiguous. I don't care if the pages are physically contiguous as long as I can see them as one piece... I mean, ok, I could break up run lists, as they are arrays of fixed size structs and the boundary cases would be predictable (but still would slow down the code with boundary checks) but this doesn't solve the generic problem later on as more things get implemented where we encounter variable width structures. It is much cleaner and easier to implement using a logically contiguous memory allocation IMHO. It's a shame Linux kernel doesn't support this kind of stuff (last time I read somewhere about this it said it doesn't anyway. I would be happy to be told wrong here). >If runlists are on-disk structures, can you look at bh->b_data instead of >keeping them in memory? They are not. They are derived from compressed on-disk structures (which are re-compressed when updating the inode). Runlists of such large size are only needed for A) huge files/directories, B) extremely fragmented files/directories, or C) a combination of A and B. (Remembering that metadata is stored as files, so the same applies for the metadata itself.) Best regards, Anton -- "Nothing succeeds like success." - Alexandre Dumas -- Anton Altaparmakov (replace at with @) Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available(1.1.15)
> I would love to move inode metadata to the page cache. That would simplify > so much in the NTFS driver and would result in an incredible speed boost > due to getting rid of all the silly copying of data back and forth inside > the driver as we could just pass pointers around instead (to locked pages > or other form of internal locking). [...] > No thank you. I would rather have it contiguous or at least logically > contiguous. I don't care if the pages are physically contiguous as long as > I can see them as one piece... I mean, ok, I could break up run lists, as > they are arrays of fixed size structs and the boundary cases would be > predictable When you talk about avoiding copying, are you thinking of not copying the runlists, as well? If so, how can you avoid uncompressing them? That seems very complicated, plus you have to concatenate the pieces of the runlist if they span multiple MFT records, anyway. > They are not. They are derived from compressed on-disk structures (which > are re-compressed when updating the inode). Runlists of such large size are > only needed for A) huge files/directories, B) extremely fragmented > files/directories, or C) a combination of A and B. (Remembering that > metadata is stored as files, so the same applies for the metadata itself.) I guess this answers my question - you will continue to uncompress the runlist when opening the inode, right? Regards, Martin - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
At 12:23 27/05/2001, Martin von Loewis wrote: > > I would love to move inode metadata to the page cache. That would simplify > > so much in the NTFS driver and would result in an incredible speed boost > > due to getting rid of all the silly copying of data back and forth inside > > the driver as we could just pass pointers around instead (to locked pages > > or other form of internal locking). >[...] > > No thank you. I would rather have it contiguous or at least logically > > contiguous. I don't care if the pages are physically contiguous as long as > > I can see them as one piece... I mean, ok, I could break up run lists, as > > they are arrays of fixed size structs and the boundary cases would be > > predictable > >When you talk about avoiding copying, are you thinking of not copying >the runlists, as well? If so, how can you avoid uncompressing them? >That seems very complicated, plus you have to concatenate the pieces >of the runlist if they span multiple MFT records, anyway. No, you can't avoid uncompressing them. But you can avoid copying all attributes out of the record into separate memory locations and then back. The run list is not part of the attribute so obviously has to be kept separate. Also, it is pointless to parse all attributes and their run lists on opening an inode, chaces are some of the attributes will never be touched at all and maybe we are just running stat on the file but don't want to start reading/writing the data. My idea involves a different run list format (as I already use in the linux-ntfs project on SF) which keeps [VCN, LCN, Length] records thus allowing you to not have the whole run list, i.e. if user accesses only beginning of the file, we only need to decompress the run list for the first extent. Also, this format allows out of line attribute extent handling, so if we access data at a certain offset we only get that extent and later as required we get the others and insert them into the run list. (This is not implemented yet, but I have been thinking about it.) Although I have to say I am not sure I want to have the run list extents fused into one. That only causes headaches when writing back the inode. But perhaps we can fuse them and then discard them without write back. Only if the attribute size changes do we need to deal with the problem but still it remains a problem. > > They are not. They are derived from compressed on-disk structures (which > > are re-compressed when updating the inode). Runlists of such large size > are > > only needed for A) huge files/directories, B) extremely fragmented > > files/directories, or C) a combination of A and B. (Remembering that > > metadata is stored as files, so the same applies for the metadata itself.) > >I guess this answers my question - you will continue to uncompress the >runlist when opening the inode, right? Yes and no. They will be uncompressed but not when opening the inode. It will be "uncompress required extent's run list on demand". The current driver with uncompressing everything and keeping copies of all attributes is A) very slow and B) eats incredible amounts of memory when you are operating on large partitions (talking tens of Gb partitions). Best regards, Anton -- "Nothing succeeds like success." - Alexandre Dumas -- Anton Altaparmakov (replace at with @) Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
> Yes and no. They will be uncompressed but not when opening the inode. It > will be "uncompress required extent's run list on demand". Are you sure this can work? Initially, I thought I could use the attribute list to only uncompress the extend that has the VCN I'm interested in. That would not work: NT would split individual runs across extends (i.e. split them in the middle). Did I misunderstand, or do you have a solution for that as well. Regards, Martin - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
At 13:53 27/05/2001, Martin von Loewis wrote: > > Yes and no. They will be uncompressed but not when opening the inode. It > > will be "uncompress required extent's run list on demand". > >Are you sure this can work? No. >Initially, I thought I could use the attribute list to only uncompress the >extend that has the VCN I'm interested in. Same idea here. >That would not work: NT would split individual runs across extends >(i.e. split them in the middle). Argh! That seems really stupid thing to do, as it makes it difficult to interpret what the highest_vcn/lowest_vcn field of attribute extents is supposed to mean!?! This does however explain some of the code uglyness I have seen (and chosen to ignore) in the ntfs.sys disassembly run list handling... >Did I misunderstand, or do you have a solution for that as well. No solution. I wasn't aware this could happen. I knew a compression block could be split in it halves but I didn't realize this braindamaged complication. I guess we will have to decode the whole run list in one go then. Anything else would be slower over all. - What we could do of course is to walk the mapping pairs array real quick not reading the numbers only to get the correct starting offset into the next extent and then only decode that one but that would mean walking the mapping pairs array repeatedly (once to get each extent) which would be overall slower than just getting the lot at once. - I maintain that we should do this on demand and not on inode open, though. - I will have another think about this, but if it is true that NT splits the records anywhere, then it would be impossible to start at any extent other than the first one. Anton -- "Nothing succeeds like success." - Alexandre Dumas -- Anton Altaparmakov (replace at with @) Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
Martin von Loewis wrote: >That would not work: NT would split individual runs across extends >(i.e. split them in the middle). Did I misunderstand, or do you have a >solution for that as well. > Are you sure that it's true? My NTFS resizer interprets parts of runlist stored in different FILE records independently and I never experienced any problems with that. Yuri - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
> >That would not work: NT would split individual runs across extends > >(i.e. split them in the middle). Did I misunderstand, or do you have a > >solution for that as well. > > > Are you sure that it's true? My NTFS resizer interprets parts of runlist > stored in different FILE records independently and I never experienced > any problems with that. I'm sure it could happen on NT 3.1. Maybe MS has fixed it since. Regards, Martin - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
At 12:10 28/05/2001, Martin von Loewis wrote: > > >That would not work: NT would split individual runs across extends > > >(i.e. split them in the middle). Did I misunderstand, or do you have a > > >solution for that as well. > > > > > Are you sure that it's true? My NTFS resizer interprets parts of runlist > > stored in different FILE records independently and I never experienced > > any problems with that. > >I'm sure it could happen on NT 3.1. Maybe MS has fixed it since. Hm, in that case I think I will implement it the way I was thinking and I will assume that it can't happen any more. - I will add some kind of sanity check at the beginning to catch problems and scream to the syslog which we can remove later. Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I know I can make sure we don't mount such beasts considering we know the driver would fail on them... - I am aware of only one person stil using NT 3.51 and he doesn't believe in the NTFS Linux driver any more, so I guess we can just say we support NT 4.0 and above only. Anton -- "Nothing succeeds like success." - Alexandre Dumas -- Anton Altaparmakov (replace at with @) Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
Anton Altaparmakov wrote: > Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I > know I can make sure we don't mount such beasts considering we know > the driver would fail on them... - I am aware of only one person stil > using NT 3.51 and he doesn't believe in the NTFS Linux driver any > more, so I guess we can just say we support NT 4.0 and above only. NT 3.51 uses exactly the same version of NTFS as NT 4.0 does. Yuri - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
At 14:08 28/05/2001, Yuri Per wrote: >Anton Altaparmakov wrote: > >>Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I >>know I can make sure we don't mount such beasts considering we know the >>driver would fail on them... - I am aware of only one person stil using >>NT 3.51 and he doesn't believe in the NTFS Linux driver any more, so I >>guess we can just say we support NT 4.0 and above only. > >NT 3.51 uses exactly the same version of NTFS as NT 4.0 does. Ok. Thanks. Anyone know about 3.1? Anton -- "Nothing succeeds like success." - Alexandre Dumas -- Anton Altaparmakov (replace at with @) Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
On Mon, May 28, 2001 at 03:49:28PM +0100, Anton Altaparmakov wrote: > At 14:08 28/05/2001, Yuri Per wrote: > >Anton Altaparmakov wrote: > > > >>Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I > >>know I can make sure we don't mount such beasts considering we know the > >>driver would fail on them... - I am aware of only one person stil using > >>NT 3.51 and he doesn't believe in the NTFS Linux driver any more, so I > >>guess we can just say we support NT 4.0 and above only. > > > >NT 3.51 uses exactly the same version of NTFS as NT 4.0 does. > > Ok. Thanks. > > Anyone know about 3.1? > > Anton > It's an HPFS variant. Try the HPFS driver, and it may work. The first cut of NT was with a hacked up HPFS driver from Microsoft OS/2. NTFS was designed internally by David G. and friends. Jeff > > -- >"Nothing succeeds like success." - Alexandre Dumas > -- > Anton Altaparmakov (replace at with @) > Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/ > ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/ > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
> > Anyone know about 3.1? > > > > It's an HPFS variant. No. NT was using NTFS right from the start. Regards, Martin - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)
On Tue, May 29, 2001 at 10:18:41AM +0200, Martin von Loewis wrote: > > > Anyone know about 3.1? > > > > > > > It's an HPFS variant. > > No. NT was using NTFS right from the start. > > Regards, > Martin No. They were not. Their first cuts of NT used an HPFS variant until NTFS could be completed. The guy who wrote NTFS for MS was a consultant for us for several years, and I got a very long briefing on the history of NT file systems over the years. I do not know which version was the cutover, but I do believe it was pre 3.X NT versions. Jeff - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
