Re: [zfs-discuss] ReiserFS4 like metadata/search
Oliver Schinagl wrote: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. You could use extended attributes for this type of data - just like HFS+ does - and then build a search tool ontop of that (like what MacOS X does with Spotlight). You can store any kind of data you like in an extended attribute, however I would caution you that storing the metadata of somethink like an MP3 file in metadata may not actually be quicker in the long run. Exactly what problem are you trying to solve and what kind of metadata are you looking for that isn't natively inside the file formats like MP3 for track info and EXIF data in JPEG etc ? Why do you believe that the file system having knowledge of this is better some how ? The other thing that ZFS has is user defineable properties on each dataset. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
[EMAIL PROTECTED] wrote on 06/27/2007 06:25:47 PM: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. So does ZFS support features like metadata and such? or is that for zfs2? :) Without getting too far into political/personal debates, Reiser has promised a lot and not done very well delivering for common case (nor does he appear to be in a position to do so any time soon). A lot can be said for having the code refused from Linux core. I much prefer the route taken by apple on this -- spotlight is fs agnostic and attaches to the kernel file update poller to know when to queue files for index/delete. It also resides in userspace with pluggable modules for extended file types (such as home brew files). One side effect of this design is that the indexing is completely pulled away from blocking any type of fs write -- they can be queued as low priority as needed. With this type of system and ZFS a pluggin could be created that indexes things like extended attributes/compression ratio/etc. What real advantages do you see doing this _in_ the filesystem layer? I can certainly see hooks being added where needed for the indexing system to interface -- but the core indexing and searching code does not seem to fit well in FS land. -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
On 28-Jun-07, at 11:37 AM, [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote on 06/27/2007 06:25:47 PM: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. So does ZFS support features like metadata and such? or is that for zfs2? :) Without getting too far into political/personal debates, Reiser has promised a lot and not done very well delivering for common case ... Do you mean a simple fs? Reiser3 certainly delivers, and R4 is stable according to the list. What real advantages do you see doing this _in_ the filesystem layer? I can certainly see hooks being added where needed for the indexing system to interface -- but the core indexing and searching code does not seem to fit well in FS land. ...except it's not really been tried. If R4 had been merged we would at least have some experiential data to work with. --Toby -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
I guess the userdefinable properties is then what i'm looking for. Well not what *I* am looking for perse. i was reading the article on Hans Reiser, the one over at wired, good read btw, (http://www.wired.com/techbiz/people/magazine/15-07/ff_hansreiser?currentPage=1). Somewhere it stated that the 'revolutionairy' new thing about reiser4 was that it tracked meta data somehow, makeing desktop searches MUCH faster. Don't ask me about the details, i'm no filesystem wiz :) but i'm sure someone familiar with Reiser3/4 hopefully ellaborate? Darren J Moffat schreef: Oliver Schinagl wrote: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. You could use extended attributes for this type of data - just like HFS+ does - and then build a search tool ontop of that (like what MacOS X does with Spotlight). You can store any kind of data you like in an extended attribute, however I would caution you that storing the metadata of somethink like an MP3 file in metadata may not actually be quicker in the long run. Exactly what problem are you trying to solve and what kind of metadata are you looking for that isn't natively inside the file formats like MP3 for track info and EXIF data in JPEG etc ? Why do you believe that the file system having knowledge of this is better some how ? The other thing that ZFS has is user defineable properties on each dataset. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
On 28-Jun-07, at 4:46 PM, Oliver Schinagl wrote: I guess the userdefinable properties is then what i'm looking for. Well not what *I* am looking for perse. i was reading the article on Hans Reiser, the one over at wired, good read btw, (http://www.wired.com/techbiz/people/magazine/15-07/ff_hansreiser? currentPage=1). Somewhere it stated that the 'revolutionairy' new thing about reiser4 was that it tracked meta data somehow, makeing desktop searches MUCH faster. Don't ask me about the details, i'm no filesystem wiz :) but i'm sure someone familiar with Reiser3/4 hopefully ellaborate? The WIRED article is technically crap^W inept. Go to http:// namesys.com/ for the R4 white papers. --Toby Darren J Moffat schreef: Oliver Schinagl wrote: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. You could use extended attributes for this type of data - just like HFS+ does - and then build a search tool ontop of that (like what MacOS X does with Spotlight). You can store any kind of data you like in an extended attribute, however I would caution you that storing the metadata of somethink like an MP3 file in metadata may not actually be quicker in the long run. Exactly what problem are you trying to solve and what kind of metadata are you looking for that isn't natively inside the file formats like MP3 for track info and EXIF data in JPEG etc ? Why do you believe that the file system having knowledge of this is better some how ? The other thing that ZFS has is user defineable properties on each dataset. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
Oh i just ment it was generally a good read :) Nicely written and the likes. It just mentions a little part about RFS4 and that the 'new' thing introduced by RFS4 was the ability to search through the metadata extremly fast. Personally, I don't really use all this search nonsense. Wait, that's a lie. I love locate. The fact that it runs once a day is fine by me. It's nice and fast, and does it's scanning in the background. I think trackerd (a linux thing) is the perfect user replacement, for those who need it. I believe it uses the inotify/dnotify fam thingers to be informed when a file changes/updates so it knows when to update/delete it's info. I guess my real question should have been, IF it turns out that quick indexing and the like are really the next hot thing, would ZFS support it (yes from what i gathered earlier on this list). rant modeCome to think of it, the biggest difference of putting this info in the FS layer or in a seperate DB would be availabilty. If the extra metadata is placed in the FS, as 'extra info' for files, then you have it in a DB right then and there, quickly accessible without having to run any daemons. Now you have to scan files keep them in a seperate db, and keep that db up to date. I can see why the guys over at namesys would want to do that, with all this 'searching' going on nowadays. I don't need to search that much myself. I usually keep my stuff organized enough to quickly find what I need. But if you look at it from a Windows point of view: Why was the 'smart start menu' introduced you think? The start menu always ends up a mess. Install 1 app, your start menu gets loaded with crap. even after a clean fresh install it's loaded with crap/unstructured. The 'smart' start menu was supposed to solve that by only showing the most rescent items and 'hiding' the rest. Then they came with the most used apps in the main menu, cause the desktop clearly wasn't usable for that anymore, as it was loaded with icons from users. And quicklaunch seems to be unused/disapeard because of that same reason. So I can see why desktop search and the like is getting so popular, people make a mess out of their system, requiring tool/methods to find stuff again. Anyway, this isn't really the right place to rant right :) /rant mode Toby Thain wrote: On 28-Jun-07, at 4:46 PM, Oliver Schinagl wrote: I guess the userdefinable properties is then what i'm looking for. Well not what *I* am looking for perse. i was reading the article on Hans Reiser, the one over at wired, good read btw, (http://www.wired.com/techbiz/people/magazine/15-07/ff_hansreiser?currentPage=1). Somewhere it stated that the 'revolutionairy' new thing about reiser4 was that it tracked meta data somehow, makeing desktop searches MUCH faster. Don't ask me about the details, i'm no filesystem wiz :) but i'm sure someone familiar with Reiser3/4 hopefully ellaborate? The WIRED article is technically crap^W inept. Go to http://namesys.com/ for the R4 white papers. --Toby Darren J Moffat schreef: Oliver Schinagl wrote: The only thing I haven't found in zfs yet, is metadata etc info. The previous 'next best thing' in FS was of course ReiserFS (4). Reiser3 was quite a nice thing, fast, journaled and all that, but Reiser4 promised to bring all those things that we see emerging now, like cross FS search, any document, audio recording etc could be instantly searched. True there is google desktop search, trackerd and what not, but those are 'afterthoughts', not supported by the underlying FS. You could use extended attributes for this type of data - just like HFS+ does - and then build a search tool ontop of that (like what MacOS X does with Spotlight). You can store any kind of data you like in an extended attribute, however I would caution you that storing the metadata of somethink like an MP3 file in metadata may not actually be quicker in the long run. Exactly what problem are you trying to solve and what kind of metadata are you looking for that isn't natively inside the file formats like MP3 for track info and EXIF data in JPEG etc ? Why do you believe that the file system having knowledge of this is better some how ? The other thing that ZFS has is user defineable properties on each dataset. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ReiserFS4 like metadata/search
I guess my real question should have been, IF it turns out that quick indexing and the like are really the next hot thing, would ZFS support it (yes from what i gathered earlier on this list). rant modeCome to think of it, the biggest difference of putting this info in the FS layer or in a seperate DB would be availabilty. If the extra metadata is placed in the FS, as 'extra info' for files, then you have it in a DB right then and there, quickly accessible without having to run any daemons. You also have now tied the indexing and index db management to fs ops along with all the baggage it brings (code and memory size for parsing documents, document parser extensions, index db btree ops, blocking or indexing thread management, index lock spinning, ...). Now you have to scan files keep them in a seperate db, and keep that db up to date. That DB would either be a new file in the fs or another type of data store managed by the fs -- either way you have the same overhead as a userspace daemon would. I can tell you that both MS and Google's desktop search product line had a _ton_ of work put in to a; make sure that the default search index priority did not slow down normal end user experience and b; is very tunable to the point of being off (no-op). When coupled directly into the fs layer you are stuck doing this in threads (if not blocking) and on a heavy used system where these are tuned down could become quite a challenge to manage. I can see why the guys over at namesys would want to do that, with all this 'searching' going on nowadays. I don't need to search that much myself. I usually keep my stuff organized enough to quickly find what I need. Interestingly enough I view the server/desktop search market as being formed behind the userspace camp already -- what seems to be on the horizon is the share search space. Servers and desktops replying to search requests as a grid. Looking at this level of search requires a userland daemon. But if you look at it from a Windows point of view: Why was the 'smart start menu' introduced you think? The start menu always ends up a mess. Install 1 app, your start menu gets loaded with crap. even after a clean fresh install it's loaded with crap/unstructured. The 'smart' start menu was supposed to solve that by only showing the most rescent items and 'hiding' the rest. Then they came with the most used apps in the main menu, cause the desktop clearly wasn't usable for that anymore, as it was loaded with icons from users. And quicklaunch seems to be unused/disapeard because of that same reason. So I can see why desktop search and the like is getting so popular, people make a mess out of their system, requiring tool/methods to find stuff again. Anyway, this isn't really the right place to rant right :) /rant mode Search (server and desktop) has many uses -- finding apps to quickly launch is for sure one. To view MS's attempt to closely tie FS and search go boot up Vista and play around with WINFS -- err nevermind. I do not hold Windows (XP or Vista) up as the bar to beat though and would prefer not to quibble about their failed attempts. -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
