Re: [slim] Re: sort order and database speed
Vidur Apparao wrote: You're right - I believe there is scope for substantial improvement here. Please file an enhancement bug at http://bugs.slimdevices.com, assign it to me and target it for 6.1. Cool. I just changed my recently filed bug 1497 from a bug to an enhancement request, targeted it for 6.1, and reassigned it to you. SBB Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
Steve Baumgarten wrote: Which brings me back to my initial post on this, and why I was so surprised to find that "Browse Music Folder" took so long in 6.0.2. You're right - I believe there is scope for substantial improvement here. Please file an enhancement bug at http://bugs.slimdevices.com, assign it to me and target it for 6.1. Thanks! --Vidur ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
Steve Baumgarten Wrote: > I kind of understand what's going on behind the scene. If you think > about it, what can SlimServer do when it descends into a directory? > > If it just spit back a raw directory listing, well, that's OK as far > as > it goes, but then none of the tracks would be "clickable". That is, it > could show you what's there, but it wouldn't understand what it was > showing you. I'm not sure I follow - what information does SlimServer need to know about the file to make it "clickable"? All it really needs is an extension to know if the file is a type that it can play. If you mean the detailed information that you see when you click the file, then the task of pulling the info from a single file should be trivial at the rate that a human being can browse and click them. The server might also need this info when it plays a track, but once again, this doesn't _have_ to come out of the database. > If a raw directory listing is fine, as I think it should be done This > would be akin to my tiny perl script that runs super > fast but has no understanding of what it's listing. I think a raw listing is fine as well. Display only the directory and file names. Restricting files shown to music files makes sense when browsing using the remote and the Squeezebox screen. When browsing through the web interface it would make sense to be able to see all files. Graphics files should be clickable, so you can view cover art, booklet scans, etc. > The alternative is for SlimServer to do a lookup for each track it > encounters -- this way it can "understand" what it's showing you in a > directory listing. > > Mixed in with this is the idea of picking up newly added tracks and/or > directories, a limited "rescan", if you will. > > I guess what's confused me is that since all I have at the top level > of > my music folder is more folders (about 800 of them) and no loose > tracks, > I don't know what takes SlimServer so long. There's really nothing > going > on there for it to try to "understand", do lookups on in the database, > etc. So what's it doing as it's reading that top level directory? In this scenario I wonder if the the server is doing 800 database lookups to tell if it's seeing any new directories? Or should it only query the database when it encounters music tracks? -- JJZolx ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
if you can read perl, the handler for browse music folder(and browse playlists) is in Pages.pm, sub browser() and browser_addtolist_done() If you have any suggestions or find any places where there is room to optimise, let me know. I would love to be able to understand that part, but I just cant put aside enough time to stare at it long enough to have it all make clear sense. -kdf ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
JJZolx wrote: > IMO, it's just plain silly to use this method of navigation to build > the database. Aren't there better options for low-cost methods of > cataloging newly added files? I kind of understand what's going on behind the scene. If you think about it, what can SlimServer do when it descends into a directory? If it just spit back a raw directory listing, well, that's OK as far as it goes, but then none of the tracks would be "clickable". That is, it could show you what's there, but it wouldn't understand what it was showing you. This would be akin to my tiny perl script that runs super fast but has no understanding of what it's listing. The alternative is for SlimServer to do a lookup for each track it encounters -- this way it can "understand" what it's showing you in a directory listing. Mixed in with this is the idea of picking up newly added tracks and/or directories, a limited "rescan", if you will. I guess what's confused me is that since all I have at the top level of my music folder is more folders (about 800 of them) and no loose tracks, I don't know what takes SlimServer so long. There's really nothing going on there for it to try to "understand", do lookups on in the database, etc. So what's it doing as it's reading that top level directory? Which brings me back to my initial post on this, and why I was so surprised to find that "Browse Music Folder" took so long in 6.0.2. SBB ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
Quoting JJZolx <[EMAIL PROTECTED]>: > IMO, it's just plain silly to use this method of navigation to build > the database. Aren't there better options for low-cost methods of > cataloging newly added files? Lots of options/plans, none implemented yet. Resources are tied up elsewhere. As always, patches are more than welcome. For all we know right now, everyone at world HW is busily hand painting paltinum SB2's in order to keep up with demand :) I've heard mention of some sort of active scan and cleanup routine, but it wouldnt' be something that could be written up overnight. If it was, it would already be there. -kdf ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
kdf Wrote: > It is a known issue, but those who have spent the most time working with > it are not sure, at this time, how to optimise it any more without > losing aspects of the design. IMO, it's just plain silly to use this method of navigation to build the database. Aren't there better options for low-cost methods of cataloging newly added files? How about implementing a slow, continuous (re)scan of the music tree? Something that might take a couple hours to complete on a fast machine with a large (30k tracks?) directory. Just let it run 24/7, with the ability to turn it off. How about the ability to do a 'selective' rescan in the web interface? Say you add a new album under your Miles Davis directory. Navigate to the directory in the web interface and select that sub-tree. Hit a "rescan" button and only that bit of the tree is scanned. Not everyone will organize their music this way, but I'm fairly certain that enough of ua are using subfolders in some manner that this would be very useful. -- JJZolx ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
Quoting SlimPvC <[EMAIL PROTECTED]>: > Thanks, that does clear things up for me! > However, in case of using the Browse Music Folder, it does not 'feel' > like an enhancement. Although technically, I'm sure it is. I can certainly understand that. The enhancement of it is not really seen so much. 5.4.x was very intermittent with finding newly added tracks, requiring frequent and costly rescans (and many complaints). The new design picks up new tracks very well, as the cost of hitting the db during folder browsing. The sort by filename option does have an effect, though again not without a tradeoff. The advantage for me is that I've not used 5.4.x in so long, I forget. Current 6.0 builds are much faster than early betas were for folder navigation. It is a known issue, but those who have spent the most time working with it are not sure, at this time, how to optimise it any more without losing aspects of the design. Patches are always welcome if anyone has some tweaks that do show improvement. I, for one, would welcome that. I do use the other modes more than I used to, but sill like folder browse when I want something specific. As annoying as it is to have to wait, I know I can't offer anything that will speed it up more than it already is. In the meantime, I'm spending my time looking at other place where I might help. If aomething does spring to mind along the way, I plan to submit a patch. Right now, I'm still trying to catch up on everythign that the server is doing with the database. of course, that doesn't help you achieve your goal of faster navigation, and I can only speak for myself as to the difficulties associated with it. Some may choose not to believe it, but that's ok. -kdf ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
kdf Wrote: > There was no database in 5.4.x. All data was loaded and stored in ram, > hence > the long startup and heavy ram usage. 6.x has a faster startup, but > the data > is primarily stored on disc in a database. scanning files, and hitting > the > database means accessing the filesystem. Browse music folder does both > of > these, whereas the other browse modes only access the db. > > -kdf Thanks, that does clear things up for me! However, in case of using the Browse Music Folder, it does not 'feel' like an enhancement. Although technically, I'm sure it is. Peter -- SlimPvC ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
There was no database in 5.4.x. All data was loaded and stored in ram, hence the long startup and heavy ram usage. 6.x has a faster startup, but the data is primarily stored on disc in a database. scanning files, and hitting the database means accessing the filesystem. Browse music folder does both of these, whereas the other browse modes only access the db. -kdf ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
Steve Baumgarten Wrote: > Jim wrote: > > > > 1 - Don't use browse folders, tag all your stuff correctly.[/color] > > My stuff is all tagged 100% correctly, and SlimServer's DB reflects > this. For what I want to be able to do, however, it's not enough. > I second this post. I use 'browse music folder' too because it gives me the most control over what I want to hear. My music is tagged OK too, the folders are neatly organised in artist\record format. It's annoying that using the Browse Music Folder halts the SB2 for about half a minute or so; database access seems slower in this case than it was in 5.4.x Hope this'll get solved -- SlimPvC ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
Quoting "max.spicer" <[EMAIL PROTECTED]>: > > Dan Sully Wrote: > > > > Artist names are shown in these listings in the 6.1 tree. We listen! :) > This is a preference in 6.1. If you want to enable it , look under > Server Settings->Formatting (a strange place to put such prefs, > IMHO!). except that this is what is defined at formatting, titleformat, time format, etc. showing artist, showing year is also formatting the list output. however, I can accept that there are as many different ways of grouping the hundreds of prefs as there are users. Maybe we need a server pref to set the grouping of prefs? ;) -kdf ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
Dan Sully Wrote: > > Artist names are shown in these listings in the 6.1 tree. We listen! :) This is a preference in 6.1. If you want to enable it , look under Server Settings->Formatting (a strange place to put such prefs, IMHO!). Max -- max.spicer The wild things roared their terrible roars and gnashed their terrible teeth and rolled their terrible eyes and showed their terrible claws but Max stepped into his private boat and waved good-bye ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
Jim Wrote: > > > 1 - Don't use browse folders, tag all your stuff correctly. > > 2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles - > White Album\" could become "D:\Music\B\Beatles\White Album" nr 1 would be nice but damn time consuming nr 2 nAH i dont use window$... i see the problem, thats why it would be nice to define settings per folder: like: -sort order -cache (yes/no) so the huge archive could be cached and the news folder will be uncached for example... -- nemstter ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Re: sort order and database speed
Jim wrote: But then you're going to be moaning about a 5-minute startup time as it would need to scan all the folders to put them into a database (kind of like a lite version of the existing rescan). It takes 15 seconds for a perl script to traverse my 800 folder, 16,000 track music library on Windows XP. (This is without doing DB access, just simply stat'ing each file or folder and printing the name to the screen using perl's File::Find package.) Why on earth do people think that file/directory access is so lousy under XP? Folders are only there for people who can't be arsed to sort out their tagging, or for quickly playing new stuff you haven't tagged. Yes and no. Folders are there, for me, so I can quickly sort by meaningful artist. That is, right now SlimServer lets me browse by artist -- but I get a huge list this way, as every compilation album that has 15 different artists contributes, resulting in an unmanageable list. (In this way SlimServer is no different than iTunes or Winamp; both suffer from exactly the same problem. All the meaningful artist names are swamped by the artists that have 1 track on 1 album.) My top-level music folder is organized by album, with names like this: Amerie - Touch (2005) M.I.A. - Arular (2005) VA [Ministry of Sound] - The Annual 2004 (2004) I have 800 albums in my music library; they are relatively easy to browse this way. SlimServer gives me a few options, none ideal: 1. Browse by artist: all artists are shown, even those that only have one track on one compilation album. This is unmanageable. 2. Browse by album: No artist names are shown. So I have to remember that Amerie's album is called "Touch". I think of it as an album by Amerie, so I expect to see her name as a memory aid. 3. New Music: This is helpful, and definitely a move in the right direction, but again, no artist names. Ideally, being able to list "meaningful" artists is what I want, and I have no doubt such a thing is now possible with the DB back end, but so far it hasn't been implemented. In the meantime, browsing my top-level music folder is a pretty fair substitute. (And in fact because of this I tend not to browse at all in SlimServer, but rather do a "Live Search" for everything. It's fast and works well. But sometimes you just want to browse, you know?) If your collection is so big that browsing folders slows it all down then do one of two things: 1 - Don't use browse folders, tag all your stuff correctly. My stuff is all tagged 100% correctly, and SlimServer's DB reflects this. For what I want to be able to do, however, it's not enough. 2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles - White Album\" could become "D:\Music\B\Beatles\White Album" Yes, I could do that. But then I lose a few things. For one, when looking at the top level folder in Windows, I can sort by date modified and see which albums I most recently added. For another I can do this with thumbnails turned on and see all the artwork, with sort order either by folder name (which is effectively by artist name given my naming scheme) or by date modified (to show me the most recently added albums on top). Really, this isn't something that's just some sort of intrinsic problem with Windows XP. We're not talking about 100,000 items in a folder, some sort of bizarre edge case, just 800. It takes less than 1 second for a perl script to read my top level directory contents and just 15 seconds to traverse my entire library and stat every single file and directory entry and print them to the screen. So rather than urge the "Doctor, doctor, it hurts when I do this..." "So don't do that" approach, I'm going to take Dan's suggestion and visit bugzilla, though only after I check into my "Sort By File" setting to see if perhaps that's playing a role in the incredibly poor music folder browsing performance I, and others, are seeing. SBB Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. ___ Discuss mailing list Discuss@lists.slimdevices.com http://li
[slim] Re: sort order and database speed
But then you're going to be moaning about a 5-minute startup time as it would need to scan all the folders to put them into a database (kind of like a lite version of the existing rescan). Windows provides no easy way to see if new folders were added to a folder, especially when in the name of speed some programs do things without updating date modified attributes of the file system. And then what happens when you copy your newly rippèd album over for a quick listen? You'll have to hit the rescan dirs button or restart the server which'll take 5 minutes. Folders are only there for people who can't be arsed to sort out their tagging, or for quickly playing new stuff you haven't tagged. If your collection is so big that browsing folders slows it all down then do one of two things: 1 - Don't use browse folders, tag all your stuff correctly. 2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles - White Album\" could become "D:\Music\B\Beatles\White Album" -- Jim ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
[slim] Re: sort order and database speed
about the slow issue: i thinks its only when i browse folder. wouldn^t it be possible to add the dirs to database aswell? -- nemstter ___ Discuss mailing list Discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
