Re: [Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
Mike, > Are shortnames still too common to make them optional? It's unfortunate > that you incur the overhead of shortname support on all clients when > only a small number of scenarios require them. They've been optional in Samba3 for a while (via the "mangled names" boolean option). Unfortunately disabling them is really just a benchmark hack for now, as the few users of them are quite important. Making cmd.exe not work on a WinXP client would be a pretty serious functionality loss :-) At Microsoft Tech-Ed 2004 they recommended "disabling 8.3 name creation" for NTFS file server performance. I was quite surprised. Eric Roseme Hewlett-Packard -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
Mike, > Are shortnames still too common to make them optional? It's unfortunate > that you incur the overhead of shortname support on all clients when > only a small number of scenarios require them. They've been optional in Samba3 for a while (via the "mangled names" boolean option). Unfortunately disabling them is really just a benchmark hack for now, as the few users of them are quite important. Making cmd.exe not work on a WinXP client would be a pretty serious functionality loss :-) Cheers, Tridge -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
On Mon, 24 Jan 2005 13:40:42 +1100 Andrew Tridgell <[EMAIL PROTECTED]> wrote: > Perhaps the most common case is where a user on the client does "del *.*". > In that case the client asks for a directory listing, then it asks to > delete each file one at a time, but it provides the mangled (8.3) name > for the file in the SMBunlink call rather than the full name. Are shortnames still too common to make them optional? It's unfortunate that you incur the overhead of shortname support on all clients when only a small number of scenarios require them. Mike -- Greedo shoots first? Not in my Star Wars. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
Mike, > If the path requested does not exist the server MUST search the files > within each directory and do a case insensitive comparison on each to > prove that the file does not exist. However, if the file does exist, > usually the case provided is correct. So as an optimization you can just > stat(2) for that component and only do the directory listing search if > it fails. Additionally if you simply stat(2) the entire path requested > at the very beginning and it exists you can skip the multiple stats and > directory searches altogether. > > I'm not terribly familar with Samba but apparently the version you're > using lacks some of these optimizations. I'm pretty sure Samba3 does have these optimisations (it did last time I looked), and certainly Samba4 does, but unfortunately there are cases where the client makes calls that don't allow this sort of optimisation. Perhaps the most common case is where a user on the client does "del *.*". In that case the client asks for a directory listing, then it asks to delete each file one at a time, but it provides the mangled (8.3) name for the file in the SMBunlink call rather than the full name. Samba tries to optimise this by keeping a reverse lookup cache from 8.3 name to long name (the so called "mangled cache"). As long as the cache doesn't fill, then this works well. If the cache does fill then things are not so nice. The code then falls back to a full directory search to do the mapping from 8.3 to full name. For the "del *.*" case this turns the whole operation into something that is O(n^2), which gets very very slow for large n. I am a proposed solution for this that I have on my todo list for Samba4. The solution is to make the cache size dynamic, but use a timer event to shrink the cache per connection when its not needed. So no matter how big the directory is Samba will always keep the whole directory in cache for a few seconds (probably 30 seconds or so), but will then shrink it after that time. It will also have to have a "number of directories in cache" limit to prevent DOS attacks on this mechanism. Cheers, Tridge -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
On Sun, 23 Jan 2005 04:40:00 +0200 "Nikos Balkanas" <[EMAIL PROTECTED]> wrote: > > jra applied some patches to ensure that the directory listing was not > > performed. > > Do you mean to say that an exact "stat" is performed in this case? > Wouldn't it be more efficient from the beginning to do 2 "stats" - 1 for > upper case, the other for lower case? Even in the case where the exact > name is known from the database, there is no guarantee that windows > filesystem will request the same case from Unix. Can one do wildcard > matching in this case? If the path requested does not exist the server MUST search the files within each directory and do a case insensitive comparison on each to prove that the file does not exist. However, if the file does exist, usually the case provided is correct. So as an optimization you can just stat(2) for that component and only do the directory listing search if it fails. Additionally if you simply stat(2) the entire path requested at the very beginning and it exists you can skip the multiple stats and directory searches altogether. I'm not terribly familar with Samba but apparently the version you're using lacks some of these optimizations. Mike -- Greedo shoots first? Not in my Star Wars. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
Hi Andrew, Thanks for the fast reply. I don't know why, your reply reached me as an attachment. A couple of questions: On Sat, 2005-01-22 at 08:53 +0200, Andrew Bartlett wrote: > On Sat, 2005-01-22 at 07:53 +0200, Nikos Balkanas wrote: >> Hello, >> >> Solution developed against samba 2.2.22. Didn't and do not have the >> opportunity to test samba 3.0.0. >> No browsing or wildmasks of files needed, only >> exact file request through the database. >> >> Putting samba through the debugger, I noticed that on every file request, it >> would scan all the files in the large directory, while converting to Unix >> filenames and building up the filename cash until it reaches 150 MB. I >> developed a configurable parameter "many files", which when set, disables >> file browsing (who needs listing of ~1,000,000 files?) and performs a "stat" >> to get the file. > Yes, this is a well-known problem with Samba's case insensitive filename > handling, on case-sensitive Unix systems. > If, as occurs in your case, the name is known exactly, such that a stat > () will determine the result, then you may set 'case sensitive = yes', > and Samba will do exactly that. This is best in 3.0.11pre2, (ie, the > current code, I'm not sure how far back the changes were applied) where > jra applied some patches to ensure that the directory listing was not > performed. Do you mean to say that an exact "stat" is performed in this case? Wouldn't it be more efficient from the beginning to do 2 "stats" - 1 for upper case, the other for lower case? Even in the case where the exact name is known from the database, there is no guarantee that windows filesystem will request the same case from Unix. Can one do wildcard matching in this case? > So, you may wish to advise your former employer that an 'out of the box' > solution should now be available. Can one also disable browsing and wildcard matching per directory? If this is not done, security is compromised. Furthermore, only half the performance boost is realized, since any time a CRM agent will try to save a bill to his own PC, Windows opens as default saving directory the source directory, killing the server in the process (and the PC). The changes I have done result to a somehow "illegal" filesystem. If you do a "dir" or open a window on it, they both come up empty and return immediately. If, however, you do a "dir filename.txt" and the file exists, it will be displayed - only that file. Siebel has a very machinelike naming of the files, so little chance an average CRM user can guess it - unless has access to the database. This is not much, but guards against accidental deletion and is as much security as can be given from the filesystem - anything more must be given through the application. > Finally, I'm pleased to see Samba used being used in such big > applications. It's a joy to hear about these kind of installations. Always a pleasure - as long as they are knowledge people to take the risk and support it in case of problems ;-) Remember, all big top systems, utilize the database and the filesystem in a similar way. Until Windows realize that there is a professional need to decouple browsing from access, samba will be a superior choice. > Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
[Samba] Re: Samba - CPU and memory usage - Proposed solution(?)
On Sat, 2005-01-22 at 07:53 +0200, Nikos Balkanas wrote: > Hello, > > Solution developed against samba 2.2.22. Didn't and do not have the > opportunity to test samba 3.0.0. > No browsing or wildmasks of files needed, only > exact file request through the database. > > Putting samba through the debugger, I noticed that on every file request, it > would scan all the files in the large directory, while converting to Unix > filenames and building up the filename cash until it reaches 150 MB. I > developed a configurable parameter "many files", which when set, disables > file browsing (who needs listing of ~1,000,000 files?) and performs a "stat" > to get the file. Yes, this is a well-known problem with Samba's case insensitive filename handling, on case-sensitive Unix systems. If, as occurs in your case, the name is known exactly, such that a stat () will determine the result, then you may set 'case sensitive = yes', and Samba will do exactly that. This is best in 3.0.11pre2, (ie, the current code, I'm not sure how far back the changes were applied) where jra applied some patches to ensure that the directory listing was not performed. So, you may wish to advise your former employer that an 'out of the box' solution should now be available. Finally, I'm pleased to see Samba used being used in such big applications. It's a joy to hear about these kind of installations. Andrew Bartlett -- Andrew Bartletthttp://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net signature.asc Description: This is a digitally signed message part -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/listinfo/samba
