[SLUG] Performance Tuning
Can somebody recommend a reasonably comprehensive but straightforward performance tuning article/HowTo/PDF/site I could read pls? Specifically, I am looking to perf-tune a dual-CPU RAID5 box used as a backup server. -- Kind Regards Kyle -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Kyle [EMAIL PROTECTED] writes: Can somebody recommend a reasonably comprehensive but straightforward performance tuning article/HowTo/PDF/site I could read pls? There isn't one, because... Specifically, I am looking to perf-tune a dual-CPU RAID5 box used as a backup server. ...what you need to do to tune your system to work effectively as a backup server is extremely different from, say, tuning to a database load, to tuning for a compute load, to tuning for anything else. You have not even given enough information about what you intend to use it for: * what parts of performance are you concerned about? * what is the backup software? * how does it get data from the clients? * where does it spool that data temporarily? * where does it store it finally? * what compression, format transformation, etc happens to the data? * what filesystem are you planning to use, and is that fixed? * can you use something other than RAID5? * is it software RAID, hardware, or FakeRAID? * What sort of dual CPU is it -- two sockets, two cores, or one HT CPU? * how much memory do you have? Regards, Daniel -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Performance depends specifically on the job at hand, and whilst short on detail, I would suspect that you want fast file systems and disks optimised for write speed, but that's probably less than half the story, and I'm guessing that whatever the rest of the details are, you will be trying to tune an IO Bound system, so perhaps googling for this will help anyway, IBM have a pretty good primer for the different types of tuning you can do that goes reasonably in depth too http://safari.ibmpressbooks.com/013144753X On Sat, Sep 6, 2008 at 9:54 AM, Kyle [EMAIL PROTECTED] wrote: Can somebody recommend a reasonably comprehensive but straightforward performance tuning article/HowTo/PDF/site I could read pls? Specifically, I am looking to perf-tune a dual-CPU RAID5 box used as a backup server. -- Kind Regards Kyle -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] X authorization
I need to understand X authorization so if anybody can explain to a bear of little brain :-) Once-upon-a-time xhost + would allow anybody to write to your display. That is no longer true I have some thin clients (ltsp) running off a server. I actually *really* need to write to the display (as opposed to sending a message ie jabber to to user) eg This POS is not in operation I cannot see how xauth would help: The POS daemon program disables terminal say, um 17 and puts a message on terminal 17 Disabled etc but neither it (The POS daemon) or the user associated with it are running a display and may not even be the same server as the LTSP server. So my basic problem: I need to put a message (say with xmessage) on a ThinClient DISPLAY whether-or-not that terminal is logged in, and I can't understand the relevant man pages / howtos. I tried asking the ltsp forums and got nervous evasion. Anybody here know? Thanks James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Tony Sceats [EMAIL PROTECTED] writes: Performance depends specifically on the job at hand, and whilst short on detail, I would suspect that you want fast file systems and disks optimised for write speed, For backups? Often you need very fast reads, either to feed a modern tape streamer at the 30 to 80 MB per second of data it wants[1], or to be able to perform read and compare operations on existing data for comparison or pooling purposes (or even just the metadata of millions of files, really.) Fast writes are typically the least important part of the backup system. They matter, sure, for beating your backup window, but the actual performance problems often pop up on the read side. In my experience, anyhow. :) Regards, Daniel Footnotes: [1] ...and more if you have two or three of these drives attached to a single server, which is more common as we have individual machines with one to two tapes capacity each -- even for the 800MB native LTO4 devices. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] X authorization
jam [EMAIL PROTECTED] writes: I need to understand X authorization so if anybody can explain to a bear of little brain :-) Once-upon-a-time xhost + would allow anybody to write to your display. That is no longer true What makes you think that? There have been some changes to X security over the years, but the fundamental mechanisms are still in place... So, presumably you have tested this and found that it doesn't work. How did you test, and what was the error you encountered? I have some thin clients (ltsp) running off a server. I actually *really* need to write to the display (as opposed to sending a message ie jabber to to user) eg Are you absolutely, positively sure that you want to allow anyone to connect to the display? This is a security nightmare, as anyone allowed to connect can also capture and forge *all* keyboard and mouse input. This POS is not in operation I cannot see how xauth would help: The POS daemon program disables terminal say, um 17 and puts a message on terminal 17 Disabled etc but neither it (The POS daemon) or the user associated with it are running a display and may not even be the same server as the LTSP server. xauth is responsible for managing the (marginally) more secure authentication mechanisms for X connections -- you don't actually /need/ a display to use it effectively on an X client. For example, ssh X forwarding uses xauth to allow X software to run on a remote machine and display relatively securely on the local client. Now, you do have a problem: So my basic problem: I need to put a message (say with xmessage) on a ThinClient DISPLAY whether-or-not that terminal is logged in, and I can't understand the relevant man pages / howtos. Getting access while a session is logged in isn't that dire: you need to arrange for your message source to have access to an X connection to the appropriate LTSP machine. That means either host-based trust (xhost +message-source.local) or a more secure authentication mechanism (xauth with an appropriate key.) You might want to generate the appropriate MIT-MAGIC-COOKIE-1 data and arrange for your message source to have access to it, if security is a concern. While the login session is active, though, you have a bigger problem: you will need to do work on your login manager to achieve the same results. For the standard xdm, gdm or kdm system this isn't too dire -- all of them run scripts, as root, to perform session setup before displaying the chooser or login panel, and you should be able to use that to achieve the same result. For a more custom display manager, which I understand LTSP to use, you will need to do something more clever. I tried asking the ltsp forums and got nervous evasion. Anybody here know? I can't say I am surprised, since this is actually reasonably challenging in many ways; the security architecture, and the trade-offs in security make this a difficult approach. Perhaps you might find another approach effective: can you have the POS software manage authentication, and be able to display the not in use banner? That would make one bit of software responsible for all of this, which would make it a lot easier to manage. Alternately, can you SSH to the machine as the logged in user (or root), then run the message software locally? Regards, Daniel -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Actually I was under the impression that this was a vanilla on-disk backup - ie, something akin to dumping tgz's onto a server with reliable disks (hence RAID5).. I must say though that I am surprised that a tape feeder is faster than disks to the point of having to maintain a large buffer, but then the last time I was responsible for any tape system was with a single DDS4 tape drive.. backup management is the curse of systems admin imho, so I avoid it at all costs - it's horribly mundane and prone to break at will (but we all thank the great bit keeper when we need them!) I was also under the (unreasearched) opinion that incremental change analysis was performed on the client side, not the backup server side, although I suppose that backing a large number of similar servers would result in a lot of the same files being written a lot of times (eg /lib), so it would be smart to only have one copy of the file and a reference for each backup that includes this file.. this would certaintly be done on the server side... but really the point of the above is that you're almost certaintly right if you're talking about enterprise backup solutions, and that more to the point, the precise backup solution/software chosen will drastically change how you tune the server, which is actually what we were both saying, so it's a case in point On Sat, Sep 6, 2008 at 11:03 AM, Daniel Pittman [EMAIL PROTECTED] wrote: Tony Sceats [EMAIL PROTECTED] writes: Performance depends specifically on the job at hand, and whilst short on detail, I would suspect that you want fast file systems and disks optimised for write speed, For backups? Often you need very fast reads, either to feed a modern tape streamer at the 30 to 80 MB per second of data it wants[1], or to be able to perform read and compare operations on existing data for comparison or pooling purposes (or even just the metadata of millions of files, really.) Fast writes are typically the least important part of the backup system. They matter, sure, for beating your backup window, but the actual performance problems often pop up on the read side. In my experience, anyhow. :) Regards, Daniel Footnotes: [1] ...and more if you have two or three of these drives attached to a single server, which is more common as we have individual machines with one to two tapes capacity each -- even for the 800MB native LTO4 devices. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Ok, a couple of responses thus far. Some further info. The software I can tune myself. I was more looking for Linux specific tuning. * Yes, I was/am concerned about I/O. * But also ensuring the OS itself (system processes) is not hindering anything otherwise. * The RAID is the storage medium. (Hardware RAID) * Incremental change analysis is done client side. * Dual P4's / 1GB RAM * Filesys is ext3 mounted with 'defaults' Kind Regards Kyle Daniel Pittman wrote: You have not even given enough information about what you intend to use it for: * what parts of performance are you concerned about? * what is the backup software? * how does it get data from the clients? * where does it spool that data temporarily? * where does it store it finally? * what compression, format transformation, etc happens to the data? * what filesystem are you planning to use, and is that fixed? * can you use something other than RAID5? * is it software RAID, hardware, or FakeRAID? * What sort of dual CPU is it -- two sockets, two cores, or one HT CPU? * how much memory do you have? -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Tony Sceats [EMAIL PROTECTED] writes: Actually I was under the impression that this was a vanilla on-disk backup - ie, something akin to dumping tgz's onto a server with reliable disks (hence RAID5).. It could be, but that is a pretty uncommon model in my experience. I must say though that I am surprised that a tape feeder is faster than disks to the point of having to maintain a large buffer, but then the last time I was responsible for any tape system was with a single DDS4 tape drive.. You poor thing. ;) Seriously, though, the tape streamers are not faster than disks in the same class: you can find plenty of enterprise level U320 / SAS SCSI disk systems that can comfortably supply an LTO4 device. You start to hit problems when folks want to do this on the cheap, so back their system with large, slow 7200 or 10K SATA disks on SAS or SATA controllers; there you really do need to stripe to keep up. backup management is the curse of systems admin imho, so I avoid it at all costs - it's horribly mundane and prone to break at will (but we all thank the great bit keeper when we need them!) I was also under the (unreasearched) opinion that incremental change analysis was performed on the client side, not the backup server side, although I suppose that backing a large number of similar servers would result in a lot of the same files being written a lot of times (eg /lib), so it would be smart to only have one copy of the file and a reference for each backup that includes this file.. this would certaintly be done on the server side... This depends enormously on your backup software; BackupPC and the various rsync-and-hard-links based backup systems tend to put a lot of load on the server side. but really the point of the above is that you're almost certaintly right if you're talking about enterprise backup solutions, and that more to the point, the precise backup solution/software chosen will drastically change how you tune the server, which is actually what we were both saying, so it's a case in point In my experience there isn't /that/ much difference between enterprise and home backup strategies, except the order of magnitude: you face the same sort of performance issues that an LTO4/SAS array user does with your SATA/DDS[1]. You just face them slower: your tape device needs less MB/second to keep it streaming rather than scrubbing, but your disks are also slower and busier, so you have the same sort of dramas with keeping the device fed. Regards, Daniel In my experience, of course, which doesn't mean that /everyone/ is going to face these same issues. Footnotes: [1] Well, maybe not with DDS, but with tape hardware that is reasonably affordable by home users in this day and age, such as LTO[12], AIT and SDLT. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Performance Tuning
Kyle [EMAIL PROTECTED] writes: The software I can tune myself. I was more looking for Linux specific tuning. * Yes, I was/am concerned about I/O. * But also ensuring the OS itself (system processes) is not hindering anything otherwise. Unless you are running other processes on the system you can be reasonably confident that early performance measurement will tell you if the OS is responsible for problems. * The RAID is the storage medium. (Hardware RAID) You /really/ need to let us know what brand and configuration; 3ware and Areca have very different performance characteristics; the presence or absence of a BBU is going to be key, also. In general, there is a reasonable chance you will find the RAID5 performance of your hardware solution disappointing, and if you can move to something closer to a RAID10 you will, in general, have better results. * Incremental change analysis is done client side. Does the data just sit on this system, or is it sent somewhere else afterwards? (In other words, is this just a big, reliable hard disk?) Does this require reading the previous backup, or is it purely date based? Do you have a backup window? * Dual P4's / 1GB RAM * Filesys is ext3 mounted with 'defaults' You will probably find performance here disappointing. XFS with a 2.6.24 or more recent kernel will do better, perhaps significantly so. Otherwise you would want to look for tips on tuning an ext3 filesystem to be less conservative about performance or, if you can carry the risk, use ext2, to deliver better write performance. For a write-only load where the server is, essentially, a big and reliable hard disk for the backup software then, basically, you don't have a lot of tuning to do. The RAID layout and filesystem choices are the only real points to consider tuning up front -- and, probably, enabling LVM for ease of future management of volume space.[1] For 3ware, at least, and probably for other hardware controllers, you can gain significantly by tuning the queue depth, in line with the vendors recommendations. Finally, test, test, test. Get something as close as possible to your real load running and identify where the bottleneck is. [...] You have not even given enough information about what you intend to use it for: * what parts of performance are you concerned about? * what is the backup software? * how does it get data from the clients? * where does it spool that data temporarily? * where does it store it finally? * what compression, format transformation, etc happens to the data? * what filesystem are you planning to use, and is that fixed? * can you use something other than RAID5? * is it software RAID, hardware, or FakeRAID? * What sort of dual CPU is it -- two sockets, two cores, or one HT CPU? * how much memory do you have? Regards, Daniel Footnotes: [1] Depending on your hardware RAID brand you may be able to grow a live array, in which case you have more or less the same flexibility that LVM would give you. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] X authorization
On Saturday 06 September 2008 09:44:12 [EMAIL PROTECTED] wrote: I need to understand X authorization so if anybody can explain to a bear of little brain :-) Once-upon-a-time xhost + would allow anybody to write to your display. That is no longer true What makes you think that? There have been some changes to X security over the years, but the fundamental mechanisms are still in place... saturn is a CentOS 5 machine: [eeyore] /home/jam [53]% ssh -X saturn xhost + access control disabled, clients can connect from any host [eeyore] /home/jam [54]% export DISPLAY=saturn:0 xmessage hello world Error: Can't open display: saturn:0 So, presumably you have tested this and found that it doesn't work. How did you test, and what was the error you encountered? All the stuff that DOES work on a RedHat 9 system tried and does NOT work here I have some thin clients (ltsp) running off a server. I actually *really* need to write to the display (as opposed to sending a message ie jabber to to user) eg Are you absolutely, positively sure that you want to allow anyone to connect to the display? This is a security nightmare, as anyone allowed to connect can also capture and forge *all* keyboard and mouse input. This in a typical hungry jacks store: private lan and *nothing* but the POS and manager machines present. I'm not adverse to being more clever, I just inherited a real not-nice setup and am trying to clean up the mess. This POS is not in operation I cannot see how xauth would help: The POS daemon program disables terminal say, um 17 and puts a message on terminal 17 Disabled etc but neither it (The POS daemon) or the user associated with it are running a display and may not even be the same server as the LTSP server. xauth is responsible for managing the (marginally) more secure authentication mechanisms for X connections -- you don't actually /need/ a display to use it effectively on an X client. More RTBM (the bloody manual) :-) I understood using xauth to get the .Xauthority from one machine and import to another For example, ssh X forwarding uses xauth to allow X software to run on a remote machine and display relatively securely on the local client. Now, you do have a problem: So my basic problem: I need to put a message (say with xmessage) on a ThinClient DISPLAY whether-or-not that terminal is logged in, and I can't understand the relevant man pages / howtos. Getting access while a session is logged in isn't that dire: you need to arrange for your message source to have access to an X connection to the appropriate LTSP machine. That means either host-based trust (xhost +message-source.local) or a more secure authentication mechanism (xauth with an appropriate key.) You might want to generate the appropriate MIT-MAGIC-COOKIE-1 data and arrange for your message source to have access to it, if security is a concern. While the login session is active, though, you have a bigger problem: you will need to do work on your login manager to achieve the same results. For the standard xdm, gdm or kdm system this isn't too dire -- all of them run scripts, as root, to perform session setup before displaying the chooser or login panel, and you should be able to use that to achieve the same result. For a more custom display manager, which I understand LTSP to use, you will need to do something more clever. I tried asking the ltsp forums and got nervous evasion. Anybody here know? I can't say I am surprised, since this is actually reasonably challenging in many ways; the security architecture, and the trade-offs in security make this a difficult approach. Perhaps you might find another approach effective: can you have the POS software manage authentication, and be able to display the not in use banner? That would make one bit of software responsible for all of this, which would make it a lot easier to manage. Alternately, can you SSH to the machine as the logged in user (or root), then run the message software locally? Daniel that was most informative and helped me lots. Thank you. James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] X authorization
jam [EMAIL PROTECTED] writes: On Saturday 06 September 2008 09:44:12 [EMAIL PROTECTED] wrote: I need to understand X authorization so if anybody can explain to a bear of little brain :-) Once-upon-a-time xhost + would allow anybody to write to your display. That is no longer true What makes you think that? There have been some changes to X security over the years, but the fundamental mechanisms are still in place... saturn is a CentOS 5 machine: [eeyore] /home/jam [53]% ssh -X saturn xhost + access control disabled, clients can connect from any host [eeyore] /home/jam [54]% export DISPLAY=saturn:0 xmessage hello world Error: Can't open display: saturn:0 Are you sure that saturn is listening for X connections via TCP, and to connections from anything but localhost? Out of the box most modern systems configure X so that remote TCP isn't possible. (For example, my current X server does not listen for TCP at all.) [...] I have some thin clients (ltsp) running off a server. I actually *really* need to write to the display (as opposed to sending a message ie jabber to to user) eg Are you absolutely, positively sure that you want to allow anyone to connect to the display? This is a security nightmare, as anyone allowed to connect can also capture and forge *all* keyboard and mouse input. This in a typical hungry jacks store: private lan and *nothing* but the POS and manager machines present. I'm not adverse to being more clever, I just inherited a real not-nice setup and am trying to clean up the mess. No worries. I figure it is better to ask (redundantly) than to have you discover the problem later. :) This POS is not in operation I cannot see how xauth would help: The POS daemon program disables terminal say, um 17 and puts a message on terminal 17 Disabled etc but neither it (The POS daemon) or the user associated with it are running a display and may not even be the same server as the LTSP server. xauth is responsible for managing the (marginally) more secure authentication mechanisms for X connections -- you don't actually /need/ a display to use it effectively on an X client. More RTBM (the bloody manual) :-) I understood using xauth to get the .Xauthority from one machine and import to another There isn't that much more to it than that -- the X server has a list of xauth managed authorities that are allowed to connect, and xauth is a command line tool to move them around and make them available to the X client libraries. :) If you already know how to move them between machines you probably actually have half the battle solved, and will need very little time with the manual. (Also, I sympathise. The manual for X security related stuff is awful, which is a large part of why it took ten years for *any* significant use of the SECURITY extension to get out into the wider distributions, and then only thanks to OpenBSD via OpenSSH. Reading them and trying to work out how this works is not fun.) [...] Daniel that was most informative and helped me lots. Thank you. No worries. I hope you can resolve your issue. One final thought: if you trust the users then an option would be to run a second X server on an additional virtual terminal on each POS register. Have that do *nothing* but display your not in service banner, and disable server zap, etc, etc. No window manager, nothing useful, just the banner. You could then disable a POS terminal with: ssh [EMAIL PROTECTED] chvt 8 Substitute 8 for the VT where the second, not in service, banner software is running. From there the console can't do much to change away. Reversing the process is as simple as the same action, back to the live VT. That is, I think, the process I would use to manage this. It makes life a *lot* easier for everyone involved, because you no longer have to worry about security -- you just have your one, single, place to set in or out of service (active VT), and full control over the banner environment (so no worries about a session being active or not.) To further reduce (or, at least, change) complexity, you could also use a graphical Linux console and a framebuffer image view/display tool rather than a second X server. Regards, Daniel -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
