Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 19, 2008 9:55 AM, chris rapier <[EMAIL PROTECTED]> wrote: > dhiggs wrote: > > If someone can split SSH into multiple threads, it should be just as > > possible to split it into multiple processes. However, I expect that > > most high-speed SSH traffic is SCP-/SFTP-based and therefore largely > > I/O bound, so it hasn't been high on anyone's requirements list. > > Based on my experience on working in the field of bulk data transfers on > high speed networks for the past 6 years I'd have to say that you are, > at the least, partly mistaken. However, maybe I'm not really > understanding what you are saying. Could you expand on this a bit more? Your experience vastly trumps mine, but I'll try to explain my thought processes: What is SSH used for? My initial breakdown was into two categories - shell/batch commands and SCP/SFTP file transport. I guessed that an application that outputs data to stdout at a rate of 10s or 100s of Mbit/s over long periods of time would quickly be changed. The output would be redirected into a file or postprocessed in some other fashion to reduce the amount of raw output a user would have to sort through. This left bulk file transfer as the primary source of SSH traffic. In retrospect I competely overlooked several very important facts. Modern disks have more than enough throughput to saturate gigabit connections - I have NO idea where my "I/O bound" statement came from. Also, SSH encrypted transport can be used for more than just 'put' and 'get'. There are probably more reasons why I'm misguided, but this is enough for now. I apologize and retract that portion of my post; you can safely disregard it as complete lunacy on my behalf. --david
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
dhiggs wrote: If someone can split SSH into multiple threads, it should be just as possible to split it into multiple processes. However, I expect that most high-speed SSH traffic is SCP-/SFTP-based and therefore largely I/O bound, so it hasn't been high on anyone's requirements list. Based on my experience on working in the field of bulk data transfers on high speed networks for the past 6 years I'd have to say that you are, at the least, partly mistaken. However, maybe I'm not really understanding what you are saying. Could you expand on this a bit more? Chris
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On 2/16/08, Marco S Hyman <[EMAIL PROTECTED]> wrote: > librthread seems to be stagnating, waiting upon kernel work that > no one is either interested in or, if interested, has the time > to do. I'm not even sure there is consensus on what kernel > changes are needed. i worked on rthreads for as long as i had interest and time. i don't really have either now. rthreads long ago met my needs, and i don't even need them anymore.
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
> Actually, this is the part of the discussion that interests me. Is > threading > a doomed hope on OpenBSD, a model of utilizing multiple cores which the > developers have zero interest supporting? Has the work on libc_r and the > like > been abandoned completely? libc_r is gone, replaced with libpthread libpthread is changed to fix specific reported problems, most recently (early Feb) a bug fix in dup2 by [EMAIL PROTECTED] librthread seems to be stagnating, waiting upon kernel work that no one is either interested in or, if interested, has the time to do. I'm not even sure there is consensus on what kernel changes are needed. For now libpthread (userland threads) are the most useful for most threaded applications. There are still bugs. libpthread based applications get no benefit from running on multi-core/processor machines. // marc
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 16, 2008 2:45 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > Marco Peereboom wrote: > > The discussion on kernel threads is irrelevant. It is not about having > > some lower level support that will magically make threads not suck. > > Actually, this is the part of the discussion that interests me. Is threading > a doomed hope on OpenBSD, a model of utilizing multiple cores which the > developers have zero interest supporting? Has the work on libc_r and the like > been abandoned completely? Firstly, threading is not doomed, stillborn, or even missing on OpenBSD. The pthread implementation works well enough, and many ports make use of it. Due to the focus on security, threaded code in OpenBSD-provided userland is frowned upon in favor of programming techniques that are easier to secure and control. The threading implementation also has some design limitations that prevent it from performing as well as other OSes. That said, the rthreads replacement is still under development, to my knowledge. The goal of rthreads is to overcome these design limitations and be portable to most/all CPU architectures, as well as maintain or improve the current state of security and functionality. If you are interested, google or someone else should be able to supply more details. If someone can split SSH into multiple threads, it should be just as possible to split it into multiple processes. However, I expect that most high-speed SSH traffic is SCP-/SFTP-based and therefore largely I/O bound, so it hasn't been high on anyone's requirements list. --david
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On 2/16/08 8:48 PM, Matthew Weigel wrote: Marco Peereboom wrote: I can take this only so long before replying... I tried to walk away too. However... The discussion on kernel threads is irrelevant. It is not about having some lower level support that will magically make threads not suck. Actually, this is the part of the discussion that interests me. Is threading a doomed hope on OpenBSD, a model of utilizing multiple cores which the developers have zero interest supporting? Has the work on libc_r and the like been abandoned completely? I got involved in this discussion because of the assertions about why OpenSSH could never be multithreaded, why no software should ever be multithreaded, and so on. Of course parallelizing work using multiple processes is a valid and reasonable approach as well, but it is not the only one, and it is not necessarily the appropriate approach in every case. As far as I see it: = Yes threads can be implemented in a good way, difficult but doable. = Threads running on cpu's that share memory and can influence each other are completely clueless from a security point of view. Since SSH's primary goal is a secure connection and current CPU's have problems with securely splitting threads multi threaded SSH isn't very interesting. +++chefren
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Wed, Feb 13, 2008 at 5:23 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > bofh wrote: > > > 2) lack of tuits to take on a major change for not really any major > > perceived gains: this is not an itch any of the developers currently have. > > ...except the developers (outside OpenBSD) who developed the patch. and they have a patch and are free to use it, as is anyone else who thinks ssh is too slow. i think it's great that pitt made a patch, but i don't expect to see this in tree. and that doesn't bother me. why does it bother you?
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
I can take this only so long before replying... POSIX threads are a complete a total disaster. They are essentially non portable. Anyone who has written threaded code that has to be portable has at one point considered suicide. We are not talking about running on OpenBSD and Linux here; we are talking about running on virtually ALL operating systems. If you honestly believe you can make that work reliably and securely you have never written any portable code of any significant size. The discussion on kernel threads is irrelevant. It is not about having some lower level support that will magically make threads not suck. The threading model is inherently flawed. Again writing a program of any size will teach you this lesson the hard way; especially if you have the honor of being part of a team of 500 people. Nothing is more fun than having short lived threads silently corrupting the heap in 100% non reproducible ways. Let the finger pointing begin... But don't believe me, try it for yourself. Write a program that listens on sockets in a thread and have them signal an external entity when they wake up. Have the external entity acknowledge the signal by returning a different signal. Oh and make that work on Linux 2.0, 2.2, 2.4 & 2.6. When you get that right go port it the BSDs. Oh and the advanced hacker does this without #ifdef. Have fun. All this aside do you honestly think that ssh is slow and needs a "boost"? I don't know about you but decrypting one character as soon as I type it works well enough for me. Benchmarking often is stupid and yields results that are wildly misinterpreted. The performance numbers sure show that it is faster and therefore better right? Quite the opposite. We are talking about one of the most attacked pieces of software in the history of the internet people. Faster != better AT ALL. On Wed, Feb 13, 2008 at 11:46:25PM -0600, Matthew Weigel wrote: > Geoff Steckel wrote: >> This discussion of threads is isomorphic to: >> >>Why do CPUs have MMUs? >>Why don't we write "good guys' timeshared programs" which run >> cooperatively and noninterfering in a shared unprotected >> environment? >> >>Answer: it's too hard to write perfect programs. >> Compartmentalized security simplifies each component and >> drastically simplifies keeping the whole system running well. > > Yes, it is. > >>> It's controlled by the code you write. >> >> That makes it implicit and/or accidental. Data sharing between any parties >> must >> be explicit and declared, otherwise it is either a bug now >> or a bug later when the next maintainer doesn't know. >> Accidental state sharing is by definition a fatal bug. > > How is it qualitatively more implicit than IPC? > they encode state implicitly in the program counters of the threads >>> >>> The hell? >> OK, you're debugging a multithreaded program. Quick, what's the state >> of all the threads now? Not encoded in data? Hmm how long >> is it going to take to discover it. > > Have you ever had to debug a multi-threaded application? You can use a > debugger to check the state just the same. > they are extremely difficult to guard against external conditions such as unexpected incoming data >>> >>> Incoming from where? >> Any useful program does I/O. Otherwise the computer is a space heater. >> So, other than pattern generators with built in data, every useful >> program must take input. Any multithreaded program which I've observed >> has some form of asynchronous inputs. Now, what happens when an unexpected >> datum arrives. > > So it's expected in the sense that "there is a thread looking for input on > this file descriptor." But it's unexpected in the sense of... how? > complete and correct error recovery requires complex and error prone code >>> >>> Welcome to "non-trivial software development." >> >> That's exactly the point. Subtle but extremely important. >> Every effort must be made to keep the software as simple as possible >> so that it is comprehensible. Incomprehensible software is by >> definition a fatal bug. > > The point is that all non-trivial software development has the problem that > complete and correct error recovery requires complex and error-prone code? > >> Any construct which causes complexity disproportionate with the >> total system gain (development & maintenance resources, run time resources, >> etc. vs useful output) must be avoided. > > Finally, something we agree on. Make the code as simple as you can - but > no simpler (to paraphrase Einstein). Look, all of these issues - *all* of > them - are exactly the sorts of problems that the developers need to get a > handle on in order to support the SMP kernel, never mind kernel threads. > If they can't get it right, then they might as well say so, and we can all > go back to uniprocessor machines for anything secure, and run Linux or > FreeBSD for anything performan
Re: Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
And I have a MULTICS o/s in my background. MULTICS was the trail blazing, secure o/s of its day. So all this hard stuff has been conceived and done well before. Granted MULTICS didn't have to suffer the 386, ISA and undisciplined hardware and their artifacts, though a killed-by-management i386 port did exist. The MULTICS code base -- written in PL/1* -- was recently returned to MIT and the public domain from Group Bull, nee Honeywell-Bull, nee Honeywell. It's down-loadable from... http://www.multicians.org/ (*I sooo wanted to point this out in the ada vs. C language for o/s dev. brewhaahaa from a while back.) -Original Message- From: Geoff Steckel <[EMAIL PROTECTED]> To: scott <[EMAIL PROTECTED]> Cc: OpenBSD Subject: Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh Date: Thu, 14 Feb 2008 03:18:33 -0500 Mailer: Thunderbird 2.0.0.6 (X11/20071022) Delivered-To: [EMAIL PROTECTED] Maybe 20 years ago Alliant Computer successfully converted 4.4BSD to MP (8 core processors and 20? peripheral processors). They developed all the locking strategies for all of the I/O and scheduling. The locking was quite fine-grained and seemed to work quite well. It would be well worth while seeing who owns that old code and seeing if a small amount of money or a large amount of persuasion could get it released under an acceptable license. Anybody have contact info for people like Craig Mundie who was one of the principals there at one time? geoff steckel
Removing One Giant Lock, was, Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Maybe 20 years ago Alliant Computer successfully converted 4.4BSD to MP (8 core processors and 20? peripheral processors). They developed all the locking strategies for all of the I/O and scheduling. The locking was quite fine-grained and seemed to work quite well. It would be well worth while seeing who owns that old code and seeing if a small amount of money or a large amount of persuasion could get it released under an acceptable license. Anybody have contact info for people like Craig Mundie who was one of the principals there at one time? geoff steckel
Engineering was Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Matthew Weigel wrote: Geoff Steckel wrote: I'm sure you're extremely bright and can do it. It's not about me. If the OpenBSD developers *can't*, they should just drop any efforts to refine the big SMP lock, any effort to provide kernel threads, any effort to make libc reentrant. And if they do that, then a) threads will have zero performance value, but b) splitting work across threads will have minimal performance value too. To put it another way, if they can deal with it in the kernel, if they can deal with the vastly wider array of security problems that can crop up in device drivers... they can handle multi-threaded application code. That argument is not good engineering. Use maximum effort at the point of maximum return: Develop the kernel SMP and process handling code so that it works very very well. Debug it once. Everything else uses those resources. Don't waste time reinventing the MP wheel in applications code. Don't add work that doesn't need to be done. Don't add work that doesn't add any significant value. Don't add work that adds avoidable risk. That is engineering. It is the art of the possible. What is the best that can be done with the resources available. That is judgment based on centuries of experience in many forms of engineering. Using judgment, measurement, history, and proven heuristics is engineering. Software is not magic. It cannot ignore those rules. If you apply these rules you will get good results. If you do not, your projects will cost more, take longer, and have worse results. It's that simple. It's engineering. Don't take my word for it. Talk to a successful electrical, mechanical, etc. engineer. Don't talk to a "software engineer". There aren't any. Don't talk to a "computer scientist". They know nothing about engineering. (Or , for that matter, human factors) You will learn a lot. If you don't bother, you're a troll. Bye.
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
You're saying, "it's hard." Are you also saying, "abandon all hope because it's hard." Are any of the parties herein indirectly suggesting or asserting that the dragonflybsd effort (visa vi api-centric model) is doomed or ridiculous or wrong minded? Either intrinsically or just because it's hard. Because dragonflybsd's site states, in part, "...What are the potential goals for the next release? Multiprocessor-safe networking without the Big Giant Lock..." Matthew Dillon, the principal, seems to be well down the path with some breaking-the-mold means and ways. dfbsd isn't finished yet. And, yes, it should be subject to a vetting along its way. But is a well principled and by all appearances a using a disciplined approach. And, in fact, I've often thought that a union (co-ompetition, if not co-operation) of obsd and dfbsd would make a whole lot of sense. -Original Message- From: Matthew Weigel <[EMAIL PROTECTED]> To: OpenBSD Subject: Re: Multi-Threaded SSH/SCP made by university of Puttsburgh Date: Thu, 14 Feb 2008 01:25:15 -0600 Mailer: Thunderbird 2.0.0.9 (Windows/20071031) Delivered-To: [EMAIL PROTECTED] Geoff Steckel wrote: > I'm sure you're extremely bright and can do it. It's not about me. If the OpenBSD developers *can't*, they should just drop any efforts to refine the big SMP lock, any effort to provide kernel threads, any effort to make libc reentrant. And if they do that, then a) threads will have zero performance value, but b) splitting work across threads will have minimal performance value too. To put it another way, if they can deal with it in the kernel, if they can deal with the vastly wider array of security problems that can crop up in device drivers... they can handle multi-threaded application code.
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Matthew Weigel wrote: but b) splitting work across threads will have minimal performance value Hrrrm. "across multiple processes". Sorry for the mistake. -- Matthew Weigel hacker unique & idempot.ent
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Geoff Steckel wrote: I'm sure you're extremely bright and can do it. It's not about me. If the OpenBSD developers *can't*, they should just drop any efforts to refine the big SMP lock, any effort to provide kernel threads, any effort to make libc reentrant. And if they do that, then a) threads will have zero performance value, but b) splitting work across threads will have minimal performance value too. To put it another way, if they can deal with it in the kernel, if they can deal with the vastly wider array of security problems that can crop up in device drivers... they can handle multi-threaded application code. -- Matthew Weigel hacker unique & idempot.ent
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Matthew Weigel wrote: This is the same model, albeit with better performance characteristics (assuming kernel threads), that Apache uses, and one of the great things about this model is that it's very easy to tightly control what memory threads share I'm sure you're extremely bright and can do it. In an absolutely perfect world where nobody has been up late the night before and the boss hasn't moved the deadline up three weeks, I'd say that our disagreements did not matter. The real world has to do with deadlines and working with people who are (1) tired (2) unfamiliar with the code in question (3) unfamiliar with the problem (4) pulled from another project (5) under great pressure to do something RIGHT NOW (6) impatient (7) want to show off how smart they are by making something very very complicated (8) trying to move the code from version 1.23A of the spec to 2.01B which has subtle differences or (9) all of the above I've made a great deal of money over my working life disproving that "easy" statement by replacing code which was intended to do as you say with what I consider to be properly isolated separate processes. The code has been shorter and performance has not suffered. And bugs have been simple to find and fix, and relatively not many. geoff steckel
Program complexity control, was: Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Tony Abernethy wrote: Geoff Steckel wrote:>> And yes, error recovery is a very significant part of any non-trivial useful program which does (for instance) network I/O, because the universe of possible errors is large. Error recover? Does anyone ever debug error recover? Is there any way anyone **COULD** debug error recovery? on order of magnitude more complicated and no tools --- predictable. Maybe I'm overly pessimistic, but if so, (try to) prove me wrong. In the general case you're absolutely correct. If you separate errors into local environment problems: sudden memory shortage, disk I/O error, hardware errors Usually these can only be dealt with by exiting as cleanly as possible as quickly as possible internal program error/inconsistencies Exit cleanly after recording as much as possible about the input provoking the problem These are (almost) always symptoms of bugs. input data malformed, inconsistent, or missing peer or server failure (no communication or bad communication) This is the area which can be dealt with. A deceptively simple strategy handles the last case. there must be only one point in the entire program which can block waiting for input from the outside world. that point must have an analysis function which decodes the message, and a scheduler functionality which vectors depending on the decoded message and explicit current state all functionality dispatches from this point and returns to this point without blocking for anything other than local disk I/O and returning a "what was done" code which the scheduler uses to compute the next state. Errors detected are reflected in the "what was done" code. setting explicit error states as needed all errors have explicit new states from an explicit state transition table - NOT from random if-then-elses all input and output messages can be traced for debugging This -can- analyzed and (to a greater or lesser extent) debugged. Some errors cannot be recovered from other than by cleaning up the debris and exiting or starting over. It is possible to be reasonably sure that the program will never hang forever or loop forever (absent internal errors of class 2 above). It is isomorphic with a state driven parser. Indeed, some fairly complex problems can be turned into explicit grammars. Tools like yacc used to generate the required control mechanisms. A great deal of code then doesn't need to be written. Doing this requires an analysis of the problem far deeper than most programmers will do or managers wait for. Implementing requires a discipline most programmers will not endure. It works, and it works well, and it can be checked by peer review and debug traces, and fixes can be tested with message replay. Programs built this way tend not to need repair and repairs tend to be simple, if not necessarily easy. geoff steckel
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Geoff Steckel wrote: This discussion of threads is isomorphic to: Why do CPUs have MMUs? Why don't we write "good guys' timeshared programs" which run cooperatively and noninterfering in a shared unprotected environment? Answer: it's too hard to write perfect programs. Compartmentalized security simplifies each component and drastically simplifies keeping the whole system running well. Yes, it is. It's controlled by the code you write. That makes it implicit and/or accidental. Data sharing between any parties must be explicit and declared, otherwise it is either a bug now or a bug later when the next maintainer doesn't know. Accidental state sharing is by definition a fatal bug. How is it qualitatively more implicit than IPC? they encode state implicitly in the program counters of the threads The hell? OK, you're debugging a multithreaded program. Quick, what's the state of all the threads now? Not encoded in data? Hmm how long is it going to take to discover it. Have you ever had to debug a multi-threaded application? You can use a debugger to check the state just the same. they are extremely difficult to guard against external conditions such as unexpected incoming data Incoming from where? Any useful program does I/O. Otherwise the computer is a space heater. So, other than pattern generators with built in data, every useful program must take input. Any multithreaded program which I've observed has some form of asynchronous inputs. Now, what happens when an unexpected datum arrives. So it's expected in the sense that "there is a thread looking for input on this file descriptor." But it's unexpected in the sense of... how? complete and correct error recovery requires complex and error prone code Welcome to "non-trivial software development." That's exactly the point. Subtle but extremely important. Every effort must be made to keep the software as simple as possible so that it is comprehensible. Incomprehensible software is by definition a fatal bug. The point is that all non-trivial software development has the problem that complete and correct error recovery requires complex and error-prone code? Any construct which causes complexity disproportionate with the total system gain (development & maintenance resources, run time resources, etc. vs useful output) must be avoided. Finally, something we agree on. Make the code as simple as you can - but no simpler (to paraphrase Einstein). Look, all of these issues - *all* of them - are exactly the sorts of problems that the developers need to get a handle on in order to support the SMP kernel, never mind kernel threads. If they can't get it right, then they might as well say so, and we can all go back to uniprocessor machines for anything secure, and run Linux or FreeBSD for anything performance-sensitive. Now, I haven't looked at the exact code written, but based on the descriptions I'm guessing it's a worker thread model where discrete units of work are parceled out, performed, and returned. This has got to be the easiest model of threading to get right, the easiest model to verify, and entirely appropriate for the problem described (processing independent blocks). This is the same model, albeit with better performance characteristics (assuming kernel threads), that Apache uses, and one of the great things about this model is that it's very easy to tightly control what memory threads share. But the response here has been an uninformed kneejerk response to 'threads' and how they're hard to get right, without anyone realizing that this particular model is actually *not* hard to get right. -- Matthew Weigel hacker unique & idempot.ent
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 13, 2008 10:05 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > > Yet, somehow, these disinterested developers with no time managed to > support and assist the newcomer to create something useful and > acceptable. How is this any different? > No idea. Since I'm not a developer, I can be very wrong. I guess we can speculate all we want, if it happens, it happens. If it doesn't, it doesn't. -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk "This officer's men seem to follow him merely out of idle curiosity." -- Sandhurst officer cadet evaluation. "Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted." -- Gene Spafford learn french: http://www.youtube.com/watch?v=j1G-3laJJP0&feature=related
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Geoff Steckel wrote: > This discussion of threads is isomorphic to: > > Why do CPUs have MMUs? > Why don't we write "good guys' timeshared programs" which run >cooperatively and noninterfering in a shared unprotected >environment? > > Answer: it's too hard to write perfect programs. >Compartmentalized security simplifies each component and > drastically simplifies keeping the whole system > running well. > > Matthew Weigel wrote: > > Geoff Steckel wrote: > > > >> IMnsHO, threads should never be used unless absolutely > necessary. They > >> are very bad software practice: > >>they share data and resources in uncontrolled ways > > > > It's controlled by the code you write. > > That makes it implicit and/or accidental. Data sharing > between any parties must > be explicit and declared, otherwise it is either a bug now > or a bug later when the next maintainer doesn't know. > Accidental state sharing is by definition a fatal bug. Probably worse, since you cannot depend on the bug showing itself. There is a big difference between probably working and failproof. > > > >>they encode state implicitly in the program counters of > the threads > > > > The hell? > OK, you're debugging a multithreaded program. Quick, what's the state > of all the threads now? Not encoded in data? Hmm how long > is it going to take to discover it. > > > >>they are extremely difficult to guard against external > conditions > >> such as unexpected incoming data > > > > Incoming from where? Incoming from all the other threads acting on common state, I'd think. In general, code goes from ---everything is consistent messing with internal state --- is not externally consistent ---everything is consistent (again)- If the object is speeding thing up, the system should be ***inconsistent*** almost everywhere (time) Now it is entirely possible to work from inconsistent data and arrive at correct results, certainly **most** of the time. There seems to be some kind of belief that threads will somehow magically take care of whatever problems there are with lots of things going on at the same time. The effort that has gone into making libraries thread-safe wouldindicate otherwise. (Application code is done better than libraries???) > Any useful program does I/O. Otherwise the computer is a space heater. > So, other than pattern generators with built in data, every useful > program must take input. Any multithreaded program which I've observed > has some form of asynchronous inputs. Now, what happens when > an unexpected > datum arrives. > > > >>complete and correct error recovery requires complex and error > >> prone code > > > > Welcome to "non-trivial software development." > > That's exactly the point. Subtle but extremely important. > Every effort must be made to keep the software as simple as possible > so that it is comprehensible. Incomprehensible software is by > definition a fatal bug. > > And yes, error recovery is a very significant part of any non-trivial > useful program which does (for instance) network I/O, because the > universe of possible errors is large. Error recover? Does anyone ever debug error recover? Is there any way anyone **COULD** debug error recovery? on order of magnitude more complicated and no tools --- predictable. Maybe I'm overly pessimistic, but if so, (try to) prove me wrong. > > It's possible to write any program a (possibly infinite) > large number of ways. > Writing a debugged and maintainable program is much harder. > Doing it in a limited time is harder still. > Any construct which causes complexity disproportionate with the > total system gain (development & maintenance resources, run > time resources, > etc. vs useful output) must be avoided. > Threads are almost always such a costly construct. > > geoff steckel
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Not that I've been constrained by CPU in the past few years, but one could easily script a parallel solution with: split -> for loop scp constrain by cores -> cat. A lot less trouble.
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Geoff Steckel wrote: IMnsHO, threads should never be used unless absolutely necessary. They are very bad software practice: they share data and resources in uncontrolled ways It's controlled by the code you write. they encode state implicitly in the program counters of the threads The hell? they are extremely difficult to guard against external conditions such as unexpected incoming data Incoming from where? complete and correct error recovery requires complex and error prone code Welcome to "non-trivial software development." -- Matthew Weigel hacker unique & idempot.ent
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
This discussion of threads is isomorphic to: Why do CPUs have MMUs? Why don't we write "good guys' timeshared programs" which run cooperatively and noninterfering in a shared unprotected environment? Answer: it's too hard to write perfect programs. Compartmentalized security simplifies each component and drastically simplifies keeping the whole system running well. Matthew Weigel wrote: Geoff Steckel wrote: IMnsHO, threads should never be used unless absolutely necessary. They are very bad software practice: they share data and resources in uncontrolled ways It's controlled by the code you write. That makes it implicit and/or accidental. Data sharing between any parties must be explicit and declared, otherwise it is either a bug now or a bug later when the next maintainer doesn't know. Accidental state sharing is by definition a fatal bug. they encode state implicitly in the program counters of the threads The hell? OK, you're debugging a multithreaded program. Quick, what's the state of all the threads now? Not encoded in data? Hmm how long is it going to take to discover it. they are extremely difficult to guard against external conditions such as unexpected incoming data Incoming from where? Any useful program does I/O. Otherwise the computer is a space heater. So, other than pattern generators with built in data, every useful program must take input. Any multithreaded program which I've observed has some form of asynchronous inputs. Now, what happens when an unexpected datum arrives. complete and correct error recovery requires complex and error prone code Welcome to "non-trivial software development." That's exactly the point. Subtle but extremely important. Every effort must be made to keep the software as simple as possible so that it is comprehensible. Incomprehensible software is by definition a fatal bug. And yes, error recovery is a very significant part of any non-trivial useful program which does (for instance) network I/O, because the universe of possible errors is large. It's possible to write any program a (possibly infinite) large number of ways. Writing a debugged and maintainable program is much harder. Doing it in a limited time is harder still. Any construct which causes complexity disproportionate with the total system gain (development & maintenance resources, run time resources, etc. vs useful output) must be avoided. Threads are almost always such a costly construct. geoff steckel
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
bofh wrote: On Feb 13, 2008 6:59 PM, <[EMAIL PROTECTED]> wrote: I'm no coding guru but I know basics about race conditions. what I don't know is why other OSs don't have these problems or how these things can I think the basic issue boils down to the following: 1) lack of complete thread support right now: the developers are still working on finishing up the work. 2) lack of tuits to take on a major change for not really any major perceived gains: this is not an itch any of the developers currently have. 3) it's complicated, and they want to make sure everything works before doing it and probably a little bit of - openbsd is known for its security, but the last security vulnerability was related to openssh. The developers are probably be a little more concerned about taking things in the direction of _well-known_ issues IMnsHO, threads should never be used unless absolutely necessary. They are very bad software practice: they share data and resources in uncontrolled ways they encode state implicitly in the program counters of the threads they are extremely difficult to guard against external conditions such as unexpected incoming data complete and correct error recovery requires complex and error prone code Any one of these is enough to forbid their use in any secure system. geoff steckel
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
bofh wrote: Which comes back to - it's not an itch any one of the developers have, and they don't have the tuits to do it. Funny, seems like a decently large number of drivers (yep, running in the kernel, which means they need to be taken seriously) have been developed by people who, at the outset, did not have CVS commit powers, and who were working on hardware no existing CVS committer had an interest in. Yet, somehow, these disinterested developers with no time managed to support and assist the newcomer to create something useful and acceptable. How is this any different? -- Matthew Weigel hacker [EMAIL PROTECTED]
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 13, 2008 8:23 PM, Matthew Weigel <[EMAIL PROTECTED]> wrote: > bofh wrote: > > > 2) lack of tuits to take on a major change for not really any major > > perceived gains: this is not an itch any of the developers currently > have. > > ...except the developers (outside OpenBSD) who developed the patch. Not sure what you mean. Surely I don't have to point out to you that if this goes in, it'll go into base, which means this will be part of the core of openbsd. Why would _I_ as a user want the developers to take this code lightly? Why would they as the developer want to take the code in lightly? Which comes back to - it's not an itch any one of the developers have, and they don't have the tuits to do it. Or are you suggesting they should take in code any tom, dick or harry wrote? -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk "This officer's men seem to follow him merely out of idle curiosity." -- Sandhurst officer cadet evaluation. "Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted." -- Gene Spafford learn french: http://www.youtube.com/watch?v=j1G-3laJJP0&feature=related
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
bofh wrote: 2) lack of tuits to take on a major change for not really any major perceived gains: this is not an itch any of the developers currently have. ...except the developers (outside OpenBSD) who developed the patch. -- Matthew Weigel hacker [EMAIL PROTECTED]
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 13, 2008 6:59 PM, <[EMAIL PROTECTED]> wrote: > I'm no coding guru but I know basics about race conditions. what I don't > know is why other OSs don't have these problems or how these things can > I think the basic issue boils down to the following: 1) lack of complete thread support right now: the developers are still working on finishing up the work. 2) lack of tuits to take on a major change for not really any major perceived gains: this is not an itch any of the developers currently have. 3) it's complicated, and they want to make sure everything works before doing it and probably a little bit of - openbsd is known for its security, but the last security vulnerability was related to openssh. The developers are probably be a little more concerned about taking things in the direction of _well-known_ issues -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk "This officer's men seem to follow him merely out of idle curiosity." -- Sandhurst officer cadet evaluation. "Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted." -- Gene Spafford learn french: http://www.youtube.com/watch?v=j1G-3laJJP0&feature=related
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
> On 2/13/08 11:17 PM, Benjamin Bennett wrote: > >> I wasn't saying "we can work on security" afterwards. This is something >> that [to our knowledge] has not been worked on previously, and what >> we're providing is code that we consider experimental (due to lack of >> review) to get the ball rolling and get some feedback from others. We >> actually haven't gotten much feedback until today, so we do appreciate >> all of your comments. > > OK, thanks for your quick response and further explanations > > .. > >> Race conditions are not unique to multi-threading. No matter how this >> is implemented, the goal is to do multiple things at once and timing >> will be an issue. > > The security problems OpenBSD people see are not in robustness of this > particular program, how well it works, (without seeing I believe that's OK > and > the code is interesting). The security problems are, for example, that > code > running in one core can access data from the other core and/or influence > the > code running in the other core. > > Very basic security, more or less out of reach of your code. > > OpenBSD puts security first and multi threading on multi cores is thought > of > as a can of worms. Dudes I wont bitch but isn't that a kind of limitation chris cappuccio was talking about on undeadly (kind of like in the meaning od "It's a limitation lets talk about how to resolv it")? I'm no coding guru but I know basics about race conditions. what I don't know is why other OSs don't have these problems or how these things can get resolved (i guess the kernel needs some serious changes..). Race conditions are not new and for sure the future wont be single core CPUs. And Networks don't get slower either.. in 6-8 years we may have 10Gbit onboard. Or a 802.11n standard... (ok, sick joke :D) But seriously: Isn't this "issue" something wich may should get resolved (if possible)?! :-/ I didn't pointed to the patch to show that OpenBSD isn't useable. In fact I hoped some developers may would take a look at the patch and send feedback so it may get included into the mainstream openssh some day (because more speed is always good to have). :) Kind regards, Sebastian
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
chefren wrote: The security problems OpenBSD people see are not in robustness of this particular program, how well it works, (without seeing I believe that's OK and the code is interesting). The security problems are, for example, that code running in one core can access data from the other core and/or influence the code running in the other core. Huh? Last I checked, the OpenBSD developers were working on supporting kernel threads so that multiple threads could run on different cores. It seems ludicrous that they would *also* believe that no one should ever, under any circumstances, *use* that functionality. -- Matthew Weigel hacker [EMAIL PROTECTED]
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On 2/13/08 11:17 PM, Benjamin Bennett wrote: I wasn't saying "we can work on security" afterwards. This is something that [to our knowledge] has not been worked on previously, and what we're providing is code that we consider experimental (due to lack of review) to get the ball rolling and get some feedback from others. We actually haven't gotten much feedback until today, so we do appreciate all of your comments. OK, thanks for your quick response and further explanations .. Race conditions are not unique to multi-threading. No matter how this is implemented, the goal is to do multiple things at once and timing will be an issue. The security problems OpenBSD people see are not in robustness of this particular program, how well it works, (without seeing I believe that's OK and the code is interesting). The security problems are, for example, that code running in one core can access data from the other core and/or influence the code running in the other core. Very basic security, more or less out of reach of your code. OpenBSD puts security first and multi threading on multi cores is thought of as a can of worms. +++chefren
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
chefren wrote: > > > On 2/13/08 10:27 PM, Benjamin Bennett wrote: > >>> It is very unlikely that this patch will be integrated - it adds threads >>> to OpenSSH, which introduce many new security considerations. >> >> Using threads is really just a means to an end, and happened to be the >> most convenient means. If that's a show stopper for you guys, we can >> work on that. > > It's about security, "we can work on that" afterwards is more the > Microsoft way of security than the way OpenBSD people look at it. I wasn't saying "we can work on security" afterwards. This is something that [to our knowledge] has not been worked on previously, and what we're providing is code that we consider experimental (due to lack of review) to get the ball rolling and get some feedback from others. We actually haven't gotten much feedback until today, so we do appreciate all of your comments. > > Are there any other issues? > > No, really!, just prove it's secure and anybody here is happy! > > > I get the idea you don't see the security implications of race > conditions in multi-core CPU's. Race conditions are not unique to multi-threading. No matter how this is implemented, the goal is to do multiple things at once and timing will be an issue. If you have a moment, please do take a look at the code and provide any comments/concerns you have (perhaps off-list). It's not very complicated and I believe it's actually easier to understand/validate in a multi-threaded form than if it were implemented other ways, I could be wrong though. thanks again for your input, --ben [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On 2/13/08 10:27 PM, Benjamin Bennett wrote: It is very unlikely that this patch will be integrated - it adds threads to OpenSSH, which introduce many new security considerations. Using threads is really just a means to an end, and happened to be the most convenient means. If that's a show stopper for you guys, we can work on that. It's about security, "we can work on that" afterwards is more the Microsoft way of security than the way OpenBSD people look at it. > Are there any other issues? No, really!, just prove it's secure and anybody here is happy! I get the idea you don't see the security implications of race conditions in multi-core CPU's. +++chefren
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Dries Schellekens wrote: [snip] A "none" cipher also should like a bad idea. Just use another file transfer protocol in that case. SSH is more than just encryption. The none switch gives the benefit of SSH authentication & MAC for data streams that do not require secrecy. --ben
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Damien Miller wrote: On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: I just wanted to bring it to your attention that the university of Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking at (include it into the base if possible? who knows..). :) Is crypto really a bottleneck for non-HPC users? My Both 1Gbps ethernet and multiple cores are pretty common on cheap systems, and even cheap laptops, these days. Most testing and measurement of this work was done with two workstations (around a couple K $US) 1Gbps, 8 cores. On those systems we did not fully utilize even four of the cores, and we've seen very significant improvement on dual-core systems which are pretty much standard [in terms of new hardware] these days. We've been working other modifications, aiming towards 10Gbps, which I would not argue are more HPC-specific. It is very unlikely that this patch will be integrated - it adds threads to OpenSSH, which introduce many new security considerations. Using threads is really just a means to an end, and happened to be the most convenient means. If that's a show stopper for you guys, we can work on that. Are there any other issues? thanks for the feedback, --ben
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
[EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include it into the base if possible? who knows..). :) fyi, this work is being done at the Pittsburgh Supercomputing Center. The Pittsburgh Supercomputing Center is a joint effort of Carnegie Mellon University and the University of Pittsburgh together with Westinghouse Electric Company. --ben [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Jussi Peltola wrote: On Wed, Feb 13, 2008 at 08:52:42AM -0500, Geoff Steckel wrote: A properly engineered solution would use separate processes and a good interprocess communication system. This is not a suggestion but a question to reduce my stupidity, but wouldn't standard unix pipes and forks/other child processes be just fine for encrypting on more than one (processor) thread? Yes! Simple, well known, and easy. A+! People will complain that it is inefficient. Ask them if they've measured it and how. I would bet a round of the best Belgian beer that the complainers have not measured or have not measured anything significant. geoff steckel
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Wed, Feb 13, 2008 at 08:52:42AM -0500, Geoff Steckel wrote: > A properly engineered solution would use separate processes and a good > interprocess communication system. This is not a suggestion but a question to reduce my stupidity, but wouldn't standard unix pipes and forks/other child processes be just fine for encrypting on more than one (processor) thread? -- Jussi Peltola
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
[EMAIL PROTECTED] wrote: On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: It seams to outperform the normal SSH a lot and this speedup looks kinda impressiv btw. The speedup looks impressive for 1 connection (because different cores are computing AES-CTR), but what happens when multiple SCP/SFTP connections are busy? In this case the normal SSH will run a process for each connection and the multiple cores are automatically active. OpenBSD is not good at multithreading so does it make sense to include this? Well things need to get improved but that's something where the devs come into the game and not me. :) Wasn't there even a discussion about the pro/contra naming such limitations on undeadly recently? Sure OpenBSD "may suck" but that doesn't mean the patch isn't usefull some day. :) A "none" cipher also should like a bad idea. Just use another file transfer protocol in that case. Of course not but it was included into the examples. Threads are inherently difficult to validate or verify for correctness. The uncontrolled sharing of memory and other resources between threads is demonstrably something that is very likely to introduce race conditions and other subtle bugs. Historically, threads were a substitute for multiple processes on systems where process switching was thought to be a performance bottleneck. A properly engineered solution would use separate processes and a good interprocess communication system. If shared memory were to be used for sufficiently demonstrated performance reasons, it would be used with a good locking system. This could be verified and maintained correctly. geoff steckel and maintained.
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
* [EMAIL PROTECTED] <[EMAIL PROTECTED]> [2008-02-13 14:09]: 1: > Threads could get secured too. 2: > A threaded OpenSSL would be an advantage for the userland crypto too 3: > I'm not skilled enough pick one. may I suggest option 3? -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
> On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: > >> It seams to outperform the normal SSH a lot and this speedup looks kinda >> impressiv btw. > > The speedup looks impressive for 1 connection (because different cores > are computing AES-CTR), but what happens when multiple SCP/SFTP > connections are busy? In this case the normal SSH will run a process > for each connection and the multiple cores are automatically active. > OpenBSD is not good at multithreading so does it make sense to include > this? Well things need to get improved but that's something where the devs come into the game and not me. :) Wasn't there even a discussion about the pro/contra naming such limitations on undeadly recently? Sure OpenBSD "may suck" but that doesn't mean the patch isn't usefull some day. :) > A "none" cipher also should like a bad idea. Just use another file > transfer protocol in that case. Of course not but it was included into the examples. Kind regards, Sebastian
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
> On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: > >> I just wanted to bring it to your attention that the university of >> Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth >> looking >> at (include it into the base if possible? who knows..). :) > > Is crypto really a bottleneck for non-HPC users? My AES128 at 0.7GBPS and RC4 at 2.25GBPS. > > It is very unlikely that this patch will be integrated - it adds threads > to OpenSSH, which introduce many new security considerations. > > -d Threads could get secured too. A threaded OpenSSL would be an advantage for the userland crypto too in some ways. I just wanted to bring it to the attention of the developers. I'm not skilled enought to say "it's good" or "it's bad" but if nobody knows about it nobody could determine it either... :-) Kind regards, Sebastian
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Feb 13, 2008 1:11 PM, <[EMAIL PROTECTED]> wrote: > It seams to outperform the normal SSH a lot and this speedup looks kinda > impressiv btw. The speedup looks impressive for 1 connection (because different cores are computing AES-CTR), but what happens when multiple SCP/SFTP connections are busy? In this case the normal SSH will run a process for each connection and the multiple cores are automatically active. OpenBSD is not good at multithreading so does it make sense to include this? A "none" cipher also should like a bad idea. Just use another file transfer protocol in that case. Cheers, Dries
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
Yay race conditions!! I'll venture to say that this will not go in... On Wed, Feb 13, 2008 at 01:11:50PM +0100, [EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include it into the base if possible? who knows..). :) > > It seams to outperform the normal SSH a lot and this speedup looks kinda > impressiv btw. > > http://www.psc.edu/networking/projects/hpn-ssh/ > > > Here's a direct link to the patch: > http://www.psc.edu/networking/projects/hpn-ssh/openssh-4.7p1-hpn13v1.diff.gz > > > Kind regards, > Sebastian
Re: Multi-Threaded SSH/SCP made by university of Puttsburgh
On Wed, 13 Feb 2008, [EMAIL PROTECTED] wrote: > I just wanted to bring it to your attention that the university of > Pittsburgh provides a HPC-Patch for OpenSSH 4.7 wich may is worth looking > at (include it into the base if possible? who knows..). :) Is crypto really a bottleneck for non-HPC users? My
