proper nt-style authentication (reactos, wine, samba tng)
[please note: due to its cross-discipline and cross-project nature, this message is going out to SEVERAL related project mailing lists. when replying, please take EXTRA caution not least because some of these lists are subscriber-only. also, please note: i _would_ post this also to the samba mailing lists but due to the fascist censorship in place since the 15th dec 2004, i am unable to do so. this is their decision and it is their loss. i am not asking you to respect that decision i am simply making you aware of it.] hi, out the woodwork i pop - not necessarily ready to chew anything because i know just how much work's involved, but what i did want to do was say hi, i'm still here and do a brain-dump of how authentication ideally needs to be implemented in reactos. at 2,600 words and 16k, this message is quite long and so i have placed a copy at http://lkcl.net/software/reports/reactos-auth.txt, just in case it doesn't make it past various mailing-list limits (i'll find out in a couple of minutes... :) it breaks down into a number of sections: 1) i describe the timescales and ways to cut them, along with some warnings and stuff. 2) amonst other stuff i outline some background as to why i am posting this to so many lists. 3) i outline a project plan and the dependencies and optional steps, 4) i describe a recommended implementation roadmap starting with the minimum requirements, and expand on some technical info and references i found, which would help with some of the nice-to-have steps. so. first. please do not be scared by how much work is involved, and how much code there is. it all hinges one ONE function and that one function... you are _not_ going to believe how much code that one function drags in, kicking and screaming, behind it. please also please i beg you DO NOT consider going bleedin 'ellfire, that's so much frakkin work we can't POSSIBLY do it the way you describe, we MUST do it our own way, starting with what we know, love and have already started tackling, and can't possibly back out of what we've already done. should you choose to exercise the NIH option, i frakkin guarantee you that you will waste about five to eight years of your (collective) lives reinventing the wheel. The Plan outlined here will shave that down to about 12 to 18 months, utilising some SIX HUNDRED THOUSAND lines of pre-existing code, and later in this message i also outline a prioritisation of the necessary work to cut down the time to maybe about ... mmm... three or so months, by leaving out some of the nice-to-have stuff. actually... _all_ of the nice-to-have stuff :) that will at least GetItWorking(tm) and the rest of the bits can be considered at leisure once people go god that's awful, we can't possibly leave it like that and hopefully hopefully things will actually progress from there. remember - please: this stuff is sufficiently complicated such that you really can't afford the niceties such as It Must Be Perfect (tm) before it can be accepted. i've seen that shit before, and it's a baad mxxxer path to go down, especially with such complex and heavily interdependant reverse-engineering projects as reactos, wine, samba and freedce. anyway. fyi - before i begin, i should mention a couple of things: 1) this message also goes out the the apache devel mailing list (specifically the APR one) because the NT style authentication thing is the sort of thing that really _should_ be in a (special?) APR util library, along with NT Named Pipes - see http://en.wikipedia.org/wiki/NamedPipes one of the reasons why NT-style NamedPipes is _not_ in an APR util is because it is believed that NT-style NamedPipes do not fit the least common denominator. by having the infrastructure outlined below, it is possible to move things up one level to the point where unix _has_ that denominononmination. also, if APR still has support for that god-awful program-running program called Win95, it would, with not much extra effort, be possible to port some reactos components to Win95 (!) such that ThePlan outlined here would make Win95 have proper NT-style authentication (!) now there's a horrible thought that will give MS and free software people nightmares both: upgrading Win95 by installing free software components at it. muhahahahah ahem. 2) i have filled in a number of pages on wikipedia.org. see http://en.wikipedia.org/wiki/User:lkcl. please take the time to review these pages, PLEASE help me with my totally biased POV (point-of-view) comments, by either editing the pages direct or by adding comments on the talk pages as appropriate. or alternatively dumping me in the nearest pond if ever you meet me. wikipedia is supposed to be encyclopedia-ic and some of my comments are anything but that. help!!! 3) please due to the quite broad distribution of this message across multiple mailing
Re: random number generator
Date: Sat, 29 Dec 2001 15:47:24 -0800 To: Ben Laurie [EMAIL PROTECTED], Justin Erenkrantz [EMAIL PROTECTED] From: Marc M. Adkins [EMAIL PROTECTED] Cc: dev@apr.apache.org Subject: RE: random number generation Message-ID: [EMAIL PROTECTED] This is almost the entire problem with random numbers - what is good enough? samba's random number generator uses /dev/[u]random whichever is available; md4 on all files in /tmp, /dev and other locations where large numbers of files are likely to be. parts of this are then supplied as input to RC4, which is very good for generating streams of random numbers, fast. the algorithm used, as it stands, however, was predictable, as demonstrated by pete chown. it was not, however, fixed before then, because pete didn't get round to doing the demonstration. if you talk to jeremy allison and also to pete chown you will be able to find out if the issue was fixed, and if so, you will have a good algorithm to [BSD-re]implement. lkcl
TNG services and FreeDCE development
development time estimates for TNG / dcerpc.net project tasks dcethreads: portability rewrite 50? (to use Apache Runtime Library for preference) pthreads rewrite 200? (2nd preference: cut out dcethreads) libapr_ntemu_util: APR NamedPipe emulation 50? NT Security emulation 50? netlogond: idl file 90 server76 client / test 76 samrd: idl file [mostly done]24 test server 120 client / test120 lsarpcd: idl file 120 server 120 client / test 80 ntlmssp and integration of ntlmssp into freedce ntlmsspauth: server [mostly done] 100 client [just beginning] 100 credential cache 16 auth (server-side)24 auth (client-side)24 nt named pipe emulation layer: tng server-side [done] 100 (it's basically done) tng client-side [done] 100 (ditto) freedce srv-side 80 (25% complete) freedce client-side 80 (25% complete) freedce auth integration 36 (related to rpc_binding_set_auth_info) i've probably missed some things out, here. times are in hours. if anyone would like to assist with any of the above development, your time and input would be greatly appreciated: please register on dcerpc.net, place an ssh public key in your user-profile and let me know which of the above projects and tasks you would like to help with. minimum requirements are to have linux on, preferably an x86 or other intel-byte-order machine (at the moment: later on we will need to test against reverse-intel-byte-order); ssh, cvs, gcc and other development tools; experience or enthusiasm with c development; lots of enthusiasm; internet connection not blocking cvs port access. thanks, luke
Re: [xad] TNG services and FreeDCE development
On Mon, Oct 08, 2001 at 06:09:32PM -0700, Howard Chu wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Luke Kenneth Casson Leighton development time estimates for TNG / dcerpc.net project tasks dcethreads: portability rewrite 50? (to use Apache Runtime Library for preference) pthreads rewrite 200? (2nd preference: cut out dcethreads) Can you give me a bit more detail on this? In particular, I need to decide whether to try to port freedce to Solaris or get Sun's commercial DCE implementation instead, and try to make it all play together. What is the issue with the current dcethreads code? (I haven't looked at the freedce source tree yet.) the current dcethreads library is an addition to the suite of POSIX draft 4 emulation libraries, and the original focus of dcethreads was to add to that suite w.r.t linux, which was _not_ included, as linux wasn't _around_ when OSF started DCE :) so you may find that you don't need dcethreads, you may be able to #define SOLARIS and compile from there, and not even _use_ dcethreads. you'll have to check the code. dcethreads is a user-space emulation library on top of sigsetjmp and siglongjump. it is horrible, but it works. personally, as i am doing this development without funding at present, and my skills are on NT / unix interoperability, as long as i have _a_ library, and _a_ platform on which that library works, i am not the best person to ask w.r.t. getting other OSes to work: i'll focus on the common bits in my area of expertise. i do, however, know of, and have been gently encouraging, some other people to develop, relace or rewrite dcethreads, plus there _is_ a possibility to do away with dcethreads and use modern posix threads instead. the disadvantage of that is that we lose thread cancellation, which POSIX draft 4, and therefore dcethreads, provides. given that most OSes don't support thread cancellation without some sort of problems such as instability and memory loss in the kernel (i.e. solaris), that's not such a big deal. luke.
Re: Shared memory and IPC
hiya jim,
thanks for the insight! prior experience always appreciated.
luke
On Mon, Sep 10, 2001 at 08:43:49AM -0400, Jim McDonough wrote:
My advice is to avoid MAP_FIXED at all costs.
absolutely love to. however, it's a speed improvement of
a potential two or maybe even three orders of magnitude,
and so cannot be ignored.
.
Consider the following code snippit
struct {
int this;
long that;
char *theOther;
...
char placeTheOtherPoints[MAXBUF];
} thingie;
base = mmap(addr, len, PROT_READ|PROT_WRITE, MAP_SHARED, 42, 0)
...
myThat = base-that;
...
base-theOther = newOther - base;
The that value is just at an offset into the mmap'd
structure, so getting it's value is trivial: most
compilers turn this into a ,base,offset
the Other value is trickier: it's a pointer to an address
in the mmapped area, so you have to add the base before
dereferencing it and subtract the base before updating it.
I think Dave's got a good idea here. I've worked on a product that did
exactly this, and the overhead of adding the offsets was trivial (on AIX).
The product was originally on AIX only, where the first mmap is guaranteed
(if you're not using the Large Program model, which you control at build
time, so we had control) to be at 0x3000. We used pointers with the
addresses in that mmap'ed region. Then, to port to other platforms, we had
to start using offsets instead. The performance drop on AIX was not
measurable. The data transfer rates we got between processes was huge
(basically was throttled by the process that put it out on the net saying
stop, I can't take any more). Yes, strictly speaking, there had to be
more overhead, but we just couldn't see it.
The only thing that helped was going from mmap() to shmat(), because we
didn't need anything to stick around in a file, and the mmap'ed file
changes were being flushed to disk every minute by the sync daemon, and
flushing 32 MB to disk doesn't happen instantaneously. Going to shmat()
and backing it in page space prevented the sync and the hiccup every
minute. But that's a different issue altogether.
Jim McDonough
IBM Linux Technology Center
6 Minuteman Drive
Scarborough, ME 04074
USA
[EMAIL PROTECTED]
Phone: (207) 885-5565
IBM tie-line: 776-9984
Re: Shared memory and IPC
On Fri, Sep 07, 2001 at 03:18:55PM -0400, Green, Paul wrote: hi paul, thanks for responding. My advice is to avoid MAP_FIXED at all costs. absolutely love to. however, it's a speed improvement of a potential two or maybe even three orders of magnitude, and so cannot be ignored. and even then, i don't think it can be ignored. this is for an implementation of DCOM (which, if you remove the D it becomes COM, too :) it is necessary to describe data structures and functions using dce/rpc IDL. it is then necessary to pass these data structures, via the function calls, between processes, whether local or remote. the remote bit we have covered. ncacn_ip_tcp and ncadg_ip_udp (ConnectioN, DataGram, tcp/ip, udp). there, it doesn't matter if the passing of data structures is slow, because the network becomes the bottleneck, with order 1ms or greater response time. the local bit, suddenly things like packing a 1mb data structure into a buffer, only to pass it over with a memcpy or five over to another program on the same host, become significant. so basically, although i don't _think_ it's essential for the first implementation (wez will be able to correct me here), it's not something that can be left as-is. the implementation has, if you to are pass over data structures between programs, presumably in exactly the same way that those DB apps are doing, you must preserve pointers / memory segment offsets. and that means using MMAP_FIXED. if anyone has any better ideas, love to hear them. all best, luke
Re: Shared memory and IPC
wez, as and when you get to this, you're probably going to need some shared-memory management ... 'stuff'. for example, as you mention, a well-known shared list of structures that describe the locations of other shared segments. that being the case, it would be extremely useful to work with sander, david, ryan, greg and justin from APR to produce an sms_shmem, for use at least by APR and the dcom project. once an sms_shmem manager is available, it can be used as the base-level on which to build the rest. luke On Fri, Sep 07, 2001 at 10:06:16AM +0100, Wez Furlong wrote: On 06/09/01, John E. Malmberg [EMAIL PROTECTED] wrote: Memory allocated using the COM IMalloc must be passable directly to another process (ie: appear at the same address), so I think we might need a COM-wide shared memory context of some kind. Is there some way that relative pointers can be used inside the shared memory, and the functions that access the pointers know what the base address is? Yes (as others have discussed), but for the purposes of the COM MEMCTX_SHARED IMalloc, if we implement it at all, it must work such that the allocated pointer works in all address spaces (because it could be passed anywhere!). Now, if the person using it is stupid enough to put a pointer to somewhere that's not accessible inside something in that shared mem, their program deserves to get a sigsegv or sigbus! It's looking like a right PITA, so I'm moving it down on my todo list. However, using shared memory to speed up marshalling seems like a good way to go (performance wise); it's staying at about the same place on my todo :) --Wez. ___ dcom-dev mailing list [EMAIL PROTECTED] http://lists.dcerpc.net/mailman/listinfo/dcom-dev
Re: Shared memory and IPC
On Fri, Sep 07, 2001 at 05:17:01AM -0500, Peter Samuelson wrote: [Wez Furlong] So there is no nice-n-easy syscall then? Even a non-portable call would be better than parsing /proc/self/maps. Don't look at me, I'm no IPC expert! (So, that out of the way...) I am not sure what you are wanting beyond mmap(MAP_FIXED). That is indeed a nice-n-easy syscall -- it will either succeed or fail. If it succeeds in one process but fails in another, just munmap() it again. The real question is, what *do* you do on failure? One possible protocol: the master process does mmap() without MAP_FIXED, reports the address to the others via some sort of existing IPC, the rest try with MAP_FIXED, report back success or failure. If anyone fails, they all unmap *except* the master, which repeats the process with another mmap() [again without FIXED] ... for a specified number of tries. Upon success, or upon giving up, the master munmaps all failed segments, if any. see, this is the bit that i was hoping that noone would suggest even be attempted, even though it's technically a correct solution :) now we have a good reason to justify why MS reserves 2gb of the physical memory address space in w95... the only issue with mmap() without MAP_FIXED is that you won't get a real physical address back, you'll get a virtual address back. if this is the case, the algorithm is sound, but the master would still need to use mmap(MAP_FIXED) with ever-increasing addresses on segment boundaries, and it could also unmap them on failure. luke
Re: Shared memory and IPC
On Fri, Sep 07, 2001 at 11:50:40AM +0100, Wez Furlong wrote: On 07/09/01, Luke Kenneth Casson Leighton [EMAIL PROTECTED] wrote: The real question is, what *do* you do on failure? One possible protocol: the master process does mmap() without MAP_FIXED, reports the address to the others via some sort of existing IPC, the rest try with MAP_FIXED, report back success or failure. If anyone fails, they all unmap *except* the master, which repeats the process with another mmap() [again without FIXED] ... for a specified number of tries. Upon success, or upon giving up, the master munmaps all failed segments, if any. Yes, but... (see below) see, this is the bit that i was hoping that noone would suggest even be attempted, even though it's technically a correct solution :) Sounds like a lot of effort on the part of all processes involved. What if a new process is introduced after the others have negotiated the address space that cannot map those address(es)?? you know, this _does_ tend to suggest that the protocol implemented by transport enumeration behind the epmapper would be useful. an additional system call which gives an array of memory ranges that will succeed, or is likely to succeed, if mmap is called immediately with MMAP_FIXED. ... or is that overkill? luke
Re: Shared memory and IPC
the only tricky bit is, and this is relevant i should imagine to doing shared, complex data structures in samba, an also a possible solution to some of the issues in APR, how do you guarantee that multiple processes - no, _all_ processes - get access to the same fixed address memory segment / segments? i wasn't aware that mmap could do fixed memory segments, otherwise i would have suggested (on apr dev) an sms_shmem that used shared memory. [fyi: sms - Smart Memory system. it's a wrapper around memory management that presents the same API to mlock()ed, mmap()ed, kernel-page-mapped, etc. memory, that has support for either malloc/realloc/free style or pool-style such as Trivial Alloc and APR pools management]. a few months back or so, doing an sms_shmem was considered [with a view to then presenting that via an apr_pool_t interface] however, the trouble with shared memory is [was!] that the offsets in the different processes for the shared memory segment are different. so, placing a linked list or any data structures with pointers in it is a waste of time. now, i know that in samba there are shmem-start-relative linked lists to data structures *without* pointers in them., so i know this can be done. and doing a linked list in MAP_FIXED segments would be an awful lot easier, in fact it would be trivial. the only thing, then, that is of concern, is how to link several segments together: it's not a good idea to mmap a massive segment of memory, have to do several smaller ones. what happens if you run out and have to get some more MMAP_FIXED segments? how do you guarantee that all processes will all successfully mmap the same shared memory segment at the same address? how do you communicate the _need_ to get more MMAPED_FIXED segments to all processes? :) [please, someone tell me if i am explaining this clearly enough] or, should the first implementation avoid this by specifying the required segment size at apr_initialise() time or as a parameter to apr_sms_shmem_create()? luke On Thu, Sep 06, 2001 at 12:37:27PM -0700, Jeremy Allison wrote: Luke Kenneth Casson Leighton wrote: is it possible to get memory allocated in one process to be shared with another process *at the same address* on unix, os/2, nt, vms etc? mmap has the MAP_FIXED flag that forces the mmap to fail if it can't be done at the given address. You could use a different IPC method to pass that address to other processes. This is UNIX/POSIX only though (as far as I know). Jeremy. ___ dcom-dev mailing list [EMAIL PROTECTED] http://lists.dcerpc.net/mailman/listinfo/dcom-dev
Re: Interposing LSA functions (was: RE: [pamldap] PAM for Windows NT/2000)
On Wed, Aug 29, 2001 at 08:45:39PM +1000, Luke Howard wrote: The URL: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/security/hh/secpack/customsecfunctions_9js1.asp is worth a look as it describes the API used to retrieve the authorization data for a user and create an LSA security token from that data. interesting. so it might be possible to write a setuid() for nt and for cygwin, after all. i'm cc'ing this to apr dev: someone there might be interested in investigating and completing the apr user-related functions. luke
[Nicolas.Williams@ubsw.com: Re: Interposing LSA functions (was: RE: [pamldap] PAM for Windows NT/2000)]
- Forwarded message from Nicolas Williams [EMAIL PROTECTED] - Delivered-To: [EMAIL PROTECTED] Date: Wed, 29 Aug 2001 10:26:47 -0400 From: Nicolas Williams [EMAIL PROTECTED] To: Luke Kenneth Casson Leighton [EMAIL PROTECTED], Luke Howard [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: Interposing LSA functions (was: RE: [pamldap] PAM for Windows NT/2000) X-Mailer: Mutt 0.93.2i In-Reply-To: [EMAIL PROTECTED]; from Luke Kenneth Casson Leighton on Wed, Aug 29, 2001 at 04:10:34PM +0200 X-WDR-Disclaimer: Version $Revision: 1.13 $ On Wed, Aug 29, 2001 at 04:10:34PM +0200, Luke Kenneth Casson Leighton wrote: On Wed, Aug 29, 2001 at 08:45:39PM +1000, Luke Howard wrote: The URL: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/security/hh/secpack/customsecfunctions_9js1.asp is worth a look as it describes the API used to retrieve the authorization data for a user and create an LSA security token from that data. interesting. so it might be possible to write a setuid() for nt and for cygwin, after all. It exists, in a way, you just have to have the client's AP-REQ! Actually, here's how it works: - if you're a system service (a daemon running as root) you can become (impersonate) any local user - if you're any kind of service, even non-priviledged, you can become (impersonate) any domain user who authenticates to you using GSS-API (SSPI). This works because: - a) the service passes the GSS tokens to the LSA, which - b) validates the tokens (and produces tokens) using any secret keys (which the service need not have access to) and - c) obtains the client's profile which the LSA then uses to provide a handle to the service which the service can then use to *locally* impersonate the client You can't adapt that to the setuid() model -- that's because the setuid model sucks. Firstly because the UIDs are a flat namespace (so no domain users) and secondly because the above model does not fit. I'd rather adapt the win2k model to Unix. i'm cc'ing this to apr dev: someone there might be interested in investigating and completing the apr user-related functions. luke Cheers, Nico -- . Visit our website at http://www.ubswarburg.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. - End forwarded message -
Re: Any interest in a apr_xml_parse_file ?
yes please on the xml parsing! On Mon, Aug 06, 2001 at 05:44:12PM -0700, Greg Stein wrote: Note that the XML Specification states that once a parse error is found, then the parser should(?) terminate; recovery is not required, nor is it even encouraged. (btw, I don't believe you can restart Expat) xml is deliberately very specific. errors, therefore, are not tolerated. what happens if someone decides to send a bad TCP packet? it's rejected. same thing. luke
SID library (was: userid is _NOT_ this...)
On Fri, Jul 27, 2001 at 12:56:34AM -0500, William A. Rowe, Jr. wrote: The code to format a sid is already in apr/user/win32/userinfo.c and I won't object to it becoming public (with the usual name.) i have been meaning to create a SID-manipulating library, if anyone's interested. we have to have SID construct / destruct code for TNG, and it's the sort of self-contained thing that could be libratised v. easily. luke
Re: proposal: apr_get_username()
On Thu, Jul 26, 2001 at 04:56:24PM -0400, Cliff Woolley wrote: If it's okay with people, I'd like to add apr_get_username(). I can implement the Unix part by calling getuid(). But somebody else would need to write the win32 equivalent. Thoughts? +1... Luke has asked for this as well. yesplease. I have no idea how it would work on Win32, but there's got to be some way or another to do it... recommend taking a look at cygwin32's sourcecode, they have to do it in there (i don't believe it's fake). the only thing that's almost impossible is setuid()-Win32, because there isn't one. luke
Re: [PATCH] Allow unrelated SMSes to reset
It seems to me that there are so many ways of doing this that we're already talking about adding more and more complexity when we may not need to. when it comes to the sms api, i'm quite happy to ward off any recommended changes that make it more complex, don't worry. think of this as a bit like craftsmanship. yes, there are loads of ways of doing things: only very few of them are worthwhile doing that you can be proud of, because [collectively] we have the experience and knowledge to pick the best. luke
sms locking revisited (was: [STATUS] (apr) Wed Jul 18 23:45:12 EDT 2001)
On Wed, Jul 18, 2001 at 11:45:12PM -0400, Rodent of Unusual Size wrote: APACHE PORTABLE RUNTIME (APR) LIBRARY STATUS: -*-text-*- Last modified at [$Date: 2001/07/17 05:43:53 $] APR Stackable Memory Code = This is just a small list of things yet to be done, or things that we may want/need to consider. - add a shared memory module. - locking needs to be addressed. The scope of the locks needs to be defined and it's likely we'll need some way of varying the scope when locking. just a reminder [yes, i'm being stubborn and stupid, here.] i suggested that the SMS locking api include the means to lock regions [apr_sms_lock(sms, void *area, size_t size) where area=NULL - global locking and size=0 - use the entire size of the allocated region you are responsible for. of course, apr_sms_general() or any other non-tracking sms says 'get lost' to this :) ] it was suggested at the time that this locking api was proposed that this should be a 'mandatory' requirement - that _all_ SMS instances must support region locking, by emulating it if the underlying SMS doesn't support it. [i didn't and don't think this is a good idea.] i suggested at the time that it would be better to 'negotiate' an SMS by having a bitfield, such that you can check whether the SMS supports the capabilities you might need: - 'i support region locking' - 'i support global locking' - 'i am threadsafe' - 'i support free' - 'i support realloc' - 'i support destroy' - 'i support mmap' - 'i support mlock' this will actually become quite important for the shared memory module interface. why? because it will be possible to add a 'helper' API to obtain handles to all SMSes that support a particular feature-set, expressed by having an and and a not bitmask pair. luke
Re: [PATCH] Allow unrelated SMSes to reset
On Wed, Jul 18, 2001 at 05:37:33PM +0200, Sander Striker wrote: yeah, well it seems to me that this would be a really smart thing to do _anyway_. throw an assert if someone tries to register the same 'object' twice, not just a child-sms. is that possible? Possible, yes :) mmm i don't like the sound of that. possible, yes. [terry pratchett readers will be familiar with this] hm, sounds like you're a dwarf who's just been asked to Botch an engineering job and Get It Done By Wednesday... ...what's the catch? :) if you have a file object, how can you check it's not the same? by using the cleanup function's address _and_ the void* as the 'key'? does that work? Yes (but you need to add the type too). That is how the matching works in sms cleanups already. We just don't do a check for uniqueness in the registration function yet. okay, okay: forgot about that. yep! about the only exception is the null cleanup [is that used in sms?] We don't have a null cleanup in sms. It is not needed here. Although I have come across a situation which could screw up the entire idea behind the sms cleanups. It was in http_log.c I believe... :( [I'll expand on that later] separate subject? lukes
Re: [PATCH] Allow unrelated SMSes to reset
On Wed, Jul 18, 2001 at 05:30:17PM +0200, Sander Striker wrote: Yup. That's the catch. It'd probably need to be a bit more sophisticated than what I've posted OR make the apr_sms_reset a bit more robust (i.e. handle SMSes that have already been cleaned up). I'm leaning towards making the apr_sms_reset more robust. -- justin I wonder how you plan on doing that. Furthermore, the method of registering the cleanup with an unrelated sms(B) will not work if the registering sms(A) is destroyed before the cleanup is run in B. In other words, if A is destroyed the cleanup is still registered in B. If B is reset/destroyed: boom. Or maybe you were planning to unregister the function when the thread exits (in which case it would probably work). manual unregistering is quite common - it's used a lot in mod_virgule.
Re: SMS Factory Re: cvs commit: apr/memory/unix apr_sms_threads.cMakefile.in
On Fri, Jul 13, 2001 at 02:53:59AM +0200, Sander Striker wrote: would it be usefull to have some kind of 'sms_factory' which you could call either with a identifier, or better yet, a set of capabilities it requires from the SMS (manysmallallocs, realeaseatend ) or ( bigallocs,tracking) which could then give you the best match for your environment? Well, at that point you already know your environment, so we might as well create a small tool asking questions about environments. Something like so: Do you have lots of small allocations [y/n]? y Do you free your malloced blocks [y/n]? n ... The best probable match is: type Where type is std, tracking, trivial, blocks, etc. Then you just put in a call to apr_sms_type_create on the spot. Each type also may have different parameters you can give it at create and a factory would make that more difficult. this fits in well with having a bit-field in every SMS [a new function]: what features do you support? see other-post, 30mins ago. luke
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
On Tue, Jul 17, 2001 at 10:12:35AM -0700, Aaron Bannert wrote: - Optionally creating a child-pool for each thread. This provides us two things in my mind: 1) Allowing the application to choose an SMS for this thread-pool (which could very well be a special SMS created just for this purpose). for example, you may wish to create a special 'security' thread. it uses an apr_sms_mlock_t instance. this special thread handles all of your security, keeping private keys and plain-text user passwords in-memory. luke
Re: [PATCH] Allow unrelated SMSes to reset
Yup. That's the catch. It'd probably need to be a bit more sophisticated than what I've posted OR make the apr_sms_reset a bit more robust (i.e. handle SMSes that have already been cleaned up). I'm leaning towards making the apr_sms_reset more robust. -- justin yeah, well it seems to me that this would be a really smart thing to do _anyway_. throw an assert if someone tries to register the same 'object' twice, not just a child-sms. is that possible? if you have a file object, how can you check it's not the same? ... by using the cleanup function's address _and_ the void* as the 'key'? does that work? that way, if it's a file object, then the file-cleanup-function pointer is the same and oh, whoops, the file object is the same too ASSERT. same for handles, sockets... ... about the only exception is the null cleanup [is that used in sms?] luke
Re: DCEthreads - cancellation / mutexes is possible
On Tue, Jul 17, 2001 at 06:49:20AM -0700, [EMAIL PROTECTED] wrote: well, in amongst all this about threads etc. i just wanted to let you know that i happened across the context_app example in the dce 1.22 codebase. it is an example client and server that maintains a context - a persistent connection across multiple remote function calls. this implies that if the connection dies without the client destroying the context, then the server must do it _for_ you [by calling context_name_t_rundown which you *have* to provide: it calls it on every outstanding context of type 'context_name_t' it is an auto-generated function.] and that means killing a thread. and _that_ implies thread cancellation and cleanups. and it works. therefore, i conclude that the dce/rpc codebase has successfully implemented thread cancellation in their POSIX/Draft4 thread library. That is quite a leap in logic there. okay, so i'm a rutherford instead of an einstein [rutherford was known for knowing what the answer was, and then cooking the books to get the missing steps between the question and the correct answer :) :) iow, i know that dce threads have cancellation. like sander says, this is pretty hard-core code used in things like the National Insurance Database, military and government-mandated applications etc., so... I can think of multiple ways that the thread itself could timeout and kill itself. You also haven't mentioned how much memory was leaked by killing that thread. I am not saying you are wrong, just that with the information you provided above, I am not convinced that they have actually successfully implemented async killing of threads. okay. i tried a little test. just before freeing the context, i instead call exit(0). i presume that this is a Bad Thing To Do? there is even a client thread so i presume doing this sort of thing is Not Good. then i do this: while 1; do ./context_client 'test message'; done; now, this generates about... difficult to tell, the screen scrolls so quickly... 50 clients per second, at a guess? i left it for about a minute, and then examined the server apps [there are 10 threads forked off]. the memory usage did NOT change / go up. ... just tried it again [2 mins]: *giggle*. oh dearie me: The Open Group is going to be so pissed :) memory usage went up from 1584 to 1608 and now the server isn't accepting connections any more. teehee :) [... ah, no: it's recovered again. hey, this is just plain odd. okay, could just be i'm overloading my computer :) ] yep: been running on-and-off for several minutes: ps auxw shows no obvious memory leaks. if you know a better way than ps axuw please let me know! luke
Re: Pools in threads WAS Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
There is *zero* benefit to having any relationship between the thread's memory pool and the parent's memory pool. You can't cleanup the thread from the parent anyway, so trust the thread to cleanup itself (as well as its pool and any memory allocations). I fail to see the problem here. Sever the imagined relationship. The code becomes simpler, too. -- justin col. problem-solving by conventions. follow these rules, everything's okay.
Re: cvs commit: apr/memory/unix apr_pools.c
so you can... make your _own_ cleanup type? say... a thread-ending-cleanup type? you can you do per-type cleanupping? then it will call the cleanups and only free [or register for reuse in the free list, depends on the sms instance implementation] areas of memory registered by that type? what happens when you destroy the sms? is _everything_ cleanedup then? luke On Sun, Jul 15, 2001 at 12:51:11AM +0200, Sander Striker wrote: Hi, rbb 01/07/14 15:31:38 Modified:include apr_pools.h memory/unix apr_pools.c Log: Add a new function to be able to cancel a child cleanup. This is necessary for some other work I am doing. I wanted to open up the discussion on cleanups, but seeing this commit triggers me to do it now. In SMS we tried to make cleanups more generic. It isn't perfect yet, but there is a good starting point. What we basically want to do is register cleanups by type. This superseeds the concept of child and general cleanups, which are just types. Everything registered as general is cleaned up at reset and/or destroy. Can you please take a quick look at the sms cleanups and tell me what you think (as a starting point). Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote: APR threads, when created, would now take an additional parameter that is the mechanism (an sms implementation) by which it should create child pools. As it is now, the pool that is passed in to apr_thread_create() serves as the parent pool for the new thread-specific sms. If this parameter were null, the apr_thread_create() function would not create a sub-pool or register cleanup routines (which satisfies my requirements). if at all possible, the behaviour should match as closely as possible the existing situation, when this parameter is NULL, even if the current situation has bugs / is not very nice. this is mainly so that people do not object to having the behaviour of existing code disrupted. luke
Re: crypt() for WIN Patch
can i recommend that people read http://advogato.org/articles/302.html okay, forget that: summary-time. basically, adding encyption for the purposes of 'authentication and authorisation' is legal. AS LONG AS it does NOT have a 'generic' interface that would allow it to be used by an inexperienced programmer for the purposes of encrypting user data. and to this end, samba has an implementation of NTLM authentication and NTLMSSP encryption, and password changing, that cannot be used for 'generic data' encryption purposes, and it IS legal. luke On Sat, Jul 14, 2001 at 11:05:12AM -0700, Justin Erenkrantz wrote: On Sat, Jul 14, 2001 at 12:53:09PM -0500, William A. Rowe, Jr. wrote: You can tell I read oldest-newest :-) I'm pretty sure I have an older ufc-crypt under a freebsd license, I'll have to look about. ITMT, I really believe that the SSL library (which nearly everyone would link in) is probably a better choice, since OpenSSL has the des methods. Does that sound 'saner' to everyone ... using OpenSSL which we will be using within Apache (and other foos: daemons or clients) anyways? I think we talked about that before and the crypto restrictions on OpenSSL scared some people. -- justin
Re: Pools in threads
In order to provide a win against the current pool code in a threaded MPM, we *need* to have thread-specific SMS that have no locks or association to anything other than a simple unlocked (from APR's perspective) malloc/free (aka std) SMS. -- justin okay. well... uhmmm... this is going to sound odd. i'm not even sure if it will help, because i am a bit out of my depth in understanding the problem. how about a 'pass-through' sms for threads? all it does is to create an apr_sms_thread_passthrough() is: memcpy(thr_pthru, someothersms_api); then OVERRIDE the create function, calling the *someothersms*-create. you then do your 'thread-specific' stuff that you need to do, e.g. locking _whatever_, i really don't know, as a wrapper around the someothersms. in other words, via this technique, you can turn a non-thread-capable SMS into a thread-safe one. ... does that sound a bit mad, but useful, or am i way off the mark. luke
Re: SMS as Pools Patch
that Does A Better Job, well... tough! go use or write something else that isn't trivial'. Ah, the naming issue :) I raised that a couple of times before commiting, and noone as of yet has come up with a better name. I started out writing apr_sms_... trivial_buffered trivial_buffer trivial_pool trivial_cache pool buffered freecached sentinel ?
Re: SMS usage patterns, hierarchies
On Wed, Jul 11, 2001 at 09:08:47AM -0700, Aaron Bannert wrote: also, would someone like to write an apr_sms_trivial_using_hashchains? the idea here would be that the size of the memory block is used as a hash-lookup into the currently-available free chain, instead of list-walking. surely, that saves time, yes? anyone care to refute this hypothesis and proposal? I'm pretty sure that's a bad idea. The whole point of pools was to take advantage of the finite lifetime of a request and it's memory requirements. Instead of having to balance all the malloc()s with free()s (where lots of differently sized and fairly small free()s followed by a number of malloc()s, over and over, can be very inefficient), pools simply dump all the memory allocated since some finite time (ie pool creation). That way we get essentially one bulk free() to replace a bunch of expensive mini-free()s. (At least that's the way I see pools being optimal inside httpd) ah ha. Ignoring the fact that a linear search through lists of ~20 (in this case) is probably going to be faster than a hash table, this case would only be useful if we are trying to optimize best-fit, and would only really work if the blocks we've already free()d are almost always *exactly* the same size as the ones we want to alloc(). Even if the block you are requesting is only slightly different, the hash would fail and we'd have to call the parent or do something else funky. ah ha! okay, now i start to get it. an explanation of the requirements specification, which is the first stage to understanding why this is needed. thank you very much, aaron. So, I'd rather see us do a next-fit implementation (a linked list of bins, search down until you find one that fits) at the thread level, and then create child-sms for each request that act like pools (and don't implement free()). ah ha! ... is that different from apr_sms_trivial, as it stands, then? by the way: another mallocator that fits the requirements is talloc.c this trivial allocator *never* frees, and it also doesn't *reuse* memory. is that a viable option? when you reset, you destroy the entire list [i.e. you never consider reuse] you only allocate large blocks and give out bits of them. use a list, because aaron and sander and equally more clued-up minds reckon these are reasonably efficient. you provide no realloc. you provide no free. only [sms_] alloc, reset and destroy. reckon? lukes
Re: lib/apr_signal.c
On Wed, Jul 11, 2001 at 08:14:15AM -0700, [EMAIL PROTECTED] wrote: okay... so... what you are saying, effectively, is that apache is vulnerable to a SIGPIPE DOS attack, amongst others. It shouldn't be. We block all signals in all processes, and only listen for those we care about. okay! fantastic! *happy*. i think that's what i understood the code to be doing. okay. ... do you have a little time spare to help me use this code, then? or, pointers to a FM to R? i have [i am afraid to say :)] another idea. i've just been helping getting SocketServer.py for Python 2.x improved / generic [yet another base class]. SocketServer.py is basically a mix-in mechanism for UDP/TCP with Threads/Fork/AnythingElseSuchAsSingleProcessMessaging and the bit i did was to add UDP/TCP/AnythingYouLike. what i suspect is happening in subversion [correct me if i'm wrong, here!] is that they are writing their own server-code. and httpd has some seriously excellent server-code. now, on my own, i don't have the experience to write server-code. not of the quality of httpd. however, what i _am_ prepared to do is to create a framework that mirrors that of SocketServer.py (the one in current CVS for the due 2.1.1 NOT the one in 2.0 NOT the one in 2.1 it has bugs in the ThreadingMixIn] i could use this for xvl, cliffs, all of the dcerpc daemons we'd like to write. etc. you could use it for httpd. subv could use it. everyone else could use it. what u think? luke
Re: SMS as Pools Patch
On Thu, Jul 12, 2001 at 11:20:59AM +0200, Sander Striker wrote: [...] we _know_ that malloc and free are pretty efficient on most operating systems. so why not rely on that efficiency instead of trying to second-guess it and round-up the malloc-block sizes? Because we most definitely get a significant speedup by allocating more mem than the user asks for, so we can serve the next request from our own block. okay... urr... so... again, you're second-guessing the capabilities of the parent-mallocator (in this case, malloc, via the default-sms. or, is the parent-mallocator another apr_[misnamed:)_trivial? It is another trivial. Oh, wait, you definitely need to take a look at apr/memory/unix/apr_sms_trivial.c. Justin added some comments so it should be a bit more clear what is happening at the moment than without the comments :) ack. cvs update and comments to follow... because if it's an apr_sms_trivial, then your speedup is _actually_ achieved by avoiding problems in another part of, or the usage of, apr_sms_trivial. It is probably the usage of. Also, problems can be avoided by using apr_sms_trivial_create_ex() which allows you to set min_free, min_alloc and max_free at runtime. ah ha :) In case of the patch we get a stack where trivial is the child of another trivial. We now have the child sms asking for mem for the parent, which adds the 4k bytes etc. We can circumvent this behaviour by doing the apr_sms_child_malloc and friends implementation. this really does sound to me like there is a problem with stacking a trivial on top of a trivial, and that the proposed solution is to make this a special case to deal with what is effectively a broken situation. Yes, this could very well be. What stings me though is that stacking trivials is giving such a penalty. So basically, what I am trying to say is that the apr_sms_child_xxx functions make it possible to stack (2 of the same, or 2 smss that make assumptions about allocations) on top of eachother, without adding extra bytes for example. ... intuitively, i can only say that i'm not entirely convinced. which means that i feel that, given the current information i know of, i can perceive a simpler and more structured solution. but as usual, i am having difficulty sieving it through my cross-wired brain into actual... like... concepts, y'know. [i'll cut with the zen, now] [...] surely there are better ways to do this that can be investigated _before_ expanding the SMS API into a more complex system? Maybe... :) But that involves changes to the httpd codebase which is not an option with both pools and pools == sms should work. It will be numerous #ifdefs in the httpd code (at the location of every apr_pool_create) if we go down that road. how about a #define that makes apr_pool_create call a new function that does what you need, especially if what you need is the same in every place [exactly what, you haven't explained, so i don't know :) ] We have that now. The first sms created is a std sms. All the others become trivial smss. Justin is doing some testing to see if we can do a trimix of std, trivial and tracking. In the long run it means that we are having to replace the apr_pool_create calls in httpd with the actual apr_sms_xxx_create call of the type of sms we need. that is what i would expect to have to happen _anyway_ - due to the other discussion-threads about considerations of doing one pool per thread, blah blah _even_ if sms is not used. same area, _probably_ the same problem :) :) sorry to be so stick-in-the-mud, my friend, esp. on-list. it's important to me that there are good reasons for things :) No prob :) Points well taken. *silly grin* lukes
Re: lib/apr_signal.c
On Thu, Jul 12, 2001 at 12:47:13PM +0200, Sander Striker wrote: [...] what i suspect is happening in subversion [correct me if i'm wrong, here!] is that they are writing their own server-code. You're wrong :) wha-heeey :) i like being wrong :) subversion is going to use httpd with mod_dav as the server :) ... *enthusiastic*... omigud, i'm 25% way through mod_dav.c when i looked at the LOC good _grief_. okay, that's irrelevent. i get the idea. Wondering what the framework like socketserver.py looks like. Summary? SocketServer.py is basically three sets of classes [with specialisations of each] that you then 'mix' together to get the required combination for your server. set one: BaseServer. specialisations: TCPServer and UDPServer [i wrote a SQLTableReadingServer where the 'request' is the data _read_ from a queueing-table!] Mixin. specialisations: ForkingMixIn and ThreadingMixIn. [on the TODO list is a single-process Mixin that keeps a list of all connections *itself* and has to deal with them one-by-one with all the associated blocking and select problems etc. on a series of file descriptors] Handlers. basically this allows you to 'Do' things to a request. there are some pre-defined UDP and TCP request-handling classes that you still have to over-ride functions to actually _do_ something with the request socket, but at least it _gives_ you a per-connection socket. the SocketServer framework basically provides a common API to create, handle, finish and close requests. you don't have to worry about how the request gets _to_ you: you only have to provide methods to deal _with_ the request. so you combine the above classes to get: ThreadingUDPServer ThreadingTCPServer ForkingUDPServer ForkingTCPServer and, with the addition of BaseServer, i created ForkingSQLTableReadingServer, which polls [yes, i know] a SQL table, locks it, reads one entry, deletes it from the table, unlocks the table, creates a handler for it, passes the entry to the handler, the handler Does Its Thing (in this case, loads from a SQL table a python script named in the request and runs it!) now, whilst this level of genericity is probably excessive, it was trivial to do, and it also provides the means to *transparently* add... SSLServer and then mix it in with Forking and Threading and you change *zero* lines of code - ssl is all done for you. redirection. loop-back testing. unix-domain-socket server. even a TransactNamedPipe server - whatever you can dream up. It seems that the apache server framework can be extended with other protocols by just adding new modules and filters. ... what if people don't want to add new modules and new filters? what if they want to go a separate route but take advantages of the best bits of httpd's code, and share the results? :) luke
Re: SMS usage patterns, hierarchies
On Tue, Jul 10, 2001 at 11:45:01AM -0700, Brian Pane wrote: Sander Striker wrote: Cliff pointed out to me that using my homedir for this stuff might be a better idea (instead of people pounding my ADSL). I saw some hits on my box and think that people were scared away by the size of the archive (~10MB). Sorry about that. In combination with the speed of my line I can imagine you didn't even try ;) Ok, now it can be found on www.apache.org/~striker/sms Sander Thanks. If I'm reading the graphs right, they show that destruction of a leaf SMS with siblings is much more common than destruction or reset of a non-leaf SMS with children. That seems to reinforce the conclusion I drew from gprof data: we could get better performance by allocating blocks for a child from something that isn't the direct parent (like a per-CPU free list, based on Dean's recommendation). okay. could someone explain to me why there are two stacks of apr_sms_trivial, such that you get the double list-walk in the first place? please? :) is there a good reason why apr_sms_trivial is not using apr_sms_general to obtain its memory? surely, to get blocks for a child directly from malloc [via apr_sms_general] would be better than going to another apr_sms_trivial, which, as the stats show, does yet another list-walk? also, would someone like to write an apr_sms_trivial_using_hashchains? the idea here would be that the size of the memory block is used as a hash-lookup into the currently-available free chain, instead of list-walking. surely, that saves time, yes? anyone care to refute this hypothesis and proposal? all best, luke
Re: SMS as Pools Patch
On Wed, Jul 11, 2001 at 04:32:54PM +0200, Sander Striker wrote: When one of us gets a chance, we can implement the child_malloc path in trivial. That should remove most of the current complaints about SMS. so, you want to add apr_sms_child_malloc etc. which makes SMS a non-trivial API, which is one solution to the problem, and the other solution is to write a better sms than apr_sms_trivial. one solution makes the SMS api quite complex. the other keeps it simple. The exported API to userland will not change (see below). ack. okay. makes me happier. i know that there have been message flying back/forth. i don't understand them. please do me a favour: could you write up a clear explanation of exactly what the purpose of apr_sms_child_malloc is? Ah, ok. The implementation of an sms calls apr_sms_child_malloc instead of apr_sms_malloc when it wants memory from its parent. ... okay... why? The same goes for calloc and free. These xxx_child_xxx functions will _not_ be exported outside the SMS code space. The xxx_child_xxx functions can be optionally present in an sms to make a distinction between a user asking memory or a child sms requesting memory. this is the bit that makes me nervous. it makes sms more complex to program to, to use, and to understand. how are you going to keep _users_ from _not_ using apr_sms_child_xxx and friends? By default the framework will use the malloc_fn, if a child_malloc_fn is present it will use this. This allows us to handle child sms allocations differently, hereby for example not letting the trivial sms add the extra 4096 bytes on the alloc each time the child requests 8192 bytes. ... urrr... not being funny or anything, but what is a trivial mallocator doing adding an extra 4096 bytes in the _first_ place? doesn't sound very trivial to me! trivial to me says 'this is really simple. it works, it probably stinks, we don't care: it proves the point, it's proof-of-concept, you can get to grips with it in under 5 minutes, and if you want more complex stuff that Does A Better Job, well... tough! go use or write something else that isn't trivial'. ... but leaving that issue aside, _why_ is the trivial_malloc 'rounding up' the sizes to the nearest min_block size? that implies that you are second-guessing the parent mallocator's ability to manage memory, and maybe even screwing _up_ the parent mallocator's ability to manage memory! in fact, that's exactly what's happening! we _know_ that malloc and free are pretty efficient on most operating systems. so why not rely on that efficiency instead of trying to second-guess it and round-up the malloc-block sizes? btw, i'm quite happy to be told that i really don't get this, and to have things [im]patiently explained to me so i shut up :) because it's very unclear [and i helped _design_ SMS, remember?] and if it's unclear, i really don't think it's a good idea to put it in, especially if there are other solutions. basically, apr_sms_child_malloc is giving me bad vibes - warning alarm bells are ringing: that's the best way i can put it. Well, for now this is the best solution to solve the problems seen in apache _without_ modifying the apache codebase. It is a way to allow stacking of advanced memory systems in a more acceptable manner when it comes to performance/waste trade-off. surely there are better ways to do this that can be investigated _before_ expanding the SMS API into a more complex system? all best, luke
Re: lib/apr_signal.c
On Wed, Jul 11, 2001 at 07:43:14AM -0700, [EMAIL PROTECTED] wrote: APR doesn't really handle signals, for a very good reason. They are incredibly non-portable, and very difficult to deal with. Having said that, there are some APR functions for dealing with signals. 1) apr_signal. Just like signal, only portable and predictable 2) apr_signal_thread puts a single thread into sigwait. Whenever ANY signal is received that thread is woken up, and a function is called. The function is passed in to the setup_signal_thread function. 3) You can get a list of signals understood by the machine. I can't remember the function, but it is there. Most of Apache specifically tries to avoid any signals, although the parent still relies on SIGWINCH, SIGTERM, and SIGHUP. And the children rely on SIGTERM and sometimes on SIGINT. okay... so... what you are saying, effectively, is that apache is vulnerable to a SIGPIPE DOS attack, amongst others. for xvl, i think... i think what i will do is simply rip all of the signal handling / fault / blocking etc. code out. xvl doesn't use apr_signal_thread() - it doesn't use threads [yet :)] when this issue has been addressed [DOS attacks possible via signals], i'll follow suit. all best, luke
Re: exploration of APR goes on
hey, guys, unless it's like bugging you or captivated your interest, don't worry about it: i have a fix, albeit not a nice one :) On Tue, Jul 10, 2001 at 01:15:48AM +0200, Sander Striker wrote: On Tue, 10 Jul 2001, Luke Kenneth Casson Leighton wrote: ImpersonateLoggedOnUser? same thing as ImpersonateNamedPipeClient. i.e. you can only impersonate an existing user IF you have a handle to that user. The other problem with ImpersonateLoggedOnUser AFAICT is that you can apparently call RevertToSelf() which does what it sounds like. That's generally undesirable in the contexts we're talking about... Ok, well maybe OpenThreadToken() and SetThreadToken() could be usefull? As you can see, I'm just going over the API, looking for leads :( Maybe someone out there knows how it's done? --Cliff Sander
Re: [PATCH] make socket timeouts work reasonably for connect()
Win32 has messaging (which is basically, send length of data
first in a 16-bit or 32-bit word then send data of EXACT
length) which is very commonly used.
so no surprise that anything else is less well supported :)
messaging is cool: guaranteed transmission and data size:
a combination of the best of TCP and the best of UDP.
luke
On Tue, Jul 10, 2001 at 01:43:49AM -0400, Jeff Trawick wrote:
(Unix at least; Win32 connect is a mess for non-blocking/timed-out
sockets... it looks to me that we wait forever; I could be confused
though :) )
concerns?
old behavior on Unix:
if timeout is set on socket before apr_connect() and connect takes a
while (normal for TCP), apr_connect() returns EINPROGRESS and app must
handle any timeout
new behavior on Unix:
if timeout is set on socket before apr_connect(), app will see
APR_ETIMEUP if it takes longer than the timeout
Index: include/arch/unix/networkio.h
===
RCS file: /home/cvspublic/apr/include/arch/unix/networkio.h,v
retrieving revision 1.45
diff -u -r1.45 networkio.h
--- include/arch/unix/networkio.h 2001/07/07 22:48:09 1.45
+++ include/arch/unix/networkio.h 2001/07/10 05:46:27
@@ -159,6 +159,7 @@
const char *apr_inet_ntop(int af, const void *src, char *dst, apr_size_t
size);
int apr_inet_pton(int af, const char *src, void *dst);
+apr_status_t apr_wait_for_io_or_timeout(apr_socket_t *sock, int for_read);
#define apr_is_option_set(mask, option) ((mask option) ==option)
Index: network_io/unix/sendrecv.c
===
RCS file: /home/cvspublic/apr/network_io/unix/sendrecv.c,v
retrieving revision 1.65
diff -u -r1.65 sendrecv.c
--- network_io/unix/sendrecv.c2001/05/03 22:38:00 1.65
+++ network_io/unix/sendrecv.c2001/07/10 05:46:29
@@ -59,7 +59,7 @@
#include fileio.h
#endif /* APR_HAS_SENDFILE */
-static apr_status_t wait_for_io_or_timeout(apr_socket_t *sock, int for_read)
+apr_status_t apr_wait_for_io_or_timeout(apr_socket_t *sock, int for_read)
{
struct timeval tv, *tvptr;
fd_set fdset;
@@ -103,7 +103,7 @@
if (rv == -1 (errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout != 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 0);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -132,7 +132,7 @@
if (rv == -1 (errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout != 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 1);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 1);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -167,7 +167,7 @@
if (rv == -1 (errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout != 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 0);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -200,7 +200,7 @@
if (rv == -1 (errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout != 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 1);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 1);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -235,7 +235,7 @@
if (rv == -1 (errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout != 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 0);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -324,7 +324,7 @@
if (rv == -1
(errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout 0) {
- arv = wait_for_io_or_timeout(sock, 0);
+ arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -423,7 +423,7 @@
* we get -1/EAGAIN/nbytes0; AFAICT it just means extra
syscalls
* from time to time
*/
-apr_status_t arv = wait_for_io_or_timeout(sock, 0);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
return arv;
@@ -577,7 +577,7 @@
if (rc == -1
(errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout 0) {
-apr_status_t arv = wait_for_io_or_timeout(sock, 0);
+apr_status_t arv = apr_wait_for_io_or_timeout(sock, 0);
if (arv != APR_SUCCESS) {
*len = 0;
@@ -710,7 +710,7 @@
if (rv == -1
(errno == EAGAIN || errno == EWOULDBLOCK)
sock-timeout 0) {
-arv = wait_for_io_or_timeout(sock,
Re: Observations on fragmentation in SMS pools
... errr.. should i be using apr_hash_t or something? :) Yes. okay. will take a look at it, to see where i could use it. i have a suspicion that a lot of instances i really _need_ the case-insensitive stuff, and also need the dual set and add capability. not least because HTTP POST args can have this: key1=value1key1=value2 and, correct me if i'm wrong, here, that results in a table: (key1, value1) (key1, value2) luke
Re: Observations on fragmentation in SMS pools
On Sun, Jul 08, 2001 at 10:19:33AM -0700, Justin Erenkrantz wrote: On Sun, Jul 08, 2001 at 10:14:16AM -0700, dean gaudet wrote: an ideal situation for free-lists (blocks of freed, but not free()d memory) is one per cpu. a less ideal situation is one per thread. an even less ideal situation is one per process (which requires locking). it's insane to have one per pool :) I think we're shooting for #2 (less ideal). Unless someone can come up with a way to dynamically tell how many CPUs we are running on and bind a free-list to a specific CPU. We're currently doing #3 (even less ideal) in apr_pools.c. And, yeah, the current trivial SMS is doing #4. =) But, don't expect it to stay like that. And, if we implement the apr_sms_child_malloc, it gets to somewhere between #2 and #3. -- justin #5 the approach taken by tdb. have a hash-chain. then you get the best of all worlds. simultaneous access only requires that you lock *one* hash chain, which allows a theoretical limit of number-of-hash-bucket simultaneous accesses. luke p.s. i'm hoping that people will not be put off by tdb's GPL license because i think that there are some really good techniques used in there that could be applied to APR code, too.
Re: multithreaded pools?
On Sun, Jul 08, 2001 at 10:36:10PM -0700, Roy T. Fielding wrote: OTOH, it seems the new sms code just needs a whack upside the head to stop it from asking for extra memory from its parent. *cackle*. just got in, saw a whole boat-load of messages about sms [grief, who'd have thought it, neh? :) will get to them in a bit]. i saw the profiling: a suggestion - why not make a less-trivial sms that uses a hash-table? i mean, the trivial sms proves the point, and people can now go 'oh yeah, it works!' [and, 'what was all the darn fuss about???']. 'now, how do we make it better, little-step-by-little-step'? sander, did you do sma as an sms yet (sma: smart memory allocator), to do a comparison? luke
Re: Observations on fragmentation in SMS pools
regarding fragmentation: is it reasonable to assume that pre-allocating larger chunks of memory and sub-dividing them yourself will prevent memory fragmentation, with the side-effect of having More Code? use hash tables to optimise the list-walking? write a better sms instance, one that isn't 'trivial'? luke
Re: Observations on fragmentation in SMS pools
On Sun, Jul 08, 2001 at 10:56:59AM -0700, Jon Travis wrote: On Sun, Jul 08, 2001 at 10:41:12AM -0700, dean gaudet wrote: On Sun, 8 Jul 2001, Jon Travis wrote: There is still all this tomfoolery with locking, though, which I think would be nice to fix with different sized buckets in the freelist. Stuff that the malloc in glibc does. I cringe at the thought of how much overhead due to abstraction this whole project is causing. haha, the abstraction problems started AGES ago :) i think the first real sign of the doom ahead was the argument of changing ap_pool_t to ap_context_t because there was this arbitrary hash table attached to it. i'd suggest that before any decisions are made regarding global free list or per thread free list someone should study what the non-SMS, pools-only server behaves like. the problems you're looking at now in the SMS server are a result of SMS's feature of allocating all the way up through the ancestry adding bytes each step of the way. this _can_ be mitigated by doing your own memory-management on top of a more normal one. which is what sander's 'smart memory allocator' does. sma is targetted at having to have massive amounts of reallocs and mallocs and then freeing large numbers of them in one blast, repeat. luke
exploration of APR goes on
hiya,
well, i'm exploring APR more and more for xvl development (we're up!
http://xmlvl.net).
i just wanted to let you know a few things:
1) as i find out more, i get more impressed. each time i want
to add a bit more code or convert some over, i find in almost
95% of cases that the functionality in APR just points the way.
i converted over the file code to apr (2 hours). the
directory-listing to apr (2 hours). i got a bit
confused about which thing to use in an apr_finfo_t:
fname or name, that _is_ really odd, guys :)
apr_proc_create()? simple! easy! love it!
2) the similarities to the data structures needed by samba,
and those created for APR usage, are freaky :) this bodes
well for cliffs (auto-generated SMB client and server - an
alternative to samba)
3) the 5% missing bits i've found so far are:
- getuid() i assume that this has been discussed? i have
to get latest httpd-2 to find out how this has been tackled.
instead i have to do a getenv('USER') [yes, yuck].
- signal handling / blocking. i am very concerned by the
recent report by todd sabin on razor.bindview.com about
80% of unix programs being vulnerable to signal attacks
(esp. SIG_PIPE). so i am going to leave in the signal
blocking - even though it will make it impossible to compile
on Win32. i can't find any equivalent functionality in
APR to stop certain kinds of signals or to trap SIG_TERM
and call a fault_cleanup(). am i missing something?
- getenv() i'm going to assume that every system has getenv()
because i can't find one in APR, but i see that the apache
1.3.x code uses getenv...
anyway, should get back to work now.
i love code that makes life easy.
luke
Re: exploration of APR goes on
On Mon, Jul 09, 2001 at 05:01:25PM -0400, Cliff Woolley wrote:
On Mon, 9 Jul 2001, Luke Kenneth Casson Leighton wrote:
3) the 5% missing bits i've found so far are:
- getuid() i assume that this has been discussed? i have
to get latest httpd-2 to find out how this has been tackled.
instead i have to do a getenv('USER') [yes, yuck].
Take a look at apr_get_userid(), which among other things is in the user
subdirectory of APR.
it doesn't do getuid() / geteuid() - it does getpwnam / getpwuid.
so there's no means to obtain _current_ user id of running
process, only a lookup from a username (or userid).
when a NULL name or NULL uid is passed in to apr_get_userid(),
it returns an APR error.
i did check :)
all best,
luke
Re: exploration of APR goes on
so there's no means to obtain _current_ user id of running
process, only a lookup from a username (or userid).
Not yet. Nobody has needed that ability so far. Feel free to implement
it though. APR follows a VERY simple rule. We don't implement a feature
until it is needed. :-)
ack!
One warning, I have no idea how this would work on Windows. In order for
this to really be useful, we have to figure that piece out.
yep.
i mean, i can get away with getenv('USER') and to be honest, it
doesn't bother me. it might bother other people though.
btw, just so you know: i know it _is_ possible else how would
cygwin work?
... and i do know that jeremy had a hell of a time getting setuid()
to work. it's almost impossible: none of the published APIs
describe how to do it. you can 'impersonate' an existing context
e.g. ImpersonateNamedPipeClient or similar but you can't
actually do a sudo. okay, it's been done, recently, and there
does exist SU.EXE, but still :)
Re: exploration of APR goes on
On Tue, Jul 10, 2001 at 12:49:38AM +0200, Sander Striker wrote:
so there's no means to obtain _current_ user id of running
process, only a lookup from a username (or userid).
Not yet. Nobody has needed that ability so far. Feel free to implement
it though. APR follows a VERY simple rule. We don't implement
a feature
until it is needed. :-)
ack!
One warning, I have no idea how this would work on Windows. In
order for
this to really be useful, we have to figure that piece out.
yep.
i mean, i can get away with getenv('USER') and to be honest, it
doesn't bother me. it might bother other people though.
btw, just so you know: i know it _is_ possible else how would
cygwin work?
and i do know that jeremy had a hell of a time getting setuid()
to work. it's almost impossible: none of the published APIs
describe how to do it. you can 'impersonate' an existing context
e.g. ImpersonateNamedPipeClient or similar but you can't
actually do a sudo. okay, it's been done, recently, and there
does exist SU.EXE, but still :)
Check out:
LogonUser -
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/security/hh
/winbase/accclsrv_9cfm.asp
ImpersonateLoggedOnUser -
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/security/hh
/winbase/accclsrv_0jle.asp
Maybe that can do the trick?
don't know about LogonUser. yes i do: it has to have a password.
ImpersonateLoggedOnUser? same thing as ImpersonateNamedPipeClient.
i.e. you can only impersonate an existing user IF you have a handle
to that user.
there is no published public API to *create* a new user context.
it's buried. i think the ntinternals, the bindview or other
security people have probably found an 'undocumented' API, but
that's not the sort of thing you put into soemthing like APR.
luke
Re: [PATCH/CONTRIB] Shared Mem Hash Table
On Sat, Jul 07, 2001 at 08:33:54AM +0100, David Reid wrote:
I'd be interested to know of other tools that people use that are available
for others here * on the list to use improving the code. So far when pools
insure-lite may help _if_ you use the following trick.
insure-lite is availabe in the non-free section of
redhat 6 cds. insure itself is about $4,000 USD and
no, they have never heard of open source projects
so they are not interested in providing licenses as
a promotional loss-leader [in complete contrast to
vmware who are very supportive of open source].
these code fragments come from samba source (GPL):
/***
close the low 3 fd's and open /dev/null in their place
/
void close_low_fds(void)
{
int fd;
int i;
close(0);
close(1);
#ifndef __INSURE__
close(2);
#endif
/* try and use up these file descriptors, so silly
library routines writing to stdout etc won't cause havoc */
for (i = 0; i 3; i++)
{
fd = open(/dev/null, O_RDWR, 0);
...
...
[except that insure writes out to stderr, so you can't close
that one]
#ifdef __INSURE__
/***
This routine is a trick to immediately catch errors when debugging
with insure. A xterm with a gdb is popped up when insure catches
a error. It is Linux specific.
/
int _Insure_trapr_error(int a1, int a2, int a3, int a4, int a5, int a6)
{
static int (*fn) ();
int ret;
char pidstr[10];
pstring cmd =
/usr/X11R6/bin/xterm -display :0 -T Panic -n Panic -e /bin/sh
-c 'cat /tmp/ierrs.*.%d ; gdb /proc/%d/exe %d';
slprintf(pidstr, sizeof(pidstr), %d, sys_getpid());
pstring_sub(cmd, %d, pidstr);
if (!fn)
{
static void *h;
h =
dlopen
(/usr/local/parasoft/insure++lite/lib.linux2/libinsure.so,
RTLD_LAZY);
fn = dlsym(h, _Insure_trapr_error);
}
ret = fn(a1, a2, a3, a4, a5, a6);
system(cmd);
return ret;
}
#endif
[this little trick _is_ actually documented in the insure
documentation. insure-lite strips out stack-debugging
info, making it almost completely useless... _unless_
you use the above little trick.]
insure is *extremely* good. if you have used purify, you will
be laughing at it.
insure detects at *run-time*:
- pointer overwriting:
a = malloc(20);
a = b;
return;
(but it doesn,t detect this:
struct foo
{
void *a;
}
f-a = malloc(20);
memset(f, 0, sizeof(struct foo));
or, it _does_, it detects it as a loss of memory f-a
but not _why_.
but _does_ detect this:
struct foo fa;
struct foo fb;
fa.a = malloc(20);
fa = fb;
cool, huh? :)
- underwriting and overwriting of memory regions and strings
- not freed, freed more than once memory
so, when you do an apr_pool_destroy, it will let you
know immediately if there are any problems AND, get this:
it tells you the stack at the point where the memory
was ALLOCATED!
etc. there's lots more. it does a little bit of
compile-time checking, too.
the down-side: a normal 3mb executable compiles up at
80mb. a 30-times increment in binary size
One of the things we really need to improve is our test suite.
yespleasetestsuitesmakeforeasycuttingandpastingtogetrealliveworkingcoderealquick
:)
luke
Re: Timezones logic...
hiya david, i just want to double-check something. i got caught out by creating cookies of 1 year which actually turned into 30 seconds because apr_time_t is in microseconds :) :) so, could you confirm at the points below for me? thanks also, i should point out some experience from seeing over-the-wire stuff from SMB, which may be relevant to NT. We wrote the apr_implode_time routine to take an exploded time structure (that is an APR only structure) and create a time_t from it. There is no ^^ do you mean a time_t or do you mean an apr_time_t? the external interface is apr_time_t. you are referring to internal implementation details, yes? This is quick and easy and makes sense (I hope) to everyone reading it. Yes it is server oriented but then it also works for client apps. Brane's addition of apr_implode_gmt helps with those apps that demand a GMT/UTC time. okay. microsoft's implementation of GMT over-the-wire is... uhhh... wrong! [they couldn't even implement a well-known algorithm correctly, having used localtimes for years. and of course, once released, they have to _keep_ it because it's too late by then, you and everyone else and their mothers have to be backwards-compatible with KludgeGMT...] what they actually do is reverse-convert-back from localtime, but they do it at the wrong times, such that for about a day or so around daylight saving time changes, GMT goes haywire +/- one hour. it may be time well worth spent investigating whether the top-level NT API that gives you GMT is as equally broken or not w.r.t file timestamps and time creation. etc. or whether you have to do the conversion _correctly_ yourselves. all best, luke
Re: Shared Memory Hash Table / SMS
On Mon, Jul 02, 2001 at 07:10:27PM +0200, Sander Striker wrote: On Mon, Jul 02, 2001 at 09:38:37AM -0700, Ian Holsman wrote: Just wondering if any has got one of these going, how does the proposed SMS shared library handle pointers? Can you please expand a bit further on what you mean? Oh, do you mean, How does the proposed SMS shared memory system handle pointers in data structures that it has? Oh, that, I dunno. Prolly with some type of offset/fixup routine. But, that may not be ideal. -- justin We basically still need to work that out :) I'm personally still not sold on the offsets idea. I'm still not getting _why_ we would need offsets instead of a simple pointer to a piece of mem. when you request that a shared memory space be mapped into real memory on different processes, it is extremely unlikely that it will be mapped to the *same* physical location in memory on each process (probability: 2^^-32 or 2^^-64...) so therefore, if you store any pointers from one process _into_ the shared memory block, when they are retrieved by _another_ process, they will of course point to garbage. there are a number of techniques for dealing with this, but basically they boil down to structure-packing, effectively treating the memory exactly as you would a very very fast file. all best, luke
Re: [PATCH] Some named pipe hacking...
hiya, didn't get a response back regarding the namedpipes / ntpipes issues, still have some questions outstanding, perhaps someone can help me understand? i asked earlier about whether it is possible to do a select() on a [unix] namedpipe: i have since discovered apr_poll() so the question should be: in the proposed [generic, unix-named-pipe-like, not the nt-named-pipe-like one!] apr_namedpipe() api: will it be possible to do an apr_poll(), fork or thread, deal with the request or requests? this is really important for the viability and useability of any proposed apr_namedpipe() implementation. as an aside, in case you're wondering, one of the reasons i haven't posted Code for an apr_ntpipe() implementation is because the infrastructure to support it / code it easily doesn't exist. namely, what i think will be needed is probably likely to be close to the apr_IOL (took me a while to work out that means input / output layer :) where can more info about the IOL be found? i have a sneaky suspicion that any namedpipe implementation - both the proposed apr_namedpipe() _and_ the proposed apr_ntpipe() APIs will need to be 'plugged in' behind an IOL, because otherwise, adding them in behind the existing code will make a horrible mess of exceptions etc. and will result in a botch-job-of-an-IOL _anyway_ :) :) all best, luke
Re: [PATCH] Some named pipe hacking...
On Sun, Jun 24, 2001 at 10:30:00AM -0700, Justin Erenkrantz wrote: On Sun, Jun 24, 2001 at 07:18:41PM +0200, Luke Kenneth Casson Leighton wrote: I'd guess that it wouldn't be too hard on your end to at least post a patch/code (within APR framework) that illustrates what you are talking about. ack. well, a _quick_ one is simple, you are right. about... 1 days' work? Yeah, that would be sufficient (Unix only). Just give us an idea of what you are talking about. The fact that on NT/OS2 it goes directly to a system call that is named pipes and on Unix goes to domain sockets (I need to read up on those) doesn't bother me in the least. The functionality would have to be equivalent (LCD) and they would have to be able to talk to each other. i have a question: can you do a select() / listen() / bind() / accept() on a *unix* namedpipe? can you do equivalent-of-select() /listen() ... on an NTnamedpipe? So the Unix implementation on the network side has to be compatible with the NT implementation - how are endian issues dealt with in SMB, for example? well, in samba, there are conversion-macros - equivalent to ntoh - but intel-byte-order not network-order :) so, you always end up, in your host, with data in the right order. but, i have to point out: it doesn't really have anything to do with creating an NTpipe api :) :) okay, maybe it does, and i'm so used to it, i forget :) a more complete one? with a 'clean' way to transfer security context? and the code _does_ need a security audit, which i even print out this fact into the log files :) :) about a week. That's what we are here for. =) We're not perfect, but enough good people looking at it will do just fine for our purposes. -- justin :)
Re: [PATCH] Some named pipe hacking...
Hi Luke, hi aaron :) IMHO, it is important to keep local and remote semantics distinct. Local and remote services (by services I mean any form of IPC) are inherently different beasts. If we allow named pipes to handle both local and remote connections under the same conditions (errors, assumptions, etc) than we may run into serious problems with reliability, error recovery, and we may even expose ourselves to security risks. reliability and error recovery are not the responsibility of the [remote] API. all applications - all the good ones, anyway - that i know of that use namedpipes or dce/rpc, pass error handling up to the application, for the application developer to deal with. in dce/rpc, this is even negotiable!!! you specify in the idl file 'i don't want to receive network errors', and well, you don't! security. well, yes, as soon as you get into distributed applications, your security risks go up. i am familiar with a fair number of the kinds of nightmare situations that can occur, having discovered and reported on large numbers of them to microsoft, until it got boring, there were so many. so, it's fair to assume that i've seen enough [an implementation of remote named pipes - the one in NT - that passes a buffer size in three separate locations, used one of them to check the received write size against the other, and the third one was the one used to allocate the data that the other one copied into. and this all in kernel-space...] If you really want to start handling remote named pipes, I am more in favor of something like what Justin suggested. That is, perhaps adding new functions for the specific handling of remote named pipes. if i was to insist, and people decided, um, well, we're not going to do it, 'cos luke's getting stroppy about it, well, then the most sensible thing for me to do would be to say, okay, go ahead: dunna bother me, and when - or more hopefully - _if_ it's noticed later it dunna do what's expected, well i help redesign. and if it works, well, it saves me some hassle and some typing! all best, luke. p.s. i now have three apache modules that are using code that could be used as an early version of the unix apr_namedpipe() code. i haven't been able to split them into separate directories, not least because they are only 300 lines each, but because the effort of putting them in separate directories and then creating a mini-library for them is not justifiable.
Re: [PATCH] Some named pipe hacking...
On Thu, Jun 21, 2001 at 10:53:57AM -0500, William A. Rowe, Jr. wrote: From: Luke Kenneth Casson Leighton [EMAIL PROTECTED] Sent: Thursday, June 21, 2001 7:35 AM thanks bill! recommendation. use the actual expected format for pipename: pass in \\.\\PIPE\pipename _not_ pipename and have it hidden / constructed by apr. APR is here to make folks cross-plaform development blindingly simple. And (for the most part) it does. What you've proposed makes things far more difficult to follow. Named unix pipes are local, no? i actually know very little about named unix pipes. but afaict, yes, _unix_ named pipes are a local-only concept. and i advise against using unix pipes to implement this proposal, which is to implement *nt* named pipes, which are, as you can see from the OS/2 and NT developer SDK documentation, are totally different from un ix named pipes. So the _Lowest Common Denominator_ says that our APR namedpipe implementation is to create local pipes. only if youconsider unix named pipes to be the same as nt named pipes, which they certainly aren't. okay. i have a question. is it unreasonable to expect Unix developers to learn an API that is identical in functionality to an NT (and an OS/2) one? think through what you are proposing. i want to implement NT named pipes, using or _in_ APR. NT already _has_ NT namedpipes (and so does OS/2). so, you are proposing to 'shackle' the NT named pipes to 'local only', because some unix developers can't be ... well ... uhh... okay, i'll be polite, but it's the same thing, 'might be confused' by something that isn't a unix API. and then, you expect me to implement 'remote' named pipes as a shim on top of a crippled, perfectly good remote named pipes implementation? [but not in apr, because it's considered too complex] *blink* uhn? :) :) i think that anyone looking at such a codepath, trying to work out what's going on, is going to do a double-take and get even more confused than if the nt impl. of apr_namedpipes just went straight through to the NT one and the unix one used some external proxying service to emulate the remote bits. IF they WANTED the remote bits. If you want to create a cross-machine pipe architecture, then you must add another function with char *machine, char *pipename semantics. If you have at least a rough outline of the implementation for this on unix. rough? better than that, i have working code. it's in use in samba tng today, has been for about a year and a half. it duplicates the functionality of NT named pipes. i was hoping to use apr to make a better version of the code in tng, having learned from the first impl. One problem with using Netbios in any public environment is that the vast majority of operators will clamp down on the Netbios ports of any public machine outside the firewall. This is for pretty good reasons, given the paucity of security on that interface. yes. there are a couple of other similar situations where proposals - or actual implementations - have provided proxying of other transports onto other transports [explanation: NetBIOS is actually a full transport, but no-one implements it on its own, now, only the proxied versions] one is DCE/RPC over HTTP [this was added to NT for exchange's benefit: it runs on port 689 and there is an IIS component that is enabled BY DEFAULT ON NT5 argh argh that proxies this to port 80]. so, now all of your entire NT domain infrastructure, including exchange etc, is vulnerable to dce/rpc-related security attacks because ms wanted their customers to be able to read email through a firewall. greaaat. the other is the new TCP over HTTP proposal (a draft rfc i understand). but anyway. so... yes? and? yes, i agree with you. i'd even recommend it. the ports are 137, 138 and 139. if you want to shut down SMB over TCP too, that's on port 445. and if you want to shut down dce/rpc portmapper, that's on 135. and the dce/rpc over http? port 689, and then disable the redirector component too, in IIS. ... but despite all this, i don't see it as a relevant point to make as a reason not to fully implement NT named pipes. If you are looking for a cross-machine semantic, yes. the underlying support must also be cross-platform. yes. now i know _you_ know that. If you propose Samba for this, [no i don't: see below] then fine, let's start discussing that. [great!] ... what i actually want to propose is a project that sander and i have been considering: an alternative implementation to samba. there _aren't_ any open source alternatives to samba, and sander and i would both like to implement one - as an Apache Software Foundation project. now, given microsoft's recent announcement regarding their SDK - You May Not Implement Or Use This In Open Source Projects Except BSD Ones, that looks like being an increasingly attractive option. anway. i apologise in advance for holding most of the designs
Re: inetd-type architecture?
On Thu, Jun 21, 2001 at 03:05:53AM +0400, Michael Tokarev wrote: Luke Kenneth Casson Leighton wrote: [] srvsvcd's job is also to handle the dependencies for service startup. we could look at the linux kernel 'Calculating module dependencies' code to get an algorithm to work out the startup order. Why, why, why this is needed Why not to use runtime linker for this? Or I completely missed the point here (probably)? not entirely: i wasn't aware of runtime linker capabilities like this. the answer to your question is because these are programs (daemons) not .so's. all best, luke
Re: More migration of code from httpd to apr-util
On Wed, Jun 20, 2001 at 10:50:03AM -0500, William A. Rowe, Jr. wrote: From: Luke Kenneth Casson Leighton [EMAIL PROTECTED] Sent: Wednesday, June 20, 2001 8:55 AM You miss my point. We at _least_ need to return a Windows Acceptable Pipe Name, instead of some /PIPE/pluming style name. it is *important* that NT-style conventions are followed. ... internally uhh... okay. okay, it's important to me, given what i would like to achieve. i explain below. sorry for not having mentioned it earlier. the 'guiding light' *must* be the NT functions. No, the 'internal light' must be the platform-appropriate function. i know that most of you may find this difficult to accept that microsoft may be able to develop an API that's really useful, and may feel 'tempted' to 'do better' by imposing 'unixy style' conventions. please don't. We aren't. And you sir, are presuming that NT is the only 'obscure' platform out there. It isn't. i wasn't, but i didn't say that, so you weren't to know, so i stand corrected for not communicating enough. I'm suggesting that we pass a pipe name to create/locate a pipe. No hokey pathing, simply a pipe name that will be normalized \\.\PIPE\name or /dev/pipe/apr/name (or however it ought to be named on win32.) what i have been hinting at, and planning, is that i will actually provide, with the TNG architecture, is the FULL \\servername\\PIPE\pipename functionality. yes, that's right: i will write code that redirects to TNG, which will farm out the data over SMB over to a remote system FOR you. that's right: a unix apr application will be able to connect to a *remote* NT apr server. that's *remote* platform independence, not just local platform independence. and if you don't expose the full pipe-name in the APR api, i can't do that. And return an opaque structure that the platform may us to open handles on that pipe using an apr_filepipe_open() call. ack to that. you may be used to doing this the other way round: making NT do what unix does. We aren't, we all agree that lcd coding is required on apr, and we offer extensions where they are needed. Heck, why could even accept unix /dev/pipe/name as a pipe, but it would be morphed into \\.\PIPE\dev_pipe_name in order to serve the data. and if that means creating files like /tmp/.apr-socket/\PIPE\samr with backslashes in them and i _know_ this _is_ possible to do on unix, and i _know_ that there are no problems with doing so because winbindd does it [i have a directory /home/DOMAIN\administrator on my linux hard disk and even an entry in /etc/passwd to match it] then so be it. Why not simply a pipe name, and leave pathing to apr? Justification? see above. does that help? luke
Re: inetd-type architecture?
On Wed, Jun 20, 2001 at 02:55:15PM +0200, Luke Kenneth Casson Leighton wrote: i was wondering. might it be better to put real functionality behind srvsvcd? after all, what you are describing is srvsvcd's job. srvsvcd's job is also to handle the dependencies for service startup. a follow-up correction so as not to confuse more people than i have already: svcctld - an implementation of \PIPE\svcctl - not srvsvcd. my mistake. luke [p.s apr people: recognise the pipe name convention? :) :)]
Re: More migration of code from httpd to apr-util
On Thu, Jun 21, 2001 at 07:58:07AM -0700, Justin Erenkrantz wrote: On Thu, Jun 21, 2001 at 02:54:43PM +0200, Luke Kenneth Casson Leighton wrote: what i have been hinting at, and planning, is that i will actually provide, with the TNG architecture, is the FULL \\servername\\PIPE\pipename functionality. yes, that's right: i will write code that redirects to TNG, which will farm out the data over SMB over to a remote system FOR you. that's right: a unix apr application will be able to connect to a *remote* NT apr server. that's *remote* platform independence, not just local platform independence. and if you don't expose the full pipe-name in the APR api, i can't do that. Heh, you attempted to answer my previous question before I asked it. whoopsie :) Teaches me to read all of your posts before replying to any of them. :) me too, i think i have enough experience at the consequences of not reading ahead, now :) But, aren't named pipes typically local-only? server-side, they can be nothing _but_ local-only. client-side, no. it's possible to connect to \\servername\PIPE\samr and make direct dce/rpc requests to it, if you feel so inclined. or to \\servername\PIPE\LANMAN, or whatever is running or whatever someone chooses to make run, listening on a pipe. Maybe what you need is something outside of traditional named pipes (i.e. remote named pipes). Add the named pipe code and then add a special function in apr that allows you to get at the full namespace, if so desired. which is why i'm advocating the NAL because then 'hooking' in 'special' functions becomes a trivial matter. *thinks. *thinks some more*. AH! bill, i think you're right, regarding server-side, about specifying _just_ the pipe name. client-side, i'm not so sure. does anyone have any working NT client / server pipe apps? i'd be a lot happier / lot more confident if i could see coding examples, native NT code. and bill, do you envisage calling apr_namedpipe_create() on _both_ the client-side _and_ the server-side, a bit like when you create a socket? because if so, i think the full syntax \\server\pipe\pipename will be needed, and if you go 'server-side', then server has to be '.' otherwise it's an error. But, it seems that this remote named pipe would only be available with an SMB library - which is TNG's domain, not APR's. i envision that anyone could write a 'redirector' daemon, and without such a daemon running, you simply don't _get_ the ability of client-side programs to connect to remote pipes, you can only connect to \\.\PIPE\pipename. I guess I'm unsure whether named pipes will work from other machines on other platforms. If this works only with Win32 (or having an SMB protocol to handle this on the Unix end), or your own 'redirector' daemon that runs on the client and on the server, but to be honest, doing it via SMB is the simplest option: take a look at the odbc-odbc-bridges that are out there, the principle is sound - for only _one_ server - but as soon as you get to multiple servers it becomes completely unworkable. this seems something that TNG could add outside of the apr space. -- justin OS/2 also supports TransactNamedPipe and friends, i asked :) all best, luke p.s you know, i really should just go and code this up :)
Re: [PATCH] Some named pipe hacking...
On Thu, Jun 21, 2001 at 07:44:25AM -0700, Justin Erenkrantz wrote: On Thu, Jun 21, 2001 at 02:35:19PM +0200, Luke Kenneth Casson Leighton wrote: thanks bill! recommendation. use the actual expected format for pipename: pass in \\.\\PIPE\pipename _not_ pipename and have it hidden / constructed by apr. Can you please give a concrete example why you need this? we would like to rewrite TNG's services using APR. that includes, but is not limited to: a NetBIOS redirector, running on ports 137, 138, 139 and 445 a NetBIOS Name Server (WINS), running on top of the NetBIOS redirector. a Network Neighbourhood server, ditto. an SMB server, ditto. a LANMAN server, running on top of the SMB IPC$ server, which is actually with an interface of \\.\PIPE\LANMAN [which is why i am advocating pipes] a SAMR server, same thing, running DCE/RPC over \\.\PIPE\samr an lsarpc server, same thing a NETLOGON server a svcctl server a srvsvc server all of the dce/rpc services are currently bounced out via a unix-domain-socket, and i would like to use an APR framework for the portability of this project. the NetBIOS and SMB ones are currently 'taking over' ports 137, 138, 139 and 445, however there are people out there who would like to make a NAL out of this stuff so that others can 'join in the fun'. I don't think any of us are seeing why you need to use the Win32 naming conventions for a pipe. What you are suggesting goes against all of the precedents in APR. -- justin whoops :) like i said, i am aware that the 'traditional' focus of APR has been to emulate unix and make it portable that way. lukes
Re: inetd-type architecture?
[this message, which originated from [EMAIL PROTECTED], is cross-posted from apr dev and samba tech. the relevance to apr dev is the issues / justification of creating an apr NamedPipe interface - exactly mirroring the NT NamedPipe one _not_ the unix namedpipe one. luke] On Wed, Jun 20, 2001 at 09:28:31AM +0100, Mark Cave-Ayland wrote: Hi everyone, hi mark :) Just thought I'd add my ideas while people are deciding what to break after the release of 2.6.1 :) Would it be an idea to give TNG a similar architecture to inetd? That is, to have a 'superdaemon', say called tngd, that handles all the SMB connections, and passes the raw pipe data over stdin to the child daemon process. The allocation of pipes to executables could be done by a tngd.conf file, e.g. like it. # Example tngd.conf //./pipe/samr /opt/samba-tng/samr //./pipe/spoolss /opt/samba-tng/spoolss //./pipe/foo /opt/samba-tng/foo the backslashes need to be the other way round for the NT-like bits. \\.\pipe\samr. this is an NT convention, and breaking it will be a pain in the neck. following this convention will also make it easier to use [the proposed] apr_named_pipe_create() and apr_named_pipe_transact(). ..and so on. I guess any other namespace is a reference to a file and should be passed by default to smb. i was wondering. might it be better to put real functionality behind srvsvcd? after all, what you are describing is srvsvcd's job. srvsvcd's job is also to handle the dependencies for service startup. we could look at the linux kernel 'Calculating module dependencies' code to get an algorithm to work out the startup order. Then if anyone wrote a windows program to send Hello World to the named pipe 'foo' on a TNG machine then it would appear on stdin when tngd launched the command /opt/samba-tng/foo //./pipe/foo. The only downside I can see with this is that it requires one process per pipe access from each client. yes. i honestly don't see this as a downside. if it's a really heavy downside, well, then the implementor of the daemon must do a threaded architecture, or a single-process model, rather than the existing 'fork' one. we _really_ need to use APR. But then again with a marshalling/unmarshalling library the programs should be very lightweight and could potentially terminate as soon as the RPC is serviced. correct. all dce/rpc services - all the good ones, anyway [so that *doesn't* include exchange, that is _so_ badly designed or rather it's the way it's used that is bad, not the interfaces themselves ] - are designed *knowing* that a function will take of the order of milliseconds or tens of milliseconds to get a response. worrying about delays by forking, memcpys etc, is really, _really_ not going to help you [okay, unless your server is hammered by thousands or tens of thousands of DCE/RPC connections, _then_ you have to worry about it :)] if you are used to optimising code that deals with 2nd-level cache but must operate out of a hard disk and that optimisation looks pretty pointless, then now extend that to optimising code to deal with a 2nd-level cache on your processor but it's actually going over a network, and you get some idea of how ridiculous it is to consider optimising DCE/RPC code for speed :) about 4 orders of magnitude away! totally different ball game, totally different programming techniques. Of course, the advantages of this would be that some changes could be made without recomiling TNG, e.g. changing the //./pipe/samr entry to point to an LDAP-aware version. Also additional RPCs could be easily added, e.g. winpopup could be handled by a //./mailslot/messgnr entry. correct! that was the *whole* point of the TNG architecture. mister dr andrew tridgell, bless him, knows better [obviously]. but that's his problem to deal with. Knowing all the people on this list, this has probably been already suggested, yes. but then dismissed for a very good reason no, it wasn't. there are in fact four totally separate samrd daemons in various stages of completion / completeness. you must run only one of them. all of them use [listen on] \\pipe\samr in a ux-dom-sock. the TNG ux-dom-sock interface, which is effectively functionally identical and i _mean_ identical to NT Named Pipes, right down to a means to transfer NT security context between the client and server, is the dividing line to ensure no other components / services are affected by an implementation _of_ another component / service. so yes, you got it, mark :) luke
fwd: [Core-Dev] for future reference: secure cvs
this may be of interest to people, so i'm fowarding it here, from joe little. I saw this today and wanted to post it here for future reference. It is likely of great interest to the members here... http://www.idealx.org/en/doc/chrooted-ssh-cvs-server/chrooted-ssh-cvs-server_monobloc.html This Howto describes how to do secure CVS via SSH with chroots and multiple repositories. It allows for both internal and external developers, limiting external developers so that they don't need to have shell access to the host. It helps solve problems with pserver ___ Core-Dev mailing list [EMAIL PROTECTED] http://open-it.org/mailman/listinfo/core-dev - End forwarded message -
Re: More migration of code from httpd to apr-util
You miss my point. We at _least_ need to return a Windows Acceptable Pipe Name, instead of some /PIPE/pluming style name. it is *important* that NT-style conventions are followed. the 'guiding light' *must* be the NT functions. i know that most of you may find this difficult to accept that microsoft may be able to develop an API that's really useful, and may feel 'tempted' to 'do better' by imposing 'unixy style' conventions. please don't. pick the NT functions that are needed, then *make* the apr/unix/xxx.[ch] functions do *exactly* the same job [and i will quite happily write that code, btw: i already have most of the components needed to do it]. you may be used to doing this the other way round: making NT do what unix does. please don't. and if that means creating files like /tmp/.apr-socket/\PIPE\samr with backslashes in them and i _know_ this _is_ possible to do on unix, and i _know_ that there are no problems with doing so because winbindd does it [i have a directory /home/DOMAIN\administrator on my linux hard disk and even an entry in /etc/passwd to match it] then so be it. luke
Re: cvs commit: apr/memory/unix apr_sms_blocks.c
is there a pointer-type-cast-and-add macro around?
#define PTR_OFFS(ptr, offset) (void*) (((size_t)((char*)ptr)) + offset)
#define PTR_DIFF(ptr1, ptr2) (size_t) (((size_t)((char*)ptr1)) +
((size_t)((char*)ptr2)) )
did i get that right?
luke
On Thu, Jun 14, 2001 at 01:58:44PM +0200, Sander Striker wrote:
Hi Jeff,
Could you back this one out and change the types in
apr_sms_blocks_t to char *?
struct apr_sms_blocks_t
{
apr_sms_theader;
apr_size_t block_sz;
char*ptr;
char*endp;
block_t *free_list;
block_t *external_list;
apr_lock_t *lock;
};
Sander
trawick 01/06/14 04:44:04
Modified:memory/unix apr_sms_blocks.c
Log:
can't add to void *; pretend it is char *
Revision ChangesPath
1.4 +2 -2 apr/memory/unix/apr_sms_blocks.c
Index: apr_sms_blocks.c
===
RCS file: /home/cvs/apr/memory/unix/apr_sms_blocks.c,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -r1.3 -r1.4
--- apr_sms_blocks.c 2001/06/14 10:59:18 1.3
+++ apr_sms_blocks.c 2001/06/14 11:44:03 1.4
@@ -113,7 +113,7 @@
}
mem = BLOCKS_T(sms)-ptr;
-BLOCKS_T(sms)-ptr += BLOCKS_T(sms)-block_sz;
+BLOCKS_T(sms)-ptr = (char *)(BLOCKS_T(sms)-ptr) +
BLOCKS_T(sms)-block_sz;
if (BLOCKS_T(sms)-ptr BLOCKS_T(sms)-endp)
return NULL;
@@ -135,7 +135,7 @@
}
mem = BLOCKS_T(sms)-ptr;
-BLOCKS_T(sms)-ptr += BLOCKS_T(sms)-block_sz;
+BLOCKS_T(sms)-ptr = (char *)(BLOCKS_T(sms)-ptr) +
BLOCKS_T(sms)-block_sz;
if (BLOCKS_T(sms)-ptr BLOCKS_T(sms)-endp)
return NULL;
Re: [RFC] Network Abstraction Layer - redux
But can we, absolutely, postively, table any change that affects an httpd 2.0 release? ack! agree! :)
Re: apr unicode-16 lib.
On Tue, Jun 12, 2001 at 11:46:30AM -0500, William A. Rowe, Jr. wrote: From: Luke Kenneth Casson Leighton [EMAIL PROTECTED] Sent: Tuesday, June 12, 2001 10:22 AM for various reasons i am prompted to ask, how would the idea of having an apr_ucs16 set of routines, apr_wstrcat, apr_wstrcpy, apr_wtolower, apr_wtoupper etc., be received? Well, since apr_isfoo apr_tofoo was 'reinvented', I don't see a huge problem. cool. on nt, it's easy: straightforward usage of the NT wstrcat, wstrcpy etc. lines. These are the folks who never read the Security Implications of ucs-8 leaving 40% of all IIS webservers still vulnerable, so I'm dubious :-) *grin*. btw, samba #defines strcpy to ERROR_USE_SAFE_STRCPY_INSTEAD etc. sorry, forgot about this. okay, rewrite that: how about an equivalent apr_pwstrcat, apr_pwstrcpy with all the safety / security / paranoia therein? Well, how about a simple question. Why restrain ourselves to ucs2? because it's what NT has: NT doesn't have 32-bit (ucs4?) unicode, afaik, only 16-bit (ucs2?) writing your own ucs4 library, forget it, might as well adopt the glib one. but iirc, the glib one _only_ does ucs4, not ucs2. (No such thing as ucs16/32, it's ucs2/4). ack. Can iconv/apr_iconv provide this in a charset-opaque manner? That is, if I want three 'characters' in shift-jis, can it give me the right number of bytes? The reason is simple, Unicode is already splintered into a multi-word character set anyways. I suspect it's easier to just get it right, knowing the apr_xlate that's been opened, and asking for the char len v.s. the byte len (sizeof) and providing the strcpy/cmp, etc. you need to be able to wtoupper, wtolower etc. that requires a lookup table. samba has an optimised lookup table of the standard ucs2 upper/lower conversion tables that is small enough to fit into the 2nd-level cache of an intel processor. luke
Re: apr unicode-16 lib.
On Wed, Jun 13, 2001 at 09:57:41AM -0500, William A. Rowe, Jr. wrote: Then let's not start adding things willy nilly. We have apr_iconv due to portability, let's build upon that. It should be across character sets, so we can handle this stuff in an opaque manner. ack. i don't mind. as long as there's something that can be used as the basis to write an APR-based SMB server, and it's capable of handling ucs2 in intel-native format off-the-wire. [i can auto-generate some code to do the conversion, it doesn't matter what the internal format is in, ultimately, as long as no information is lost, and there's a secondary consideration to speed. samba is full of code that converts ucs2 to ascii by dropping the high byte.] that's the driving factor, here. lukes
Re: More migration of code from httpd to apr-util
If the server spawns the programs, then you can also use regular pipes. I have been thinking of creating apr/rpc/... to handle stuff like this. However, right now, we have named pipes. They need to be implemented on more platforms, and that may require changing the API a bit, but please let's stick with what we have already. The only thing we can't do with named pipes today is communicate with different machines. IMHO, calling any cross-machine communication medium a named pipe is just going to confuse any unix programmer. Give it a different name. hiya ryan, i promised i'd get back to you after having had to leave quickly. RPC. okay. building RPC directly into apr libs is something i can strongly recommend you don't do: it could easily get out of hand, make the code bulky, and most people would wonder what drugs you were taking today [i.e. it goes against what i am beginning to understand to be the APR ethos]. however, what i _can_ recommend is a slim wrapper - a *local* named-pipe (ux-dom-sock-based, tcp-loopback-based, NT-NamedPipe-based, doesn't matter what the implementation is) that feeds to a separate program that is responsible for locating and communicating remote procedures. a bit like the epmapper on dce/rpc but also a transport as well. it will take a bit of work to design, however what you end up with is a very small library that apps have to link with, they also have to run this [entirely separate] program, and voila you have RPC. now, if you're concerned about this design, well... uh... how do you think NT provides RPC? :) :) :) worse than that, they put these mechanisms *in the kernel*! [services running as System, and of course they _have_ to be optimised and of _course_ that means running parts in the kernel, agh!] [i.e., if you're concerned, be concerned about using NT :) ] all best, lukes
Re: More migration of code from httpd to apr-util
On Mon, Jun 11, 2001 at 10:33:17AM -0500, William A. Rowe, Jr. wrote: Ok. Here are the not-so-subtle differences. First, I understand 9x actually supports reading and writing to a named pipe created on an NT/2000 machine, but they don't support them internally. And they are always blocking. Second, the naming conventions are entirely different. On win32, the name is \\.\pipe\somepipename while on unix (if I understand it) any directory can 'host' a named pipe. On unix, my pipe can be apache2.0/connectors/foo, while on win32 it is all within that 'pipe' pseudofolder. On Win32, we can actually open the \\somemachine\pipe\somepipename from either 9x or WinNT, but the blocking features change. All this implies a common support for 'flat' entries (no directory tree, or the caviat that the 'path' will be ignored on win32.) It implies one of; an alternate file open call, a 'open pipe' flag bit, or returning the handles on open. Which is also a requirement. When we call namedpipe_create, we have to RETURN SOMETHING! Win32 will create a pipe handle (not the same as the read/write file handle.) Every (NT/2000) machine could Create or Connect to get that pipe handle. But once that pipe handle is closed, the pipe evaporates, they are not persistant. So some semantics change. We don't 'rm' a pipe when we are finished with it, we just close the last handle. And pipes on win32 carry a 'suggested size', but the default is a paultry 8kb. the reasons for this are historical, and to do with the fact that the implementation uses SMB [even on loopback, i believe] on top of this meagre data size, you can add your own protocol that provides you with more data, and that has the added benefit of guaranteeing your communication and latency timing without clogging up the network. If this is what we want to use, let's start talking about what api we want. I'm happy to propose the changes to the _namedpipe_ functions that will get this working in the next day or two on Win32. the basic functionality that i really need is to be able to do 'messages'. definition of messages: - send some data on a pipe, it is guaranteed to all be transferred. followed by: - receive some data from the pipe, even if the data received is zero bytes [yes, this _is_ important], and it is guaranteed to be all transferred, and the amount i am told that is being received _is_ received. i.e. on NT, TransactNamedPipe. on ux-dom-sock, this can be achieved by sending / receiving a 4-byte header indicating the size of the data to be transferred. even if that size (send or receive) is zero bytes, the transaction *has* occurred, which is communication in itself [hi, please put this in a local file, and acknowledge it by sending me back zero bytes to indicate that the transaction is completed]. i'm labouring the point, here, i'll shut up now. The only thing we can't do with named pipes today is communicate with different machines. IMHO, calling any cross-machine communication medium a named pipe is just going to confuse any unix programmer. Give it a different name. No, it effectively is a named pipe. They *really* don't differ all that much. The differences are in the naming rules, and Win9x compatibility. Implementors are just going to have to accept that 9x aren't Operating Systems in today's sense, but consumer appliances/gaming consoles, seconded! 9x is a 'program-running-program' where the people developing it burn out in 6 months, leave after 2 years MAX. NT is a mt-os where the people developing it stay with MS for an average of 8 years, most of them are now millionaires and are there out of a sense of responsibility and duty to develop the best OS they can. pick your poison... luke
apr unicode-16 lib.
for various reasons i am prompted to ask, how would the idea of having an apr_ucs16 set of routines, apr_wstrcat, apr_wstrcpy, apr_wtolower, apr_wtoupper etc., be received? on nt, it's easy: straightforward usage of the NT wstrcat, wstrcpy etc. lines. on unix, it's slightly more tricky, but easily doable. [and example code exists in samba, anyway: they've tried it there, but never yet completed it satisfactorily] iirc, glib has a unicode library, however it is ucs32 not ucs16, and depends on glib, which is an N-mbytes install, and not what i need, iow. how about it? :) luke
Re: mem locking (was: Re: cvs commit: apr/memory/unix apr_sms.c)
On Mon, Jun 11, 2001 at 11:45:07AM +0100, David Reid wrote: This thread is getting rather long, but I don't see the motivation. What is the requirement for this range locking? multiple threads accessing same memory range. see tdb for the motivation. tdb is gdbm-like simultaneous reader, simultaneous WRITER [unlike all of the other *dbm databases, which have funny things like, if you write to this db, all bets are off. ??? :)] db-access, with optional mmap support. the implementation for the data-access is hash-chains. each hash-chain is *individually* locked, with a byte-range posix lock. they have some intriguing double-dealing with enumeration, and are not recommending the use of the first/next as a result, but recommend using the tdb_traverse() instead. anyway. if an sms-implementor of an mmap or thread-accessed memory module decided to optimise it by having the admin-memory byte-range-locked, that should be their choice to be able to do that. by not providing an sms-range_lock() fn, or equiv., that optimisation possibility is pretty much guaranteed ruled out. in a stackeable manner, anyway. We don't have to do everything imaginable simply because we can. Every time we load features into APR(UTIL), that just means we need to maintain them. Why do we *need* this feature? And will it be widely useful? do you think that allowing [sms] Developers to optimise memory-administrative management is good? and if so, does that help justify providing range locking in sms? I'm not seeing it. that's because i didn't explain/justify why i was recommending it. which would help :) luke
Re: More migration of code from httpd to apr-util
On Mon, Jun 11, 2001 at 03:04:10AM -0700, Greg Stein wrote: On Mon, Jun 11, 2001 at 12:38:23AM +0200, Luke Kenneth Casson Leighton wrote: On Sat, Jun 09, 2001 at 08:30:23AM -0700, [EMAIL PROTECTED] wrote: ... BTW, APR already has named pipes. Just what are we trying to solve here? We only have them implemented for Unix. OS/2 returns an ENOTIMPL and Windows simply doesn't have an implementation for it. i looked at the named pipe code: i believe that you are thinking of 'unix' named pipes, which are a totally diffrerent beast from nt named pipes. Right. I don't recall all the bits about creating named pipes in Windows, but it may be feasible to use the filename in the apr_file_namedpipe_create function for the host/name of the pipe in NT. In other words, it may be a simple matter of writing the function for Windows. But then again, the function params may not be rich enough for what is needed. ... - a means to set up a server and have other programs (not remote programs) connect to and communicate with that server. the semantics must be identical to those of unix-domain-sockets, namely a listen, bind, accept, read, write and close. I would think you could use named pipes, don't know anything about them: are they portable? TCP sockets, known to be slow, even on 127.0.0.1 loopback. or Unix sockets. not portable. [would do the trick, imo, for a ux-impl. of NT NamedPipes.] If the server spawns the programs, then you can also use regular pipes. - a means to transfer security context between the server and the 'other programs'. i.e. if the client program is running as 'foo', then it must be possible for the server to be 'told' this - _if_ it needs to know. I do not believe there is a way in any OS to say what *program* you are the program is irrelevant: it's the named pipe that's important. exactly in the same way as in ux-dom-socks, the program is irrelevant [i have to say, i don't quite follow what you mean or think i meant that prompted you to mention 'programs, here: perhaps you could elaborate so we don't mis-communicate?] e.g i will call pp = apr_create_np(\PIPE\xvl\xmlvl.org, pool) and it will create, on ux, a domsock /tmp/.apr/PIPE\xvl\xmlvl.org. i will then call apr_bind(pp), apr_listen(pp) etc blah blah. i will then, in a client-program (mod_virgule), call apr_open_np( \PIPE\xvl\xmlvl.org) etc. which opens me a connection, causes the server to call apr_connect() blah blah, in all the usual ways. now, on NT, of course, this will not work: NT doesn't _have_ ux-dom-sock. but it _does_ have NamedPipes. and i expect that using them could provide exactly the same functionality [as described in above example]. running (in an authenticated fashion). Reading man pages, it appears Linux 2.2 added a way to send another process the PID, UID, and GID. But that would obviously be non-portable. the technique used in samba TNG's IPC mechanism [which is ux-dom-sock-based], is to transfer the uid, gid and ux-groups [plus the NT auth equivalents as well] over the socket as ... 'out-of-band' data at setup time. this info is recorded in a state-structure that can be obtained by a get_sec_ctxt() -get security context - function. so, the impl. would be, if you detect that lx 2.2 supports transfer like this, use it. if not, send the uid, gid, ux-groups at setup time. simple :) [btw, it so happens that the pid is also transferred, but that's for other reasons that have nothing to do with emulating NamedPipes, it's to do with SMB and the implementation of Samba TNG.] all best, luke
Re: mem locking (was: Re: cvs commit: apr/memory/unix apr_sms.c)
or, you mandate providing locking, emulating regions by ... errr providing ref-counting on a global lock (yuck!) hm, that don't work. nope: have to do the lock and lock_region, then have a means for Users to detect that a lock_region function is available / supported. you know, this is all tending to suggest having a 'apr_sms_get_features()' fn, returning a bitfield of supported features. You can emulate regions with global locks and shared memory, of course. And still tell people which you support natively. Don't know whether its worth it - does anyone have any views? hiya ben, well... yes, i do: i wouldn't recommend making anything such as emulation of range locking 'mandatory' in sms. it's quite complex, i guarantee you: see samba's emulation of NT byte-locking on top of posix range locks! given that... okay, it's a long story. trust me: you don't ever want to try it :) :) anyway - the point of sms is KISS. to the extent that we're having quite complex discussions right now, as experienced experts and developers, to stave off and help justify _not_ adding in complexity. the fine line between simplicity/complexity and necessary/nice-to-have _can_ be walked :) so, my recommendation is, make range_locking() either non-existent [favoured by some because i didn't mention a justification for adding it, *duuh* :)] or optional. i mean, to be honest, the number of smses that are likely to _use_ range_locking() is probably going to be quite small, although i can't speak for the people whose job it is to worry about speedspeedspeedoptimiseoptimiseoptimise in apr/httpd, esp. in the area memory handling :) i only recommend range_locking() as a possible way to provide that speedoptimising if it's thought necessary. otherwise, it doesn't bother me particularly if it's not added. all best ben, luke
Re: More migration of code from httpd to apr-util
On Mon, Jun 11, 2001 at 08:04:53AM -0700, [EMAIL PROTECTED] wrote: BTW, APR already has named pipes. Just what are we trying to solve here? We only have them implemented for Unix. OS/2 returns an ENOTIMPL and Windows simply doesn't have an implementation for it. So, the solution is to implement them on other platforms, not to re-invent them. For historical completeness however, allow me to explain how we got where we are with named pipes. Unix has them becaues they are easy to implement. Windows doesn't have them because only certain versions of Windows supports named pipes. Namely, NT and 2000 have named pipes, 9x doesn't. grep NamedPipes */*.c shows some code that uses NPs. there is a nice comment about 9x not having some feature [blocking or something], and it says, well, uhhh TOUGH! :) so, there does already exist code in apr that uses CreateNamedPipe() - for different purposes. Originally, APR supported named pipes on Windows, great!!! they were removed in revision 1.11. From looking at the log, I don't understand why. *curious*. i looked at the named pipe code: i believe that you are thinking of 'unix' named pipes, which are a totally diffrerent beast from nt named pipes. Right. I don't recall all the bits about creating named pipes in Windows, but it may be feasible to use the filename in the apr_file_namedpipe_create function for the host/name of the pipe in NT. In other words, it may be a simple matter of writing the function for Windows. But then again, the function params may not be rich enough for what is needed. See above, this has worked in the past, all we need to do is copy the code back into place, and possibly tweak it. yesplease! less work! ... - a means to set up a server and have other programs (not remote programs) connect to and communicate with that server. the semantics must be identical to those of unix-domain-sockets, namely a listen, bind, accept, read, write and close. I would think you could use named pipes, TCP sockets, or Unix sockets. If the server spawns the programs, then you can also use regular pipes. I have been thinking of creating apr/rpc/... to handle stuff like this. However, right now, we have named pipes. They need to be implemented on more platforms, and that may require changing the API a bit, but please let's stick with what we have already. The only thing we can't do with named pipes today is communicate with different machines. IMHO, calling any cross-machine communication medium a named pipe is just going to confuse any unix programmer. Give it a different name. okay. i am familiar with how remote IPC works, from a few perspectives, having implemented a subset of DCE/RPC that is MS-compatible, over SMB, for Samba (ISBN 1578701503). [eek, have bus to catch soon, gotta run]. i can attest to the complexity of implementing such: it's not something to add to a simple library like APR without a *lot* of prior thought - an implementation may be about 7,000 to 10,000 LOC - not trivial. gotta run, more later. all best, luke
Re: apr-util, crypto and openssl
On Sat, Jun 09, 2001 at 06:38:58PM +0100, Ben Laurie wrote: Luke Kenneth Casson Leighton wrote: Well, I'm already questioning the whole MD4 thing. I can see that Samba needs it, but who else? I'm not entirely sure why we put that into APRUTIL because I'm not sure who *else* might want to have it there. Can anybody name one or two other apps that use MD4? rsync. librsync. Blech! *cackle*. forgot: so that includes mod_rproxy (which uses librsync). lukes
Re: apr-util, crypto and openssl
rsync. librsync. Blech! *cackle*. forgot: so that includes mod_rproxy (which uses librsync). Hmm. Any chance that those apps would switch to use APR(UTIL)? (and thus, our MD4) sander would like to do rsync2. martin pool (a known apache contributer) wrote mod_rproxy: he'd have to be asked. For new apps, why would somebody choose MD4 over MD5 or SHA1? Are there particular benefits to using it? md4 is 16 times faster, but cryptographically (academically) less secure. don't know the theory, though. luke
Re: cvs commit: apr/memory/unix apr_sms.c
On Sat, Jun 09, 2001 at 06:34:21PM +0100, Ben Laurie wrote: Luke Kenneth Casson Leighton wrote: uh... i thought i should point out: the original proposal i recommended for the sms locking routine should pass in the area of memory to be locked and its length, and ditto for the unlocking. +1. in this way, let's say you do mmap on a file, you can use posix locking as the implementation of the locking on the memory. [or can you? i dunno, for sure :) ] I believe so. and, if the argument is NULL, it means ' do a Global Lock'. if the unlock() argument is NULL, it means 'do a Global Unlock - i.e. free all locks'. And if the locking mechanism doesn't support regions? (This is particularly important if you allow unlocking of subregions). The obvious answer being that we implement them in APR, of course. well, that is what made me think that perhaps a separate lock and lock_region be done, such that one or other cold be supported. or, you mandate providing locking, emulating regions by ... errr providing ref-counting on a global lock (yuck!) hm, that don't work. nope: have to do the lock and lock_region, then have a means for Users to detect that a lock_region function is available / supported. you know, this is all tending to suggest having a 'apr_sms_get_features()' fn, returning a bitfield of supported features. ... ? luke
Re: More migration of code from httpd to apr-util
On Fri, Jun 08, 2001 at 11:37:46AM -0700, Greg Stein wrote: On Fri, Jun 08, 2001 at 10:52:34AM -0700, Justin Erenkrantz wrote: On Fri, Jun 08, 2001 at 10:47:10AM -0700, dean gaudet wrote: On Fri, 8 Jun 2001, Luke Kenneth Casson Leighton wrote: what i perhaps should recommend is that a series of independent libraries be created. e.g. libaprhttputil (i know it's a bit long). why? because i forgot about te extract only the needed bits thing :)
Re: More migration of code from httpd to apr-util
btw, this message goes primarily to the developers of mod_cgid, but is related to the apr_create_named_pipe proposal. my question is, does mod_cgid compile under win32, and would you like it to? so far i count 2 immediate projects that need an apr_create_named_pipe (xvl and mod_cgid) and two long-term ones (tng and dce/rpc). an apr_wait_named_pipe_handle_state() which emulates .h.. WaitNamedPipeHandle will also be needed. i know how to do this in unix-domain-socket land, i'm eager to hear if anyone wants to do the win32 version.[btw plare please remember i'm not subscribed to new-httpd, thx!] luke
Re: More migration of code from httpd to apr-util
my preference is to do the unix-implementation of the apr_named_pipe API, if there's anyone else willing to help parallel-develop the unix implementation, using xvl as a test to prove the case. I'd certainly look over the code, but I'm not terribly certain if I'd use it. I'd just have to see and evaluate if named pipes make sense in my cases. If it does in yours, great. -- justin ack. se my other post/question: do you want mod_cgid to work on win32? :) :) all best, luke
Re: SMS stuff
Um. Hello? Have you counted the number of third party modules in existence? Tossing the pools means tossing the basic framework for a couple hundred modules. apr_pool_t and apr_p* will remain (effectively) forever. hiya greg, you know that, i know that, we all know that, and so can cater for it: it means providing a set of functions that provides exactly the same job, exactly the same semantics, behaviour and function arguments. that's the _easy_ bit :) no-one in their right mind is suggesting, not here, not ever, never has, [but probably will, despite all the messages on the topic], advocated doing anything that would require third party modules to modify their code. it's just simply not.. going... to happen. so we deal with that and take it into account in the design and implementation. agreed? thx, luke
Re: SMS stuff
Greg's argument (and I'm leaning that way) says 'pools are now widely deployed'. Grow a pool into an sms, don't break anyone's code along the way (by making the default sms our beloved pool schema) and we are in strong shape. yaaay :) [btw, i'm sorry i got so stressed yesterday: seeing so many messages that looked like they had the wrong end of the stick, i threw progressively harder words about as i read more and more. sorry for any fingers burnt. luke]
Re: cvs commit: apr/threadproc/unix thread.c
On Thu, Jun 07, 2001 at 09:27:00AM -0700, Justin Erenkrantz wrote: On Thu, Jun 07, 2001 at 03:54:24PM +0200, Luke Kenneth Casson Leighton wrote: only an SMS instance should know exactly how to deal with its memory - _including_ locking - IF it is needed. Ah. I see the logic in that. Thanks for explaining. =) np. Yes, the SMS should be the only one dealing with locking. I'm sold on that now. cool. *happy*. Late for class now. -- justin eek! :)
Re: SMS stuff
Here is why I thought that: David Reid wrote: When we're done we'll have locks - apr_sms_t sms - locks - apr_sms_t So personally I don't see the problem and thus I made the change! I guess maybe it's because people keep saying that we're going to change the pools to use sms. Why? To get the maximum flexibility we'll need to use sms throughout so while we may have apr_pstrdup(apr_pool_t *pool, char *str) we'll end up with apr_pstrdup(apr_sms_t *mem_sys, char *str) So no wonder I'm a bit concerned. ack. ... okay. doing a half-way-house isn't going to satisfy you, me, or anybody. david clarified in a later post, on the above. sander's written up a roadmap that outlines what me, sander and david agree on, and it takes into consideration everyone's concerns: that's my take on it, anyway. I haven't complained about the code one bit. Yes, it is simple, but reading it isn't change what I'm talking about... I have an issue with what we're going to *do* with it. And given that I believe SMS with providing storage for a pool, ack. then I question why we have a pool underneath an SMS. me, too, don't worry! :) okay, there are two approaches to do SMS with providing storage for a pool 1) implement apr_pool.c to call functions in apr_sms 2) provide exactly the same functionality and then #define. if 1) just becomes a thin wrapper, then it's pointless function overhead and you might as well do 2) _anyway_! luke
Re: cvs commit: apr/threadproc/unix thread.c
On Thu, Jun 07, 2001 at 08:03:46AM -0700, [EMAIL PROTECTED] wrote: As it happens, this is exactly what I envisioned when SMS was first discussed. Namely, a new back-end for pools that can use any allocator. ack. sorry for thinking otherwise. luke
Re: apr-util, crypto and openssl
Well, I'm already questioning the whole MD4 thing. I can see that Samba needs it, but who else? I'm not entirely sure why we put that into APRUTIL because I'm not sure who *else* might want to have it there. Can anybody name one or two other apps that use MD4? rsync. librsync. PS. I do have a crc32 patch laying about (isn't in openssl or apr(-util) and it is pretty generic. I think it has some use in projects other than ours, but I'm not sure). Interested? I haven't seen a need for it, so no. I'd prefer to at least have some *requirements* for stuff like this. And preferably from more than one application. rsync uses crc. well, actually, it uses a rolling crc - a ... oh, bugger, what's it called... adler! adler32. [sander and i] have some plans to do an advanced librsync: there are improvements that can be made to the rsync algorithm. and the codebase we'd like to use to do it is, surprise-surprise: apr. luke
Re: XMLvL, apache, apr, mod_virgule origins: advice needed [LONG]
On Thu, Jun 07, 2001 at 02:14:58AM -0700, Greg Stein wrote: On Wed, Jun 06, 2001 at 04:28:43PM +0200, Luke Kenneth Casson Leighton wrote: ... it doesn't use apache authentication, in fact there's a whole boat-load of stuff it can't use. it provides its own authentication, from cookies stored in xml-based user-profile files. Side question: why don't you write mod_auth_FOO to authenticate against those files? Then you wouldn't need to bother with authentication in xvl. Users could also swap in/out their own authentication as they desired. Or, of course, use the default mod_auth_FOO mechanism. ... the xvl code (and the original mod_virgule) have zero hooks back to using other mod_xxxes. yes, i would dearly love to be able to hook into other authentication mechanisms, and to that end i have already split authentication from authorisation (acct.c and profile.c) and user credentials and profile information. once i have a clear idea of how to proceed to hook into mod_auth_xxxes, i'll probably do it real quick :) so. any suggestions? what components already exist? Well, you obviously have an entire httpd server to grab code from. yep!!! But in terms of isolated components? Nothing beyond APR and APRUTIL. ack. if i make a [rather short] list of functions i am literally duplicating line-for-line, would that suffice as motivation to, say, add them to aprutil? ahh, heck, why not just cp the file over now :) these are the functions that i am duplicating - verbatim - from httpd. needless to say, duplicating code makes me very unhappy, and if anything can be done about that, i'd be willing to help make it happen. API_EXPORT(char *) ap_escape_html(apr_pool_t *p, const char *s); API_EXPORT(char *) ap_os_escape_path(apr_pool_t *p, const char *path, int partial); API_EXPORT(char *) ap_make_full_path(apr_pool_t *a, const char *src1, const char *src2); API_EXPORT(char *) ap_ht_time(apr_pool_t *p, time_t t, const char *fmt, int gmt); API_EXPORT(char *) ap_gm_timestr_822(apr_pool_t *p, time_t sec); API_EXPORT(char *) ap_getword(apr_pool_t *atrans, const char **line, char stop); API_EXPORT(int) ap_count_dirs(const char *path); API_EXPORT(char *) ap_make_dirstr_prefix(char *d, const char *s, int n); API_EXPORT(char *) ap_make_dirstr_parent(apr_pool_t *p, const char *s); API_EXPORT(int) ap_regexec(const regex_t *preg, const char *string, size_t nmatch, regmatch_t pmatch[], int eflags); API_EXPORT(size_t) ap_regerror(int errcode, const regex_t *preg, char *errbuf, size_t errbuf_size); API_EXPORT(char *) ap_pregsub(apr_pool_t *p, const char *input, const char *source, size_t nmatch, regmatch_t pmatch[]); API_EXPORT(void) ap_str_tolower(char *str); i'm using a brain-dead installation of mandrake 7.0 with glibc 2.2 (so i get error, cannot find symbol dl_init_next@@GLIBC_2_0 whenever i compile a shared library with -ldl, that includes libxml2, apr, pretty much damn well everything: anyone any clues? i'm installing a lot of RPMs recently to overcome this, _when_ they're available...) Hmm. No clue on that link error. Never seen it. Of course, I have glibc 2.1 on my box. Have you tried dropping the -ldl? Maybe 2.2 includes that already. yes i tried dropping -ldl: it says error, cannot find dlopen [and friends] at link-time instead. i have found a temporary solution: compile everything with ./configure --enable-shared=no which doesn't exactly make me very happy, but there y'go. i shouldn't be so damn stupid as to stuff up my development machine, _should_ i? :) xvl uses the ap_pool code, the table code, list code, ap_psprintf, i need to exec / spawn programs, ideally i also need a portable version of unix-domain-sockets, which i understand doesn't exist in APR, yet. APR does Unix domain sockets. We use them in mod_cgid.c. hooray! i tried glib for this job, a while back, but its 'object-orientated' nature just... got in the way :) APRUTIL also locates Expat on the system for you, or builds its bundled copy of Expat. You could drop your libxml2 dependency. ah, uh... libxml2 is a thorougly harsh dependency in xvl. when i say harsh, i mean, in a 12,000-line codebase there are 1,000 occurrences of lines with xml in it [grep xml *.c */*.c | wc] one of the modules - the socket module - uses the SAX2 interface in such a way so as to provide an easily intuitive [and not original] way to make data transfer look like XML that took *four weeks* to implement. i really don't have the kind of time available on my hands that took up so much concentration to get that stupid code right. Even better, there has been a call for a SAX-like API in APRUTIL, backed by Expat, libxml, or Xerces. If you're motivated, you could implement that API for APRUTIL :-) :) interesting concept. would keep _me_ happier too because i'm not entirely ok with the clash between libxml
Re: SMS stuff
Well, we'll agree to differ on these. The basic problem is that they're all the same method for allocating memory, all that varies is where the memory comes from. Hell, if that's all we want to do then why are we bothering with sms? sms adds a lot of baggage if all we really want is to be able to slot in a backend! remember that one of the suggestions i made was to create a mem-locked SMS, for security of encryption / storage of private keys (using techniques / code already in GPG). the basic premise is that pretty much all code uses pools. therefore, in order to fulfil the requirements, you back-end pools with the appropriate sms as it's a self-contained location in the code dependency-order. luke
Re: cvs commit: apr/threadproc/unix thread.c
That implies that somebody has to alloc a lock outside, but we know that we can do that: an SMS around malloc(), a pool around that, then a lock using that pool. We can then pass that lock to SMS(my-custom-bugger). this lock would be passed in as an argument to the initialisation / creation of an SMS instance, yes? oh, and btw, i'm glad to see that someone else gets this sms stuff :) luke
Re: cvs commit: apr/threadproc/unix thread.c
On Wed, Jun 06, 2001 at 05:35:43PM -0700, Justin Erenkrantz wrote: On Thu, Jun 07, 2001 at 01:19:27AM +0100, David Reid wrote: This isn't possible with pools. They limit you to a single way of getting at your memory regardless of how it was obtained. What I basically said in my email to David was that I could see having a pool take in an option as to which SMS it should use. I've said this before a few weeks ago (probably when I first looked at the SMS code), but let me reiterate that suggestion. Obviously, there would be a default SMS that would be used, if one isn't provided when the pool is created. I think that this allows the SMS code to be kept simple, while the pool code can be kept relatively similar to what it does now. I'll shut up for a while on this. -- justin :) perhaps i should re-iterate / confirm that what justin says, above, is exactly how i envisioned SMS to be used. more than that, it can even be hidden behind apr_initialise(). the initial call to create the initial pool does not even need to _take_ an extra sms argument: by _default_ the existing apr_pool_create() function just... uses the standard-memory sms. now, if you don't _want_ a pool that use the standard-memory sms, you call, uh... say, apr_pool_create_from_sms() - which is what justin above recommends be actually called apr_pool_create(). i'll shut up too because i'm probably confusing matters by not exactly knowing the total internal workings of apr. yet. luke
Re: SMS stuff
Point out that requirement, and you can change a lot of minds here. But until then, I think you'll continue to see confused/concerned people, not understanding why you are suggesting we toss all of the memory management in APR in favor of SMS. uh, sorry to have to point this out like this, but your understanding of the original plan for SMS usage is plain wrong. i have no idea what makes you think that anyone is suggesting that all mman in APR is 'tossed' in favour of sms, and if anyone else recommends it i will bitch at them persistently until they give a decent justification, or give up. :) _one_ of the suggestions was to create functions in sms that are identical to those of pools except with different names and different types. _one_ [then you add into apr_compat.h #defines to move them to the new types] you don't have to take or do that suggestion, esp. as this is an easy-does-it approach. please, like david did, if you don't like or don't understand the explanations, please read the code. it's really short, it's really simple - and it's short and simple _because_ we [collectively - all of us] have enough experience to realise that anything else will cause us to have nightmares until the code's ripped out and burnt. ritually. :) luke
Re: cvs commit: apr/threadproc/unix thread.c
On Wed, Jun 06, 2001 at 04:34:22PM -0700, Justin Erenkrantz wrote: On Wed, Jun 06, 2001 at 11:47:53PM +0100, David Reid wrote: This is the crux of the issue methinks. We don't yet have a module that would allow us to even get close to replacing pools. We need a lot of things from it and Sander and I have had some good early discussions about how it could work. Basically we want to have a fast, stable tracking allocator that has a smaller memory footprint than pools. Is it possible? I don't honestly know but we're going to give it a good try. Why haven't we opened up our discussions? Because we haven't even got any code and are still bashing around the early design which is probably better done privately. Once we have something we like we'll post. See, I think this is the difference. I see that the pools are on top of sms. (Gee, this is what Cliff said...) The sms doesn't need to know anything about refcounting or anything special. What does refcounting give you? I'm still not also sure why locking needs to be in the SMS. because, by handing over responsibility for locking of memory to SMS, and by having the pools code use SMS, you make the pools code a) simpler b) more powerful. only an SMS instance should know exactly how to deal with its memory - _including_ locking - IF it is needed. as people have already pointed out. if you want the same thing _without_ using SMS, but you still want pools as an interface to shared memory, to mlock()/GlobalLock()ed memory, to this memory, to that memory, to even _files_ implemented as memory - _whatever_, then you must code into pools the capability to do locking of shared memory, locking of this memory, locking of that memory, and it's just such a messy nightmare that no-one in their right mind wishes to consider it, and i _agree_ with them :) (I think I asked for clarification on this, but I received none...) whoops :) does the above help? :) All an sms knows how to do is to get a chunk and free a chunk of memory. uh... not quite. and it knows how to manage it. keep track of it. lock it [_if_ needed: that is up to the implementation]. _and_ the management can be done by using some _other_ sms [parent sms] to use, which may be quicker, better, niftier, sliced-breadier whatever. None of the pool logic needs to be in sms. ??? what code are you looking at? :) you've examined the tracking memory system, yes? and the destroy and reset? I saw that the sms were just an abstraction around allocating memory. correct. The pool will actually handle the cleanups. well, we have to provide cleanups in sms _anyway_. if, as the implementor of pools, you [whoever] choose not to leverage the functionality [sms cleanups] to make the pools code simpler, well, then... that is your choice. Everyone still uses apr_pool_t. that is correct. i would not imagine for one _second_ that this would not be considered unless it could be done as a #define in apr_compat.h, and even _then_ i would, as david recommends, only slate it for some distant future release. The pool itself uses apr_sms_t to allocate memory. This enables us to have a shared memory pool, a file-backed memory pool, a heap-backed memory pool - whatever we want. correct. The sms doesn't need to do any locking - the pool will guarantee that the allocation is done atomically by its own locking mechanisms (what it does now - albeit the pool locking is a bit coarser than it really needs to be). wrong. how can pools know what type of locking is required for each type of sms it _might_ use, without turning pools into a nightmare of spaghetti #ifdefs and #ifthatdefs that the sms API is designed *exactly* to avoid??? see above: i explain further. I think the thing is that I've seen the sms as slightly different than what it was originally posted as. i think that is the case, yes. either that, or we haven't explained it enough. concepts tend to spring from sander and i like fully-formed hydras. getting them across without bashing heads together is an interesting learning experience - for me, at least :) So, I might be in the minority here. I think we are seeing two different views of what an sms should be. i hope not. i'll try to keep an eye / time on these discussions to make sure everyone who _wants_ to know does know. to that end, the idea was floated that perhaps those people who wish to get together head-to-head to discuss /implement this contact me? i have two possible venues in the u.k. where it will be possible to invite up to... 5 people at one venue, and ... 10, maybe? at the other [50 _would_ be pushing it, a bit: all the rooms are empty but it'd be feasible if uncomfortable :)] - some time in july, august or september? [hm, time to get a cable-modem, i think :) ] all best, luke
Re: cvs commit: apr/threadproc/unix thread.c
On Wed, Jun 06, 2001 at 03:51:55PM -0700, Ian Holsman wrote: -Original Message- From: Sander Striker [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 06, 2001 3:13 PM To: Cliff Woolley; David Reid Cc: APR Development List Subject: RE: cvs commit: apr/threadproc/unix thread.c On Wed, 6 Jun 2001, David Reid wrote: There is no pleasing some people is there?? Nope. =-) Argh!!! [getting abit frustrated now, because there seems to be no way of moving forward and thus proving the system] maybe you could take a bit at a time for example.. modify the pool code so that it has a 'sms' memory pointer in the structure, on init/destroy you also clean it up .. then you could takle parts of the apr, for example strings replacing the 'standard' pool memory functions it uses with sms memory calls in their place, as long as the 'standard' and SMS memory have the same lifetime. i think that putting #define apr_pool_t apr_sms_t and #define apr_pool_create apr_sms_pool_create ... etc. stands a much better chance of a) success b) being accepted because a) the developers can write tests that prove that the existing code that uses pools - UNMODIFIED - is unaffected by using apr_sms_pool_create or apr_pool_create whatever-you-like _and_ you can write a test suite that does random / specific memory tests using both systems and comparing the results b) if you don't _like_ the implementation of apr_sms_pool_create and friends, well, uhh.. you remove the #defines in apr_compat.h and you're done: you're back to where you started, no harm done. luke
Re: cvs commit: apr/threadproc/unix thread.c
then you could takle parts of the apr, for example strings replacing the 'standard' pool memory functions it uses with sms memory calls in their place, as long as the 'standard' and SMS memory have the same lifetime. ... i also should point out that your comment leads me to believe that you misunderstand how sms works. can i recommend that, like david did, you read the code, and also read my comments i sent just a fewmins ago as reply to justin's comments? thanks, luke
Re: cvs commit: apr/threadproc/unix thread.c
See, that's where my overall view of where we hope to get to differs. shrug In my mind, APR depends on pools. Period. It would require a major overhaul for most APR operations to be safe WITHOUT pools (ie, lots of apr_sms_free operations would have to be added, which is exactly what the pools are meant to avoid). no, no, _wrong_! that's exactly what you _don't_ do, and anyone who proposes it, or thinks that that is what is being proposed, i will stamp on their fingers or break their keyboard in frustration. or something. no. we are _not_ proposing that pools be replaced [with something other than what looks exactly like pools]. same API [pools], different implementation. doing otherwise is just.. _nuts_! you know that, _i_ know that. what on earth makes you think _i_ think otherwise??? darn :) luke
Re: cvs commit: apr/memory/unix apr_sms.c
uh... i thought i should point out: the original proposal i recommended for the sms locking routine should pass in the area of memory to be locked and its length, and ditto for the unlocking. in this way, let's say you do mmap on a file, you can use posix locking as the implementation of the locking on the memory. [or can you? i dunno, for sure :) ] and, if the argument is NULL, it means ' do a Global Lock'. if the unlock() argument is NULL, it means 'do a Global Unlock - i.e. free all locks'. luke On Wed, Jun 06, 2001 at 05:54:22PM -0400, Cliff Woolley wrote: On 6 Jun 2001 [EMAIL PROTECTED] wrote: If we don't have a lock and unlock function in an sms module then it's not an error as this is allowed. Add a comment to this effect to make it clearer. if (!mem_sys-lock_fn) -return APR_ENOTIMPL; +return APR_SUCCESS; return mem_sys-lock_fn(mem_sys); Ahh, that makes sense. My bad. Thanks, David. --Cliff -- Cliff Woolley [EMAIL PROTECTED] Charlottesville, VA
Re: cvs commit: apr/threadproc/unix thread.c
As for locking, well the logical level for locking to be implemented is in the sms module. Look at the standard module. malloc should be thread safe so no locking should be required, hence we don't have any. In a tracking system we can't guarantee that so we lock. I don't really have a problem with the locking end of things, per se... there will probably be some SMS's that need to lock, so we might as well give them a convenient means to do so. I think. I haven't really looked into this, though. i've been investigating the design: a lock global and lock region (and unlocking equivs) will be needed, even if the sms implementor decides to implement the lock region as a global lock with ref-counting or something stupid. luke
Re: file/mmap buckets, subrequests, pools, 2.0.18
On Wed, Jun 06, 2001 at 06:30:30PM +0200, Sander Striker wrote: what's the profile of the memory allocation / reallocation / deallocation like? sander has written sma - smart memory allocator. it basically allows you to have more control over the block size etc for management etc of memory allocation / reallocation. it's wrapped through the apr_sms memory API [i think!] so you'd be able to use a different mallocator if you decided you didn't like sma. use the tracking sms, or even the simple standard-lib (malloc/free/realloc) sms instead. It isn't an sms yet :-) But if people are interested, I can implement it as an sms. up to you: the sms API framework is there to allow you to experiment and definitively make these kinds of development decisions. Yes, but everything now has a dependency on pools, not on sms. ah: i naturally assumed, but should make it explicity clear, that it should go like this: sma depends on nothing pools depend on sma everything else depends on pools. given this [logical] proviso, the above experimentation is possible. luke
Re: XMLvL, apache, apr, mod_virgule origins: advice needed [LONG]
On Thu, Jun 07, 2001 at 11:19:18AM +0200, Daniel Veillard wrote: On Thu, Jun 07, 2001 at 02:14:58AM -0700, Greg Stein wrote: APRUTIL also locates Expat on the system for you, or builds its bundled copy of Expat. You could drop your libxml2 dependency. Even better, there has been a call for a SAX-like API in APRUTIL, backed by Expat, libxml, or Xerces. If you're motivated, you could implement that API for APRUTIL :-) Actually I doubt mod_virgule would stick at the SAX level. I didn't looked at the code but from the exchanges I had with Raph Levien when he developped it, I'm pretty sure it uses the DOM tree build of libxml2. DOM tree, libxml 1.8.x. the version on http://virgule.sourceforge.net - xvl [a stand-alone executable] - makes heavy usage of the SAX interface in 2.x. i even use one of the _internal_ SAX functions, and i'm praying you don't delete it from the library! Now whether switching to Xerces makes sense or not is a question I'm probably too biased to answer. The irony is that I started libxml in 99 when trying to build a small WebDAV implementation for Apache. funny, neh :)
Re: [PATCH] apr-util hmac md5
raaay, good for sander. HMAC_MD5 is used in NTLMv2 security to help guarantee against replay attacks on different sessions [NTLMv2 doesn't stop replay attacks on the _same_ session :)] HMAC_xxx is used to generate one-way hashes from secret keys and public data, basically. if you have to one-way hash, it's pretty much O(N ^^ -128) likely that you will be able to obtain the secret key. sander, just a thought: would it be possible for to write a general HMAC_xx that accepts an xx? and then HMAC_MD5 being a specialisation of that? or, is it simply worth saying,well, uh, if you're gonna do that, forget it: use openssl. ? On Tue, Jun 05, 2001 at 07:34:33PM +0100, Ben Laurie wrote: Justin Erenkrantz wrote: On Tue, Jun 05, 2001 at 01:54:05AM +0200, Sander Striker wrote: Hi, This patch adds HMAC MD5 to apr-util. Where would we use this? Is this algorithm of sufficient usage that it would benefit being in apr-util? I've never heard of HMAC before - I had to look it up on rfc-editor.org. Maybe I live in a paper bag. Please be assured that you _do_ live in a paper bag. HMACs are good things if you care about security. :-) Cheers, Ben. -- http://www.apache-ssl.org/ben.html There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff
Re: cvs commit: apr/locks/win32 locks.c
The surprise comes in just how far reaching the apr_pool_t goes. please: identify the dependency chain. please. if there's some automated code tools that help do this - i believe that even VC++ may help, here - then all the better. or source navigator, or code warrior, or your own perl scripts, or _something_. please, make an ordered tree of data structures that use-and-depend on apr_pool_t, and then you can decide which bits to target first. if i was working on this project, and i've had experience at doing it the _other_ way with 300,000-line codebases, where everybody needs to know what's going on in order to approve it, this is what i would do. the other way, btw, is just to sit in front of a screen 10x7 for 4 weeks and morph the code into submission [ala extreme programming]. this tends to get results very fast and leaves everyone else gawping - and nervous, 'cos it look _nothing_ like it used to. luke
