Re: cvs commit: apr/network_io/unix inet_ntop.c
On 2 Mar 2001, Jeff Trawick wrote: > [EMAIL PROTECTED] writes: > > > rbb 01/03/02 10:27:42 > > > > Modified:network_io/unix inet_ntop.c > > Log: > > need apr_strings for apr_snprintf. > > ouch... thanks! Yeah 'cause you've _never_ had to fix one of my bugs. :-) Ryan ___ Ryan Bloom [EMAIL PROTECTED] 406 29th St. San Francisco, CA 94131 ---
Re: cvs commit: apr/network_io/unix inet_ntop.c
[EMAIL PROTECTED] writes: > rbb 01/03/02 10:27:42 > > Modified:network_io/unix inet_ntop.c > Log: > need apr_strings for apr_snprintf. ouch... thanks! -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
Re: configure as root?
Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> writes: > question. samba, in its search for info during ./configure-time, fails > certain tests (e.g. the setuid ones) if not run as root. > > same for apr? nope; no such tests -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
Re: [RFC] Network Abstraction Layer
On Fri, Mar 02, 2001 at 02:17:05AM -0800, Greg Stein wrote: [...] > My personal, uninformed opinion :-) would tend towards adding new > OS-provided socket types to apr_socket_t (allowing all apps the benefit of > the new socket types; not just those that can fill in a func table), and [...] That gave me a good hint. I investigate apr_socket_t closely. I like the _external_ API! It's realy quite good. It fits quite exactly, what we need. What I don't like is the "internal" way, because it doesn't allow an application/library to add apr_socket_ts at runtime. I would like something like in the buckets area: Having some apr_socket_type_t with function pointers and the like. apr_send() will then mostly only call the appropiate function pointers and the old "hardcoded" apr_send() will be that fn-ptr. Most of this is quite straightforward to implement and it wont even break any applications (Apache!?), that already depend on the current API, because most is only internal rewriting and exposing some things. The thing, that asks for some proper design is apr_pollfd_t related stuff: sockets might have a filedescriptor, that the base poll()/select() can use, but they might find out, that there actualy is no data, that they could hand over to the application. In the worst case, there might exist sockets, that can't even provide with a nice filedescriptor, some *arg* polling or other good means would be needed. This requires a good thought out API in the apr_socket_type_t-fn-struct. BTW: apr already has some "own" sockets: one can convert an apr_file_t into a socket... I doubt apr_setsockopt (or what it is named) does any good on it... Having a proper apr_socket_type_t apr_file_socket_type_t would help here. Elrond
Re: pool stacking
I've some further notes about pools. The reason, why I asked for refcounting: An object might want to live in two pools. Of course it should only die, if both pools do not anymore reference that object. Of course, the object can do the refcounting itself, pointing the pools destroy-fn-ptr to its unref-fn. Is there a way to remove a object from a pool, without knowing the pool it was added to? Pools somewhat resemble, what I once called "rmm - referencing memory manager", it was somewhat even more general, apr_pool would have been just another object in the whole thing and the like... I'm just curious, if most is there, that I came up with. (it was never implemented, and I doubt it will ever, nor do I know, if it is actually necessary) Elrond
configure as root?
question. samba, in its search for info during ./configure-time, fails certain tests (e.g. the setuid ones) if not run as root. same for apr? - Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> - "i want a world of dreams, run by near-sighted visionaries" "good. that's them sorted out. now, on _this_ world..."
Re: FreeBSD and gnu m4
On Fri, 2 Mar 2001, Greg Stein wrote: > On Thu, Mar 01, 2001 at 10:32:21PM -0800, [EMAIL PROTECTED] wrote: > > On Thu, 1 Mar 2001, Roy T. Fielding wrote: > > > On Thu, Mar 01, 2001 at 09:40:44PM -0800, [EMAIL PROTECTED] wrote: > > > > We still need to APR namespace protection. We tried to not namespace > > > > protect things to begin with, and Apache and APR were conflicting > > > > horribly. > > > > > > Because the method I described was not used. > > Goody for you, Oh Omniscient One. We are just poor souls who don't have your > wisdom, so we fucked up the code. > > > Doesn't your model require that all APR applications define their > > configuration macros the same way? If an APR application is forced to do > > all of their macros in a given way, then I am against this model. If this > > works regardless of how the app defines it macro's, then cool. > > If an application includes an APR header, and then includes its own header, > then it will receive errors/warnings. Invariably, I always structure my > headers (say) in Subversion like so: > > #include > #include > > #include "public_svn_header.h" > #include "another.h" > > #include "module_private_header.h" > > > In the above scenario, my SVN headers would create duplicates. "So fix SVN." > Okay, I happen to be able to do that. Now, let's add an Expat include in > there. Ooops. Hey, I have privs to fix that, too. Now SVN includes Berkeley > DB headers. Fuck. > > autoconf headers are just not meant to be mixed. If APR is intending to > export the values, then they must be namespace protected. That implies that > a plain old AC_CHECK_HEADER(foo.h) is not sufficient. We need to follow that > up with setting a shell variable (fooh=0/1), then using that to def/undef or > 0/1 an APR-namespace symbol. > > > > > Add to that, that we define our own macros that no other > > > > package should have. > > > > > > That is a separate issue -- anything that is defined by an APR-specific > > > macro should be name-protected. I am only talking about the standard > > > names that every autoconf package defines. > > We don't have a problem with the standard names. Those go into > include/arch/unix/apr_private.h(.in). The issue is the generation of > APR_HAVE_FOO_H in the public apr.h header. For those, we need logic beyond a > simple AC_CHECK_HEADER(). > > > What happens with a package that doesn't use autoconf? > > Presumbly, they wouldn't be using HAVE_STDIO_H, so a conflict won't occur. True, but didn't Apache 1.3 have macros like HAVE_STDIO_H? It is a boundary condition, but I believe it is an issue. That format for the macros is pretty obvious. > I'm a big proponent of revamping the build system for Apache (more like > APR(UTIL)), and a basic cleaning of its config system. ++1. And I would add that a lot of the Apache autoconf stuff could just go away and mvoe into APR. I don't think Apache should be checking for headers personally. Ryan ___ Ryan Bloom [EMAIL PROTECTED] 406 29th St. San Francisco, CA 94131 ---
Re: FreeBSD and gnu m4
Roy T. Fielding wrote: > > Correct me if I am wrong, but all these changes and problems are > simply to reduce the three parameters of a standard autoconf macro > to one parameter in an APR-specific macro? If so, then -1 --> that > kind of change is why our buildconf setup is so much less portable > than typical configure scripts, and what I have been trying to get > rid of over the past few weeks. Do it the autoconf way or don't do > it at all. > There's nothing anti-autoconf in creating macros that use and extend AC macros, in a way that results in fewer bugs. IMO the reason why our present system is so much less portable is because instead of having a simple but brain-dead Sys7-shell based script, we have a m4/shell combo where we have to worry about all the various "enhancements" and "flavors" of both. Instead of working on the actual script that runs, we work on a script that calls m4 which then calls sh, and all those layers create portability concerns that don't show up "immediately". I *still* can't get PHP to build on some admittedly older systems (that use the old SysV 'ar') without major pain, because libtool assumes that 'ar' doesn't truncate filenames > 14 chars. So the use of autoconf doesn't imply that everything will automatically be portable :) -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "Hell is hot, that's never been disputed by anybody."
Re: FreeBSD and gnu m4
Roy T. Fielding wrote: > > Then we should fix the cause of that goofyness, which is due to all > APR defines being set to "0" or "1" instead of undefined/defined. > Fixing that would remove the cause of all those other subtle typos and > bugs, in addition to these above, and leave us with a standard autoconf > function with one parameter. > isn't "0"/"1" the "preferred" way? It certainly is more bug proof, since the build will complain if a macro isn't defined, whereas if we use #ifdef subtle bugs, where macros aren't defined, can silently effect us. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "Hell is hot, that's never been disputed by anybody."
Re: [RFC] Network Abstraction Layer
> > so, one bucket can deal with the NetBIOS header.
>
> Careful. We may be getting some terminology mixed up here.
we? nahh, just me
> I think we're
> definitely on the same page :-), but I'd like to clarify...
appreciate it.
> *) in the above example, the brigade has a HEAP or POOL or some other kind
>of simple bucket inserted as the first element. It refers to some memory
>with the appropriate data.
>
> *) it also has a FILE bucket and an EOS bucket
>
> *) a filter inserted the bucket into the brigade
>
> *) in the Samba case, I suspect your NetBIOS header is just another HEAP
>bucket, inserted by a NetBIOS *filter*
u *headache*. okay.
and the implementation on NT would do NBT but that bypasses IPX, NETBEUI,
TCP/IP, DECnet 3.0, carrier pigeons etc. whilst on samba (or any other
user-level implementation) you have to do it as buckets/filters.
... urr... :)
> > one with the SMB header.
> >
> > one with the IPC$ layer.
> >
> > one with the SMBtrans layer.
> >
> > one with the DCE/RPC pipe layer.
> >
> > one with the DCE/RPC header layer.
> >
> > one with the DCE/RPC data.
>
> Again, I would think *filters* are inserting standard bucket types into the
> brigade. You wouldn't necessarily need to create new bucket types for each
> of the above concepts. You could definitely create filters for each one,
> however.
okay.
getting educated, here.
> >...
> > ... we still need an apr_bucket_NAL which can "bounce" in-and-out of
> > buckets, _even though it may actually be implemented as buckets itself,
> > underneath!_ and handle the possibility that on one particular OS - e.g.
> > NT - is in fact a call to a kernel-level function, e.g. CreateNamedPipe().
>
> Hmm. I'm thinking that a filter would just choose the right kind of bucket
> to insert into the brigade. It could insert a SOCKET bucket, a PIPE bucket,
> or it could read data from the OS over an OS-specific API and then insert a
> HEAP bucket pointing to that memory.
ah ha. so... a pulling-approach.
... hmmm...
so, you'd be quite happy to educate people that if we want someone to do,
say, the IPC$ and the SMBtrans layer, above (OOPS! i missed one out!!!
the SMBtrans "named pipe" layer!!! that's 8 layers not 7 :) :) that they
have to know how to deal with three layers and their interactions. four,
because they're implementing two of them.
what i'd really like to have is that they are told, "you write this
program, it is implemented as a unix-domain-socket, it's a
'NamedPipe'Demon. to get data to /from the layer below you, call these
functions (SMBipcread, SMBipcwrite, or apr_nal_create("smbipc$")
preferably). provide _us_ with functions we can call that get data to /
from _you_."
that is a much more obvious, isolated [self-contained], rewarding task,
that i think is... a safer and less daunting task.
now, if apr_nal_create("smbipc$") _happens_ to be implemented in terms of
filters / buckets, and there happens to be a performance-hit because a)
you're doing ux-dom-sock file/io instead of filling in data into the same
memory block, well, i'll take the hit as the payoff is code
maintainability as this project turns from a 300,000 line deal into the
order of millions of lines.
> > or even, later on, let's say that someone decides to provide a linux
> > kernel-level SMB layer or even a NetBIOS kernel API.
>
> Yup. When this happens, I'd say there are a couple choices:
>
> 1) write a new bucket type. your "read from network" filter could be
>configured to insert the new bucket type, rather than a standard type.
> 2) write a new filter which is inserted as the "read from network" filter.
>it does the job of talking to the new kernel API, and populating the
>brigade with standard bucket types containing the data.
>
> 3) a combination of the above two: rather than a configurable filter, you
>simply insert a new filter which always uses the new custom bucket type.
...
i'm not getting it, then, am i :)
can this be done across several different programs?
i.e. you have one user-space daemon that handles the NetBIOS layer for
you (which is direct-to-kernel if you have linux kernel netbios support,
or you call NT's NBT routines)
and you have one user-space program that _uses_ the NetBIOS layer?
etc.?
>
> > as you can see, seven levels of buckets like that - all of which a
>
> Shouldn't this be seven levels of filters?
whoops: sorry :)
> In the Apache case, each little black box is a filter. It gets a brigade,
> it operates on that brigade, then it passes it. It doesn't have to know
> where the brigade came from, or who is next in line. It usually doesn't have
> to know anything about the bucket types (just that it can read()).
putting everything into one program, where one program is turing into 10
million lines of code, _that_ scares me. i dunno about you.
:)
> > what you think?
>
> It sounds great! I'm encouraged by your interest in the filters, buckets,
> and brigades. It seem
Re: [RFC] Network Abstraction Layer
On Fri, Mar 02, 2001 at 09:16:27PM +1100, Luke Kenneth Casson Leighton wrote:
> On Fri, 2 Mar 2001, Greg Stein wrote:
>...
> > BRIGADE = { FILE, EOS }# FILE bucket and EOS (End Of Stream) bucket
> >
> > becomes
> >
> > BRIGADE = { "packet header bytes", FILE, EOS }
> >
> > We inserted a header without touching the file. The output filter generates
> > a sendfile which contains an iovec for those header bytes. Blam! Out the
> > network it goes... :-)
>...
> so, one bucket can deal with the NetBIOS header.
Careful. We may be getting some terminology mixed up here. I think we're
definitely on the same page :-), but I'd like to clarify...
*) in the above example, the brigade has a HEAP or POOL or some other kind
of simple bucket inserted as the first element. It refers to some memory
with the appropriate data.
*) it also has a FILE bucket and an EOS bucket
*) a filter inserted the bucket into the brigade
*) in the Samba case, I suspect your NetBIOS header is just another HEAP
bucket, inserted by a NetBIOS *filter*
> one with the SMB header.
>
> one with the IPC$ layer.
>
> one with the SMBtrans layer.
>
> one with the DCE/RPC pipe layer.
>
> one with the DCE/RPC header layer.
>
> one with the DCE/RPC data.
Again, I would think *filters* are inserting standard bucket types into the
brigade. You wouldn't necessarily need to create new bucket types for each
of the above concepts. You could definitely create filters for each one,
however.
But I do see a great correlation in the models!
>...
> ... we still need an apr_bucket_NAL which can "bounce" in-and-out of
> buckets, _even though it may actually be implemented as buckets itself,
> underneath!_ and handle the possibility that on one particular OS - e.g.
> NT - is in fact a call to a kernel-level function, e.g. CreateNamedPipe().
Hmm. I'm thinking that a filter would just choose the right kind of bucket
to insert into the brigade. It could insert a SOCKET bucket, a PIPE bucket,
or it could read data from the OS over an OS-specific API and then insert a
HEAP bucket pointing to that memory.
Another approach is a custom bucket that does the OS-specific read. The
filter would then simply insert the custom bucket rather than do the
read-into-HEAP-bucket stuff.
> or even, later on, let's say that someone decides to provide a linux
> kernel-level SMB layer or even a NetBIOS kernel API.
Yup. When this happens, I'd say there are a couple choices:
1) write a new bucket type. your "read from network" filter could be
configured to insert the new bucket type, rather than a standard type.
2) write a new filter which is inserted as the "read from network" filter.
it does the job of talking to the new kernel API, and populating the
brigade with standard bucket types containing the data.
3) a combination of the above two: rather than a configurable filter, you
simply insert a new filter which always uses the new custom bucket type.
> as you can see, seven levels of buckets like that - all of which a
Shouldn't this be seven levels of filters?
Hmm. I can also see a case where you have a bucket that implements its
read() function by reading from another bucket, performing something on that
data, and returning it. In this sense, the chaining of buckets would be
similar to a chain of filters.
However, I'd think the filters are a slightly better approach, as they
operate on a brigade level rather than a single bucket level. Let's say that
you have a low-level bucket that reads from the network and you get a
partial packet. What should the upper level bucket do? Where should it put
that data while it asks for more data? In the brigade approach, the various
bits of data are accumulated into a brigade until "enough" is available.
> programmer will need to be "aware" of - is exactly what makes me very,
> very wary of introducing "newbie" programmers - no matter _how_ skilled or
> advanced or experienced they are - into this kind of environment.
Agreed. This is why we all try to encapsulate processing between tightly
controlled APIs. Nice little black boxes :-)
In the Apache case, each little black box is a filter. It gets a brigade,
it operates on that brigade, then it passes it. It doesn't have to know
where the brigade came from, or who is next in line. It usually doesn't have
to know anything about the bucket types (just that it can read()).
Then, some "higher level magic" organizes the filters into a useful
sequence, and puts the right data into the filter stack.
> if we can split things out so that an individual programmer only has to
> deal with *at most* two layers - the one they're coming from and the one
> they're going to - i'll be well-happy.
Get a brigade. Muck with it. Send it. ... nothing more :-)
> and to do _that_, in TNG, i decided to split at the SMBtrans / DCE/RPC
> pipe layer with a unix-domain-socket. and on NT, that is going to be
> CreateNamedPipe() that will do that for us, bypassing the
> SMBtrans-
Re: pool stacking
> > i'm going over to sander's at the w/e, we'll see if we can thrash it out. > > Please let us know your result. We've talked about types of pools before. > One that keeps the current semantics, one that maps straight to malloc/free, > and one that handled shared memory. > > I didn't know about a GPG memory type :-) Sounds way cool, and I definitely > bet that we could use that within Apache. well, it's not so much a GPG memory type as that they have their own wrappers on memory allocation to make sure that it gets locked (i forget the system call to do it) and stays in-memory and never swapped to disk. you don't want your unencrypted private key in a swap file, do you? :) > We have had some problems with the pool types, however. Consider that you > pass a function a malloc-based pool. Who calls the free? ah ha :) we worked this out. the sma_mem_sys has pointers to free, destroy, reset, and alloc (which have corresponding functions - except for free - in apr_pool-ing) the _users_ of an sma_mem_sys (apache pools, for example), must check if the parent-sma_mem_sys has either a free or a destroy, and call them if they exist. > The function > doesn't know whether it is a standard pool, or a malloc pool. And inserting > a bunch of apr_pool_free(ptr) calls (which are noops for some pools) defeats > the purpose of using pools to avoid carefully tracking free(). ... you still have to call free() in the existing apr_pool code, right??? :) :) so you call pool->sma_parent->free() instead(). well, actually - implementation of apr_pool_destroy() something like this: foreach pool alloc'd block: if (pool->sma_parent->free != NULL) pool->sma_parent->free(pool->sma_parent_data, pool_alloc'd_block) followed up by: if (pool->sma_parent->free_all != NULL) pool->sma_parent->free_all(pool->sma_parent_data) in this way, you cover _both_ cases where the sma_parent instance can either be capable of doing freeing _or_ is in fact _yet_ another "pool" allocation system and doesn't have a free at all. then, you can have an sma_parent which is in fact another apr_pool. uhm... a little tricky, i know. > For shared memory, I think we were having some issues with grabbing the > right amounts of shared memory, then doing the suballocations. I think this > can be solved, though. *scared* :) l - Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> - "i want a world of dreams, run by near-sighted visionaries" "good. that's them sorted out. now, on _this_ world..."
Re: pool stacking
On Fri, Mar 02, 2001 at 09:03:59PM +1100, Luke Kenneth Casson Leighton wrote: >... > in xmlvl, i investigated splitting out the pool code into stackable pools. > sander and i came up with sma_mem_sys as a result. > > imagine that you want to do secure memory stacking, using the gpg > memory-locking method, but you don't want to have to rewrite entire > sections of apr and apache to do it. > > passing in a sma_mem_sys (which is an array of parent-pool functions) > which does gpg-style memory-page-locking into apr_pool_creeate(), you can > do this. > > the "default" parent-pool sma_mem_sys of course should be one that wraps > malloc and free. > > you can then define sub-pools to use an apache-pool sma_mem_sys, and in > fact, the only difference then between a sub-pool and the global pool is > that the sub-pool uses an apr-sma_mem_sys instance whilst the global pool > uses a malloc/free-sma_mem_sys. > > i'm going over to sander's at the w/e, we'll see if we can thrash it out. Please let us know your result. We've talked about types of pools before. One that keeps the current semantics, one that maps straight to malloc/free, and one that handled shared memory. I didn't know about a GPG memory type :-) Sounds way cool, and I definitely bet that we could use that within Apache. We have had some problems with the pool types, however. Consider that you pass a function a malloc-based pool. Who calls the free? The function doesn't know whether it is a standard pool, or a malloc pool. And inserting a bunch of apr_pool_free(ptr) calls (which are noops for some pools) defeats the purpose of using pools to avoid carefully tracking free(). For shared memory, I think we were having some issues with grabbing the right amounts of shared memory, then doing the suballocations. I think this can be solved, though. I do like the idea of pool types, and I know others do, too. At least from me, there is a big +1 on integrating GPG pools into APR. Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: [RFC] Network Abstraction Layer
> I'd also like to take a moment just to say that I'm describing what we > currently have. much appreciate it. it's all network data, and you['ve basically already implemented an advanced version of what i was prating-about-with, two and a half years ago. > Pick your poison, and if the brigades and filters are not an appropriate > model for your data, then we can examine how to add an NAL concept to APR. brigades/filters are _very_ appropriate: see that other message where i described just how many layers we're having to deal with! extending those so that you can jump in/out at any point by making it look like it's terminated with a "socket", when in fact that "socket" is _yet another_ bucket brigade [on one OS, but not on another], would be a seriously powerful combination. luke - Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> - "i want a world of dreams, run by near-sighted visionaries" "good. that's them sorted out. now, on _this_ world..."
Re: [RFC] Network Abstraction Layer
On Fri, 2 Mar 2001, Greg Stein wrote:
> Getting long here. Watch out! :-)
other-thread.
> BRIGADE = { FILE, EOS }# FILE bucket and EOS (End Of Stream) bucket
>
> becomes
>
> BRIGADE = { "packet header bytes", FILE, EOS }
>
> We inserted a header without touching the file. The output filter generates
> a sendfile which contains an iovec for those header bytes. Blam! Out the
> network it goes... :-)
EXCELLENT. this is exactly the kind of thing that we'll need to do / use.
i started on something like this - memory-based only. it involved
chaining of bits of memory together, and you didn't have to worry about
how long they each were.
like it a lot.
so, one bucket can deal with the NetBIOS header.
one with the SMB header.
one with the IPC$ layer.
one with the SMBtrans layer.
one with the DCE/RPC pipe layer.
one with the DCE/RPC header layer.
one with the DCE/RPC data.
that's the sort of depth we're going into, here! and for simplicity, i've
left out all the authentication and encryption stages!
so...
... we still need an apr_bucket_NAL which can "bounce" in-and-out of
buckets, _even though it may actually be implemented as buckets itself,
underneath!_ and handle the possibility that on one particular OS - e.g.
NT - is in fact a call to a kernel-level function, e.g. CreateNamedPipe().
or even, later on, let's say that someone decides to provide a linux
kernel-level SMB layer or even a NetBIOS kernel API.
as you can see, seven levels of buckets like that - all of which a
programmer will need to be "aware" of - is exactly what makes me very,
very wary of introducing "newbie" programmers - no matter _how_ skilled or
advanced or experienced they are - into this kind of environment.
if we can split things out so that an individual programmer only has to
deal with *at most* two layers - the one they're coming from and the one
they're going to - i'll be well-happy.
and to do _that_, in TNG, i decided to split at the SMBtrans / DCE/RPC
pipe layer with a unix-domain-socket. and on NT, that is going to be
CreateNamedPipe() that will do that for us, bypassing the
SMBtrans-and-below layers altogether.
and still get good code reuse / abstraction.
what you think?
luke
- Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> -
"i want a world of dreams, run by near-sighted visionaries"
"good. that's them sorted out. now, on _this_ world..."
pool stacking (was Re: [RFC] Network Abstraction Layer)
On Fri, 2 Mar 2001, Greg Stein wrote: > Getting long here. Watch out! :-) split in two :) > Even the connection pool is a child of another. The pools can be described > as a tree of pools, with a single global pool created at startup. greg, in xmlvl, i investigated splitting out the pool code into stackable pools. sander and i came up with sma_mem_sys as a result. imagine that you want to do secure memory stacking, using the gpg memory-locking method, but you don't want to have to rewrite entire sections of apr and apache to do it. passing in a sma_mem_sys (which is an array of parent-pool functions) which does gpg-style memory-page-locking into apr_pool_creeate(), you can do this. the "default" parent-pool sma_mem_sys of course should be one that wraps malloc and free. you can then define sub-pools to use an apache-pool sma_mem_sys, and in fact, the only difference then between a sub-pool and the global pool is that the sub-pool uses an apr-sma_mem_sys instance whilst the global pool uses a malloc/free-sma_mem_sys. i'm going over to sander's at the w/e, we'll see if we can thrash it out. luke - Luke Kenneth Casson Leighton <[EMAIL PROTECTED]> - "i want a world of dreams, run by near-sighted visionaries" "good. that's them sorted out. now, on _this_ world..."
Re: [RFC] Network Abstraction Layer
On Fri, Mar 02, 2001 at 06:25:56AM +1100, Luke Kenneth Casson Leighton wrote:
> > The issue here is, that the protocols, that are below SMB,
> > are more like sockets. From a clean point of view, they
> > should be implemented in the kernel, but none of us is
> > willing to write kerneldrivers for each OS out there.
>
> however, on NT, some functions already exist, such as CreateNamedPipe.
>
> and there is no way in hell i'm writing a kerneldriver for NT when there's
> a perfectly good function already out there.
Yah... that would just suck :-) Have no fear...
> however, i _am_ prepared to "mirror" that functionality in systems that
> _don't_ have a CreateNamedPipe function.
>
> so, the implementation of apr_nal_create_socket("namedpipe:foo", ...) on
> NT will call CreateNamedPipe. which i should imagine will be utterly
> trivial to do.
>
> and the implementation on non-NT systems will go through hoops and layers
> like there's no tomorrow [i.e. emulating what NT does in the kernel.
> *yuck* :)].
>
> if the buckets principle can hide this from a developer, so be it.
Yes, you could create a two types of buckets: one for NT using true named
pipes, and one for "those other guys" :-)
The appropriate bucket is inserted into the "input brigade" and the rest of
the input processing is blissfully ignorant. They simply read content from a
bucket.
I'd also like to take a moment just to say that I'm describing what we
currently have. I'm not going to try to advocate or force particular choices
on you guys(!) It's your problem domain, and you know best. I'm just hoping
for a complete description of what we already have, so that you guys can
make an informed decision on the approach to take.
I certainly see no problem with an NAL type concept in APR itself (note that
buckets are "one step higher" in APRUTIL). Or possibly APR just grows more
types of sockets within its existing apr_socket_t public type. It is just
that we may have some other stuff that can help :-)
Pick your poison, and if the brigades and filters are not an appropriate
model for your data, then we can examine how to add an NAL concept to APR.
My personal, uninformed opinion :-) would tend towards adding new
OS-provided socket types to apr_socket_t (allowing all apps the benefit of
the new socket types; not just those that can fill in a func table), and
then building the "selection" mechanisms above that using buckets, brigades,
and filters.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: [RFC] Network Abstraction Layer
Getting long here. Watch out! :-)
On Thu, Mar 01, 2001 at 08:17:10PM +0100, Elrond wrote:
> On Wed, Feb 28, 2001 at 07:40:21AM -0800, Greg Stein wrote:
>...
> > It might be interesting to examine the filters that we have in Apache right
> > now. They provide for the protocol-stacking, buffering, and (hey!)
> > filtering.
>
> The filters are currently in Apache itself?
Yup.
> That might explain, why I didn't find anything relating to
> filter in apr-util.
>
> If filters actualy are, what we're looking for, it would be
> nice, if the base of the filters (not the filters
> themselves) be moved into apr-util.
> This might be a good idea anyway.
As I stated elsewhere, there is actually not a lot of stuff related to
filters. Most of the code deals with registration of the filter, rather than
their use. I'm not sure how much has broad utility.
In a nutshell, a filter function has this prototype:
typedef apr_status_t (*ap_out_filter_func)(ap_filter_t *f,
apr_bucket_brigade *bb);
ap_filter_t has a couple pieces of metadata, the filter function pointer,
and a pointer to the "next" filter. A filter function looks something like:
filter_func(f, bb)
{
/* manipulate bb in interesting ways */
return ap_pass_brigade(f->next, bb);
}
Where ap_pass_brigade is essentially:
{
return f->func(f, bb);
}
The meat of the filter code deals with naming the filters and then creating
the linked list of filters (we insert by name, and filters have an inherent
ordering).
>...
> I've read the complete apr_buckets.h.
> From a _very_ high point of view, bucket brigades are a
> list of buckets, and buckets are simply referring to static
> read-only data, which was (or will be) created.
This is correct. A brigade's contents are read-only, but may be generated
*during* a bucket's read() function (e.g. read the content from a pipe,
socket, or file).
"Manipulating" a brigade involves splitting buckets, inserting or removing
portions, or simply replacing portions. For example, let's say you are a
filter handling server-side-includes (SSI), and you get a brigade with a
single bucket with the string:
BRIGADE
\
BUCKET
"duration = seconds"
The filter will split this into three buckets:
BRIGADE
\
BUCKET -- BUCKET -- BUCKET
"duration = " """ seconds"
It then replaces the middle bucket with a new bucket representing the
request values. Note that the first and last buckets do not copy the value;
they simply point to different parts of the original bucket's value.
> So this is a nice way of moving around data basicaly.
It is a way to do zero-copy of content (presuming that is feasible within
the constraints of the filters' manipulations; a gzip filter simply can't
retain the original data :-)
Since one of the bucket types represents a FILE, you can also inject a file
into the brigade and send that down the output stack. Let's say that nothing
in the chain happens to need the file contents. The file then arrives at the
bottom "filter" of the chain, and sendfile() is used to shove the thing out
onto the network.
This zero-copy is quite feasible when you consider a brigade transform such
as this:
BRIGADE = { FILE, EOS }# FILE bucket and EOS (End Of Stream) bucket
becomes
BRIGADE = { "packet header bytes", FILE, EOS }
We inserted a header without touching the file. The output filter generates
a sendfile which contains an iovec for those header bytes. Blam! Out the
network it goes... :-)
>...
> Some of our protocol layers might be actualy doable as
> filters. (SMB itself quiet likely, at least for the
> fileserving part, SMB does more than fileserving, but it
> probably resembles http in some ways)
Possibly doable. I merely bring it up as something for you to look at. The
brigades are a great way to move around content.
> The part, where Sander's NAL comes in, is the one filter
> that writes stuff to the network:
>
> In Apache this is most likely a simple unix tcp socket,
> into which you write data.
> NAL tries to abstract sockets.
Right!
> The issue here is, that the protocols, that are below SMB,
> are more like sockets. From a clean point of view, they
> should be implemented in the kernel, but none of us is
> willing to write kerneldrivers for each OS out there.
>
> If you look through Sander's proposal, nearly all typical
> Unix socket operations are represened as function pointers,
> and actualy nearly each of them needs to be implemented,
> and most of them will actualy do something complex.
Yep, I believe that I understood that part. Note that, in the filter stack
approach, the last "filter" is the thing that delivers to the network. Using
this point, it is possible to define each of those custom "sockets" as one
of these end-point filters.
In Apache, that end-point is a plain socket, as you describe.
In Samba, it could be a filter that delivers to a kernel-level SMB
Re: FreeBSD and gnu m4
On Thu, Mar 01, 2001 at 10:32:21PM -0800, [EMAIL PROTECTED] wrote: > On Thu, 1 Mar 2001, Roy T. Fielding wrote: > > On Thu, Mar 01, 2001 at 09:40:44PM -0800, [EMAIL PROTECTED] wrote: > > > We still need to APR namespace protection. We tried to not namespace > > > protect things to begin with, and Apache and APR were conflicting > > > horribly. > > > > Because the method I described was not used. Goody for you, Oh Omniscient One. We are just poor souls who don't have your wisdom, so we fucked up the code. > Doesn't your model require that all APR applications define their > configuration macros the same way? If an APR application is forced to do > all of their macros in a given way, then I am against this model. If this > works regardless of how the app defines it macro's, then cool. If an application includes an APR header, and then includes its own header, then it will receive errors/warnings. Invariably, I always structure my headers (say) in Subversion like so: #include #include #include "public_svn_header.h" #include "another.h" #include "module_private_header.h" In the above scenario, my SVN headers would create duplicates. "So fix SVN." Okay, I happen to be able to do that. Now, let's add an Expat include in there. Ooops. Hey, I have privs to fix that, too. Now SVN includes Berkeley DB headers. Fuck. autoconf headers are just not meant to be mixed. If APR is intending to export the values, then they must be namespace protected. That implies that a plain old AC_CHECK_HEADER(foo.h) is not sufficient. We need to follow that up with setting a shell variable (fooh=0/1), then using that to def/undef or 0/1 an APR-namespace symbol. > > > Add to that, that we define our own macros that no other > > > package should have. > > > > That is a separate issue -- anything that is defined by an APR-specific > > macro should be name-protected. I am only talking about the standard > > names that every autoconf package defines. We don't have a problem with the standard names. Those go into include/arch/unix/apr_private.h(.in). The issue is the generation of APR_HAVE_FOO_H in the public apr.h header. For those, we need logic beyond a simple AC_CHECK_HEADER(). > What happens with a package that doesn't use autoconf? Presumbly, they wouldn't be using HAVE_STDIO_H, so a conflict won't occur. >... > > As I mentioned when I started the build blues thread, I read the archives > > first. Most of the decisions back then were made because APR had to be > > integrated with a non-configure-based httpd. I have the benefit of > > That's not true. Most of the problems weren't even discovered until we > integrated APR into a configure-based Apache. Exactly. Mix one autoconf'd guy with another, and you're in for trouble. > > hindsight and a holistic view of the build system, so it shouldn't be > > too surprising that I can think of a solution that may not have been > > possible back then. And you're saying that we don't have that either? That only you possess that knowledge? That isn't fair, Roy. The current spate of problems was due to some M4 coding that I was doing to simplify and optimize our configuration. Unfortunately, despite M4's age, it appears to be rather non-portable and prone to flakiness. We backed off a bit, but we're still in a very good and stable spot. No more deep magic, and it should all work quite fine. We don't do anything beyond normal autoconf setups. > I hope you have found new ideas, but the current APR configuration system > is much more autoconf-like than the Apache one, and most of the problems > haven't been with the APR configure system. Exactly. This recent spate was M4. My fault. Our autoconf system has been working quite well. > > The reason I bring it up right now is because every step we have taken away > > from a standard autoconf setup has resulted in significant problems on > > one platform or another, leading to another step away and more problems, > > ad nauseum. Some steps, like not using the standard generating mechanism > > for the Makefile files, have benefits that outweigh the pain. Others do > > not. > > I look at the APR configuration system, and I see a system that is very > close to what autoconf expects. Exactly. It is much more complex than many because it touches "all" of an operating system's features. Many apps just don't have that kind of breadth, so their configure scripts are quite short (look at Expat). > Apache, on the other hand is completely > non-autoconf. However, this model is not completely broken, and it is > what PHP uses, which is why it was originally chosen. I am 100% for > fixing this mess, but I am concerned by the number of times that this > issue has been raised, and never resolved. When the Apache stuff was first put in, based on the PHP stuff, I wasn't paying much attention to it. It seemed to work, so that was fine. But over the past year or so (or however long it has been), I've had to deal with it more and more. Gad
Re: FreeBSD and gnu m4
On Thu, 1 Mar 2001, Roy T. Fielding wrote: > On Thu, Mar 01, 2001 at 09:40:44PM -0800, [EMAIL PROTECTED] wrote: > > We still need to APR namespace protection. We tried to not namespace > > protect things to begin with, and Apache and APR were conflicting > > horribly. > > Because the method I described was not used. Doesn't your model require that all APR applications define their configuration macros the same way? If an APR application is forced to do all of their macros in a given way, then I am against this model. If this works regardless of how the app defines it macro's, then cool. > > Add to that, that we define our own macros that no other > > package should have. > > That is a separate issue -- anything that is defined by an APR-specific > macro should be name-protected. I am only talking about the standard > names that every autoconf package defines. What happens with a package that doesn't use autoconf? > > We have been down the road of exposing non-namespace protected macros > > before, it didn't work then, and I don't believe it will work now. Please > > take a look at the archive from when we first put autoconf into APR, > > and how it conflicted with Apache. > > As I mentioned when I started the build blues thread, I read the archives > first. Most of the decisions back then were made because APR had to be > integrated with a non-configure-based httpd. I have the benefit of That's not true. Most of the problems weren't even discovered until we integrated APR into a configure-based Apache. > hindsight and a holistic view of the build system, so it shouldn't be > too surprising that I can think of a solution that may not have been > possible back then. I hope you have found new ideas, but the current APR configuration system is much more autoconf-like than the Apache one, and most of the problems haven't been with the APR configure system. > The reason I bring it up right now is because every step we have taken away > from a standard autoconf setup has resulted in significant problems on > one platform or another, leading to another step away and more problems, > ad nauseum. Some steps, like not using the standard generating mechanism > for the Makefile files, have benefits that outweigh the pain. Others do not. I look at the APR configuration system, and I see a system that is very close to what autoconf expects. Apache, on the other hand is completely non-autoconf. However, this model is not completely broken, and it is what PHP uses, which is why it was originally chosen. I am 100% for fixing this mess, but I am concerned by the number of times that this issue has been raised, and never resolved. Ryan ___ Ryan Bloom [EMAIL PROTECTED] 406 29th St. San Francisco, CA 94131 ---
Re: FreeBSD and gnu m4
On Thu, Mar 01, 2001 at 09:40:44PM -0800, [EMAIL PROTECTED] wrote: > > We still need to APR namespace protection. We tried to not namespace > protect things to begin with, and Apache and APR were conflicting > horribly. Because the method I described was not used. > Add to that, that we define our own macros that no other > package should have. That is a separate issue -- anything that is defined by an APR-specific macro should be name-protected. I am only talking about the standard names that every autoconf package defines. > We have been down the road of exposing non-namespace protected macros > before, it didn't work then, and I don't believe it will work now. Please > take a look at the archive from when we first put autoconf into APR, > and how it conflicted with Apache. As I mentioned when I started the build blues thread, I read the archives first. Most of the decisions back then were made because APR had to be integrated with a non-configure-based httpd. I have the benefit of hindsight and a holistic view of the build system, so it shouldn't be too surprising that I can think of a solution that may not have been possible back then. The reason I bring it up right now is because every step we have taken away from a standard autoconf setup has resulted in significant problems on one platform or another, leading to another step away and more problems, ad nauseum. Some steps, like not using the standard generating mechanism for the Makefile files, have benefits that outweigh the pain. Others do not. Roy
Re: FreeBSD and gnu m4
We still need to APR namespace protection. We tried to not namespace protect things to begin with, and Apache and APR were conflicting horribly. Add to that, that we define our own macros that no other package should have. We have been down the road of exposing non-namespace protected macros before, it didn't work then, and I don't believe it will work now. Please take a look at the archive from when we first put autoconf into APR, and how it conflicted with Apache. Ryan On Thu, 1 Mar 2001, Roy T. Fielding wrote: > > Changing from "1" and "0" to def, undef wouldn't change anything at all. > > The exact same logic is required, and the same general variables are > > required. The differences between the two are incredibly minor, because > > we want to use namespace protected macros. > > If we changed to def/undef and we used the standard autoconf macros > with the default behavior, then we don't need namespace protected symbols. > Defining them as "0" or "1" doesn't allow us to use > >#ifndef HAVE_UNISTD_H >#undef HAVE_UNISTD_H >#endif > > as the template for the generated file, which gets translated to > >#ifndef HAVE_UNISTD_H >#define HAVE_UNISTD_H 1 >#endif > > by configure if AC_CHECK_HEADERS(unistd.h) succeeds, which won't result > in errors if the same symbol is defined by includes in other packages, > which was the original justification for the APR_ prefix on those symbols. > > Of course, this means we'd have to use a template instead of autoheader, > but that is how I prefer to set up autoconf in any case, since it also > allows us to add documentation to the generated file. Doing it this way > has the nice side-effect of allowing the user to override a configure > definition by simply doing a > >env CFLAGS="-DHAVE_UNISTD_H=1" make > > I wasn't kidding when I said that our problems with autoconf > are due to the way we are using it. > > Roy > > ___ Ryan Bloom [EMAIL PROTECTED] 406 29th St. San Francisco, CA 94131 ---
Re: FreeBSD and gnu m4
> Changing from "1" and "0" to def, undef wouldn't change anything at all. > The exact same logic is required, and the same general variables are > required. The differences between the two are incredibly minor, because > we want to use namespace protected macros. If we changed to def/undef and we used the standard autoconf macros with the default behavior, then we don't need namespace protected symbols. Defining them as "0" or "1" doesn't allow us to use #ifndef HAVE_UNISTD_H #undef HAVE_UNISTD_H #endif as the template for the generated file, which gets translated to #ifndef HAVE_UNISTD_H #define HAVE_UNISTD_H 1 #endif by configure if AC_CHECK_HEADERS(unistd.h) succeeds, which won't result in errors if the same symbol is defined by includes in other packages, which was the original justification for the APR_ prefix on those symbols. Of course, this means we'd have to use a template instead of autoheader, but that is how I prefer to set up autoconf in any case, since it also allows us to add documentation to the generated file. Doing it this way has the nice side-effect of allowing the user to override a configure definition by simply doing a env CFLAGS="-DHAVE_UNISTD_H=1" make I wasn't kidding when I said that our problems with autoconf are due to the way we are using it. Roy
Re: FreeBSD and gnu m4
On Thu, 1 Mar 2001, Roy T. Fielding wrote: > On Thu, Mar 01, 2001 at 09:39:22PM -0500, Cliff Woolley wrote: > > On Thu, 1 Mar 2001, Roy T. Fielding wrote: > > > > > Correct me if I am wrong, but all these changes and problems are > > > simply to reduce the three parameters of a standard autoconf macro > > > to one parameter in an APR-specific macro? > > > > That's one way to look at it, yeah, but that doesn't capture all of the > > benefits. Those three parameters have given us *numerous* subtle bugs due > > to typos and the like that go undetected on the committer's platform but > > cause complete build breakage on other platforms. Reducing to one > > parameter saves the work that causes these bugs. > > Then we should fix the cause of that goofyness, which is due to all > APR defines being set to "0" or "1" instead of undefined/defined. > Fixing that would remove the cause of all those other subtle typos and > bugs, in addition to these above, and leave us with a standard autoconf > function with one parameter. Changing from "1" and "0" to def, undef wouldn't change anything at all. The exact same logic is required, and the same general variables are required. The differences between the two are incredibly minor, because we want to use namespace protected macros. Ryan ___ Ryan Bloom [EMAIL PROTECTED] 406 29th St. San Francisco, CA 94131 ---
Re: FreeBSD and gnu m4
On Thu, Mar 01, 2001 at 09:39:22PM -0500, Cliff Woolley wrote: > On Thu, 1 Mar 2001, Roy T. Fielding wrote: > > > Correct me if I am wrong, but all these changes and problems are > > simply to reduce the three parameters of a standard autoconf macro > > to one parameter in an APR-specific macro? > > That's one way to look at it, yeah, but that doesn't capture all of the > benefits. Those three parameters have given us *numerous* subtle bugs due > to typos and the like that go undetected on the committer's platform but > cause complete build breakage on other platforms. Reducing to one > parameter saves the work that causes these bugs. Then we should fix the cause of that goofyness, which is due to all APR defines being set to "0" or "1" instead of undefined/defined. Fixing that would remove the cause of all those other subtle typos and bugs, in addition to these above, and leave us with a standard autoconf function with one parameter. Roy
Re: FreeBSD and gnu m4
On Thu, 1 Mar 2001, Roy T. Fielding wrote: > Correct me if I am wrong, but all these changes and problems are > simply to reduce the three parameters of a standard autoconf macro > to one parameter in an APR-specific macro? That's one way to look at it, yeah, but that doesn't capture all of the benefits. Those three parameters have given us *numerous* subtle bugs due to typos and the like that go undetected on the committer's platform but cause complete build breakage on other platforms. Reducing to one parameter saves the work that causes these bugs. --Cliff
Re: FreeBSD and gnu m4
Correct me if I am wrong, but all these changes and problems are simply to reduce the three parameters of a standard autoconf macro to one parameter in an APR-specific macro? If so, then -1 --> that kind of change is why our buildconf setup is so much less portable than typical configure scripts, and what I have been trying to get rid of over the past few weeks. Do it the autoconf way or don't do it at all. Roy
