Re: new zero copy sockets patches available
Kenneth D. Merry writes: > > As a related question, will this work with the broadcom gigabit (bge) > > driver, which is the Tigon III? If not, what would it take to get > > it working? > > Unfortunately, it won't work with the Tigon III. > > If you can get firmware source for the Tigon III, I can probably get header > splitting working. (The only way it wouldn't work is if they've offloaded > most of the packet processing into the hardware.) I don't want this to sound too much like an advertisement, but since I now work for Myricom, I feel obligated to point out that the other adapters that this can be made to work with are Myricom Myrinet interfaces (*) Anyway, if this gets committed, I will enhance the Myricom GM-2 firmware to support page-flipping receives. I think my boss will approve the work. With FreeBSD & gm-2, we now see roughly 1.8Gb/sec from iperf, but the receiver's CPU is maxed out. It would be nice to have some room left over to actually do something with the data ;) > The send side code will work on any NIC, and you can kludge up special case > header splitting on the receive side if the NIC allows you to break jumbo > frames into multiple chunks of data. (This is what Drew originally did for > the Tigon II -- you just size the initial chunk of data so that it'll just > hold the ethernet header, IP header, TCP header and TCP options, and the > payload will "magically" end up page aligned. This doesn't work for > protocols other than TCP, and it won't work if your TCP header changes in > length.) This mostly doesn't work for TCP at all. It does work for the client-side zero-copy NFS read-response work I did where I page-flip the received frame into the buffer cache. This was something of a hack, because I don't actually parse the rpcs, I just guess that the data starts at a certain offset into the packet (which it does, 99% of the time). Cheers, Drew (*) 2Gb/sec + 2Gb/sec. See http://www.myri.com/myrinet/product_list.html. Before your jaw hits the floor at the price per adapter, take into account how cheap the switches are when compared to high-density gigabit ethernet switches. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
Terry Lambert writes: > To do the work, you'd have to do it on your own, after licensing > the firmware, after signing an NDA. Unlike the rather public > Tigon II firmware, the Tigon III doesn't have a lot of synergy > or interesting work going for it. Most people doing interesting > work tend to use Tigon II cards, because of this. It also requires a good contact at Broadcom. Some people I know at another institution are willing to sign an NDA and still cannot get the firmware from Broadcom. Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On Sat, May 18, 2002 at 13:15:43 -0400, Don Bowman wrote: > > Andrew Gallatin writes: > >> Kenneth D. Merry writes: > >> > > >> > I have released a new set of zero copy sockets patches, against > -current > >> > from today (May 17th, 2002). > > > > Hi Ken, > > > > I'm glad to see that you're still maintining this! > > > > Assuming the mutex issues get sorted out, what do you think the odds > > are of getting this into the tree? The only possible issue I see is > > with the tigon firmware. Is the firmware you're using of the same > > vintage as what's in the tree now? Does it contain all the same > > fixes? > > As a related question, will this work with the broadcom gigabit (bge) > driver, which is the Tigon III? If not, what would it take to get > it working? Unfortunately, it won't work with the Tigon III. If you can get firmware source for the Tigon III, I can probably get header splitting working. (The only way it wouldn't work is if they've offloaded most of the packet processing into the hardware.) The send side code will work on any NIC, and you can kludge up special case header splitting on the receive side if the NIC allows you to break jumbo frames into multiple chunks of data. (This is what Drew originally did for the Tigon II -- you just size the initial chunk of data so that it'll just hold the ethernet header, IP header, TCP header and TCP options, and the payload will "magically" end up page aligned. This doesn't work for protocols other than TCP, and it won't work if your TCP header changes in length.) Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On Sat, May 18, 2002 at 13:12:09 -0400, Andrew Gallatin wrote: > > Kenneth D. Merry writes: > > > > I have released a new set of zero copy sockets patches, against -current > > from today (May 17th, 2002). > > > > The main change is to deal with the vfs_ioopt changes that Alan Cox made in > > kern_subr.c. (They conflicted a bit with the zero copy receive code.) > > > > The patches and the FAQ are available here: > > > > http://people.freebsd.org/~ken/zero_copy/ > > > > Comments, questions and reviews are all welcome! > > > > Hi Ken, > > I'm glad to see that you're still maintining this! > > Assuming the mutex issues get sorted out, what do you think the odds > are of getting this into the tree? The only possible issue I see is > with the tigon firmware. Is the firmware you're using of the same > vintage as what's in the tree now? Does it contain all the same > fixes? I think the odds of getting into the tree are pretty good. I certainly plan to give it a shot. The firmware is basically identical to what we currently have in the tree (12.4.11 plus some fixes Bill Paul picked out from 12.4.13), plus the header splitting patches. There have been some newer firmware releases, up to 12.4.18 I think, but I haven't yet been able to get anything from 3Com. I know 12.4.18 at least fixes some checksum offloading issues in a few strange cases. > Thanks again for keeping this alive, > > Drew Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On Sat, May 18, 2002 at 09:03:38 -0400, John Baldwin wrote:
>
> On 18-May-2002 Kenneth D. Merry wrote:
> > On Fri, May 17, 2002 at 23:02:55 -0700, Alfred Perlstein wrote:
> >> * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 22:40] wrote:
> >> >
> >> > I have released a new set of zero copy sockets patches, against -current
> >> > from today (May 17th, 2002).
> >> >
> >> > The main change is to deal with the vfs_ioopt changes that Alan Cox made
> >> > in
> >> > kern_subr.c. (They conflicted a bit with the zero copy receive code.)
> >> >
> >> > The patches and the FAQ are available here:
> >> >
> >> > http://people.freebsd.org/~ken/zero_copy/
> >> >
> >> > Comments, questions and reviews are all welcome!
> >>
> >> jumbo_vm_init() has a bunch of bugs
> >>
> >> first it doesn't work right if called concurrently.
> >> you need to protect the initialization of jumbo_vm_object otherwise
> >> bad things can happen. my suggestion is to store the results of
> >> vm_object_allocate into a temporary, grab the mutex and then check
> >> to see if jumbo_vm_object has been initialized again if it has then
> >> free the object, otherwise store the allocated object into the
> >> global and continue.
> >
> > The problem here is that the mutex needs to be initialized before I can
> > acquire it, and there's going to be a race between checking to see
> > whether it has been initialized and actually initializing it.
> >
> > e.g.:
> >
> > if (!mtx_initialized(&jumbo_mutex))
> > mtx_init(&jumbo_mutex, "jumbo mutex", NULL, MTX_DEF);
> >
> > mtx_lock(&jumbo_mutex);
> >
> > if (jumbo_vm_object != NULL) {
> > mtx_unlock(&jumbo_mutex);
> > return (1);
> > }
> >
> > /* allocate our object */
> > jumbo_vm_object = vm_object_allocate(OBJT_DEFAULT, JUMBO_MAX_PAGES);
> >
> > The above would work, I think, if it weren't for the race in the mutex
> > initialization, and assuming I can allocate a vm object while holding
> > the jumbo mutex.
> >
> > Suggestions?
>
> Either use a sysinit or something gross like this:
>
> static int jumbo_init = 0;
>
> ...
> while (jumbo_init != 2) {
> if (atomic_cmpset_acq_int(&jumbo_init, 0, 1) {
> mtx_init(&jumbo_mutex, "jumbo_mutex", NULL, MTX_DEF);
> atomic_store_rel_int(&jumbo_init, 2);
> }
> }
>
> mtx_lock(&jumbo_mutex);
>
> ...
>
> a sysinit is probably best though.
Sounds like a sysinit is probably the way to go.
I'll give it a try, thanks!
Ken
--
Kenneth Merry
[EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
John Baldwin wrote:
> > This is actually what I was saying was bad: a static function
> > per mutex declaration.
>
> Umm, no, there is _one_ global function that we call. Why not check
> the actual code?
Are you talking about a P4 branch, and not the main repository?
> Why don't you read the code?
>
> Here, I'll quote it for you:
>
> struct mtx_args {
> struct mtx *ma_mtx;
> const char *ma_desc;
> int ma_opts;
> };
>
> #define MTX_SYSINIT(name, mtx, desc, opts) \
> static struct mtx_args name##_args = { \
> mtx,\
> desc, \
> opts\
> }; \
> SYSINIT(name##_mtx_sysinit, SI_SUB_LOCK, SI_ORDER_MIDDLE, \
> mtx_sysinit, &name##_args)
This is *EXACTLY* what I thought should be avoided. The linker
set should not contain the address of a bunch of static mtx_args
structs, because there shouldn't *be* a bunch of static mtx_args
structs.
And the ability to pass different initialization values to different
mutex instances is broken, in that it makes it impossible to look at
a mutex, and know what you need to know about it, merely because it's
a mutex.
Minimally, you are going to add 12 bytes per mutex (24 on 64 bit
machines), plus however much memory is taken up by the "ma_desc",
plus it makes the memory unrecoverable, because the address of the
string that's in the supposedly "recoverable" data segment can't be
recovered any more, because making the data segment "go away" would
invalidate dereferences of the pointer.
This is about as ugly the mutex code taking parameters (we've had
that discussion before).
> Note no static function, instead we use the global function mtx_sysinit()
> in kern_mutex.c:
>
> /*
> * General init routine used by the MTX_SYSINIT() macro.
> */
> void
> mtx_sysinit(void *arg)
> {
> struct mtx_args *margs = arg;
>
> mtx_init(margs->ma_mtx, margs->ma_desc, NULL, margs->ma_opts);
> }
On a per instance basis. For which you create not only an mtx_args
structure, but also a sysinit structure, to act as a container for
the reference to the mtx_args struct address, and the mtx_sysinit
function, etc..
Bleh. 8-).
-- Terry
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On 18-May-2002 Terry Lambert wrote:
> John Baldwin wrote:
>> On 18-May-2002 Terry Lambert wrote:
>> > John Baldwin wrote:
>> >> > God, it's annoying that a statically declared mutex is not
>> >> > defacto initialized.
>> >>
>> >> Is it in solaris?
>> >
>> > It isn't in FreeBSD because of the need to link mutex'es into
>> > the "witness protection program". 8-).
>>
>> Actually, there is more to it than that. Or at least, there will be
>> when turnstiles are added (turnstiles require some function callouts
>> to work properly).
>
> It's the function callouts that are the problem. 8-(.
Unfortunately there aren't easy solutions to this. Even solaris 7
uses callouts. We would use fewer of them at least.
>> > MUTEX_DECLARE(mutex_name).
>>
>> Umm, yes, like MTX_SYSINIT(). :)
>
> No. An MTX_SYSINIT() that wrapped a declaration and a SYSINIT()
> would als imply the definition of an static function to do the
> initialization, that then got called by the SYSINIT() itself.
This is a false implication.
> This is actually what I was saying was bad: a static function
> per mutex declaration.
Umm, no, there is _one_ global function that we call. Why not check
the actual code?
> So, among other things, you want to use the same initializer function
> instance for all statically declared mutexes.
Um, we already do this.
> So you have to use something other than a SYSINIT() for the declarations,
> but you could use *one SYSINIT()* to do the pre-use initialization of
> *all* declarations.
Why don't you read the code?
Here, I'll quote it for you:
struct mtx_args {
struct mtx *ma_mtx;
const char *ma_desc;
int ma_opts;
};
#define MTX_SYSINIT(name, mtx, desc, opts) \
static struct mtx_args name##_args = { \
mtx,\
desc, \
opts\
}; \
SYSINIT(name##_mtx_sysinit, SI_SUB_LOCK, SI_ORDER_MIDDLE, \
mtx_sysinit, &name##_args)
Note no static function, instead we use the global function mtx_sysinit()
in kern_mutex.c:
/*
* General init routine used by the MTX_SYSINIT() macro.
*/
void
mtx_sysinit(void *arg)
{
struct mtx_args *margs = arg;
mtx_init(margs->ma_mtx, margs->ma_desc, NULL, margs->ma_opts);
}
--
John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve!" - http://www.FreeBSD.org/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
John Baldwin wrote: > On 18-May-2002 Terry Lambert wrote: > > John Baldwin wrote: > >> > God, it's annoying that a statically declared mutex is not > >> > defacto initialized. > >> > >> Is it in solaris? > > > > It isn't in FreeBSD because of the need to link mutex'es into > > the "witness protection program". 8-). > > Actually, there is more to it than that. Or at least, there will be > when turnstiles are added (turnstiles require some function callouts > to work properly). It's the function callouts that are the problem. 8-(. > > MUTEX_DECLARE(mutex_name). > > Umm, yes, like MTX_SYSINIT(). :) No. An MTX_SYSINIT() that wrapped a declaration and a SYSINIT() would als imply the definition of an static function to do the initialization, that then got called by the SYSINIT() itself. This is actually what I was saying was bad: a static function per mutex declaration. You really want to avoid the pthreads "mutex initializer" thing; the compiler doesn't have code that can attach to data declarations to fix the problem. So, among other things, you want to use the same initializer function instance for all statically declared mutexes. So you have to use something other than a SYSINIT() for the declarations, but you could use *one SYSINIT()* to do the pre-use initialization of *all* declarations. > > You could do this with a SYSINIT(), as has been suggested, but > > that would add a relatively large per mutex overhead for each > > one you want to declare, since you'd basically be repeating the > > common components for doing the job for each and every mutex, > > instead of sharing them. > > Umm, this is a boot-up thing so it's not that big of a deal, plus > the actual code isn't duplicated, we call a common function for > the actual mutex initialization. Maybe. I don't think a large sysinit structure is as useful as an array of addresses of mutexes needing initialization. There would be a serious tempation in the former case to provide for some type of non-uniform initialization of statically declared sysinit objects. It would be nice if the non-uniformity could be limited to the fact that if the WITNESS code was there, it needed to be linked onto the list. With the WITNESS code not compiled into the kernel, I'd prefer not to include the initialization code at all. It would encorage bad practice, if it weren't conditional on WITNESS. > > Technically, some later programmer could come along and recover > > the linker set memory, actually, since it's only used once, for > > the traversal, at kernel startup. > > Erm, they could do the same with the little mtx_args structs if > they want to as well, but I think it's more effor than its worth. Actually, I don't think they could. The intermediate stage for that would really suck. The only way to handle it is to use different ELF sections for that datam so that you could discard the section later. This is worthwhile for device probe code, maybe attach code, any linker set contents, but not really for individial small structures each getting jammed into their own page size object. 8-). You'd need to do a lot more work, including KVA defragmentation by being able to sectionally attribute anything in the paging path, to keep it from being moved around, or the move-it-around code itself. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
Don Bowman wrote: > > Andrew Gallatin writes: > >> Kenneth D. Merry writes: > >> > > >> > I have released a new set of zero copy sockets patches, against > -current > >> > from today (May 17th, 2002). > > > > Hi Ken, > > > > I'm glad to see that you're still maintining this! > > > > Assuming the mutex issues get sorted out, what do you think the odds > > are of getting this into the tree? The only possible issue I see is > > with the tigon firmware. Is the firmware you're using of the same > > vintage as what's in the tree now? Does it contain all the same > > fixes? > > As a related question, will this work with the broadcom gigabit (bge) > driver, which is the Tigon III? If not, what would it take to get > it working? Broadcom is a bit more anal about people having access to their firmware to make their products meet their design capabilities. Really annoying about the whole Broadcom/Alteon thing... To do the work, you'd have to do it on your own, after licensing the firmware, after signing an NDA. Unlike the rather public Tigon II firmware, the Tigon III doesn't have a lot of synergy or interesting work going for it. Most people doing interesting work tend to use Tigon II cards, because of this. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On 18-May-2002 Terry Lambert wrote: > John Baldwin wrote: >> > God, it's annoying that a statically declared mutex is not >> > defacto initialized. >> >> Is it in solaris? > > It isn't in FreeBSD because of the need to link mutex'es into > the "witness protection program". 8-). Actually, there is more to it than that. Or at least, there will be when turnstiles are added (turnstiles require some function callouts to work properly). > MUTEX_DECLARE(mutex_name). Umm, yes, like MTX_SYSINIT(). :) > You could do this with a SYSINIT(), as has been suggested, but > that would add a relatively large per mutex overhead for each > one you want to declare, since you'd basically be repeating the > common components for doing the job for each and every mutex, > instead of sharing them. Umm, this is a boot-up thing so it's not that big of a deal, plus the actual code isn't duplicated, we call a common function for the actual mutex initialization. > Technically, some later programmer could come along and recover > the linker set memory, actually, since it's only used once, for > the traversal, at kernel startup. Erm, they could do the same with the little mtx_args structs if they want to as well, but I think it's more effor than its worth. > -- Terry -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
John Baldwin wrote: > > God, it's annoying that a statically declared mutex is not > > defacto initialized. > > Is it in solaris? It isn't in FreeBSD because of the need to link mutex'es into the "witness protection program". 8-). > > Yeah, I understand the "witness" crap (if it's there); that > > doesn't make it any less annoying. > > > > Actually, a linker set (not a SYSINIT) could fix that... you > > would still need one sysinit to do the linkage of the statically > > declared structures, but it's at least doable. > > a SYSINIT just is a linker set, and there is a convenience SYSINIT > MTX_SYSINIT() or what not that just registers a sysinit to initialize > a mutex. What you want to do is implement a: MUTEX_DECLARE(mutex_name). This would implicitly add the mutex into the limker set containing the addresses of statically declared mutex'es. The SYSINIT()'s purpose would be to traverse this linker set, calling the moral equivalent of "mutex_init" on each one of them. You could do this with a SYSINIT(), as has been suggested, but that would add a relatively large per mutex overhead for each one you want to declare, since you'd basically be repeating the common components for doing the job for each and every mutex, instead of sharing them. Technically, some later programmer could come along and recover the linker set memory, actually, since it's only used once, for the traversal, at kernel startup. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
RE: new zero copy sockets patches available
> Andrew Gallatin writes: >> Kenneth D. Merry writes: >> > >> > I have released a new set of zero copy sockets patches, against -current >> > from today (May 17th, 2002). > > Hi Ken, > > I'm glad to see that you're still maintining this! > > Assuming the mutex issues get sorted out, what do you think the odds > are of getting this into the tree? The only possible issue I see is > with the tigon firmware. Is the firmware you're using of the same > vintage as what's in the tree now? Does it contain all the same > fixes? As a related question, will this work with the broadcom gigabit (bge) driver, which is the Tigon III? If not, what would it take to get it working? To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
Kenneth D. Merry writes: > > I have released a new set of zero copy sockets patches, against -current > from today (May 17th, 2002). > > The main change is to deal with the vfs_ioopt changes that Alan Cox made in > kern_subr.c. (They conflicted a bit with the zero copy receive code.) > > The patches and the FAQ are available here: > > http://people.freebsd.org/~ken/zero_copy/ > > Comments, questions and reviews are all welcome! > Hi Ken, I'm glad to see that you're still maintining this! Assuming the mutex issues get sorted out, what do you think the odds are of getting this into the tree? The only possible issue I see is with the tigon firmware. Is the firmware you're using of the same vintage as what's in the tree now? Does it contain all the same fixes? Thanks again for keeping this alive, Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
:Alfred Perlstein wrote: :> * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 23:31] wrote: :> > The problem here is that the mutex needs to be initialized before I can :> > acquire it, and there's going to be a race between checking to see :> > whether it has been initialized and actually initializing it. :> > :> ... :> > Suggestions? :> :> *slaps forhead* :> :> Probably a SYSINIT? mutex(9) and sx(9) describe two macros for this purpose. -- Andrew R. Reiter [EMAIL PROTECTED] [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On 18-May-2002 Terry Lambert wrote: > Alfred Perlstein wrote: >> * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 23:31] wrote: >> > The problem here is that the mutex needs to be initialized before I can >> > acquire it, and there's going to be a race between checking to see >> > whether it has been initialized and actually initializing it. >> > >> ... >> > Suggestions? >> >> *slaps forhead* >> >> Probably a SYSINIT? > > God, it's annoying that a statically declared mutex is not > defacto initialized. Is it in solaris? > Yeah, I understand the "witness" crap (if it's there); that > doesn't make it any less annoying. > > Actually, a linker set (not a SYSINIT) could fix that... you > would still need one sysinit to do the linkage of the statically > declared structures, but it's at least doable. a SYSINIT just is a linker set, and there is a convenience SYSINIT MTX_SYSINIT() or what not that just registers a sysinit to initialize a mutex. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On 18-May-2002 Kenneth D. Merry wrote:
> On Fri, May 17, 2002 at 23:02:55 -0700, Alfred Perlstein wrote:
>> * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 22:40] wrote:
>> >
>> > I have released a new set of zero copy sockets patches, against -current
>> > from today (May 17th, 2002).
>> >
>> > The main change is to deal with the vfs_ioopt changes that Alan Cox made
>> > in
>> > kern_subr.c. (They conflicted a bit with the zero copy receive code.)
>> >
>> > The patches and the FAQ are available here:
>> >
>> > http://people.freebsd.org/~ken/zero_copy/
>> >
>> > Comments, questions and reviews are all welcome!
>>
>> jumbo_vm_init() has a bunch of bugs
>>
>> first it doesn't work right if called concurrently.
>> you need to protect the initialization of jumbo_vm_object otherwise
>> bad things can happen. my suggestion is to store the results of
>> vm_object_allocate into a temporary, grab the mutex and then check
>> to see if jumbo_vm_object has been initialized again if it has then
>> free the object, otherwise store the allocated object into the
>> global and continue.
>
> The problem here is that the mutex needs to be initialized before I can
> acquire it, and there's going to be a race between checking to see
> whether it has been initialized and actually initializing it.
>
> e.g.:
>
> if (!mtx_initialized(&jumbo_mutex))
> mtx_init(&jumbo_mutex, "jumbo mutex", NULL, MTX_DEF);
>
> mtx_lock(&jumbo_mutex);
>
> if (jumbo_vm_object != NULL) {
> mtx_unlock(&jumbo_mutex);
> return (1);
> }
>
> /* allocate our object */
> jumbo_vm_object = vm_object_allocate(OBJT_DEFAULT, JUMBO_MAX_PAGES);
>
> The above would work, I think, if it weren't for the race in the mutex
> initialization, and assuming I can allocate a vm object while holding
> the jumbo mutex.
>
> Suggestions?
Either use a sysinit or something gross like this:
static int jumbo_init = 0;
...
while (jumbo_init != 2) {
if (atomic_cmpset_acq_int(&jumbo_init, 0, 1) {
mtx_init(&jumbo_mutex, "jumbo_mutex", NULL, MTX_DEF);
atomic_store_rel_int(&jumbo_init, 2);
}
}
mtx_lock(&jumbo_mutex);
...
a sysinit is probably best though.
--
John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve!" - http://www.FreeBSD.org/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
Alfred Perlstein wrote: > * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 23:31] wrote: > > The problem here is that the mutex needs to be initialized before I can > > acquire it, and there's going to be a race between checking to see > > whether it has been initialized and actually initializing it. > > > ... > > Suggestions? > > *slaps forhead* > > Probably a SYSINIT? God, it's annoying that a statically declared mutex is not defacto initialized. Yeah, I understand the "witness" crap (if it's there); that doesn't make it any less annoying. Actually, a linker set (not a SYSINIT) could fix that... you would still need one sysinit to do the linkage of the statically declared structures, but it's at least doable. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
* Kenneth D. Merry <[EMAIL PROTECTED]> [020517 23:31] wrote: > > The problem here is that the mutex needs to be initialized before I can > acquire it, and there's going to be a race between checking to see > whether it has been initialized and actually initializing it. > ... > Suggestions? *slaps forhead* Probably a SYSINIT? > Cool, thanks for the feedback! np, this is promising work! -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductible donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
On Fri, May 17, 2002 at 23:02:55 -0700, Alfred Perlstein wrote:
> * Kenneth D. Merry <[EMAIL PROTECTED]> [020517 22:40] wrote:
> >
> > I have released a new set of zero copy sockets patches, against -current
> > from today (May 17th, 2002).
> >
> > The main change is to deal with the vfs_ioopt changes that Alan Cox made in
> > kern_subr.c. (They conflicted a bit with the zero copy receive code.)
> >
> > The patches and the FAQ are available here:
> >
> > http://people.freebsd.org/~ken/zero_copy/
> >
> > Comments, questions and reviews are all welcome!
>
> jumbo_vm_init() has a bunch of bugs
>
> first it doesn't work right if called concurrently.
> you need to protect the initialization of jumbo_vm_object otherwise
> bad things can happen. my suggestion is to store the results of
> vm_object_allocate into a temporary, grab the mutex and then check
> to see if jumbo_vm_object has been initialized again if it has then
> free the object, otherwise store the allocated object into the
> global and continue.
The problem here is that the mutex needs to be initialized before I can
acquire it, and there's going to be a race between checking to see
whether it has been initialized and actually initializing it.
e.g.:
if (!mtx_initialized(&jumbo_mutex))
mtx_init(&jumbo_mutex, "jumbo mutex", NULL, MTX_DEF);
mtx_lock(&jumbo_mutex);
if (jumbo_vm_object != NULL) {
mtx_unlock(&jumbo_mutex);
return (1);
}
/* allocate our object */
jumbo_vm_object = vm_object_allocate(OBJT_DEFAULT, JUMBO_MAX_PAGES);
The above would work, I think, if it weren't for the race in the mutex
initialization, and assuming I can allocate a vm object while holding
the jumbo mutex.
Suggestions?
> you may not call malloc(9) with M_WAITOK while holding a mutex.
Ahh, okay.
> entry = jumbo_kmap_inuse.slh_first;
>
> I'm sure that should use a list macro.
Yes, SLIST_FIRST(), thanks.
> That's all I see off the bat. :) Looks cool though.
Cool, thanks for the feedback!
Ken
--
Kenneth Merry
[EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: new zero copy sockets patches available
* Kenneth D. Merry <[EMAIL PROTECTED]> [020517 22:40] wrote: > > I have released a new set of zero copy sockets patches, against -current > from today (May 17th, 2002). > > The main change is to deal with the vfs_ioopt changes that Alan Cox made in > kern_subr.c. (They conflicted a bit with the zero copy receive code.) > > The patches and the FAQ are available here: > > http://people.freebsd.org/~ken/zero_copy/ > > Comments, questions and reviews are all welcome! jumbo_vm_init() has a bunch of bugs first it doesn't work right if called concurrently. you need to protect the initialization of jumbo_vm_object otherwise bad things can happen. my suggestion is to store the results of vm_object_allocate into a temporary, grab the mutex and then check to see if jumbo_vm_object has been initialized again if it has then free the object, otherwise store the allocated object into the global and continue. you may not call malloc(9) with M_WAITOK while holding a mutex. --- entry = jumbo_kmap_inuse.slh_first; I'm sure that should use a list macro. --- That's all I see off the bat. :) Looks cool though. -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductible donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
