Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Thu, 15 Jul 1999 11:09:01 -0700 (PDT), Matthew Dillon dil...@apollo.backplane.com said: Umm... how are you getting the reserved numbers? pstat -s on SunOS4, and swap -s on SunOS5. From Solaris man page: :-s Print summary information about total swap : space usage and availability: : : allocated The total amount of swap space : (in 1024-byte blocks) : currently allocated for use as : backing store. : : reservedThe total amount of swap space : (in 1024-bytes blocks) not : currentlyallocated,but : claimed by memory mappings for : possible future use. : : usedThe total amount of swap space : (in 1024-byte blocks) that is : either allocated or reserved. -- soda To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:pstat -s on SunOS4, and swap -s on SunOS5. From Solaris man page: : ::-s Print summary information about total swap :: space usage and availability: :: :: allocated The total amount of swap space :: (in 1024-byte blocks) :: currently allocated for use as :: backing store. :: :: reservedThe total amount of swap space :: (in 1024-bytes blocks) not :: currentlyallocated,but :: claimed by memory mappings for :: possible future use. :: :: usedThe total amount of swap space :: (in 1024-byte blocks) that is :: either allocated or reserved. :-- :soda Yah, that's what I thought. A solaris expert could tell us for sure but I am pretty sure those are simply cached swap blocks after-the-fact, not actual reservations on potentially swappable space. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Advice on deriving accurate time values from the kernel?
If you only want to timestamp events and not generate the event, you can use microtime() or nanotime(). On a 400MHz PII non-SMP you should get 2.5 ns resolution with nanotime(). On a normal kernel with kern.timecounter.method at the default of 0, the get... versions give you time at the last tick or even worse, so they are no good for that. John -- John Hay -- john@mikom.csir.co.za Hi, I am in the process of developing a device driver for the purpose of stepper motor control. The timing of each pulse is determined by external timing hardware on an I/O board, which will fire an interrupt after the time requested. Using this method, I am able to generate streams of pulses at approximately 5000Hz on a Pentium II 400MHz system. Everything seems to be working well, but I'd really like to gather some accurate timing data in order to derive some statistics to from the system. Intuition tells me I'll need a clock with a tick rate of at least 2 Hz to derive this. So, is such a thing available in the kernel? I've searched through various mailing list archives and have found reference to the HZ option to the kernel, which works to a point. However, it is not ideal as setting HZ to high values generates far too much kernel overhead. Also being considered is additional external timing hardware, but this is something I'd rather avoid for many reasons. What I am after is not a timer as such - all I need to do is derive a time value at an initial time, and a subsequent value at a later time. I've used getmicrouptime, but this appears dependent on the Hz option, and as such is of limited use. I've just had some input from a colleauge who has suggested using the Pentium profiling registers, which we are currently investigating... Any advice gratefully received, -- Jennifer Clark http://telepresence.dmem.strath.ac.uk http://www.crmjewellery.co.uk http://www.furniturenet.co.uk To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
::-s Print summary information about total swap :: space usage and availability: :: :: allocated The total amount of swap space :: (in 1024-byte blocks) :: currently allocated for use as :: backing store. :: :: reservedThe total amount of swap space :: (in 1024-bytes blocks) not :: currentlyallocated,but :: claimed by memory mappings for :: possible future use. :: :: usedThe total amount of swap space :: (in 1024-byte blocks) that is :: either allocated or reserved. :-- :soda It would be really easy to test this. Write a program that malloc's 32MB of space and touches it, then sleeps 10 seconds and forks, with both child and parent sleeping afterwords. ( the parent and the forked child should not touch the memory after the fork occurs ). Do a pstat -s before, after the initial touch, and after the fork. If you do not see the reserved swap space jump by 32MB after the fork, it isn't what you thought it was. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
650 MB MFS?
Are there any design limits to mfs? I want to use cdrecord to write to a dozen or so CD's at once, and fear making lots of coasters if I run them all off a single on-disk file. However, a CD only holds 650 MB, so it seems like I could have the image on mfs and sleep well sans coasters. Would FreeBSD handle an mfs of this size? Thanks! --- David Miller To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: 650 MB MFS?
: :Are there any design limits to mfs? I want to use cdrecord to write to a :dozen or so CD's at once, and fear making lots of coasters if I run them :all off a single on-disk file. However, a CD only holds 650 MB, so it :seems like I could have the image on mfs and sleep well sans coasters. : :Would FreeBSD handle an mfs of this size? : :Thanks! : :--- David Miller Well, if you have 650MB of ram available... I suppose. Otherwise MFS is just going to shove the data into swap. The answer is, yes you can create an MFS partition that large. You have to make sure that you have sufficient swap available and that your datasize resource limit is big enough. So, checklist: * You will need 650MB of swap, possibly even more. (unless you have 650MB+ of ram in your system) * from csh, 'unlimit data' then type 'limit' to see what your maximum datasize limit is. You may have to reconfigure your kernel to increase it: optionsMAXDSIZ=(1024*1024*1024) * Look into using the VN device instead of MFS. VN allows you to create a 'disk file' and then turn it into a partition that you can then mount and use normally. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: gdb instead of adb
Mike Smith wrote: Is the reason why adb hasn't been ported to freebsd because the source is proprietary? You make no sense. If gdb should suffice for my debugging needs, how can a breakpoint be set at a particular interrupt, or even at any interrupt? The break command only seems to accept functions, offsets, linenumbers and addresses... I'm all out of ideas and the gdb info file isn't helping. You make little more sense, unless you are talking about using gdb-remote on a running kernel, in which case you should know that interrupts are vectored through functions, and thus the entire issue is moot. Note also that debugging through interrupt handlers can be problematic on PC hardware. Logic analyzers and/or ICEs are your friends here. Expensive friends. -- Where am I, and what am I doing in this handbasket? Wes Peters Softweyr LLC http://softweyr.com/ w...@softweyr.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: printing
On Thu, Jul 15, 1999 at 10:44:57AM +0100, Alex Knowles wrote: I hope this is the right place to post, sorry if it's not. I'm really sorry to post what is probably a repeat question, but I've just upgraded to freebsd 3.2-release and I'm having real problems getting the kernel to see my printer ports: here is my kernel device ppc0at isa? port? flags 0x40 net irq 7 controller ppbus0 device lpt0at ppbus? device plip0 at ppbus? device ppi0at ppbus? and here is my dmesg output: ppc0 at 0x3bc irq 7 flags 0x40 on isa ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode ppi0: generic parallel i/o on ppbus 0 plip0: PLIP network interface on ppbus 0 whenever I try to access lpt0 it says that the device is not configured. If I try and use the old configuration of lpt and I try and build the kernel I get a whole load of make errors. Quick guess: Remove your device entries in /dev/ and recreate them with /dev/MAKEDEV. There may well be a different major/minor number for the new device. -- Dom Mitchell -- Palmer Harvey McLane -- Unix Systems Administrator In Mountain View did Larry Wall Sedately launch a quiet plea: That DOS, the ancient system, shall On boxes pleasureless to all Run Perl though lack they C. -- ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. ** To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Wed, 14 Jul 1999, John Nemeth wrote: On Jul 15, 2:40am, Daniel C. Sobral wrote: } Garance A Drosihn wrote: } At 12:20 AM +0900 7/15/99, Daniel C. Sobral wrote: } In which case the program that consumed all memory will be killed. } The program killed is +NOT+ the one demanding memory, it's the one } with most of it. } } But that isn't always the best process to have killed off... } } Sure it is. :-) Let's see... This statement is absurd. Only a comptetant admin can decide which process can be killed. No arbitrary decision is going to be correct. } It would be nice to have a way to indicate that, a la SIGDANGER. How about assigning something like a class to process, which gives VM a hint which processes should be killed first without much thinking, and which the last (or never)? In other words, let's say class 10 means totally disposable, kill whenever you want, and class 1 means never try to kill me. Of course, most processes would get some default value, and superuser could renice them to more resistant class. This way both sides of the discussion would be satisfied :-) Andrzej Bialecki // ab...@webgiro.com WebGiro AB, Sweden (http://www.webgiro.com) // --- // -- FreeBSD: The Power to Serve. http://www.freebsd.org // --- Small Embedded FreeBSD: http://www.freebsd.org/~picobsd/ To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: seg fault in mutex_queue_enq
I don't care one way or the other. I could leave out the wrapped poll() very easily and avoid the issue all together. This would provide -stable with everything -current has, except of course poll(). I'd prefer to add poll, though... I'm OK with adding poll(), it just seemed odd that the version number bumped with no API interface changes taking place. Handle it however you best see fit. :) - Jordan To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Help with PCI code understanding
Can someone outline the initialization process of PCI devices in FreeBSD? I know many of the basic stuff of PCI introduced in the book PCI System Architecture. I just want to know how each driver is registered into some linker set and its probe routine gets called. In other words, I want to know the major data structures and routines and their relationship. I wonder if there is already a document somewhere. Any help is appreciated. -Zhihui To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
[Hijacked from freebsd-security]
On Thu, 15 Jul 1999 17:33:29 -0400, Garance A Drosihn wrote:
What I wanted to do was have estr routines, where the destination
is specified as the starting point and the ending point of the area
available for the string (as two parameters). The routines would
return the position of the current string-terminator. So you could
do things like:
As I understand it, the goal here is to return to the caller the number
of bytes copied (however you represent it), so that the caller can
easily determine whether or not dst is safe for operations demanding a
null-terminated string.
If that is true, then I think the interface you propose is overly
complex. Looking at the existing functions, their only flaw is that they
return known (and therefore useless) information, wasting the return
value. All we need is:
size_t
foocpy(char *dst, const char *src)
size_t
fooncpy(char *dst, const char *src, size_t len)
size_t
foocat(char *s, const char *append)
size_t
fooncat(char *s, const char *append, size_t count)
where the return value is the number of bytes {copied,appended}.
Since the goal is simply to make it easier to do what is already
possible, I think that this approach is better than what you're
suggesting, since the pointer arithmetic required in the caller is
simpler.
And since the prototypes for fooncpy and fooncat above match exactly
those of the proposed strlcpy and strlcat respectively (just had a
look before I hit the send button), I'd say that the latter two are
definitely the functions you want.
Ciao,
Sheldon.
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: make fails in 3.2-RELEASE for netboot
According to Gregory A. Carter: I'm assuming that might have something to do with it. The file scrt0.c This is the old a.out crt code. The one in 3.0+ is crt1.c, look into /usr/src/lib/csu/i386-elf/. -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- robe...@keltia.freenix.fr FreeBSD keltia.freenix.fr 4.0-CURRENT #72: Mon Jul 12 08:26:43 CEST 1999 To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Fri, Jul 16, 1999 at 12:15:31AM +0200, Sheldon Hearn wrote:
As I understand it, the goal here is to return to the caller the number
of bytes copied (however you represent it), so that the caller can
easily determine whether or not dst is safe for operations demanding a
null-terminated string.
[...]
size_t
fooncat(char *s, const char *append, size_t count)
where the return value is the number of bytes {copied,appended}.
Eeks! This will quickly lead to code like
if (fooncat(string, append, sizeof(string)) != strlen(append))
...
which is rather evil, given that the second strlen(append) would be
completely gratuitous if it weren't for the interface you're
suggesting.
--
This is my .signature which gets appended to the end of my messages.
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:Before program start: :total: 2k bytes allocated + 4792k reserved = 24792k used, 191048k available : :After malloc, before touch: :total: 18756k bytes allocated + 37500k reserved = 56256k used, 159580k available : :After malloc + touch: :total: 52804k bytes allocated + 4852k reserved = 57656k used, 158184k available : :After fork: :total: 52928k bytes allocated + 37644k reserved = 90572k used, 125264k available : :[there has been a little background activity, but the numbers speak for themselves] : : :Daniel Assuming the allocated field is not inclusive of real memory, what we have is swap reservation under solaris for clean pages, and allocation and assignment for dirty pages. The grand total will tell you the total VM potential for malloc'd space but does not appear to tell you how much swap is actually active - i.e. was written to and contains valid data. It would be interesting to see if the stack segment is included in the reservation. Try setting the stack resource limit to 32m and run the same program, except without bothering to malloc() or touch anything. See if the stack segment is included in the reservation field. It would also be interesting to see how solaris deals with MAP_PRIVATE mmap's. If this is correct, then solaris is using a VMSPACE = SWAPSPACE model. FreeBSD uses a VMSPACE = SWAPSPACE + REALMEM model. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
In article local.mail.freebsd-hackers/199907151825.laa11...@apollo.backplane.com you write: ::-s Print summary information about total swap :: space usage and availability: :: :: allocated The total amount of swap space :: (in 1024-byte blocks) :: currently allocated for use as :: backing store. :: :: reservedThe total amount of swap space :: (in 1024-bytes blocks) not :: currentlyallocated,but :: claimed by memory mappings for :: possible future use. :: :: usedThe total amount of swap space :: (in 1024-byte blocks) that is :: either allocated or reserved. :-- :soda It would be really easy to test this. Write a program that malloc's 32MB of space and touches it, then sleeps 10 seconds and forks, with both child and parent sleeping afterwords. ( the parent and the forked child should not touch the memory after the fork occurs ). Do a pstat -s before, after the initial touch, and after the fork. If you do not see the reserved swap space jump by 32MB after the fork, it isn't what you thought it was. aladdin[5:32pm] prtconf System Configuration: Sun Microsystems i86pc Memory size: 128 Megabytes aladdin[5:41pm] uname -a SunOS aladdin 5.6 Generic_105182-14 i86pc i386 total: 67280k bytes allocated + 28668k reserved = 95948k used, 196460k avail malloced 32MB... total: 67320k bytes allocated + 61460k reserved = 128780k used, 163592k avail touched... total: 100084k bytes allocated + 28696k reserved = 128780k used, 163732k avail forking... total: 100092k bytes allocated + 61520k reserved = 161612k used, 130864k avail touching again (parent)... touching again (child)... total: 132864k bytes allocated + 28748k reserved = 161612k used, 130760k avail exiting... exiting... total: 67248k bytes allocated + 28700k reserved = 95948k used, 196448k avail -- Jonathan To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
If this is correct, then solaris is using a VMSPACE = SWAPSPACE model. FreeBSD uses a VMSPACE = SWAPSPACE + REALMEM model. AFAIK it has been stated quite explicitly by the Solaris folks that Solaris 2.x uses VMSPACE = SWAPSPACE + REALMEM. This is *different* from SunOS 4.1.x. Steinar Haug, Nethelp consulting, sth...@nethelp.no To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Fri, Jul 16, 1999 at 12:15:31AM +0200, Sheldon Hearn wrote:
As I understand it, the goal here is to return to the caller the number
of bytes copied (however you represent it), so that the caller can
easily determine whether or not dst is safe for operations demanding a
null-terminated string.
[...]
size_t
fooncat(char *s, const char *append, size_t count)
where the return value is the number of bytes {copied,appended}.
Eeks! This will quickly lead to code like
if (fooncat(string, append, sizeof(string)) != strlen(append))
...
which is rather evil, given that the second strlen(append) would be
completely gratuitous if it weren't for the interface you're
suggesting.
What's really stupid is that most of the time you're trying to use
these functions to fix code that looks like:
strcpy(buf, str1);
strcat(buf, str2);
strcat(buf, str3);
without overflowing buf. This is dumb! Use asprintf instead:
asprinf(buf, %s%s%s, str1, str2, str3);
If you can't keep all of the string elements together at once, try:
asprinf(buf, %s%s, str1, str2);
...
asprintf(buf2, %s%s, buf, str3);
free(buf);
No, it's not fast, but it _is_ robust.
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ msm...@freebsd.org
\\-- Joseph Merrick \\ msm...@cdrom.com
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999 18:34:42 -0400, Tim Vanderhoek wrote:
if (fooncat(string, append, sizeof(string)) != strlen(append))
...
which is rather evil, given that the second strlen(append) would be
completely gratuitous if it weren't for the interface you're
suggesting.
Tim, you're doing that I'm right, but too concise to be understood
thing again. What are you saying here? :-)
If all you're saying is that you want an API that doesn't require a test
against the known length of src (append in your example), then you won't
like strl*. :-)
You'd probably prefer the functions to return the number of bytes which
they did not manage to {copy,append}, yes? Lazy bastard [1]. :-)
While this might be something we add, it shouldn't be called
strl{cpy,cat}. And the original question was whether or not we'd add the
strl{cpy,cat} functions to libc. If we do, I seriously hope I'll be
given the opportunity to submit a replacement manpage, since theirs
sucks.
Ciao,
Sheldon.
[1] It's usually the lazy guy who demands the best API, provided his
demands are tempered by the pedantic guy.
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999, Mike Smith wrote: What's really stupid is that most of the time you're trying to use these functions to fix code that looks like: strcpy(buf, str1); strcat(buf, str2); strcat(buf, str3); without overflowing buf. This is dumb! Use asprintf instead: There was a talk on these (strlcpy(3) and strlcat(3)) at USENIX. The logic as to their design was presented and I agree totally with the way that the logic was played out into the functions. They are described in the FreeNIX proceedings on page 175. I feel they make a lot more sense that teh present version sand we should support OpenBSD's application to Posix to make them standard. They lead to faster overall code, (people using present functions often re-count the size of result strings, or check the end of the buffer for a 0) and lead to more readable code. All of these age a Good Thing (TM). asprinf(buf, %s%s%s, str1, str2, str3); Very nice but not always applicable. Particularly when compiling strings under algorythmic control. If you can't keep all of the string elements together at once, try: asprinf(buf, %s%s, str1, str2); ... asprintf(buf2, %s%s, buf, str3); free(buf); No, it's not fast, but it _is_ robust. Why not be fast AND robust? :-) I like the new functions and would like to see them supported. julian To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999, Julian Elischer wrote: There was a talk on these (strlcpy(3) and strlcat(3)) at USENIX. The logic as to their design was presented and I agree totally with the way that the logic was played out into the functions. They are described in the FreeNIX proceedings on page 175. I feel they make a lot more sense that teh present version sand we should support OpenBSD's application to Posix to make them standard. Yes, this discussion started out on freebsd-security and when I first wrote about it, I mentioned the paper at USENIX by Todd Miller and Theo de Raadt. It was later mentioned that the paper and accompanying slides are available at: http://www.openbsd.org/papers/strlcpy-paper.ps http://www.openbsd.org/papers/strlcpy-slides.ps I think each function is well thought out and I think they'd be a great addition to FreeBSD. Paul Hart -- Paul Robert Hart8 8 8Verio Web Hosting, Inc. h...@iserver.com8 8 8http://www.iserver.com/ To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Here is what I get from one of BEST's mail www proxy machines. ~dillon/br adds the object size's together. 'swap' and 'default' objects refers to unbacked VM objects - and none of the processes running fork shared unbacked objects so we don't have to worry about that. The 'swap' designation means that at least one page in the object has been assigned swap. The default designation means that no pages have been assigned swap. The pages can be dirty or clean. Typical /proc/PID/map output looks like this (taken from one of the sendmail processes). The lines I've marked are the ones being counted as unbacked/swap-backed VM. The rest are vnode-backed and not counted. 0x1000 0x4b000 66 0 r-x COW vnode 0x4b0000x4e0003 3 rwx COW vnode 0x4e0000x87000 5343 rwx COW swap --- 0x870000x373000 738 738 rwx default --- 0x2004b000 0x2005a000 2 0 r-x COW vnode 0x2005a000 0x2005c000 2 0 rwx COW vnode 0x2005c000 0x20065000 6 2 rwx COW swap --- 0x20068000 0x2006d000 3 0 r-x COW vnode 0x2006d000 0x2006e000 1 1 rwx COW vnode 0x2006e000 0x200cc00070 0 r-x COW vnode 0x200cc000 0x200d 4 4 rwx COW vnode 0x200d 0x200e7000 8 6 rwx COW swap --- 0xefbde000 0xefbfe0001414 rwx COW swap --- proxy1:/tmp# cat /proc/*/map | egrep 'swap|default' | ~dillon/br 639168K proxy1:/tmp# pstat -s Device 1K-blocks UsedAvail Capacity Type /dev/sd0b 52428812596 511628 2%Interleaved This machine has 256MB of ram of which around 200MB is in use, we will assume the entire 200MB is used by VM spaces for processes. It is an active machine with around 205 processes at the time of the test. So. 200MB of ram + 12MB of swap = 212MB of actual storage being used out of 639MB of total swap-backable VM. About a factor of 3.2:1. Actual swap utilization is sitting at 2%. If no overcommit were allowed, and assuming a VMSPACE = REALMEM + SWAP model, 200MB of ram would be active and 439MB worth of swap would be either allocated or reserved ( though only 12MB would be actually written, that part doesn't change ). 439MB of swap verses 12MB of swap. In that scenario, the 512MB of swap I assigned to this machine would be dangerously low. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
What's really stupid is that most of the time you're trying to use these functions to fix code that looks like: strcpy(buf, str1); strcat(buf, str2); strcat(buf, str3); without overflowing buf. This is dumb! Use asprintf instead: There was a talk on these (strlcpy(3) and strlcat(3)) at USENIX. The logic as to their design was presented and I agree totally with the way that the logic was played out into the functions. I'm inclined to disagree, simply because they take too fine-grained an approach to the problem. asprinf(buf, %s%s%s, str1, str2, str3); Very nice but not always applicable. Particularly when compiling strings under algorythmic control. I'm not sure what compiling strings under algorithmic control is actually meant to mean. The only addition I'd want to make to asprintf() is reasprintf() which reallocs and appends to the end of an already existing string. No, it's not fast, but it _is_ robust. Why not be fast AND robust? :-) The two are generally not compatible. I like the new functions and would like to see them supported. Obviously, I'm not in agreement here. The original trend (assembling strings piecemeal) was expedient but not really a great idea in the long term. Likewise there are a plethora of hand-rolled string parsers that could be replaced with sscanf() these days. Adding fancier piecemeal operators just continues to encourage the old, unsafe style. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999, Julian Elischer wrote: There was a talk on these (strlcpy(3) and strlcat(3)) at USENIX. The logic as to their design was presented and I agree totally with the way that the logic was played out into the functions. They are described in the FreeNIX proceedings on page 175. I feel they make a lot more sense that teh present version sand we should support OpenBSD's application to Posix to make them standard. Yes, this discussion started out on freebsd-security and when I first wrote about it, I mentioned the paper at USENIX by Todd Miller and Theo de Raadt. It was later mentioned that the paper and accompanying slides are available at: http://www.openbsd.org/papers/strlcpy-paper.ps http://www.openbsd.org/papers/strlcpy-slides.ps I think each function is well thought out and I think they'd be a great addition to FreeBSD. Ugh. Take the first example in the paper; it rewrites as len = asprintf(path, %s/.foorc); as opposed to strlcat(path, homedir, sizeof(path)); strlcat(path, /, sizeof(path)); strlcat(path, .foord, sizeof(path)); len = strlen(path); Yes, they're a better str*cat/cpy, but they're not the solution that they claim to be. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
Ugh. Take the first example in the paper; it rewrites as len = asprintf(path, %s/.foorc); ^ , homedir Whoops. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999, Mike Smith wrote: Ugh. Take the first example in the paper; it rewrites as len = asprintf(path, %s/.foorc); as opposed to strlcat(path, homedir, sizeof(path)); strlcat(path, /, sizeof(path)); strlcat(path, .foord, sizeof(path)); len = strlen(path); Yes, they're a better str*cat/cpy, but they're not the solution that they claim to be. I don't think that anyone has intended them to be anything other than a better replacement for strcpy/strcat than strncpy/strncat (which they certainly are). Sure, you could go around telling people use snprintf instead or use asprintf instead, but is that the issue at hand? Paul Hart -- Paul Robert Hart8 8 8Verio Web Hosting, Inc. h...@iserver.com8 8 8http://www.iserver.com/ To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Fri, Jul 16, 1999 at 12:53:13AM +0200, Sheldon Hearn wrote:
If all you're saying is that you want an API that doesn't require a test
against the known length of src (append in your example), then you won't
like strl*. :-)
Well, if I read your message correctly, the difference between fooncat()
and strlcat() would be that strlcat() returns the total number of
chars in (or would be in) the destination string, whereas fooncat()
returns the total number of chars copied. The former, strlcat(),
avoids the problem I was noting.
Looking at OpenBSD's actual definition of strlcat() which returns the
number of chars that would have been in the final string is
potentially non-useful, but not really too terrible.
[If I'm using strlcat() in the first place, am I _really_ going to
care how many chars would have been copied? Maybe in legacy code,
but in anything newer...]
You'd probably prefer the functions to return the number of bytes which
they did not manage to {copy,append}, yes? Lazy bastard [1]. :-)
Hmm... That's an interesting idea...
strl{cpy,cat}. And the original question was whether or not we'd add the
strl{cpy,cat} functions to libc. If we do, I seriously hope I'll be
Ahh, well, you did hijack this off of the -security list. Adding
strlcpy() and strlcat() is probably a good idea.
given the opportunity to submit a replacement manpage, since theirs
sucks.
Bah. You're in avail now. Just commit ontop of whatever manpage gets
imported. ;-) If your replacement is good, no one will object. :)
--
This is my .signature which gets appended to the end of my messages.
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
At 12:15 AM +0200 7/16/99, Sheldon Hearn wrote: [Hijacked from freebsd-security] For those who missed the original article, here's the initial topic (from Paul Hart, but truncated a bit): I was just reviewing the proceedings from the USENIX 1999 Annual Technical Conference where Todd Miller and Theo de Raadt presented a paper on two new functions that OpenBSD has integrated into libc. The new functions, strlcpy(3) and strlcat(3), are intended to provide an easily understood means of safe string copying and concatenation to programmers. I was impressed by the paper and wondered if anyone besides myself would be amenable to including them in FreeBSD's libc. If you are a USENIX member you can access the text of the paper at: http://www.usenix.org/events/usenix99/millert.html (or check:) Paper: http://www.openbsd.org/papers/strlcpy-paper.ps Slides (worth looking at too): http://www.openbsd.org/papers/strlcpy-slides.ps In reply to that, I mentioned some routines that I have been meaning to write, to address what I saw as the problems with using strncat and strncpy (before I had heard of the strl* routines). Sheldon is arguing against the routines I was talking about, not the strl* routines that Paul referred to... On Thu, 15 Jul 1999 17:33:29 -0400, Garance A Drosihn wrote: What I wanted to do was have estr routines, where the destination is specified as the starting point and the ending point of the area available for the string (as two parameters). The routines would return the position of the current string-terminator. So you could do things like: As I understand it, the goal here is to return to the caller the number of bytes copied (however you represent it), so that the caller can easily determine whether or not it is safe for operations demanding a null-terminated string. Um, no. that certainly was not my intention with the estr* ideas... It was noticed as a side-effect of what I ended up with, but it wasn't the main objective. And since the prototypes for fooncpy and fooncat above match exactly those of the proposed strlcpy and strlcat respectively (just had a look before I hit the send button), I'd say that the latter two are definitely the functions you want. Well, they aren't exactly the functions *I* would want, but that isn't really the point. I do think the strl* routines are a good idea to have. I would like to see them included in standard C (or at least FreeBSD), because they are better (IMO) than using strncat and strncpy to avoid buffer overflows. Even looking over my OWN code, I come across times that I've used strncat or strncpy wrong. So, while I still SLIGHTLY prefer my estr* ideas over the strl* ideas, it isn't enough that I would argue against the strl* routines being standard. (and the more platforms that have them, the better). --- Garance Alistair Drosehn = g...@eclipse.acs.rpi.edu Senior Systems Programmer or dro...@rpi.edu Rensselaer Polytechnic Institute To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
In that scenario, the 512MB of swap I assigned to this machine would be dangerously low. With 13GB disks available for a couple of hundred bucks, my machines aren't going to run out of swap space any time soon, even if I commit to disk. All I want for Christmas is a knob to disable overcommit. --lyndon To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: make fails in 3.2-RELEASE for netboot
Ollivier Robert [Re: make fails in 3.2-RELEASE for netboot] 7.16.1999 .. . According to Gregory A. Carter: . I'm assuming that might have something to do with it. The file scrt0.c . . This is the old a.out crt code. The one in 3.0+ is crt1.c, look into . /usr/src/lib/csu/i386-elf/. . -- . Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- robe...@keltia.freenix.fr . FreeBSD keltia.freenix.fr 4.0-CURRENT #72: Mon Jul 12 08:26:43 CEST 1999 .. I found it when I went searching however I still can't get the netboot to compile as it was designed for aout. Any ideas of why it wasn't moved to elf along with the rest of the OS? Or if not how *I* can port it to elf instead? Greg +(o...@dynmc.net)--+ | Dynamic Networking Solutions InterX Technologies | | Senior Network Administratorbits/keyID 1024/7DF9C285 | | o...@interx.net o...@itstudio.net o...@undernet.org o...@webpop3.com | +[ DC 50 57 59 C3 76 46 E8 EB 75 A8 94 FE 96 9E D3 ]--+ To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Thu, 15 Jul 1999 17:53:52 CST, lyn...@orthanc.ab.ca wrote: All I want for Christmas is a knob to disable overcommit. And what I'm pretty sure the majority of the readers on this list want is for those of you who really think it's necessary to do it yourselves. What? Nobody who wants to disable the policy knows how to do it? Hmmm, I wonder whether that's significant... Ciao, Sheldon. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999, Mike Smith wrote:
Ugh. Take the first example in the paper; it rewrites as
len = asprintf(path, %s/.foorc);
as opposed to
strlcat(path, homedir, sizeof(path));
strlcat(path, /, sizeof(path));
strlcat(path, .foord, sizeof(path));
len = strlen(path);
Yes, they're a better str*cat/cpy, but they're not the solution that
they claim to be.
I don't think that anyone has intended them to be anything other than a
better replacement for strcpy/strcat than strncpy/strncat (which they
certainly are). Sure, you could go around telling people use snprintf
instead or use asprintf instead, but is that the issue at hand?
In context, yes it is.
The paper talks about dealing with the construction of composite
strings into static buffers, and points out that there's a real problem
when you hit the edge of the buffer (overflow, truncation, etc.)
But they don't address the core of the problem, which is the use of a
static buffer in the first place. This and other habitual programming
style items are what's at the real core of the C is an insecure
language argument; people are so used to these idiomatic programming
techniques they refuse to look for better solutions to the larger
problem.
Going back to the example they gave, let's put it in context. You
probably have something like this:
{
struct passwd *pw;
charbuf[MAXPATHLEN];
FILE*fp;
pw = getpwuid(getuid());
strlcpy(buf, pw-dir, sizeof(buf));
strlcat(buf, /.appname/, sizeof(buf));
strlcat(buf, conffilename, sizeof(buf));
if (strlen(buf) = sizeof(buf))
return(error);
fp = fopen(buf, r);
...
That works, as long as MAXPATHLEN is actually long enough. In this
particular case it will be (because fopen will fail otherwise), but
there's no guarantee that you're going to know in advance.
OTOH, here it is with asprintf:
{
struct passwd *pw;
char*buf;
FILE*fp;
pw = getpwuid(getuid());
if (asprintf(buf, %s/.appname/%s, pw-dir, conffilename) == -1)
return(error);
fp = fopen(buf, r);
free(buf);
...
The latter has a few really clear advantages:
- you can see what the string is meant to look like.
- it doesn't matter how long any of the components are.
- the constructed value is on the heap, so you can return it (just
imagine how much nicer ctime() would be if it did this).
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ msm...@freebsd.org
\\-- Joseph Merrick \\ msm...@cdrom.com
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
On Thu, Jul 15, 1999 at 11:48:41PM +0900, Daniel C. Sobral wrote: Actually, applications are written assuming that malloc() will not fail, generally speaking. Is this really the case? I'm pretty sure I've _never_ ignored the possibility of a NULL return from malloc, and I've been using it for nearly 20 years. I usually print a message and exit, but I never ignore it. I thought that was pretty standard practise. This is just a random comment, orthogonal to the overcommit issue, but I've seen both you and Matthew say this now, and I was surprised both times. -- Andrew To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
but what about
While ( more data items)
{
copy data items onto end of buffer
if full{
write out buffer
clear buffer, copy in rest of last item.
}
}
I'd certainly not want to use xxprintf() for that
On Thu, 15 Jul 1999, Mike Smith wrote:
On Thu, 15 Jul 1999, Mike Smith wrote:
Ugh. Take the first example in the paper; it rewrites as
len = asprintf(path, %s/.foorc);
as opposed to
strlcat(path, homedir, sizeof(path));
strlcat(path, /, sizeof(path));
strlcat(path, .foord, sizeof(path));
len = strlen(path);
Yes, they're a better str*cat/cpy, but they're not the solution that
they claim to be.
I don't think that anyone has intended them to be anything other than a
better replacement for strcpy/strcat than strncpy/strncat (which they
certainly are). Sure, you could go around telling people use snprintf
instead or use asprintf instead, but is that the issue at hand?
In context, yes it is.
The paper talks about dealing with the construction of composite
strings into static buffers, and points out that there's a real problem
when you hit the edge of the buffer (overflow, truncation, etc.)
But they don't address the core of the problem, which is the use of a
static buffer in the first place. This and other habitual programming
style items are what's at the real core of the C is an insecure
language argument; people are so used to these idiomatic programming
techniques they refuse to look for better solutions to the larger
problem.
Going back to the example they gave, let's put it in context. You
probably have something like this:
{
struct passwd *pw;
charbuf[MAXPATHLEN];
FILE*fp;
pw = getpwuid(getuid());
strlcpy(buf, pw-dir, sizeof(buf));
strlcat(buf, /.appname/, sizeof(buf));
strlcat(buf, conffilename, sizeof(buf));
if (strlen(buf) = sizeof(buf))
return(error);
fp = fopen(buf, r);
...
That works, as long as MAXPATHLEN is actually long enough. In this
particular case it will be (because fopen will fail otherwise), but
there's no guarantee that you're going to know in advance.
OTOH, here it is with asprintf:
{
struct passwd *pw;
char*buf;
FILE*fp;
pw = getpwuid(getuid());
if (asprintf(buf, %s/.appname/%s, pw-dir, conffilename) == -1)
return(error);
fp = fopen(buf, r);
free(buf);
...
The latter has a few really clear advantages:
- you can see what the string is meant to look like.
- it doesn't matter how long any of the components are.
- the constructed value is on the heap, so you can return it (just
imagine how much nicer ctime() would be if it did this).
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ msm...@freebsd.org
\\-- Joseph Merrick \\ msm...@cdrom.com
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
but what about
While ( more data items)
{
copy data items onto end of buffer
if full{
write out buffer
clear buffer, copy in rest of last item.
}
}
I'd certainly not want to use xxprintf() for that
This is what stdio does, funnily enough. See fwrite() etc.
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ msm...@freebsd.org
\\-- Joseph Merrick \\ msm...@cdrom.com
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907152358.qaa01...@dingo.cdrom.com Mike Smith writes: : if (strlen(buf) = sizeof(buf)) : return(error); This can never be true with the strl functions They don't run off the end, so strlen(buf) is always going to be sizeof(buf) since it doesn't include the traling null. Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907152244.paa01...@dingo.cdrom.com Mike Smith writes: : What's really stupid is that most of the time you're trying to use : these functions to fix code that looks like: : strcpy(buf, str1); : strcat(buf, str2); : strcat(buf, str3); : without overflowing buf. This is dumb! Use asprintf instead: : : asprinf(buf, %s%s%s, str1, str2, str3); : : If you can't keep all of the string elements together at once, try: : : asprinf(buf, %s%s, str1, str2); : ... : asprintf(buf2, %s%s, buf, str3); : free(buf); : : No, it's not fast, but it _is_ robust. That is true for this case, but not always true. I think these APIs have an excellent role to play. Sure, there are other ways to do it, but there are a growing number of systems that have strl* on them (OpenBSD, Linux and Solaris), which is reason enough to improve our portability by using them. The asprintf isn't completely robust becuase you must free() the routine, or face a memory leak. It won't overflow, but it might introduce another bug. The whole point of these APIs was to transition old code to new in a safe manner that isn't prone to potentiall programming errors. Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907152329.qaa01...@dingo.cdrom.com Mike Smith writes: : Ugh. Take the first example in the paper; it rewrites as : : len = asprintf(path, %s/.foorc); : : as opposed to : : strlcat(path, homedir, sizeof(path)); : strlcat(path, /, sizeof(path)); : strlcat(path, .foord, sizeof(path)); : len = strlen(path); : : Yes, they're a better str*cat/cpy, but they're not the solution that : they claim to be. You've forgotten the free(path) sometime later in your code... That's a can of warms you conveniently ignore... And can be big problems for library routines whose API is defined to return stuff into a static buffer... Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907152358.qaa01...@dingo.cdrom.com Mike Smith writes: : if (strlen(buf) = sizeof(buf)) : return(error); This can never be true with the strl functions They don't run off the end, so strlen(buf) is always going to be sizeof(buf) since it doesn't include the traling null. I actually should have copied the original example from the paper, which was disgustingly more verbose; I mistransposed it for the abovem which should probably have been (strlen(buf) == (sizeof(buf) - 1)) or similar. They recommend using: len = strlcpy(path, homedir, sizeof(path)); if (len = sizeof(path)) return(ENAMETOOLONG) etc. I still think this is the wrong way to deal with the problem. 8) -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: gdb instead of adb
On Wednesday, 14 July 1999 at 22:45:32 -0700, Mike Smith wrote: Is the reason why adb hasn't been ported to freebsd because the source is proprietary? You make no sense. What don't you understand? It makes plenty of sense to me (and the answer is: yes). If gdb should suffice for my debugging needs, how can a breakpoint be set at a particular interrupt, or even at any interrupt? The break command only seems to accept functions, offsets, linenumbers and addresses... I'm all out of ideas and the gdb info file isn't helping. You make little more sense, unless you are talking about using gdb-remote on a running kernel, in which case you should know that interrupts are vectored through functions, and thus the entire issue is moot. Translation: to set a breakpoint on an interrupt, set the breakpoint on its interrupt handler. Note also that debugging through interrupt handlers can be problematic on PC hardware. On all hardware. But if Marc has used adb to set breakpoints on interrupt handlers, he'll know that. Greg -- Finger g...@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 19990715194203.a54...@mad Tim Vanderhoek writes: : Looking at OpenBSD's actual definition of strlcat() which returns the : number of chars that would have been in the final string is : potentially non-useful, but not really too terrible. No. It is useful. If you look at the return value, you can detect that an overflow would have happened and bail w/o having the overflow actually happen. That is useful (and even documented in the man page by a nice example). : given the opportunity to submit a replacement manpage, since theirs : sucks. : : Bah. You're in avail now. Just commit ontop of whatever manpage gets : imported. ;-) If your replacement is good, no one will object. :) I'm planning on committing their man page. I don't see problems with it, purhaps people could point them out to me so that both our man pages and theirs could be better. Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907160023.raa02...@dingo.cdrom.com Mike Smith writes: : I still think this is the wrong way to deal with the problem. 8) We mildly disagree here. The strl* functions are the end all, be all of security. They are just designed to make the existing code that uses static buffers easy to make more robust w/o radically altering that code. Of course, strings have always been weak in 'C'. You make them static and they overflow. You malloc them, and often people forget to free them later leading to other problems... Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907160032.saa01...@harmony.village.org Warner Losh writes: : We mildly disagree here. The strl* functions are the end all, be all : of security. NOTE: This should have read: We mildly disagree here. The strl* functions are NOT the end all, be all of security. which changes its meaning quite a bit... Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
On Thu, 15 Jul 1999 18:28:52 CST, Warner Losh wrote: I'm planning on committing their man page. I don't see problems with it, purhaps people could point them out to me so that both our man pages and theirs could be better. As I've said already, there's too much in DESCRIPTION that should be in HISTORY or IMPLEMENTATION NOTES. I almost get the feeling that the manpage doubles as a marketing brochure. The reason I see this as a problem is that there is a great deal of consistency in the string(3) family manpages. This consistency makes it very easy to move from one manpage to the next and spot differences. The strlcpy manpage breaks this style badly. While that's fine when you read it on its own, it's a PITA when you're missioning around the strings(3) manpages. If you see my point, let me know and I'll send you an alternative strlcpy.3 . Ciao, Sheldon. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 199907160023.raa02...@dingo.cdrom.com Mike Smith writes: : I still think this is the wrong way to deal with the problem. 8) We mildly disagree here. The strl* functions are the end all, be all of security. They are just designed to make the existing code that uses static buffers easy to make more robust w/o radically altering that code. Of course, strings have always been weak in 'C'. You make them static and they overflow. You malloc them, and often people forget to free them later leading to other problems... With the addition of a not in your first paragraph, I actually think we're in agreement here. I'm just maintaining that in most of the in-tree cases where static buffers are used, a dynamic buffer would have been a better design choice; you might want to disagree there too of course. 8) Regardless, we should definitely adopt these functions for no other reason than portability, no argument there. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
In message 81768.932085...@axl.noc.iafrica.com Sheldon Hearn writes: : If you see my point, let me know and I'll send you an alternative : strlcpy.3 . I can see your point. I don't know if I'll like your man pages better or not, but I'd be willing to give them a spin. Warner To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: make fails in 3.2-RELEASE for netboot
I found it when I went searching however I still can't get the netboot to compile as it was designed for aout. Any ideas of why it wasn't moved to elf along with the rest of the OS? Or if not how *I* can port it to elf instead? The intention is that loader(8) will provide the same functionality, and the framework is already in place for this. I suggest you use etherboot in the ports collection, at least until the loader support is completed. -- Robert Nordier To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
Andrew Reilly wrote: On Thu, Jul 15, 1999 at 11:48:41PM +0900, Daniel C. Sobral wrote: Actually, applications are written assuming that malloc() will not fail, generally speaking. Is this really the case? I'm pretty sure I've _never_ ignored the possibility of a NULL return from malloc, and I've been using it for nearly 20 years. I usually print a message and exit, but I never ignore it. I thought that was pretty standard practise. You are always free to inspect how applications deal with malloc(), as far as open source software goes. Anyway, your usual behavior is to expect malloc() will not fail. To print a message and exit is to treat it as a fatal error, don't you agree? -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org Would you like to go out with me? I'd love to. Oh, well, n... err... would you?... ahh... huh... what do I do next? To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
but what about
While ( more data items)
{
copy data items onto end of buffer
if full{
write out buffer
clear buffer, copy in rest of last item.
}
}
I'd certainly not want to use xxprintf() for that
This is what stdio does, funnily enough. See fwrite() etc.
I do like the idea of these functions... just ran into a problem a day
or two ago which would be completely awful to do with *printf, since it
involved assembling a line of text through two functions and a for()
loop... I thought briefly about trying to do it with strncat but I
gagged a bit at the obtuse way it would have needed to have been done.
... Joe
---
Joe Greco - Systems Administrator jgr...@ns.sol.net
Solaria Public Access UNIX - Milwaukee, WI 414/342-4847
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
All I want for Christmas is a knob to disable overcommit. And what I'm pretty sure the majority of the readers on this list want is for those of you who really think it's necessary to do it yourselves. What? Nobody who wants to disable the policy knows how to do it? Hmmm, I wonder whether that's significant... that's an impressively bold statement to make. by my reconning, at least 4 people who have posted wanting no overcommit are more than capable of programming this for NetBSD. .mrg. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Advice on deriving accurate time values from the kernel?
I've done some work on measuring things like interrupt response times and the interval between two interesting events or steps in processing. A cheap way to do this is to use the TSC register in the CPU, though you then need to calibrate the frequency that the CPU really runs at. If you're willing to spend some money, you can get hardware to plug into a PCI slot that can return timestamps in 100ns units, as well as generating interrupt at some time in the future, etc. See http://www.bancomm.com/cbc637PCI.htm for one example of such hardware. I have a device driver for FreeBSD for this board which even allows user processes to get precise timing by mmap()'ing the device registers into user space for easy access. The driver will be contributed to the FreeBSD project soon. I was pretty close to doing so just prior to the newbus conversion and now need to update the driver for a more recent -current. louie To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: matcd on an SB16
Mike Smith wrote: Is the matcd driver known to work on FreeBSD 3.2 ? If not, does anyone have any estimate of the amount of effort that'd be required to fix it? It works for some definitions of work. Firstly, there are three different CDROM interfaces that can be hung off an SB16; one is the Matsushita drive, there's also a Mutsumi interface (I don't think we support it) and a Sony interface (also, I believe, unsupported). Ghods, you're going through some old mail! [ and how was DEFCON, btw? :-) ] FWIW, the guy I was talking about embarked on a network install from another machine with a CD-ROM drive and an NFS server; the network install failed for slightly related reasons, having to do with the idea the hardware in this box is generally crap. The disappointing thing is that Linux works on it, though :-/ - mark Mark Newton Email: new...@internode.com.au (W) Network Engineer Email: new...@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82232999 Network Man - Anagram of Mark Newton Mobile: +61-416-202-223 To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
re: HELP!! Slice info disappeared
Hello Niall and Josef, Thanks for your great help. Finally I get all data on the disk back after the struggle on the weekend :) I run the program you sent me, but I got nothing from it. I guess the reason as follows: (1) The program fetches each chunks of 16 blocks from the disk and check if they are superblocks. (2) My disk is sliced into 3 parts. The fist one is for MS-DOS (FAT format), and the rest of them are FFS. It's highly possible that the offset block of the beginning of FFS slice is not a multiple of 16. (3) So I did: # for i in 0 1 2 3 ... 15; do # ./findsb /dev/wd2 $[ (estimated end of block for FAT slice)*512 + $i*512 ] # done After I got the info from the superblocks, getting back data was easy. Again, thanks much for your great help. You saved a lot of my time to reconstruct the data from scratch. Sincerely yours, -- Kazukiyo Ueda From: Josef Karthauser j...@pavilion.net Subject: re: HELP!! Slice info disappeared Date: Fri, 9 Jul 1999 10:01:16 +0100 Hi Kazukiyo, This is certainly possible. I've enclosed a hack from Niall Smart that should generated enough information to for you to reconstruct it. I'm working on a general solution to this for inclusion FreeBSD as shipped, but it's at home and I'm at work, that said it's Niall's basic code saved my harddisk a few weeks ago :) Joe To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
VMWare plug/quickie tests.
I've been running VM Ware under NT for a few days now, booting FreeBSD and other OS's. In some quick testing: The host machine is NT 4.0, SP5, 384MB RAM, dual 450 PII's. The guest OS is FreeBSD 3.2-RELEASE, configured with the VMWARE 512MB disk, and 32MB RAM allocated Compiling a generic kernel from scratch came out: 156.7u 64.3s 5:04.63 72.5%, ...149pf On a PII 300 with IDE disks, 128MB RAM, it was: (3.2-stable) 205.7u 17.9s 6:56.37 53.7%, ...32pf+0w All in all, not too shabby. So far it hasn't crashed, although I am not running X in the guest session yet, just using it in console mode. It has some weird pauses and stuff when it boots up, and the boot loader spinner creeps along, but once it's up and running, it's fine. I can see the utility of this pretty easily. Just wish it wasn't 400 bucks. I'm going to do some testing between VM's and such, and see how it goes. Next stop, 4.0-current. It doesn't consume any CPU cycles that I can see while idle, and running it at the standard priorities isn't affecting my NT stuff at all... I could grow to like it. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
predictability in an unpredictable world
If there were a mechanism whereby one could opt out of the SIGKILL,
most if not all of the complaints would go away. SIGDANGER would
suffice, but even a rude hack would do in a pinch, such as the one
included below (untested). If you mmap real disk instead of sbrk'ing,
and use this procfs control then the overcommit world won't interfere
with your process (but take care of thrashing stack/text of course).
No more need to fight the holy war!
3.2-STABLE:
===
diff -u vm/vm_pageout.c ../../src3.2/sys/vm/vm_pageout.c
--- vm/vm_pageout.c Thu Jul 15 20:40:56 1999
+++ ../../src3.2/sys/vm/vm_pageout.cThu Mar 18 17:28:39 1999
@@ -1144,7 +1144,7 @@
/*
* if this is a system process, skip it
*/
- if ((p-p_flag
(P_PRECIOUS|P_SYSTEM|P_NOSWAP)) || (p-p_pid == 1) ||
+ if ((p-p_flag (P_SYSTEM|P_NOSWAP)) ||
(p-p_pid == 1) ||
((p-p_pid 48) (vm_swap_size != 0)))
{
continue;
}
===
diff -u sys/proc.h ../../src3.2/sys/sys/proc.h
--- sys/proc.h Thu Jul 15 20:41:50 1999
+++ ../../src3.2/sys/sys/proc.h Fri May 14 01:32:41 1999
@@ -268,8 +268,6 @@
#defineP_NOCLDWAIT 0x40 /* No zombies if child dies
*/
-#defineP_PRECIOUS 0x80 /* Please, please let me live
*/
-
/*
* MOVE TO ucred.h?
===
diff -u miscfs/procfs/procfs_ctl.c
../../src3.2/sys/miscfs/procfs/procfs_ctl.c
--- miscfs/procfs/procfs_ctl.c Thu Jul 15 20:40:59 1999
+++ ../../src3.2/sys/miscfs/procfs/procfs_ctl.c Sat Aug 2 09:32:10
1997
@@ -69,8 +69,6 @@
#define PROCFS_CTL_STEP3
#define PROCFS_CTL_RUN 4
#define PROCFS_CTL_WAIT5
-#define PROCFS_CTL_PRECIOUS6
-#define PROCFS_CTL_NOPRECIOUS 7
static vfs_namemap_t ctlnames[] = {
/* special /proc commands */
@@ -79,8 +77,6 @@
{ step, PROCFS_CTL_STEP },
{ run,PROCFS_CTL_RUN },
{ wait, PROCFS_CTL_WAIT },
- { precious, PROCFS_CTL_PRECIOUS },
- { noprecious, PROCFS_CTL_NOPRECIOUS },
{ 0 },
};
@@ -255,14 +251,6 @@
}
}
return (error);
-
- case PROCFS_CTL_PRECIOUS:
- p-p_flag |= P_PRECIOUS;
- return 0;
-
- case PROCFS_CTL_NOPRECIOUS:
- p-p_flag = ~P_PRECIOUS;
- return 0;
default:
panic(procfs_control);
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
And what I'm pretty sure the majority of the readers on this list want is for those of you who really think it's necessary to do it yourselves. What? Nobody who wants to disable the policy knows how to do it? Hmmm, I wonder whether that's significant... Sheldon, if you can't contribute something useful, then shut up. If I have to do it myself, I will. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
: fail, generally speaking. : :Is this really the case? I'm pretty sure I've _never_ ignored the :possibility of a NULL return from malloc, and I've been using it :for nearly 20 years. I usually print a message and exit, but I :never ignore it. I thought that was pretty standard practise. : :This is just a random comment, orthogonal to the overcommit issue, :but I've seen both you and Matthew say this now, and I was surprised :both times. : :-- :Andrew The are dozens of libc routines which call malloc internally and return allocated storage. strdup(), opendir(), fopen(), setvbuf(), asprintf(), and so forth. Dozens. And while we might check some of these for NULL, we don't check them all, and the ones we do check we tend to conclude a failure other then a memory failure. We would assume that the directory or file does not exist, for example. How many programmers check errno after such a failure? Very few. How many programmers bother to even *clear* errno before making these calls (since some system calls do not set errno if it already non-zero). Virtually nobody. Having malloc() return NULL due to some *unrelated* process running the system out of swap is unnacceptable as it would result in serious instability to a great many programs that were never designed to deal with the case. Rather then crying about the system killing your favorite process, you would be crying about half a dozen processes that are still running no longer being stable. As a sysop, I would reboot a system in such a state instantly. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
: In that scenario, the 512MB of swap I assigned to this machine would be : dangerously low. : :With 13GB disks available for a couple of hundred bucks, my machines aren't :going to run out of swap space any time soon, even if I commit to disk. : :All I want for Christmas is a knob to disable overcommit. : :--lyndon If your machines aren't going to run out of swap, then the overcommit isn't going to hurt you in a million years. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: OpenBSD's strlcpy(3) and strlcat(3)
Any use of str{,n}cat makes me gag. In the past I have used
a composable function that may be of interest. Composable in
the sense that the result can be immediately used as an arg
to another call and it doesn't have the O(N^2) behavior of
strcat. Such a function can be totally safe. Something like:
char* r;
r = scpy(char* dst, char* const dst_end, const char* src)
where the following post-conditions hold:
dst_end = dst
r == MIN(dst + strlen(src)), dst_end)
r[0] == '\0'
That is, the returned ptr points in `dst' _just_ past the
copied data. Note that `dst_end' points to the _last_ char
of `dst'.
Example:
char* string[N];
...
char str[STRSIZE];
char* const strend = str + sizeof str - 1;
char* t = str;
for (int i = 0; i N t strend; i++)
t = scpy(t, strend, string[i]);
or
scpy(scpy(str, strend, this), strend, and that));
Here is the implementation:
char*
scpy(char* d, char* const e, const char* s)
{
while (*s d e)
*d++ = *s++;
*d = '\0';
return d;
}
This is far too simple to merit a paper or a long name :-)
And I am sure a zillion others have come up with the same idea.
-- bakul
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
On Thu, 15 Jul 1999, Matthew Dillon wrote: The are dozens of libc routines which call malloc internally and return allocated storage. strdup(), opendir(), fopen(), setvbuf(), asprintf(), and so forth. Dozens. And while we might check some of these for NULL, we don't check them all, and the ones we do check we tend to conclude a failure other then a memory failure. We would assume that the directory or file does not exist, for example. How many programmers check errno after such a failure? Very few. How many programmers bother to even *clear* errno before making these calls (since some system calls do not ^^ We're not supposed to have to clear errno unless we have to explicitly test if it has changed. We're not supposed to clear it before any system call which could possibly fail and set errno. set errno if it already non-zero). Virtually nobody. Erm... WTF?!?! If so, why the HELL are we doing that?!? -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message Brian Fundakowski Feldman _ __ ___ ___ ___ ___ gr...@freebsd.org _ __ ___ | _ ) __| \ FreeBSD: The Power to Serve!_ __ | _ \._ \ |) | http://www.FreeBSD.org/ _ |___/___/___/ To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Technical follow-up: Contrary to what I previously said, a number of tests reveal that Solaris, indeed, does not overcommit. All non-read only segments, and all malloc()ed memory is reserved upon exec() or fork(), and the reserved memory is not allowed to exceed the total memory. It makes extensive use of read only DATA segments, and has a NON_RESERVE mmap() flag. Though the foot firmly planted in my mouth ought to prevent me from saying anything else, I must say that it does explain a few things to me... -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org Would you like to go out with me? I'd love to. Oh, well, n... err... would you?... ahh... huh... what do I do next? To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
: The are dozens of libc routines which call malloc internally and return : allocated storage. strdup(), opendir(), fopen(), setvbuf(), asprintf(), : and so forth. Dozens. And while we might check some of these for NULL, : we don't check them all, and the ones we do check we tend to conclude : a failure other then a memory failure. We would assume that the directory : or file does not exist, for example. How many programmers check errno : after such a failure? Very few. How many programmers bother to even : *clear* errno before making these calls (since some system calls do not : ^^ :We're not supposed to have to clear errno unless we have to explicitly :test if it has changed. We're not supposed to clear it before any system :call which could possibly fail and set errno. : : set errno if it already non-zero). Virtually nobody. : :Erm... WTF?!?! If so, why the HELL are we doing that?!? No, wait, I got that wrong I think. Oh yah, I remember now. Hmm. How odd. I came across a case where read() could return -1 and not set errno properly if errno was already set, but a perusal of the kernel code seems to indicate that this can't happen. Very weird. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:Technical follow-up: : :Contrary to what I previously said, a number of tests reveal that :Solaris, indeed, does not overcommit. All non-read only segments, :and all malloc()ed memory is reserved upon exec() or fork(), and the :reserved memory is not allowed to exceed the total memory. It makes :extensive use of read only DATA segments, and has a NON_RESERVE :mmap() flag. : :Though the foot firmly planted in my mouth ought to prevent me from :saying anything else, I must say that it does explain a few things :to me... : :-- :Daniel C. Sobral (8-DCS) :d...@newsguy.com Something is weird here. If the solaris people are using a SWAPSIZE + REALMEM VM model, they have to allow the allocated + reserved space go +REALMEM bytes over available swap space. If not they are using only a SWAPSIZE VM model. Wait - does Solaris normally use swap files or swap partitions? Or is it that weird /tmp filesystem stuff? If it normally uses swap files and allows holes then that explains everything. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
: The are dozens of libc routines which call malloc internally and return : allocated storage. strdup(), opendir(), fopen(), setvbuf(), asprintf(), : and so forth. Dozens. And while we might check some of these for NULL, : we don't check them all, and the ones we do check we tend to conclude : a failure other then a memory failure. We would assume that the directory : or file does not exist, for example. How many programmers check errno : after such a failure? Very few. How many programmers bother to even : *clear* errno before making these calls (since some system calls do not :^^ :We're not supposed to have to clear errno unless we have to explicitly :test if it has changed. We're not supposed to clear it before any system :call which could possibly fail and set errno. : : set errno if it already non-zero). Virtually nobody. : :Erm... WTF?!?! If so, why the HELL are we doing that?!? No, wait, I got that wrong I think. Oh yah, I remember now. Hmm. How odd. I came across a case where read() could return -1 and not set errno properly if errno was already set, but a perusal of the kernel code seems to indicate that this can't happen. Very weird. I thought I saw this somewhere too, but I thought it was more of a case that it was somewhere *inside* read that errno had to be preserved. i.e. errno gets set somewhere at the top of the code, and if it was already set at a certain point, failure was expected, and to pass along the original errno, not the new one. Or perhaps we're sharing a hallucination. :) Kevin To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
implementing poll() in a device driver (fwd)
Got no response from freebsd-questions. can anyone here help ? Please cc your reply to my email a/c. thanks --Vasudha -- Forwarded message -- Date: Thu, 15 Jul 1999 14:42:34 +0800 (SGT) From: Vasudha Ramnath v...@krdl.org.sg To: freebsd-questi...@freebsd.org Subject: implementing poll() in a device driver I'm running FreeBSD 3.1-RELEASE. Could someone explain what the poll() function in a device driver should do ? Can it return POLLERR or POLLHUP ? I have a test driver that returns these values from the poll() function. However, the application that called the select() is not getting an error. Instead, the select is returning that the particular file descriptor is, in this case, 'readable' ! Am I missing something ? Any insight appreciated. thank you --Vasudha To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-questions in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Swap overcommit
No, wait, I got that wrong I think. Oh yah, I remember now. Hmm. How odd. I came across a case where read() could return -1 and not set errno properly if errno was already set, but a perusal of the kernel code seems to indicate that this can't happen. Very weird. I thought I saw this somewhere too, but I thought it was more of a case that it was somewhere *inside* read that errno had to be preserved. i.e. errno gets set somewhere at the top of the code, and if it was already set at a certain point, failure was expected, and to pass along the original errno, not the new one. Or perhaps we're sharing a hallucination. :) Well, set/getpriority(2), certainly can return -1 and not be an error. You would need to clear out errno before that call and check it on return. This is where excpetions would be a great gain. It could also be used to force programmers to check their system calls more closely. Oops, you didn't handle excpetion foo? SIGBADPROGRAMMER. -- David Cross | email: cro...@cs.rpi.edu Systems Administrator/Research Programmer | Web: http://www.cs.rpi.edu/~crossd Rensselaer Polytechnic Institute, | Ph: 518.276.2860 Department of Computer Science| Fax: 518.276.4033 I speak only for myself. | WinNT:Linux::Linux:FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
