Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED] Peter Seebach writes:
: I searched through the standard extensively to see if "allocates
: space" is defined and couldn't find anything other than 'the poitner
: can be used to access the space allocated.'
:
: EXACTLY!
:
: If it can't actually be used, then something has gone horribly wrong.
It says can, not must. I disagree with you that you can't overcommit.
Also, access is ill defined. Does that mean read access? write
access? How about execute access? It just says access and leaves the
meaning up to the implementor.
You fail to draw a distinction between "reading" the allocated area,
and "writing" to the allocated area. The standard makes no promises
that you can do both, and only has the vaguely defined access as to
what you can do with it.
Reasonable people can differ as to what this means.
: There appear to be no
: guarantees that you can always use all of the memory that you've
: allocated, the performance characterstics of accessing said memory or
: anything else.
:
: Performance characteristics are not at issue; I don't think anyone on the
: committee would complain about an implementation which provided "committed"
: space by backing it all to disk, all the time.
No one would. However, the standard does not preclude the ability to
overcommit. Many different systems do this already.
: Do not read too much into the above words. They mean exactly what
: they say. FreeBSD is in compliance.
:
: Assuming I make the Copenhagen meeting, I'll bring this up again as a DR
: or something. This gets debated every few years, and the consensus seems
: to be that malloc *MUST* return a valid pointer to space which can actually
: be used.
Really, I guess I missed those debates. I've not seen it in the
literature. I've not seen this addressed in the last 20 years that
systems have been overcomitting in any meaningful way.
: The space is indeed allocated to
: the process address space. System resource issues might get in the
: way of being able to use that, but that's true of anything you
: allocate.
:
: That's not what "allocate" means. A resource which is *allocated* is one
: you actually have access to.
Not necessarily. Allocate means, in the absense of resource
shortages, you get the memory. You might get the memory, you might
not. Consider a system that has a quota on dirty pages. If I malloc
a huge array, and then only use part of it, I can stay under my
quota. But if I use most of it by dirtying it, then the system can
kill the process.
: Basically, if your interpretation had been intended, the committee would have
: put in words to the effect of "access to an allocated object invokes undefined
: behavior".
:
: It doesn't.
Yes it does. You do not understand. If there's an ECC error, then
you do not have access to the memory without errors. This is no
different, just a different class of error.
: Therefore, the following code:
: #include stdlib.h
: int main(void) {
: unsigned char *p;
: if (2 SIZE_MAX) {
: p = malloc(2)
: if (p) {
: size_t i;
: for (i = 0; i 2; ++i) {
: p[i] = i % 5;
: }
: }
: }
: return 0;
: }
: *MUST* succeed and return 0. The program does not invoke undefined behavior;
: therefore, the compiler is obliged to ensure that it succeeds. It's fine for
: malloc to fail; it's not fine for the loop to segfault.
That's not necessarily true. There's nothing in the standard that
states that you have to be able to write to all parts of an allocated
region. It is a perfectly reasonable implementation to return the
memory from malloc. The system might enforce an upper bound on the
number of dirty pages a process has. malloc would not necessarily
have any way of knowing what those limits are, and those limits and
quotas might be dynamic or shared amoung classes of users. The limit
might be 100 right now, but 10 later. This is a host environment
issue. The malloc call gets a resource, but in the future you might
not be able to access that resource due to circumstances beyond the
control of the program.
Also, the program does not *HAVE* to succeed. If you have a low CPU
quota, for example, and the program exceeds it, the host environment
will signal SIGXCPU to the process. Another process might send this
one a SIGKILL. The system may crash in the middle of its execution.
There might be an unrecoverable memory parity error. There might be
an unrecoverable disk error on the swap partition. There might be any
number of things that preclude it from reaching completion, beyond
your control.
I don't see how this is significantly different than a dirty page
quota or a dirty page limit. That's what we're talking about here.
There's
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Warner Losh writes: It says can, not must. But if it "can be used", and no one says "undefined behavior", then we're done; no undefined behavior is allowed. I disagree with you that you can't overcommit. Also, access is ill defined. It's defined better in C99 than it was in C89. Does that mean read access? write access? How about execute access? Within "defined behavior", read and write; all jumps to data are undefined behavior. The standard makes no promises that you can do both, and only has the vaguely defined access as to what you can do with it. Once again, if it had been intended to allow only limited access, it would say so. Reasonable people can differ as to what this means. Yes. : Performance characteristics are not at issue; I don't think anyone on the : committee would complain about an implementation which provided "committed" : space by backing it all to disk, all the time. No one would. However, the standard does not preclude the ability to overcommit. It is certainly read that way by a number of people, and I think we're currently a majority on the committee. No one cares; it's not as though an implementation is required to be in the maximally conforming mode at all times. Many different systems do this already. Yes, they do. That doesn't mean it's correct. I don't believe there's a single completely correct implementation of the fully ammended C89 standard, let alone C99. Really, I guess I missed those debates. I've not seen it in the literature. I've not seen this addressed in the last 20 years that systems have been overcomitting in any meaningful way. It's an old flamewar on comp.std.c. :) : That's not what "allocate" means. A resource which is *allocated* is one : you actually have access to. Not necessarily. Allocate means, in the absense of resource shortages, you get the memory. No. To allocate is to get. Allocation *fails* in the case of a resource shortage. You might get the memory, you might not. Consider a system that has a quota on dirty pages. If I malloc a huge array, and then only use part of it, I can stay under my quota. But if I use most of it by dirtying it, then the system can kill the process. Sure. Now imagine a quota on file I/O. If I start writing to a file, I'm under my quota, but if I exceed it, then the system can kill the process. If the standard had meant to say "access to allocated space invokes undefined behavior", it would have done so. Would you argue that the system which simply *kills* a process that tries to write to a full disk is correct? If not, can you show how that's any different? Yes it does. You do not understand. If there's an ECC error, then you do not have access to the memory without errors. This is no different, just a different class of error. Hardware errors do, indeed, produce a non-conforming implementation. However, on a system where the hardware works, overcommit-and-kill is *STILL* a conformance problem; killing on ECC errors isn't a conformance problem unless there are some. That's not necessarily true. There's nothing in the standard that states that you have to be able to write to all parts of an allocated region. Sure, and there's nothing in the standard that explicitly says "a call to frwrite with valid arguments should not raise SIGKILL", but everyone knows that would be a mistake. It is a perfectly reasonable implementation to return the memory from malloc. It's very reasonable, but it does not provide the environment described in the standard, which is an environment in which the memory returned by malloc is all available for use. might be 100 right now, but 10 later. This is a host environment issue. The malloc call gets a resource, but in the future you might not be able to access that resource due to circumstances beyond the control of the program. The same argument applies to killing programs when they write to a full disk. Also, the program does not *HAVE* to succeed. If you have a low CPU quota, for example, and the program exceeds it, the host environment will signal SIGXCPU to the process. Another process might send this one a SIGKILL. The system may crash in the middle of its execution. There might be an unrecoverable memory parity error. There might be an unrecoverable disk error on the swap partition. There might be any number of things that preclude it from reaching completion, beyond your control. Yes, and *EVERY ONE OF THOSE* is a conformance failure. I'm not saying we can produce an environment which is guaranteed to conform in all cases; it's obviously impossible. We *could* avoid breaking things. I don't see how this is significantly different than a dirty page quota or a dirty page limit. Very simple: We don't have control over hardware, and the Unix environment says you're allowed to do all sorts of things which, from the point of view of the spec, magically render the execution environment non-conforming.
Re: Compaq e500, 3com cardbus card help
Kenneth Wayne Culver [EMAIL PROTECTED] writes: On 24 Feb 2001, Dag-Erling Smorgrav wrote: Kenneth Wayne Culver [EMAIL PROTECTED] writes: FreeBSD supports cardbus in -CURRENT, but I wouldn't expect it to ever support cardbus in 4.x. If you are daring you can get -CURRENT, but from what I hear right now, it's not working very well. It works just fine, thank you very much, but it takes some hand-holding. Must not be cardbus then. I beg your pardon? Were you trying to say that -CURRENT does not work very well, or that Cardbus does not work very well? I assumed the former, which is partly true (-CURRENT works fine if you keep close enough track of things to know when it's safe to upgrade and how to fix or work around whatever bugs you hit). If you meant the latter, I beg to differ - Cardbus itself works fine and dandy for me. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
diskless-boot/PXE/dhcp
hi all, I've made some changes to libstand/bootp.c and they are available for evaluation/comments/critics at: ftp://ftp.cs.huji.ac.il/users/danny/freebsd/diskless-boot short desc.: o the bootp request now does a vendor-specific-request FreeBSDc instead of PXEClient. o dhcp options are placed in the env. as dhcp.option-name o FBSD options are placed in the env as is. long description: see README :-) danny To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Dag-Erling Smorgrav wrote: [EMAIL PROTECTED] (Peter Seebach) writes: Is there any hope that, some day, a setting could be provided where a program could request that malloc *NOT* overcommit? There are programs which would rather know in advance, and clean up, than be killed abruptly. Malloc() does not overcommit - the kernel does. Malloc() doesn't know and doesn't care. I imagine you could add a compile-time option that forces obreak() (why do we still use brk()/sbrk()?) to prefault the newly allocated address space. Or you could write a malloc() implementation that only uses mmap(), and add a flag value that makes mmap() prefault pages, and fail if there isn't enough memory available. None of these solutions are portable, however; the portable way around memory overcommit is to write a malloc() wrapper that installs a SIGSEGV handler, then tries to dirty the newly allocated memory, and fails gracefully if this causes a segfault. Untested code: None of these solutions will work. In all of the above cases, it is still possible for an application to be killed due to out of memory conditions caused by other applications. The main point of non-overcommitting applications is to avoid that. Personally, I like AIX solution of a SIGDANGER signal, and having applications that do not have a handler installed be killed first. But that isn't what is being asked for. OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who *wants* that feature to sit down and code it. It won't happen otherwise. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Dag-Erling Smorgrav wrote: None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. Show me a modern OS (excluding real-time and/or embedded OSes) that makes this guarantee. Solaris and AIX (on AIX this is optional on a global or per-application level). -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Julian Elischer wrote: to not be caught by surprise, simply touch every page after you allocate it. It doesn't work. The application killed by reason of insufficient resources is not (necessarily) the one that causes the page fault leading to that. Id est, if my application allocates all available memory, touching every page, and then some other application touches a pre-allocated page not backed by memory or swap, it is quite likely mine which will get killed. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Listing configured md(4) devices
I think my commit to md yesterday torpedoed your patch, because one part of it rej'ected. I applied the .rej by hand and found a few nits here and there. I change the list function extensively, because the "512" size of the array were not picked from where it should come, sys/disklabel.h, because it is under #ifdef _KERNEL there, so instead I did a list based sorting instead. Thanks! In message [EMAIL PROTECTED], Dima Dorfman write s: The "md" problem should really be solved by adding #define MD_NAME "md" to sys/mdioctl.h and using this in both the driver and the mdconfig program. Can I get you to incorporate that in your patch ? Certainly! The updated patch is at http://www.unixfreak.org/~dima/home/md-list4.diff (it's getting a little big, so I thought I'd not attach it here). The changes include, - "md" (and variants thereof) - MD_NAME, - "mdctl" (and variants thereof) - MDCTL_NAME (for consistency), and - update the mdconfig(8) manual page to mention the -l option. Thanks Dima Dorfman [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
"Daniel C. Sobral" [EMAIL PROTECTED] writes: It doesn't work. The application killed by reason of insufficient resources is not (necessarily) the one that causes the page fault leading to that. This is arguably a bug which needs to be fixed. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
NFS behaviour
NFS experts out there, I have a question about synchronisation: Imagine two hosts A (NFS server), host B (NFS client). Process on A modifies a file. When does process on B get notification about the change? Does it depend on the time set on the different hosts? Is it a caching issue or what? With two Linux systems we are seeing all sorts of strange effects with such a setup. Just curious if FreeBSD behaves better here. Well, I could try it out, but if it is a general problem anyway with NFS I'd prefer to discuss it theoretically :-) -- Chris Christoph P. U. Kukulies [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who *wants* that feature to sit down and code it. It won't happen otherwise. So, out of idle curiousity: If, somewhere down the road, I know the kernel well enough to attempt such a thing, what would the interest level be in merging such a feature? -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Compaq e500, 3com cardbus card help
What I was saying is that cardbus only works on current. = | Kenneth Culver | FreeBSD: The best NT upgrade| | Unix Systems Administrator | ICQ #: 24767726 | | and student at The | AIM: muythaibxr | | The University of Maryland, | Website: (Under Construction) | | College Park. | http://www.wam.umd.edu/~culverk/| = On 25 Feb 2001, Dag-Erling Smorgrav wrote: Kenneth Wayne Culver [EMAIL PROTECTED] writes: On 24 Feb 2001, Dag-Erling Smorgrav wrote: Kenneth Wayne Culver [EMAIL PROTECTED] writes: FreeBSD supports cardbus in -CURRENT, but I wouldn't expect it to ever support cardbus in 4.x. If you are daring you can get -CURRENT, but from what I hear right now, it's not working very well. It works just fine, thank you very much, but it takes some hand-holding. Must not be cardbus then. I beg your pardon? Were you trying to say that -CURRENT does not work very well, or that Cardbus does not work very well? I assumed the former, which is partly true (-CURRENT works fine if you keep close enough track of things to know when it's safe to upgrade and how to fix or work around whatever bugs you hit). If you meant the latter, I beg to differ - Cardbus itself works fine and dandy for me. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Converting Perforce to CVS
Dear Anton, If you still have the Perforce-CVS conversion script, I would be very grateful if you could e-mail it to me. Thanks in advance John Wilson On Tue, 9 Jan 2001 09:13:39 +0100, Anton Berezin wrote: On Sat, Jan 06, 2001 at 03:06:20PM -0800, John Wilson wrote: I apologize in advance, as this is not strictly a FreeBSD-related question, but I know that a lot of FreeBSD'ers use CVS as well as Perforce, so here goes... What is the easiest way to convert a P4 source repository to CVS, while preserving revisions, history, log messages, etc? Both systems seem to use RCS, but is it as simple as copying the files? Are there any caveats? I have one script, but it does not handle branches (the project I was converting did not have any). I can mail it to you if you want. The branch handling should be rather similar to binary files handling, which the script already performs. Cheers, ^Anton. -- May the tuna salad be with you. ___ Send a cool gift with your E-Card http://www.bluemountain.com/giftcenter/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
[hackers] Re: Large MFS on NFS-swap?
"Matt" == Matt Dillon [EMAIL PROTECTED] writes: Matt Use the vn device to create a swap-backed filesystem. 'man Matt vnconfig'. (In current VN functionality has been merged into MD Matt and MD in swap-backed mode should be used instead ). If you Matt turn softupdates on on the VN based filesystem, it should work Matt quite nicely for small files and yet still be able to handle the Matt very large files (assuming you have sufficient swap). Hmmm... I was just having a little fun, and I think that someone's using the wrong type of integer somewhere: [1:23:323]root@news:~ vnconfig -e -s labels -S 1t vn0 [1:24:324]root@news:~ disklabel -r -w vn0 auto [1:25:325]root@news:~ newfs /dev/vn0c preposterous size -2147483648 Dave. -- |David Gilbert, Velocet Communications. | Two things can only be | |Mail: [EMAIL PROTECTED] | equal if and only if they | |http://www.velocet.net/~dgilbert | are precisely opposite. | =GLO To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Large MFS on NFS-swap?
:Hmmm... I was just having a little fun, and I think that someone's :using the wrong type of integer somewhere: : :[1:23:323]root@news:~ vnconfig -e -s labels -S 1t vn0 :[1:24:324]root@news:~ disklabel -r -w vn0 auto :[1:25:325]root@news:~ newfs /dev/vn0c :preposterous size -2147483648 : :Dave. Heh heh. Yes, newfs has some overflows inside it when you get that big. Also, you'll probably run out of swap just newfs'ing the metadata, you need to use a larger block size, large -c value, and a large bytes/inode (-i) value. But then, of course, you are likely to run out of swap trying to write out a large file even if you do manage to newfs it. I had a set of patches for newfs a year or two ago but never incorporated them. We'd have to do a run-through on newfs to get it to newfs a swap-backed (i.e. 4K/sector) 1TB filesystem. Actually, this brings up a good point. Drive storage is beginning to reach the limitations of FFS and our internal (512 byte/block) block numbering scheme. IBM is almost certain to come out with their 500GB hard drive sometime this year. We should probably do a bit of cleanup work to make sure that we can at least handle FFS's theoretical limitations for real. vnconfig -e -s labels -S 900g vn0 newfs -i 1048576 -f 8192 -b 65536 -c 100 /dev/vn0c mobile:/home/dillon pstat -s Device 1K-blocks UsedAvail Capacity Type /dev/ad0s1b524160 188976 33518436%Interleaved -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED] Peter Seebach writes: : In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Large MFS on NFS-swap?
On Sun, Feb 25, 2001 at 05:44:13PM -0500, David Gilbert wrote: "Matt" == Matt Dillon [EMAIL PROTECTED] writes: [... my newfw bomb deleted ...] Matt I had a set of patches for newfs a year or two ago but never Matt incorporated them. We'd have to do a run-through on newfs to Matt get it to newfs a swap-backed (i.e. 4K/sector) 1TB filesystem. Matt Actually, this brings up a good point. Drive storage is Matt beginning to reach the limitations of FFS and our internal (512 Matt byte/block) block numbering scheme. IBM is almost certain to Matt come out with their 500GB hard drive sometime this year. We Matt should probably do a bit of cleanup work to make sure that we Matt can at least handle FFS's theoretical limitations for real. That and the availability of vinum and other raid solutions. You can always make multiple partitions for no good reason (other than filesystem limitations), but we were planning to put together a 1TB filesystem next month. From what you're telling me, I'd need larger block sizes to make this work? With 512 Byte blocksizes you are limited to 1T because the physical block number is a signed 32bit. FFS uses 32bit (I wouldn't count on the high bit) frag numbers. A fragment defaults to 1k so even with 1k fragments the limit is at least 2T. It is possible to reformat most SCSI disks to 1k or 2k block sizes, but I'm not shure if vinum handles non 512 byte blocks correctly and I don't know if the buffer code always uses 512 or physical sizes. Maybe ccd is an option. AFAIK the same limit is there for SCSI as SCSI uses 32bit block numbers. Using a RAID controller will show the same limits and it's usually untested with block sizes != 512. IMHO, we might reconsider that. With SAN-type designs, you're probably going to find the distribution of filesizes on multi-terrabyte filesystems that are shared by 100's of computers to be roughly the same as the filesize distributions on today's filesystems. Making the run for larger block sizes puts us in the same league as DOS. While it will stave off the wolves, it will only work for so long give Moore's law. Noone is telling that that's the way the world will go. But is a goable workaround until the world is more perfect. The base design should allow using 64bit values some day without the scaling problem and consistency risks as DOS. The steps have to go increasing the limits of the buffer system and the filesystem after that. I'm shure the persons with knowledge about that will do what is possible after -current stabilizes a bit. Changing everything at the same time is not a good way to go. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Large MFS on NFS-swap?
:Making the run for larger block sizes puts us in the same league as :DOS. While it will stave off the wolves, it will only work for so :long give Moore's law. : :Dave. : :-- : :|David Gilbert, Velocet Communications. | Two things can only be | Ultimately the limit we face is the fact that we use 32 bit quantities to store block numbers. At 512 bytes per block, and assuming only 31 bits of useful space, we are limited to 2G x 512 = 1TB. (filesystems such as FFS use 'negative' block numbers for special purposes). Within the kernel we have already moved to 64 bit offsets for everything that is offset-based. Block numbers however are still 32 bits. There have been a number of proposals on how to solve the problem and the one that will probably win in the end will be to pass 64 bit offsets all the way through to the low level disk I/O. e.g. we would still guarentee appropriate block boundries for the offsets when passing them through to the device level, but we would not do the divide that we currently do. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Large MFS on NFS-swap?
:With 512 Byte blocksizes you are limited to 1T because the physical :block number is a signed 32bit. :FFS uses 32bit (I wouldn't count on the high bit) frag numbers. :A fragment defaults to 1k so even with 1k fragments the limit is :at least 2T. Yes, the FFS limit is essentially the frag size limitation. However, internally our device I/O path normalizes blocks to 512 byte quantities. I've heard of people running larger filesystems (16TB) with larger fragment sizes, but I'm not exactly sure how it can work under FreeBSD due to the normalization that the FreeBSD device layer makes. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
ata-disk ioctl and atactl patch
As I promised on -mobile earlier this week, I've cleaned up my patches
to port the {Net,Open}BSD atactl utility, including a simplistic
ata-disk ioctl. They apply cleanly against this afternoon's -stable
(including Soren's latest commit bringing -stable up to date with
-current). I've been running them for some time and they ''work great
here''.
Before announcing this in a broader context, I wanted to get a bit of
feedback on the ioctl implementation. In particular, is it safe to
just do an ata_command inside adioctl() without any further checking?
(e.g., can this cause bad things to happen under heavy i/o load?)
Here's a snippet from the newly added adioctl() in ata-disk.c:
switch (cmd) {
case ATAIOCCOMMAND:
{
struct atareq *req = (struct atareq *)addr;
if (!req)
return EINVAL; /* request not valid */
/* issue an ata command */
switch (req-command) {
case ATAPI_STANDBY_IMMEDIATE:
case ATAPI_IDLE_IMMEDIATE:
/* no count argument */
error = ata_command(adp-controller, adp-unit, req-command,
0, 0, 0, 0, 0, ATA_WAIT_INTR);
The full diffs are at http://www.renfro.org/scott (at the bottom of
the page). These patches only implement the idle, setidle, standby,
and setstandby commands. I haven't tackled sleep, identify, or
checkpower commands. (Sleep isn't hard to implement, but I haven't
tested whether the driver would actually reset all state properly when
coming out of sleep or whether more code is required to do this).
For those that question the usefulness of atactl: I do find this
extremely useful. On a laptop with noatime mounts, softupdates, and
syslog, cron, and atrun disabled, I can go a long time (an hour or
more) without the drive spinning up. It also cuts the noise level of
my Z505L in half. (Now if I can just get the fan to shut up ;-)
Thanks,
-scott
--
Scott Renfro [EMAIL PROTECTED] +1 650 906 9618
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Converting Perforce to CVS
On Sun, Feb 25, 2001 at 09:17:38AM -0800, John Wilson scribbled: | If you still have the Perforce-CVS conversion script, I would be very | grateful if you could e-mail it to me. Such a script is available for download on www.perforce.com. | On Tue, 9 Jan 2001 09:13:39 +0100, Anton Berezin wrote: | On Sat, Jan 06, 2001 at 03:06:20PM -0800, John Wilson wrote: |I apologize in advance, as this is not strictly a FreeBSD-related | question, |but I know that a lot of FreeBSD'ers use CVS as well as Perforce, so | here |goes... | |What is the easiest way to convert a P4 source repository to CVS, while |preserving revisions, history, log messages, etc? Both systems seem | to use |RCS, but is it as simple as copying the files? Are there any caveats? | | | I have one script, but it does not handle branches (the project I was | converting did not have any). I can mail it to you if you want. The | branch handling should be rather similar to binary files handling, which | the script already performs. -- +--+ | [EMAIL PROTECTED] | [EMAIL PROTECTED] | | http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. | +--+ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Warner Losh writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. This thread reminds me of what happened when I brought up the same issue a few years ago, arguing that the kernel shouldn't overcommit memory (i.e., the same thing, everybody though I was nuts :) For me it helps to understand people's underlying motivation. Here's mine... my perspective probably comes from using FreeBSD in the embedded world, which is very different from using FreeBSD in the rack-mounted server world. One important way to gain confidence that you're little box won't silently crash at the worst possible time for the customer is to be able to *prove* to yourself that it can't happen, given certain assumptions. Those assumptions usually include things like "the hardware is working properly" (e.g., no ECC errors) and "the compiler compiled my C code correctly". Given these basic assumptions, you go through and check that you've properly handled every possible case of input (malicious or otherwise) from the outside world. Part of the "proof" is verifying that you've checked all of your malloc(3) return values for NULL.. and assuming that if malloc(3) returns != NULL, then the memory is really there. Now, if malloc can return NULL and the memory *not* really be there, there is simply no way to prove that your code is not going to crash. This memory overcommit thing is the only case that I can think of where this happens, given the basic assumptions of correctly functioning hardware, etc. That is why it's especially annoying to (some) people. -Archie __ Archie Cobbs * Packet Design * http://www.packetdesign.com To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Archie Cobbs wrote: Warner Losh writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. This thread reminds me of what happened when I brought up the same issue a few years ago, arguing that the kernel shouldn't overcommit memory (i.e., the same thing, everybody though I was nuts :) For me it helps to understand people's underlying motivation. Here's mine... my perspective probably comes from using FreeBSD in the embedded world, which is very different from using FreeBSD in the rack-mounted server world. One important way to gain confidence that you're little box won't silently crash at the worst possible time for the customer is to be able to *prove* to yourself that it can't happen, given certain assumptions. Those assumptions usually include things like "the hardware is working properly" (e.g., no ECC errors) and "the compiler compiled my C code correctly". Given these basic assumptions, you go through and check that you've properly handled every possible case of input (malicious or otherwise) from the outside world. Part of the "proof" is verifying that you've checked all of your malloc(3) return values for NULL.. and assuming that if malloc(3) returns != NULL, then the memory is really there. Now, if malloc can return NULL and the memory *not* really be there, there is simply no way to prove that your code is not going to crash. This memory overcommit thing is the only case that I can think of where this happens, given the basic assumptions of correctly functioning hardware, etc. That is why it's especially annoying to (some) people. I still think that in such a case it should be possible to 'test the commitment' by touching all the allocated memory while trapping page faults. and fault all your memory from 'potential' to 'allocated'. As someone said. it is not sure which program when you run out of swap, but I think you might be able to somehow change this behaviour to "the program making the request" fails (or gets a fault). You could allocate your memory. trap faults. touch all of the allocated memory. if it faults, you can remap some file to that location to allow the instruction to continue.. continue and abort the check.. exit as needed, OR continue with secure knowledge that all your memory is there. Alternatively you could allocate your own on-disk swapspace for a program by telling malloc to use a file for all your memory needs. -Archie __ Archie Cobbs * Packet Design * http://www.packetdesign.com To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message -- __--_|\ Julian Elischer / \ [EMAIL PROTECTED] ( OZ) World tour 2000-2001 --- X_.---._/ v To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
:... : : merging such a feature? : : Assuming that it doesn't break anything, that it doesn't introduce a : severe performance penalty and works, there would be interest. There : are times that this is a desirable feature. : :This thread reminds me of what happened when I brought up the same :issue a few years ago, arguing that the kernel shouldn't overcommit :memory (i.e., the same thing, everybody though I was nuts :) : :For me it helps to understand people's underlying motivation. Here's The memory overcommit thread comes up once or twice a year. So this time around I am going to try to take a different tact in trying to explain the issue. One could argue about making the OS not overcommit until one is blue in the face. One could argue that every single routine that allocates memory must be prepared to handle a memory failure in a graceful manner. One could argue all sorts of high-and-mighty value things. But its all a crock. It simply isn't possible to gracefully handle an out of memory condition. All sorts of side effects occur when the system runs out of memory, even *with* overcommit protection. In fact, all sorts of side effects occur even when the system *doesn't* run out of memory, but instead just starts to thrash swap. All sorts of side effects occur if the system starts to get cornered memory-wise even if you don't *have* any swap. The number of possible combinations of effects is near infinite and nearly impossible to program against. Simply put, the 'overcommit' argument doesn't actually solve the problem in any meaningful way. Any significantly sized program that actually handled every possible combination and side effect of an out-of-memory condition gracefully would have so much conditional garbage cluttering it up that the rest of the code would be obscured in a haze of if()'s. There ain't no magic bullet here, folks. By the time you get a memory failure, even with no overcommit, it is far too late to save the day. There is only one correct way to handle an out of memory condition and that is to make sure it doesn't happen... so when it does happen you can treat it as a fatal error and scream bloody murder. It's a whole lot easier to design a program with bounded, deterministic memory use (e.g. database X requires Y kilobytes of memory per client instance) and control that use at the edges rather then to try to deal with memory failures gracefully in the deepest darkest corners of the program. And it's a whole lot more reliable, too. When I write such a program, if I care about bounding memory use I do it at the edges. I don't pollute low level allocation routines (like strdup(), small structural allocations, etc...) with all sorts of conditionals to gracefully back out of a memory failure. It's a huge waste of time. I just wrap the routines... safe_strdup(), safe_malloc(), safe_asprintf()... and the wrappers scream bloody murder and exit if they get a failure. It's that simple... and far more reliable. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
:In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: :OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who :*wants* that feature to sit down and code it. It won't happen otherwise. : :So, out of idle curiousity: If, somewhere down the road, I know the kernel :well enough to attempt such a thing, what would the interest level be in :merging such a feature? : :-s The problem is a whole lot more complex then you think. Dealing with overcommit is not simply counting mapped pages, there are all sorts of issues involved. But the biggest gotcha is that putting in overcommit protection will not actually save your system from dying a terrible death. It in fact makes it *more* likely that the system will die a horrible death, because with overcommit protection you wind up pre-reserving resources that would otherwise be reserved on demand (and often might not ever be used by the programs mapping those resources). Not only that, but overcommit protection does not protect you from thrashing, and quite often the system will become unusable long before it actually runs out of memory+swap. This is true whether you have configured swap or not... it is perfectly easy to thrash a swapless system if the vast majority of pages are clean (e.g. you mmap() a lot of files read-only). A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE), or even simply the static data and bss space associated with a program binary. Such memory only takes up physical space if you dirty it...you can *read* the memory all you want without having to reserve any resources, the OS can throw the physical page away at any time and reload it form the backing file. But the moment you dirty the memory the OS must allocate real resources to it ... either a physical page, or swap (and it can flip between the two). Many such mappings never actually require dirtying the underlying page and thus never actually take any resources. But with overcommit protection you have to reserve the resources immediately, even if the program will never use them. The result is that your system is likely to run out of memory long before it would without the overcommit protection. There are many other issues involved as well that are more complex, such as what you have to do when a program fork()s. So before you get into tring to 'protect' yourself by implementing overcommit protection, you need to think long and hard on what you are actually protecting yourself from... and what you aren't. People seem to believe that edge cases such as allocating memory and then the system faulting the process later because it couldn't allocate the actual backing store later is of paramount importance but they seem to forget that by the time you reach that situation, you are generally already past the point of no return. Overcommit protection doesn't even come close to being able to prevent your system from getting into an unrecoverable situation and it certainly is no substitute for writing the software correctly (such that the memory failure can't occur in the first place). -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: ata-disk ioctl and atactl patch
It seems Scott Renfro wrote:
As I promised on -mobile earlier this week, I've cleaned up my patches
to port the {Net,Open}BSD atactl utility, including a simplistic
ata-disk ioctl. They apply cleanly against this afternoon's -stable
(including Soren's latest commit bringing -stable up to date with
-current). I've been running them for some time and they ''work great
here''.
Before announcing this in a broader context, I wanted to get a bit of
feedback on the ioctl implementation. In particular, is it safe to
just do an ata_command inside adioctl() without any further checking?
(e.g., can this cause bad things to happen under heavy i/o load?)
No its not safe at all, you risk trashing an already running command...
Anyhow, I have an atacontrol thingy in the works for attach/detach,
raid control etc, etc, I'll try to merge this functionality into that
(the ioctl's will change etc, but the functionality is nice)...
-Sren
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Matt Dillon wrote: But its all a crock. It simply isn't possible to gracefully handle an out of memory condition. All sorts of side effects occur when the system runs out of memory, even *with* overcommit protection. In fact, all sorts of side effects occur even when the system *doesn't* run out of memory, but instead just starts to thrash swap. All sorts of side effects occur if the system starts to get cornered memory-wise even if you don't *have* any swap. The number of possible combinations of effects is near infinite and nearly impossible to program against. Simply put, the 'overcommit' argument doesn't actually solve the problem in any meaningful way. Any significantly sized program that actually handled every possible combination and side effect of an out-of-memory condition gracefully would have so much conditional garbage cluttering it up that the rest of the code would be obscured in a haze of if()'s. That's an assumption. There ain't no magic bullet here, folks. By the time you get a memory failure, even with no overcommit, it is far too late to save the day. Scientific computation. If, at one point, no more memory can be allocated, you back off, save the present results, and try again later. There is only one correct way to handle an out of memory condition and that is to make sure it doesn't happen... so when it does happen you can treat it as a fatal error and scream bloody murder. It's a whole lot easier to design a program with bounded, deterministic memory use (e.g. database X requires Y kilobytes of memory per client instance) and control that use at the edges rather then to try to deal with memory failures gracefully in the deepest darkest corners of the program. And it's a whole lot more reliable, too. When I write such a program, if I care about bounding memory use I do it at the edges. I don't pollute low level allocation routines (like strdup(), small structural allocations, etc...) with all sorts of conditionals to gracefully back out of a memory failure. It's a huge waste of time. I just wrap the routines... safe_strdup(), safe_malloc(), safe_asprintf()... and the wrappers scream bloody murder and exit if they get a failure. It's that simple... and far more reliable. You assume too much. Quite a few of the more efficient garbage collection algorithms depend on knowing when the memory has become full. You keep allocating, and when malloc() returns NULL, *then* you run the garbage collector, free some space, and try again. If malloc() doesn't work, quite a few very efficient garbage collection algorithms become impossible to implement. Just because *you* don't see how one thing can work, it doesn't mean it can't work. As I have trivially shown above. Honestly, I think non-overcommit is a mistake and your approach is much better, but it's not the only approach and there _are_ valid approaches that depend on not overcommitting, and I really hate having to defend non-overcommit against such bogus arguments. You don't implement it because you don't think it's worth your time and you are not being paid to do so. That's a perfectly good reason, and if people would just stick with it the threads about this would die much sooner. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
