Re: FreeBSD vice OS X memory management
does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? IMHO no. OSX is somehow-microkernel based, they did take things from FreeBSD but not this IMHO. anyway - who cares Something is deeply broken in OS X memory management http://workstuff.tumblr.com/post/20464780085/something-is-deeply-broken-in-os-x- memory-management One of the problems that caught my eyes was inactive memory reclamation. I remember some time ago there was a thread here with similar topic. http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239121.html jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
most importantly networking but certainly not memory subsystem. On Wed, 25 Apr 2012, Chuck Swiger wrote: On Apr 25, 2012, at 5:31 AM, jb wrote: does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? The simple answer is no. A more complex answer: % grep -ri freebsd xnu-1699.24.23 | wc -l 520 % grep -ril freebsd xnu-1699.24.23 | sort | uniq ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
2) Inactive memory (which is memory that has been recently used but is no longer) is supposed to be seamlessly reclaimed automatically by the OS when needed for new programs. In practice, I?ve found that this isn?t the case, and my system slows to a crawl and starts paging out to disk when free memory drops to zero, even as half of the available RAM (which is a lot) is marked as inactive. ... Well, this is not a case of a BSD is dying troll (you can safely ignore those). yes it is, just search a bit to know what inactive memory in FreeBSD is. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
If you really are having a problem with FreeBSD you are going to have to do a lot better than this in terms of providing some data points which define the problem. I am in agreement with Adam here: either you can work the problem or you can troll. I don't see any indication yet of any real problem analysis, only a wild mix of stuff non-related to FreeBSD sprinkled with some magic 'memory management' dust. The fact that FreeBSD DOES NOT page excessively on the same workload relative to other OS (linux, netbsd) is one of most important thing i decided to use it. If his system is heavily paging then simply he have too large working set. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
is relatively new. My guess is that if there is a problem it's ZFS specific. If it were a more general problem I think we'd see a lot more complaints, whereas ZFS already has a reputation for needing lots of memory. you may precisely set up a limits of memory that ZFS would use at most. or just don't use it which i do. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
Wojciech Puchar wojtek at wojtek.tensor.gdynia.pl writes: 2) Inactive memory (which is memory that has been recently used but is no longer) is supposed to be seamlessly reclaimed automatically by the OS when needed for new programs. In practice, I?ve found that this isn?t the case, and my system slows to a crawl and starts paging out to disk when free memory drops to zero, even as half of the available RAM (which is a lot) is marked as inactive. ... Well, this is not a case of a BSD is dying troll (you can safely ignore those). yes it is, just search a bit to know what inactive memory in FreeBSD is. His description (the quoted text) is at least of intuitive nature, and in fact in may be correct as it referrs to OS X MM subsys, which may be based on least-recently used pageout algorithm (as FreeBSD originally used to be too). jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
Wojciech Puchar wojtek at wojtek.tensor.gdynia.pl writes: does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? IMHO no. OSX is somehow-microkernel based, they did take things from FreeBSD but not this IMHO. anyway - who cares Well, I quoted the source in my 2nd post in this thread. But I will repeat it once again: I'm quite sure that the memory manager of OSX wasn't derived from BSD, but from Mach. Actually, FreeBSD has adapted that memory manager, so it's rather the other way around If so, both OS X and FreeBSD share the same MM subsys base. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
On Thu, Apr 26, 2012 at 12:04 AM, jb jb.1234a...@gmail.com wrote: If so, should FreeBSD adopt NetBSD's MM subsys, or just improve itself surgically ? You ought first establish there is a problem. What you have cited is recently reinvigorated trend that has taken on the air of the BDS is dying troll. What you have is a set of computer users with no understanding of kernel internals attempting to diagnose some sort of possibly legitimate problem by reaching conclusion via rumor and guesswork. These people can be taken about as seriously as those who insist the moon landing was fake and other bizarre ignorant pseudo-science. http://workstuff.tumblr.com/post/19036310553/two-things-that-really-helped-speed-up-my-mac-and http://dywypi.org/2012/02/back-on-linux.html When you have a test case illustrating your feared FreeBSD VM shortcomings, you may at that point begin to attract developer interest. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
Adam Vande More amvandemore at gmail.com writes: ... http://workstuff.tumblr.com/post/19036310553/two-things-that-really-helped- speed-up-my-mac-and http://dywypi.org/2012/02/back-on-linux.html 2) Inactive memory (which is memory that has been recently used but is no longer) is supposed to be seamlessly reclaimed automatically by the OS when needed for new programs. In practice, I’ve found that this isn’t the case, and my system slows to a crawl and starts paging out to disk when free memory drops to zero, even as half of the available RAM (which is a lot) is marked as inactive. ... Well, this is not a case of a BSD is dying troll (you can safely ignore those). The above and the past FreeBSD thread here, both I referred to, have something in common - the system seems to progressively come under stress due to what one user experienced as missing memory, and other two users experienced (as shown here above) as inefficient (or lack of) early reclamation of inactive pages. We just want the devs and users make aware of things. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
Adam Vande More wrote: On Thu, Apr 26, 2012 at 12:04 AM, jb jb.1234a...@gmail.com wrote: If so, should FreeBSD adopt NetBSD's MM subsys, or just improve itself surgically ? You ought first establish there is a problem. What you have cited is recently reinvigorated trend that has taken on the air of the BDS is dying troll. What you have is a set of computer users with no understanding of kernel internals attempting to diagnose some sort of possibly legitimate problem by reaching conclusion via rumor and guesswork. These people can be taken about as seriously as those who insist the moon landing was fake and other bizarre ignorant pseudo-science. http://workstuff.tumblr.com/post/19036310553/two-things-that-really-helped- speed-up-my-mac-and http://dywypi.org/2012/02/back-on-linux.html When you have a test case illustrating your feared FreeBSD VM shortcomings, you may at that point begin to attract developer interest. To the OP: A potential first test case where the symptom is my system slows to a crawl and starts paging out to disk might be to build a kernel with the SCHED_4BSD scheduler. There have been a couple of edge/corner cases that sound like this. That is, if you really have a problem and want to try eliminating one possibility. Another thing that shows up in things like top is it breaks and does not report accurate values for anything when userland and kernel are out of sync, that is if it runs at all without segfaulting. World and kernel being out of sync would be operator error. In this case the values you are using to somehow relate the symptom to memory management would be false. As far as all the rest, such as something being deeply broken in OS X memory management, mentions of NetBSD memory management, etc, are all irrelevant. It is this wild mix of stuff seemingly non-related to any problem in FreeBSD per se, that makes this look like a troll. If you really are having a problem with FreeBSD you are going to have to do a lot better than this in terms of providing some data points which define the problem. I am in agreement with Adam here: either you can work the problem or you can troll. I don't see any indication yet of any real problem analysis, only a wild mix of stuff non-related to FreeBSD sprinkled with some magic 'memory management' dust. Sorry if this comes across the wrong way, but this really looks like troll material to me too - it has a great resemblance to a pattern trolls have used for many years. -Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
On Thu, 26 Apr 2012 08:32:39 + (UTC) jb wrote: Adam Vande More amvandemore at gmail.com writes: ... http://workstuff.tumblr.com/post/19036310553/two-things-that-really-helped- speed-up-my-mac-and http://dywypi.org/2012/02/back-on-linux.html 2) Inactive memory (which is memory that has been recently used but is no longer) is supposed to be seamlessly reclaimed automatically by the OS when needed for new programs. In practice, I’ve found that this isn’t the case, and my system slows to a crawl and starts paging out to disk when free memory drops to zero, even as half of the available RAM (which is a lot) is marked as inactive. ... That's not a good description of inactive memory, most of which contains useful data. The situation described is undesirable, but not abnormal. It can happen when your physical memory is spread thinly, but most of it isn't being frequently accessed. In that case the inactive queue can be dominated by dirty swap-backed pages. The above and the past FreeBSD thread here, both I referred to, have something in common - the system seems to progressively come under stress due to what one user experienced as missing memory, The FreeBSD link involved ZFS which manages its own disk caching and is relatively new. My guess is that if there is a problem it's ZFS specific. If it were a more general problem I think we'd see a lot more complaints, whereas ZFS already has a reputation for needing lots of memory. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
RW rwmaillists at googlemail.com writes: ... ... 2) Inactive memory (which is memory that has been recently used but is no longer) is supposed to be seamlessly reclaimed automatically by the OS when needed for new programs. In practice, I’ve found that this isn’t the case, and my system slows to a crawl and starts paging out to disk when free memory drops to zero, even as half of the available RAM (which is a lot) is marked as inactive. ... That's not a good description of inactive memory, most of which contains useful data. The situation described is undesirable, but not abnormal. It can happen when your physical memory is spread thinly, but most of it isn't being frequently accessed. In that case the inactive queue can be dominated by dirty swap-backed pages. ... Would implementing the VM pageout algorithm in such a way that it would mix in equal proportion the current least-actively used algo and the old least-recently used algo help the situation ? jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
FreeBSD vice OS X memory management
Hi, does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? Something is deeply broken in OS X memory management http://workstuff.tumblr.com/post/20464780085/something-is-deeply-broken-in-os-x- memory-management One of the problems that caught my eyes was inactive memory reclamation. I remember some time ago there was a thread here with similar topic. http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239121.html jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
On Apr 25, 2012, at 5:31 AM, jb wrote: does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? The simple answer is no. A more complex answer: % grep -ri freebsd xnu-1699.24.23 | wc -l 520 % grep -ril freebsd xnu-1699.24.23 | sort | uniq % grep -ril freebsd xnu-1699.24.23 | sort | uniq ~/Downloads xnu-1699.24.23/EXTERNAL_HEADERS/stdbool.h xnu-1699.24.23/bsd/bsm/audit.h xnu-1699.24.23/bsd/bsm/audit_domain.h xnu-1699.24.23/bsd/bsm/audit_errno.h xnu-1699.24.23/bsd/bsm/audit_fcntl.h xnu-1699.24.23/bsd/bsm/audit_kevents.h xnu-1699.24.23/bsd/crypto/aes/gen/aesopt.h xnu-1699.24.23/bsd/crypto/blowfish/bf_enc.c xnu-1699.24.23/bsd/crypto/blowfish/bf_locl.h xnu-1699.24.23/bsd/crypto/blowfish/bf_pi.h xnu-1699.24.23/bsd/crypto/blowfish/bf_skey.c xnu-1699.24.23/bsd/crypto/blowfish/blowfish.h xnu-1699.24.23/bsd/crypto/cast128/cast128.c xnu-1699.24.23/bsd/crypto/cast128/cast128.h xnu-1699.24.23/bsd/crypto/cast128/cast128_subkey.h xnu-1699.24.23/bsd/crypto/des/des.h xnu-1699.24.23/bsd/crypto/des/des_ecb.c xnu-1699.24.23/bsd/crypto/des/des_enc.c xnu-1699.24.23/bsd/crypto/des/des_locl.h xnu-1699.24.23/bsd/crypto/des/des_setkey.c xnu-1699.24.23/bsd/crypto/des/podd.h xnu-1699.24.23/bsd/crypto/des/sk.h xnu-1699.24.23/bsd/crypto/des/spr.h xnu-1699.24.23/bsd/crypto/rc4/rc4.c xnu-1699.24.23/bsd/crypto/rc4/rc4.h xnu-1699.24.23/bsd/crypto/sha2/sha2.c xnu-1699.24.23/bsd/crypto/sha2/sha2.h xnu-1699.24.23/bsd/dev/dtrace/blist.c xnu-1699.24.23/bsd/dev/dtrace/blist.h xnu-1699.24.23/bsd/dev/memdev.c xnu-1699.24.23/bsd/dev/vn/vn.c xnu-1699.24.23/bsd/hfs/hfs_lookup.c xnu-1699.24.23/bsd/hfs/hfscommon/headers/RedBlackTree.h xnu-1699.24.23/bsd/kern/kern_event.c xnu-1699.24.23/bsd/kern/kern_mib.c xnu-1699.24.23/bsd/kern/kern_newsysctl.c xnu-1699.24.23/bsd/kern/kern_resource.c xnu-1699.24.23/bsd/kern/makesyscalls.sh xnu-1699.24.23/bsd/kern/sys_pipe.c xnu-1699.24.23/bsd/kern/syscalls.master xnu-1699.24.23/bsd/kern/tty.c xnu-1699.24.23/bsd/kern/uipc_socket.c xnu-1699.24.23/bsd/kern/uipc_socket2.c xnu-1699.24.23/bsd/libkern/strsep.c xnu-1699.24.23/bsd/man/man2/aio_cancel.2 xnu-1699.24.23/bsd/man/man2/aio_error.2 xnu-1699.24.23/bsd/man/man2/aio_read.2 xnu-1699.24.23/bsd/man/man2/aio_return.2 xnu-1699.24.23/bsd/man/man2/aio_suspend.2 xnu-1699.24.23/bsd/man/man2/aio_write.2 xnu-1699.24.23/bsd/man/man2/audit.2 xnu-1699.24.23/bsd/man/man2/auditctl.2 xnu-1699.24.23/bsd/man/man2/auditon.2 xnu-1699.24.23/bsd/man/man2/getaudit.2 xnu-1699.24.23/bsd/man/man2/getauid.2 xnu-1699.24.23/bsd/man/man2/getdtablesize.2 xnu-1699.24.23/bsd/man/man2/getlcid.2 xnu-1699.24.23/bsd/man/man2/getpgrp.2 xnu-1699.24.23/bsd/man/man2/getsid.2 xnu-1699.24.23/bsd/man/man2/i386_get_ldt.2 xnu-1699.24.23/bsd/man/man2/issetugid.2 xnu-1699.24.23/bsd/man/man2/kqueue.2 xnu-1699.24.23/bsd/man/man2/mmap.2 xnu-1699.24.23/bsd/man/man2/mprotect.2 xnu-1699.24.23/bsd/man/man2/msync.2 xnu-1699.24.23/bsd/man/man2/read.2 xnu-1699.24.23/bsd/man/man2/semctl.2 xnu-1699.24.23/bsd/man/man2/semget.2 xnu-1699.24.23/bsd/man/man2/semop.2 xnu-1699.24.23/bsd/man/man2/sendfile.2 xnu-1699.24.23/bsd/man/man2/setaudit.2 xnu-1699.24.23/bsd/man/man2/setauid.2 xnu-1699.24.23/bsd/man/man2/setlcid.2 xnu-1699.24.23/bsd/man/man2/setregid.2 xnu-1699.24.23/bsd/man/man2/setreuid.2 xnu-1699.24.23/bsd/man/man2/sigaction.2 xnu-1699.24.23/bsd/man/man2/undelete.2 xnu-1699.24.23/bsd/man/man2/utimes.2 xnu-1699.24.23/bsd/man/man2/write.2 xnu-1699.24.23/bsd/man/man3/queue.3 xnu-1699.24.23/bsd/man/man4/aio.4 xnu-1699.24.23/bsd/man/man4/audit.4 xnu-1699.24.23/bsd/man/man4/auditpipe.4 xnu-1699.24.23/bsd/man/man4/bpf.4 xnu-1699.24.23/bsd/man/man4/divert.4 xnu-1699.24.23/bsd/man/man4/dummynet.4 xnu-1699.24.23/bsd/man/man4/faith.4 xnu-1699.24.23/bsd/man/man4/gif.4 xnu-1699.24.23/bsd/man/man4/ifmib.4 xnu-1699.24.23/bsd/man/man4/inet6.4 xnu-1699.24.23/bsd/man/man4/ipfirewall.4 xnu-1699.24.23/bsd/man/man4/ipsec.4 xnu-1699.24.23/bsd/man/man4/stf.4 xnu-1699.24.23/bsd/man/man4/tty.4 xnu-1699.24.23/bsd/man/man9/copy.9 xnu-1699.24.23/bsd/man/man9/fetch.9 xnu-1699.24.23/bsd/man/man9/intro.9 xnu-1699.24.23/bsd/man/man9/store.9 xnu-1699.24.23/bsd/man/man9/style.9 xnu-1699.24.23/bsd/miscfs/devfs/README xnu-1699.24.23/bsd/miscfs/devfs/devfs.h xnu-1699.24.23/bsd/miscfs/devfs/devfs_tree.c xnu-1699.24.23/bsd/miscfs/devfs/devfs_vfsops.c xnu-1699.24.23/bsd/miscfs/devfs/devfs_vnops.c xnu-1699.24.23/bsd/miscfs/devfs/devfsdefs.h xnu-1699.24.23/bsd/net/bpf.c xnu-1699.24.23/bsd/net/bpf.h xnu-1699.24.23/bsd/net/bpf_compat.h xnu-1699.24.23/bsd/net/bpf_filter.c xnu-1699.24.23/bsd/net/bpfdesc.h xnu-1699.24.23/bsd/net/bridgestp.c xnu-1699.24.23/bsd/net/bridgestp.h xnu-1699.24.23/bsd/net/if.c xnu-1699.24.23/bsd/net/if.h xnu-1699.24.23/bsd/net/if_arp.h xnu-1699.24.23/bsd/net/if_bridge.c xnu-1699.24.23/bsd/net/if_bridgevar.h xnu-1699.24.23/bsd/net/if_dl.h xnu-1699.24.23/bsd/net/if_gif.c xnu-1699.24.23/bsd/net/if_loop.c
Re: FreeBSD vice OS X memory management
Chuck Swiger cswiger at mac.com writes: On Apr 25, 2012, at 5:31 AM, jb wrote: does OS X kernel share any code with FreeBSD kernel's memory management subsystem ? The simple answer is no. A more complex answer: % grep -ri freebsd xnu-1699.24.23 | wc -l 520 % grep -ril freebsd xnu-1699.24.23 | sort | uniq % grep -ril freebsd xnu-1699.24.23 | sort | uniq ~/Downloads xnu-1699.24.23/EXTERNAL_HEADERS/stdbool.h xnu-1699.24.23/bsd/bsm/audit.h ... Right, MM subsys is not part of BSD import. But XNU kernel is a combo of Mach and old BSD kernel parts. There was some discussion here: http://www.osnews.com/comments/25861 where two comments are of interest: I'm quite sure that the memory manager of OSX wasn't derived from BSD, but from Mach. Actually, FreeBSD has adapted that memory manager, so it's rather the other way around. But Apple might learn from the way FreeBSD does things. If it is feasible, as the kernel is quite different. The related implementation in FreeBSD seems to have a similar problem: NetBSD users have also reported that UVM’s im- provements have had a positive effect on their applica- tions. This is most noticeable when physical memory becomes scarce and the VM system must page out data to free up memory. Under BSD VM this type of paging causes the system to become highly unresponsive, while under UVM the system slows while paging but does not become unresponsive. http://static.usenix.org/event/usenix99/full_papers/cranor/cranor.p... Should be easy to fix: just start to page out some stuff in time before there is no memory left. When I browsed the USENIX paper (dated 1999) I understood that it is indeed possible that FreeBSD may have imported some Mach's MM code in those early BSD VM days. And over time since then some ideas (if not exact code) may have migrated between OS X and FreeBSD MM subsystems. If so, the problems experienced may be similar or identical even today. jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD vice OS X memory management
jb jb.1234abcd at gmail.com writes: ... The related implementation in FreeBSD seems to have a similar problem: NetBSD users have also reported that UVM’s im- provements have had a positive effect on their applica- tions. This is most noticeable when physical memory becomes scarce and the VM system must page out data to free up memory. Under BSD VM this type of paging causes the system to become highly unresponsive, while under UVM the system slows while paging but does not become unresponsive. http://static.usenix.org/event/usenix99/full_papers/cranor/cranor.p... Should be easy to fix: just start to page out some stuff in time before there is no memory left. ... Would this mean that FreeBSD's (and Mach's ?) MM subsys are behind NetBSD's ? If so, should FreeBSD adopt NetBSD's MM subsys, or just improve itself surgically ? jb ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
