Hi all:
We had a meet of the Bangalore OpenSolaris User Group meet this Saturday.
We didn't cover everything on the agenda, since the first two topics
took longer time than planned for. However, both of these were very
exciting in themselves.
a. The new memory allocator in Solaris (and Linux too, today).
All applications need memory during usage. When we load a Word
document into memory, Word asks Windows for some memory and used that
memory for working with the document. This happens with other
applications and with other operating systems too.
Did you know that just like files on hard disks get fragmented,
memory can get fragmented too ? And that just as comp. sci.
researchers are working on file systems with low fragmentation, comp.
sci. researchers are working on optimized memory allocators too ? In
fact, writing and selling memory allocation libraries is a huge
industry in itself.
Today, Surya Pakki from Sun Microsystems spoke about the new
generation memory allocator that the Solaris Kernel provides. This is
called the Slab Memory Allocator. It is now available on the Linux
kernel too (It's a clean room implementation based on a Usenix paper
authored by Jeff Bonwick from Sun - the same person who wrote the slab
memory allocation as well as the ZFS file system). He described how
the Slab Memory allocator improves memory allocation and deallocation
speeds, is safer, and can be used by either explicit functions calls,
or (on Solaris) by using libumem instead of libc.
Surya applied DTrace to quickly show us how memory allocation was
working in the kernel.
I initially assumed that this would be a rather technically involved
talk which I'd not understand, etc, but this was a very interactive
and engaging talk.
b. Industry report on memory allocation issues:
Shiv, a long time member of the BOSUG and a tech person at Nokia
Siemens Networks, pointed out how memory allocation across Unixes is
misleading because when the developer asks the OS to take back memory,
the OS does not necessarily do so ! So on phone systems which cater to
something like a million calls a sec during Christmas, they need
absolute control over memory, and diagnostic tools sometimes report
insufficient memory, though the OS itself would be able to allocate
memory if needed. Surya clarified that when there is a demand for
memory, the memory allocator gives up memory on its own. Shiv also
revealed that despite the app running on Solaris 10, the resource
requirements were so critical that even enabling memory allocator
debugging would be expensive (leave alone using the memory allocator
which imposes much more demand).
The real problem here is once an an app calls free() memory is not
returned to the system. So massive demand apps can grow their memory
usage to a large size during peak load and then remain pegged at that
mem usage unless there is a memory crunch in which case the page
scanner kicks in and retires unused pages from application's heap to
swap. This behavior can be a problem for high-end telecom apps as
noted above.
We had an engaging discussion on this - I found this an interesting
problem to solve.
c. A new package management system for Belenix
Moinak, the creator of Belenix - an OpenSolaris based distribution -
spoke about our plans to put in a new package management systems for
Belenix (and eventually, OpenSolaris too). The current packaging spec
(SVR4) is very well thought out, but poorly implemented. The
replacement from Sun is riddled with issues (you can't mirror the
repository, for e.g.), the present package management tool that we're
using turned out to _not_ be open source, and the author has a
restrictive license. So, we cannot adapt and modify it, and are not
interested in contributing to a non-opensource tool . For the short
term (the next six months), we're going to put in place our own
package management. The aim is to provide a mirrorable repository, and
some python scripts that permit us to fetch files from the repository
with automatic dependency resolution, install, upgrade, multi-site
support, etc. Once this is in place (ETA two months), we'll focus our
energy on moving to Nexenta's port of dpkg to OpenSolaris, and add
functionality to the Smart Package Manager to integrate it better with
OpenSolaris
The Smart Package Manager is sponsored by Canonical (the company
behind Ubuntu), and receives advice from the maintainers of RPM
(Redhat/Fedora format), dpkg (Debian format), apt (the Debian/Ubuntu
packagement management tool), and the freebsd ports maintainers.
The Belenix group is very friendly with Nexenta. We're focussed on
the desktop, while they're focussed on enterprise grade storage using
the ZFS file system.
Anyone who is interested in helping out is welcome. The source is
available in the Belenix SVN repository, while the dpkg port source is
available in the Nexenta source repository.
All in all, this was a very interesting session today. We could go
ahead and have the next BOSUG meet too in the Thoughtworks office :)
-- Ram
