Re: Idea: disposable memory
Wouldn't stackable filesystems solve this rather neatly? -- Ben Rosengart UNIX Systems Engineer, Skunk Group StarMedia Network, Inc. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
On Thu, 23 Sep 1999, Matthew Dillon wrote: : Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. Don't you mean MADV_FREE? -Alfred The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: until the system reuses it, at which point it reverts to zero-fill. : :Don't you mean MADV_FREE? : :-Alfred Oops. Yes. Sorry. MADV_DONTNEED maintains data integrity. MADV_FREE doesn't. What I described in my last message applies to MADV_FREE. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. Either I'm not properly explaining what I want, or I'm misunderstanding you guys. :) Let me try to rexplain. The way things work now for us is that I mmap() decompressed video files in, and send them straight to our blitter. This works well since the system will nicely keep as much as is possible in ram, and rather than swap this out, it'll just discard what it is in memory if more ram is needed. When I need that bit of data again, it's fetched back off the disk for me. I'm now playing with compressed data streams. The decompression is slow, so I'd like to cache the *decompressed* version of these files. I end up allocating large amounts of ram in one process to cache the decompressed data. This is a disavantage over the above scenario, since now the system swaps out my decompressed data when more ram is needed elsewhere. Swapping out then swapping back in my decompressed data is about 4x slower than just re-reading my compressed stream and decompressing it again. Why don't I just allocate a predefined amount of memory and use that for a cache all the time? Most of the time we have about 20MB free on our system. Sometimes we end up with about 2MB free though, and what's happening now is that I start paging out data that I could recreate in less time than the page-in/page-out takes. What I want is to maintain a largish in-memory cache of decompressed data, and tell the kernel that "I'm still using this memory, but if memory runs short somewhere else, you're allowed to take it back from me. This ram has approximately the same priority as data in the disk cache". However, since I could decide to re-use anything in my cache at any time, I *need* to know if the kernel took some of this back from me, so I can then recreate it when needed. I could see two ways (from a userland perspective) of doing this. Either I have to check (somehow) every time I use a region of ram to make sure it's still there by some syscall, or the kernel somehow tells me that I need to get rid of x amount of ram, and it's up to me to decide what I want to throw away. I don't think MADV_FREE is what I want, since it makes my memory go away very quickly, *and* I have no way of knowing that the kernel did it. Make any more sense? Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Lo and behold, Kevin Day once said: I don't think MADV_FREE is what I want, since it makes my memory go away very quickly, *and* I have no way of knowing that the kernel did it. You do have a way of knowing the kernel did it - your memory is suddenly full of zeros. You don't have an *asynchronous* way of knowing that the kernel did it, however, which I believe is more what you're aiming at. (Something like a "SIG_I_STOLE_YOUR_MEMORY"), no? -Dave -- work: [EMAIL PROTECTED] me: [EMAIL PROTECTED] MIT LCS http://www.angio.net/ "If you haul a geek up a crack, you will bloody their fingers for a day... If you teach a geek to climb, you will bloody their fingers for life." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
In message [EMAIL PROTECTED] Kevin Day writes: : I'm now playing with compressed data streams. The decompression is slow, so : I'd like to cache the *decompressed* version of these files. I end up : allocating large amounts of ram in one process to cache the decompressed : data. This is a disavantage over the above scenario, since now the system : swaps out my decompressed data when more ram is needed elsewhere. Swapping : out then swapping back in my decompressed data is about 4x slower than just : re-reading my compressed stream and decompressing it again. Sounds like a short term fix might be to store the decompressed files on the md device that phk just checked in. However, while better in some ways than what you are doing now, it is worse in others. There is also a memory filesystem as well. However, neither is quite what you are aksing for because the cache doesn't get tossed. automatically. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
:I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] :Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
It sounds like what he wants is some sort of userland swapper. In this case, the implementation would be to decompress when pages are swapped in, and simply drop the page when it's swapped out. Given the current constraints, and the fact that decompression will touch the entire dataset _anyhow_, it would make sense for the decompression pass to prime a data structure with pointers to non-zero data within each page (probably int-aligned for performance reasons), and mark it disposable as suggested elsewhere. Skip any page which is all zeros. Then when the data is to be used, mlock() it, check to see if any of the non-zero pointers now point to zeros, decompress those pages as needed, blit them, munlink(), and mark them disposable again. Actually, that might be better than a userland swapper, because in that case there's nothing to prevent you from blitting half the dataset, and then hitting a swap. Later, scott - Original Message - From: Matthew Dillon [EMAIL PROTECTED] To: Kevin Day [EMAIL PROTECTED] Cc: Kevin Day [EMAIL PROTECTED]; Daniel C. Sobral [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, September 23, 1999 9:54 AM Subject: Re: Idea: disposable memory :I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Another idea might be to enhance the swapper. Using interleaved swap across a number of SCSI disks is a poor-man's way of getting serious disk bandwidth. My seacrate's can do around 15MB/sec to the platter. My test machine's swap is spread across three of them, giving me 45MB/sec of swap bandwidth. Of course, that's with relatively expensive drives and a U2W/LVD SCSI bus (80MB/sec bus). Another possibility is to purchase a single large, cheap DMA/IDE drive. IBM has a number of 20+ GB drives that can transfer (I believe) 20MB/sec+ from the platter. You get one of those babies and you can use a raw partition to hold part of your decompressed video stream. No memory is used at all in this case, you depend entirely on the disk's bandwidth to push the data out and pull it in as needed. If the disk is dedicated, this should be doable. Using a raw partition (e.g. like /dev/rda5a) is beneficial if you intend to do your own cache management. Using a block partition (e.g. like /dev/da5a) is beneficial if you want the system to manage the caching for you but will result in lower I/O bandwidth due to the extra copy. You can also implement a multi-process scheme to stage the data in and out of memory. One process would be responsible for the cache management and would do lookaheads and other requests, with the cache implemented as a shared-memory segment (see shmat, shmdt, shmctl, shmget). The other process would map the same shared memory segment and use the results. You could use SysV semaphores (see semctl, semget, and semop) for locking. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
I think what is needed is something similar to what we used to use at TFS. A device driver that controled a large number of pages. it had ioclts to allocate 'buffers' from these pages. each buffer was given a handle by whichthe user process refered to it. multiple processes could refer to them. the kernel could dealocate them, in which case an attempt to use that handle in an ioclt (all operatiosn were via ioctls) would return an error and the user knew it was no longer valid. device drivers were taught to be able to take a UIO of physical addresses for IO, and ioctls were added to these devices which took a buffer handle as an argument. e.g. "write buffer 0x34823 to disk location 0x125434" (the length was implied by the length of the buffer..) ioctls included.. (protocol stack). send buffer xyz to address 12.34.56.78 (buffer device) allocate a buffer of size x. (buffer device) free buffer y (buffer device) allow discard of buffer z (can discard if short of ram) (disk driver)write buffer z to location A. (disk driver)read buffer z from location A (sized by buffer size) (there was also a protocol which delivered a received buffer as a result of a recvmesg() call in the auxhiliary data) there was reference counting so that a buffer could never be freed until all users had released it. (e.g. disk requwsts held a reference, as did send requests). you could also mmap buffers into address space. julian On Thu, 23 Sep 1999, Matthew Dillon wrote: :I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] :Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Kevin Day [EMAIL PROTECTED] wrote: I'd like a way to be able to specify that a region of malloc'ed data is 'disposable' and has approximately the same weight as disk cached data. As others have pointed out, this is almost (but not quite) the same as madvise(MADV_FREE). I don't think there is any way to achieve what Kevin wants with the current kernel. The simplest solution would appear to be a new madvise option which marks the relevant pages as `throw away instead of swapping out'. This is similar to MADV_FREE, but does not immediately free the page. It will also need to flag the page so the the user is warned if the page _is_ reused - the most logical way is by returning SIGBUS or SIGSEGV the first time the page is referenced after it has been released. The user process then just reads data normally. If it gets a signal, it knows it needs to re-create that page - this is presumably relatively trivial to do on a page-by-page basis (since you have no control over which page gets reused). The code to handle the `throw away' bit should be fairly trivial - it's just the existing MADV_FREE code, without the actual `free' bit. I'm less certain about being able to mark a page mapping so that a a page fault gets passed back to userland as well as mapping a zero- filled page to that location. Peter To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Idea: disposable memory
Perhaps this is already possible somehow, but... In working with a graphical based embedded system (non-xwin), I'll typically mmap the graphic files and bcopy them straight to our hardware blitter. This works very nicely, since the kernel caches what it can off the disk, but when more ram is needed by other processes, things get thrown away and can be retrieved easily. I've been experimenting with various kinds of compression. Most of them decompress somewhat quickly, but it still takes time. So, I'd like to decompress them once, and put them in ram, in case I'd ever like them again. This has the side effect of sucking too much ram at times, and not using enough ram as a cache in other times. I don't know if another process springs up suddenly and needs half my system ram for a short time. The system will throw away anything in the disk cache, but will instead swap out the ram being used as a cache. Swapping the much larger decompressed data out, then bringing it back is about 4 times slower than just throwing it away, and re-reading the compressed data and decompressing it again. I'd like a way to be able to specify that a region of malloc'ed data is 'disposable' and has approximately the same weight as disk cached data. (I don't even mind running some syscall to tell the vm system that i just used a page, to update a lru counter or something). If the system needs ram back for some other process, either give me a way of knowing this when it steals the ram back, or signal me with something and tell me how much ram you want me to give back and let the userland process do it. Is anyone aware of anything even similar to this now, or can anyone think of a better way of doing this? Yes, I could make my cache smarter, but without constantly watching what other processes are doing, I could be not using ram that I could be using as a texture cache, or I might be using so much ram that my cache gets swapped out, which is much slower than me just recreating what's in the cache. Thoughts? Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
