Re: buildfailure -current on Alpha?
Wilko Bulte wrote... On a freshly supped -current I get: location type 0 in non-PLT relocations ./make_keys /usr/src/lib/libncurses/../../contrib/ncurses/ncurses/tinfo/keys.list init_keytry.h /usr/libexec/ld-elf.so.1: /usr/obj/usr/src/tmp/usr/lib/libc.so.3: Unsupported relocation type 0 in non-PLT relocations *** Error code 1 1 error * I have no idea what a PLT-relocation is supposed to be ;-) Any other Alpha users seen this? I take it you haven't rebuilt world in the past few months? There was a change that went in in June, I think, that requires that you reinstall ld-elf.so.1 manually before you can buildworld on an Alpha. Kinda annoying, I think. Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: StarOffice 5.1 - infinite setup ?
On Wed, Sep 22, 1999 at 08:46:42PM -0500, Kevin Day wrote: Hi, I have got a surprising problem with StarOffice 5.1 for Linux on FreeBSD 4.0-current, the latest snapshot. The CD-ROM installation went fine (after I configured the Posix real-time thread support and linked the additional libraries to the Linux compatibility directory and slightly corrected the startup scripts) but when I start the installed program "soffice" it just starts the setup screen again! Tried to go through it, select no additional components, the setup says that everything is installed and exits. I was not able to get anything else from it. Has anyone met this problem before ? Any ideas would be welcome. Thank you! -SB We've got a similar problem. Instals fine as root, runs fine a 'joe', but if anyone else tries to run it they get the setup screen! My hunch is that it's something to do with permissions on Sys5 IPC queues or something. A Ktrace of both showed that different things were going on, but I lost the plot before working out exactly what. Joe -- Josef KarthauserFreeBSD: How many times have you booted today? Technical Manager Viagra for your server (http://www.uk.freebsd.org) Pavilion Internet plc. [[EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED]] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
On Thu, 23 Sep 1999, Matthew Dillon wrote: : Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. Don't you mean MADV_FREE? -Alfred The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: until the system reuses it, at which point it reverts to zero-fill. : :Don't you mean MADV_FREE? : :-Alfred Oops. Yes. Sorry. MADV_DONTNEED maintains data integrity. MADV_FREE doesn't. What I described in my last message applies to MADV_FREE. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: StarOffice 5.1 - infinite setup ?
We've got a similar problem. Instals fine as root, runs fine a 'joe', but if anyone else tries to run it they get the setup screen! My hunch is that it's something to do with permissions on Sys5 IPC queues or something. A Ktrace of both showed that different things were going on, but I lost the plot before working out exactly what. I wasn't paying attention to the beginning of this thread, so forgive me if I'm repeating something already stated... Are you sure you initially installed StarOffice with the /net option (./setup /net)? There is also a StarOffice setup file that is created in the home directory of each person that runs setup (don't recall what the name of it is, .soffice?). You have to be sure to delete this file before running setup. Each user then needs to run setup once (without /net) and select a workstation install, which just installs a few files in their home directory. Dan Eischen [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
: Thoughts? : : man madvise? : : :Yeah, but MADV_FREE doesn't really do what I need. I have no idea if the :system actually did free my ram or not. I want to hang on to the data, but :if more ram is needed, then it can be discarded, but I need to know that it :did, so that I can recreate it. Checking every time I blit an object to see :if the page is zero'ed won't work. : :Kevin madvise ... MADV_DONTNEED is what you want. The data will remain mapped until the system reuses it, at which point it reverts to zero-fill. The system will reuse the data fairly quickly, even if the system is not all that loaded. You can lock the page back in simply by writing to something in the page. The system implements this madvise feature by marking the pages clean. If you happen to write to the page before the system reuses it, it of course gets redirtied. If you don't and the system reuses the page, it goes bye bye (turns into zero-fill) from the point of view of your process. Either I'm not properly explaining what I want, or I'm misunderstanding you guys. :) Let me try to rexplain. The way things work now for us is that I mmap() decompressed video files in, and send them straight to our blitter. This works well since the system will nicely keep as much as is possible in ram, and rather than swap this out, it'll just discard what it is in memory if more ram is needed. When I need that bit of data again, it's fetched back off the disk for me. I'm now playing with compressed data streams. The decompression is slow, so I'd like to cache the *decompressed* version of these files. I end up allocating large amounts of ram in one process to cache the decompressed data. This is a disavantage over the above scenario, since now the system swaps out my decompressed data when more ram is needed elsewhere. Swapping out then swapping back in my decompressed data is about 4x slower than just re-reading my compressed stream and decompressing it again. Why don't I just allocate a predefined amount of memory and use that for a cache all the time? Most of the time we have about 20MB free on our system. Sometimes we end up with about 2MB free though, and what's happening now is that I start paging out data that I could recreate in less time than the page-in/page-out takes. What I want is to maintain a largish in-memory cache of decompressed data, and tell the kernel that "I'm still using this memory, but if memory runs short somewhere else, you're allowed to take it back from me. This ram has approximately the same priority as data in the disk cache". However, since I could decide to re-use anything in my cache at any time, I *need* to know if the kernel took some of this back from me, so I can then recreate it when needed. I could see two ways (from a userland perspective) of doing this. Either I have to check (somehow) every time I use a region of ram to make sure it's still there by some syscall, or the kernel somehow tells me that I need to get rid of x amount of ram, and it's up to me to decide what I want to throw away. I don't think MADV_FREE is what I want, since it makes my memory go away very quickly, *and* I have no way of knowing that the kernel did it. Make any more sense? Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Lo and behold, Kevin Day once said: I don't think MADV_FREE is what I want, since it makes my memory go away very quickly, *and* I have no way of knowing that the kernel did it. You do have a way of knowing the kernel did it - your memory is suddenly full of zeros. You don't have an *asynchronous* way of knowing that the kernel did it, however, which I believe is more what you're aiming at. (Something like a "SIG_I_STOLE_YOUR_MEMORY"), no? -Dave -- work: [EMAIL PROTECTED] me: [EMAIL PROTECTED] MIT LCS http://www.angio.net/ "If you haul a geek up a crack, you will bloody their fingers for a day... If you teach a geek to climb, you will bloody their fingers for life." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Max M Queue Message Size
It appears that I can only send a maximum of 4096 bytes per message into a SYSV Message Queue. How do I increase this limit? Much thanks, Wayne To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
In message [EMAIL PROTECTED] Kevin Day writes: : I'm now playing with compressed data streams. The decompression is slow, so : I'd like to cache the *decompressed* version of these files. I end up : allocating large amounts of ram in one process to cache the decompressed : data. This is a disavantage over the above scenario, since now the system : swaps out my decompressed data when more ram is needed elsewhere. Swapping : out then swapping back in my decompressed data is about 4x slower than just : re-reading my compressed stream and decompressing it again. Sounds like a short term fix might be to store the decompressed files on the md device that phk just checked in. However, while better in some ways than what you are doing now, it is worse in others. There is also a memory filesystem as well. However, neither is quite what you are aksing for because the cache doesn't get tossed. automatically. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
NFS and RPC
NFS is said to be built on RPC. However, NFS daemons seldom uses RPC library calls before they enter into the kernel forever (The nfsd daemon only calls pmap_set(). The nfsiod daemon invokes no RPC call at all! The mountd daemon uses a lot of RPC calls, but it does not enter the kernel forever and it is not used during normal reads/writes). Once inside the kernel, the NFS daemons can not use RPC library any more, they must create/interprete RPC format messages themselves. My guess this is for performance reason and because there is no kernel-to-kernel RPC. Since the kernel part of NFS code does not use RPC library routines, why FreeBSD still conforms to the RPC format for NFS requests/replies? Is this for compatibily with other NFS servers/clients that are implemented entirely as user-level code and with RPC library routines? One more question is about how to assembly a RPC request from several mbufs? I notice that there is a check for 0x8000 in the routine nfsrv_getstream() for the last fragment. But I assume that all mbufs linked together via their m_next field should consist of one RPC request. I also assume that the protocol layer has ripped off any TCP/IP headers from the mbufs before it pass the mbufs up to the socket layer. Any help is appreciated. -- Zhihui Zhang. Please visit http://www.freebsd.org -- To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: buildfailure -current on Alpha?
In article [EMAIL PROTECTED], Kenneth D. Merry [EMAIL PROTECTED] wrote: I take it you haven't rebuilt world in the past few months? There was a change that went in in June, I think, that requires that you reinstall ld-elf.so.1 manually before you can buildworld on an Alpha. Kinda annoying, I think. Annoying but unavoidable. There's no way to override the location of the dynamic linker at make world time, for security-related reasons. An important bugfix to the Alpha binutils required an updated dynamic linker to handle the new relocation type. The new dynamic linker was committed well before the binutils changes that required it. But still it can bite people who aren't tracking -current very closely. That's life in currentland. John -- John Polstra [EMAIL PROTECTED] John D. Polstra Co., Inc.Seattle, Washington USA "No matter how cynical I get, I just can't keep up."-- Nora Ephron To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Out of swap handling and X lockups in 3.2R
:Matthew Dillon wrote: : : How about this - add an 'importance' resource. The lower the number, : the more likely the process will be killed if the system runs out of : resources. We would also make fork automatically decrement the number : by one in the child. : :Well, that's one thing people have asked for. It can be useful, and :doesn't sound particularly hard to code, nor too intrusive or :resource-hog. Would make some people, on both camps. : :Alas, some people will never let go until we have a no overcommit :switch, and *then* they'll start asking for us to go to the lengths :Solaris does to reduce the disadvantages. : :-- :Daniel C. Sobral (8-DCS) My feeling is that adding an importance mechanism would not preclude adding more sophisticated mechanisms on top of it. The real question is: Are enough people interested in me doing the *basic* (word emphasized) importance mechanism for me to do it? What it all comes down to is a juxtaposition of what people believe is appropriate verses what people are actually willing to code up. I'm willing to code up my importance mechanism idea. The question is whether it's a good enough idea to throw into the tree. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
:I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] :Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
It sounds like what he wants is some sort of userland swapper. In this case, the implementation would be to decompress when pages are swapped in, and simply drop the page when it's swapped out. Given the current constraints, and the fact that decompression will touch the entire dataset _anyhow_, it would make sense for the decompression pass to prime a data structure with pointers to non-zero data within each page (probably int-aligned for performance reasons), and mark it disposable as suggested elsewhere. Skip any page which is all zeros. Then when the data is to be used, mlock() it, check to see if any of the non-zero pointers now point to zeros, decompress those pages as needed, blit them, munlink(), and mark them disposable again. Actually, that might be better than a userland swapper, because in that case there's nothing to prevent you from blitting half the dataset, and then hitting a swap. Later, scott - Original Message - From: Matthew Dillon [EMAIL PROTECTED] To: Kevin Day [EMAIL PROTECTED] Cc: Kevin Day [EMAIL PROTECTED]; Daniel C. Sobral [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, September 23, 1999 9:54 AM Subject: Re: Idea: disposable memory :I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Another idea might be to enhance the swapper. Using interleaved swap across a number of SCSI disks is a poor-man's way of getting serious disk bandwidth. My seacrate's can do around 15MB/sec to the platter. My test machine's swap is spread across three of them, giving me 45MB/sec of swap bandwidth. Of course, that's with relatively expensive drives and a U2W/LVD SCSI bus (80MB/sec bus). Another possibility is to purchase a single large, cheap DMA/IDE drive. IBM has a number of 20+ GB drives that can transfer (I believe) 20MB/sec+ from the platter. You get one of those babies and you can use a raw partition to hold part of your decompressed video stream. No memory is used at all in this case, you depend entirely on the disk's bandwidth to push the data out and pull it in as needed. If the disk is dedicated, this should be doable. Using a raw partition (e.g. like /dev/rda5a) is beneficial if you intend to do your own cache management. Using a block partition (e.g. like /dev/da5a) is beneficial if you want the system to manage the caching for you but will result in lower I/O bandwidth due to the extra copy. You can also implement a multi-process scheme to stage the data in and out of memory. One process would be responsible for the cache management and would do lookaheads and other requests, with the cache implemented as a shared-memory segment (see shmat, shmdt, shmctl, shmget). The other process would map the same shared memory segment and use the results. You could use SysV semaphores (see semctl, semget, and semop) for locking. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
mbufs, external storage, and MFREE
I have the following question: Let's say that I have a block of user memory which I've mapped into the kernel, and would like to send on a network socket. I'd like to simply grab an mbuf, point to the memory as external storage, and queue it up for transmission. This would work fine, except that when MFREE gets called, I have to write an deallocator that maintains a table of all the different cases where I've done this, and do a reverse mapping back to the original block, and then deal with sending more, unmapping, etc. In other words, having MFREE call a deallocator with just the data pointer and the size is inconvenient (actually, it would make my scenario quite inefficient given the number of mappings back to the original block that would have to be done). Am I missing another mechanism to handle this? Does it not come up enough to matter? -Chris To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
using the ppp driver
Hi, I'm trying to write a driver which sends ppp packetsthrough the ethernet NIC. In order avoid implementing all the ppp protocol I want to use the existing ppp driver in FreeBSD. Is there a wayto send to the ppp driver data and get it back covered with the appropriate ppp frame? If there is, how can I indicate the different stages in the ppp protocol? I looked in the implementation of the ppp which uses the ppp driver but I think it does some assumptions (like using ppp with the modem device) that confuses me. please send your replies to me. Thank you, --Daniel Hilevich mailto:[EMAIL PROTECTED]Charlotte's Web Networks LTD.Tel: +972-4-9592203 ext. 214
Re: mbufs, external storage, and MFREE
:I have the following question: Let's say that I have a block of user :memory which I've mapped into the kernel, and would like to send on a :network socket. I'd like to simply grab an mbuf, point to the memory as :external storage, and queue it up for transmission. This would work fine, :except that when MFREE gets called, I have to write an deallocator that :maintains a table of all the different cases where I've done this, and do :a reverse mapping back to the original block, and then deal with sending :more, unmapping, etc. In other words, having MFREE call a deallocator :with just the data pointer and the size is inconvenient (actually, it :would make my scenario quite inefficient given the number of mappings back :to the original block that would have to be done). : :Am I missing another mechanism to handle this? Does it not come up enough :to matter? : :-Chris This is almost precisely the mechanism that the sendfile() system call uses. In that case it maps VMIO-backed data rather then user memory, but it is a very similar problem. There has been talk of implementing this type of mechanism not only for sockets, but for file read()/write() as well. In fact, John Dyson had delved into the issue with his vfs.ioopt stuff before he ran out of time. The one problem with using direct VM page mappings is that currently there is no way for the socket to prevent the underlying data from being modified in the middle of a transmission. And, in the same respect for vfs.ioopt, no way to prevent the data the user ostensibly read() into his 'private' buffer from changing out from under the user if the underlying file is modified. For user memory, the only way such a mechanism can currently be implemented is by obtaining the underlying pages and busy'ing them for the duration of their use by the system, causing anyone trying to access them while the system operation is in progress to block. This can cause a potential problem with TCP in that the mbuf data you send to TCP sticks around until it gets pushed out the door *and* acknowledged by the other end. i.e. the data is not disposed of as when read() or write() returns but instead goes directly into TCP's outgoing queue. If the TCP connection hangs, the process may hang. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
I think what is needed is something similar to what we used to use at TFS. A device driver that controled a large number of pages. it had ioclts to allocate 'buffers' from these pages. each buffer was given a handle by whichthe user process refered to it. multiple processes could refer to them. the kernel could dealocate them, in which case an attempt to use that handle in an ioclt (all operatiosn were via ioctls) would return an error and the user knew it was no longer valid. device drivers were taught to be able to take a UIO of physical addresses for IO, and ioctls were added to these devices which took a buffer handle as an argument. e.g. "write buffer 0x34823 to disk location 0x125434" (the length was implied by the length of the buffer..) ioctls included.. (protocol stack). send buffer xyz to address 12.34.56.78 (buffer device) allocate a buffer of size x. (buffer device) free buffer y (buffer device) allow discard of buffer z (can discard if short of ram) (disk driver)write buffer z to location A. (disk driver)read buffer z from location A (sized by buffer size) (there was also a protocol which delivered a received buffer as a result of a recvmesg() call in the auxhiliary data) there was reference counting so that a buffer could never be freed until all users had released it. (e.g. disk requwsts held a reference, as did send requests). you could also mmap buffers into address space. julian On Thu, 23 Sep 1999, Matthew Dillon wrote: :I'm now playing with compressed data streams. The decompression is slow, so :I'd like to cache the *decompressed* version of these files. I end up :allocating large amounts of ram in one process to cache the decompressed :data. This is a disavantage over the above scenario, since now the system :swaps out my decompressed data when more ram is needed elsewhere. Swapping :out then swapping back in my decompressed data is about 4x slower than just :re-reading my compressed stream and decompressing it again. : :Why don't I just allocate a predefined amount of memory and use that for a :cache all the time? Most of the time we have about 20MB free on our system. :Sometimes we end up with about 2MB free though, and what's happening now is :that I start paging out data that I could recreate in less time than the :page-in/page-out takes. Hmm. Well, you can check whether the memory has been swapped out with mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing something that has been swapped out frees the swap and turns it back into zero-fill). That doesn't get rid of the swapout bandwidth, though. I think, ultimately, you need to manage the memory used for your cache manually. That means using mlock() and munlock() to lock your cache into memory. For example, choose a cache size that you believe the system can support without going bonkers, like 5MB. mmap() 5MB of ram and mlock() it into memory. From that point on until you munlock() it or exit, the memory will not be swapped out. If the purpose of the box is to maintain the flow of video, then the cache is a critical resource and should be treated as such. -Matt Matthew Dillon [EMAIL PROTECTED] :Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mbufs, external storage, and MFREE
On Thu, 23 Sep 1999, Matthew Dillon wrote: :I have the following question: Let's say that I have a block of user :memory which I've mapped into the kernel, and would like to send on a :network socket. I'd like to simply grab an mbuf, point to the memory as :external storage, and queue it up for transmission. This would work fine, :except that when MFREE gets called, I have to write an deallocator that :maintains a table of all the different cases where I've done this, and do :a reverse mapping back to the original block, and then deal with sending :more, unmapping, etc. In other words, having MFREE call a deallocator :with just the data pointer and the size is inconvenient (actually, it :would make my scenario quite inefficient given the number of mappings back :to the original block that would have to be done). : :Am I missing another mechanism to handle this? Does it not come up enough :to matter? : :-Chris This is almost precisely the mechanism that the sendfile() system call uses. In that case it maps VMIO-backed data rather then user memory, but it is a very similar problem. There has been talk of implementing this type of mechanism not only for sockets, but for file read()/write() as well. In fact, John Dyson had delved into the issue with his vfs.ioopt stuff before he ran out of time. This is good--it seems a shame to copy things around all the time, though I'm not sure where the crossover is between copying and mapping into kernel space. (And, as a side note, what's up with struct buf? The thing is bloody huge if you only want to map user memory into kernel space :) The one problem with using direct VM page mappings is that currently there is no way for the socket to prevent the underlying data from being modified in the middle of a transmission. And, in the same respect for vfs.ioopt, no way to prevent the data the user ostensibly read() into his 'private' buffer from changing out from under the user if the underlying file is modified. Isn't this a case that the programmer has to handle? That is, if you mess with the data before it actually gets written, that's your problem. I take it that vfs.ioopt stuff is something like a temporary mmap() effect, since in the socket case once the data had been put in the buffer, I'd remove the kernel mapping and thus not be able to tweak it. For user memory, the only way such a mechanism can currently be implemented is by obtaining the underlying pages and busy'ing them for the duration of their use by the system, causing anyone trying to access them while the system operation is in progress to block. This can cause a potential problem with TCP in that the mbuf data you send to TCP sticks around until it gets pushed out the door *and* acknowledged by the other end. i.e. the data is not disposed of as when read() or write() returns but instead goes directly into TCP's outgoing queue. If the TCP connection hangs, the process may hang. I had been thinking about this in the context of async io operations, where its OK to have the operation not complete until the data has actually been ack'd by the remote end. With synchronous write() calls, this can be more problematic since it would significantly increase latency in cases where the original coder might not expect it. It might actually be nice to (optionally) have the same effect with async writes to disk, where the operation wouldn't actually complete until the data was known to be on the platter. -Chris To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mbufs, external storage, and MFREE
: vfs.ioopt, no way to prevent the data the user ostensibly read() into : his 'private' buffer from changing out from under the user if the : underlying file is modified. : :Isn't this a case that the programmer has to handle? That is, if you mess :with the data before it actually gets written, that's your problem. I :take it that vfs.ioopt stuff is something like a temporary mmap() effect, :since in the socket case once the data had been put in the buffer, I'd :remove the kernel mapping and thus not be able to tweak it. Yes and no. Sometimes changing data out from under the kernel can cause bad things to happen. For example, changing TCP data out from under the TCP protocol will screw up checksums, and making changes to a buffer undergoing DMA might screwup a device protocol crc or do something worse. The kernel thus needs to ensure that nothing the user does can screw it (the kernel) up. : For user memory, the only way such a mechanism can currently be : implemented is by obtaining the underlying pages and busy'ing them : for the duration of their use by the system, causing anyone trying to :... : : :I had been thinking about this in the context of async io operations, :where its OK to have the operation not complete until the data has :actually been ack'd by the remote end. With synchronous write() calls, :this can be more problematic since it would significantly increase latency :in cases where the original coder might not expect it. It might actually :be nice to (optionally) have the same effect with async writes to disk, :where the operation wouldn't actually complete until the data was known to :be on the platter. : :-Chris I think asynchronous I/O is the way to go with this too. An asynchronous API formally disallows changing data backing an I/O while the I/O is in progress (though it may not necessarily physically prevent the process from doing so). There is much less chance of the programmer making a mistake. Almost all of my embedded projects use asynchronous event-oriented I/O and simply eat the data out of the user process's memory space directly. Buffer copying is *really* expensive on a 68K cpu. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: using the ppp driver
Daniel Hilevich wrote: Hi, I'm trying to write a driver which sends ppp packets through the ethernet NIC. In order avoid implementing all the ppp protocol I want to use the existing ppp driver in FreeBSD. Is there a way to send to the ppp driver data and get it back covered with the appropriate ppp frame? The user-mode ppp system in FreeBSD can already do this, if you want to skip implementing ANY of ppp. ;^) -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC [EMAIL PROTECTED] http://softweyr.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mbufs, external storage, and MFREE
Matthew Dillon wrote... :I have the following question: Let's say that I have a block of user :memory which I've mapped into the kernel, and would like to send on a :network socket. I'd like to simply grab an mbuf, point to the memory as :external storage, and queue it up for transmission. This would work fine, :except that when MFREE gets called, I have to write an deallocator that :maintains a table of all the different cases where I've done this, and do :a reverse mapping back to the original block, and then deal with sending :more, unmapping, etc. In other words, having MFREE call a deallocator :with just the data pointer and the size is inconvenient (actually, it :would make my scenario quite inefficient given the number of mappings back :to the original block that would have to be done). : :Am I missing another mechanism to handle this? Does it not come up enough :to matter? : :-Chris This is almost precisely the mechanism that the sendfile() system call uses. In that case it maps VMIO-backed data rather then user memory, but it is a very similar problem. There has been talk of implementing this type of mechanism not only for sockets, but for file read()/write() as well. In fact, John Dyson had delved into the issue with his vfs.ioopt stuff before he ran out of time. The one problem with using direct VM page mappings is that currently there is no way for the socket to prevent the underlying data from being modified in the middle of a transmission. And, in the same respect for vfs.ioopt, no way to prevent the data the user ostensibly read() into his 'private' buffer from changing out from under the user if the underlying file is modified. How about marking the page copy-on-write? That way, if the user modifies the page while it is being transmitted, it'll just be copied, so the original data will be intact. Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: mbufs, external storage, and MFREE
: :How about marking the page copy-on-write? That way, if the user modifies :the page while it is being transmitted, it'll just be copied, so the :original data will be intact. : :Ken If it were a normal page we could, but the VM system currently cannot handle pages associated with vnodes themselves being marked copy-on-write. This is kinda hard to explain, but I will try. When a process maps a file MAP_PRIVATE, the VM object held by the process is not actually a vnode. Instead it is holding what is called a default object. The default object shadows the VM object representing the vnode. When a fault occurs, vm_fault knows to copy-on-write the page from the read-only backing VM object to the the front VM object and so from the point of view of the process, the page is copy-on-write. From the system's point of view, a new page has been added to the default VM object and no changes have been made to the vnode's VM object. When a process maps a file MAP_PRIVATE or MAP_SHARED and doesn't touch any of the pages, and some other process goes in and write()'s to the file via a descriptor, the process's view of the file will change because the pages associated with the underlying vnode have changed. The problem that occurs when we try to optimize read by mapping a vnode's page into a user address space is that some other process may go and modify the underlying file, modifying the data that the user process sees *after* the read() has returned. But the user process is expecting that data not to change because it thinks it has read() it into a private buffer when, in fact, the OS optimized the read by replacing the private memory with the file map. i.e. our problem is not so much the user process making a change to its buffer -- that case is handled by copy-on-write, but of another process writing directly to the vnode causing the data the first process read() to appear to change in its buffer. -Matt Matthew Dillon [EMAIL PROTECTED] :-- :Kenneth Merry :[EMAIL PROTECTED] : To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
A new package fetching utility, pkg_get
Hello, I have made this little program named pkg_get. It's decided to ease fetching and installing of FreeBSD (and why not Open- or NetBSD) binary packages, by making a database out of packages at a ftp server's packages directory. I had the idea from Debian Linux's atp-get utility, which my friend praised a lot. The source tarball can be fetched from the following URL: http://www.saunalahti.fi/~jsalomaa/pkg_get.tar.gz The program has already been tested by a few geeks (hope no-one gets hurt) at IRCNET, but of course I'd like some more testers, creative ideas how things should really be done (I'm not much of a C-guru nor UN*X guru) and flame about segmentation faults and uselessnes. And yes, I know what ports are. But fetching and installing packages is much faster than fetching the bigger source tarball and compiling it. And yes, I know about the remote fetching ability of pkg_add, but it is pretty poor in my opinion. I think we need something like this. I'll make a port out of this if you want, or if it's my lucky day, perhaps it should even be added to the base distribution (damn, I'm ambitious :-]). For those of you without enough time/interest to try it out, I add some "screen shots" (things typed by the user are surrounded by asterisks.) % ** ./pkg_get -h ** Syntax: ./pkg_get [options] [package] Available options: -h : Get help -i : Install package -U : Upgrade database -s : Server (if not set, first check PKGSERVER environment variable, then the server database (./database) -p : Port to use with ftp server, default 21 -d : FTP directory at server (if not set, first check PKGDIR environment variable, then appropriate entry in the server db -P : Use passive ftp -n : Username at ftp server, default anonymous -f : Only fetch package to CWD, don't install. -a : Automatically fetch any not previously installed or fetched dependancys of the package to be installed. The package-argument may be a regular expression. -- % ** ./pkg_get -i gtk ** Current server is the main distribution site, ftp.freebsd.org. Do you want to use it? (Y/N) ** n ** Enter hostname of your FreeBSD mirror site: nic.funet.fi No entry found in database for server 'nic.funet.fi'. Enter ftp directory at server (D for default, /pub/FreeBSD/packages/All) :** D ** URL to packages dir: ftp:[EMAIL PROTECTED]:21/%2Fpub/FreeBSD/packages/All No database entry for server nic.funet.fi, making one. Done. No package entry for server nic.funet.fi, making one. Making tcp connection to nic.funet.fi:21 Sending username anonymous...done. Sending password [EMAIL PROTECTED] Changing working dir to /pub/FreeBSD/packages/All...done. Sending PORT command...done. Listing ftp directory done. Terminating control connection...done. Matching packages: 1. asclock-gtk-beta-2.1.10.tgz 2. gtk---1.0.0.tgz 3. gtk-1.0.6.tgz 4. gtk-1.1.2.tgz clippety clip 17. gtksql-0.3.tgz 18. gtkstep-1.8.tgz 19. gtkyahoo-0.16.tgz 20. ja-gtkicq-0.60.tgz Press enter to continue. ** enter ** 21. ko-gtk-1.0.6.tgz Enter too big or non-numeral value to re-show choices. Select package [1-21]: ** 5 ** Receiving gtk-1.2.3.tgz (1474260 bytes): 100% 1474260 bytes transferred in 194.6 seconds (7.40 Kbytes/s) New dependancy: glib-1.2.3 Package gtk-1.2.3.tgz has dependancys (glib-1.2.3). Do you want to: (A)utomatically fetch them (F)etch this one only (Q)uit, already fetched packages will besaved in the packages directory (./packages) Enter your choice [AFQ]: ** a ** Receiving glib-1.2.3.tgz (166214 bytes): 100% 166214 bytes transferred in 21.9 seconds (7.40 Kbytes/s) /usr/sbin/pkg_add gtk-1.2.3.tgz - The last line meant that it echoed what it would do. Both of the files were fetches to the packages/ subdir, and the gettext dependancy was ignored, because it's already installed. This was the first run of the program, of course next time it will know more, for example the last used ftp server, the directory at it and stuff. So, there it was. May you live long and prosper. Jaakko Salomaa To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Idea: disposable memory
Kevin Day [EMAIL PROTECTED] wrote: I'd like a way to be able to specify that a region of malloc'ed data is 'disposable' and has approximately the same weight as disk cached data. As others have pointed out, this is almost (but not quite) the same as madvise(MADV_FREE). I don't think there is any way to achieve what Kevin wants with the current kernel. The simplest solution would appear to be a new madvise option which marks the relevant pages as `throw away instead of swapping out'. This is similar to MADV_FREE, but does not immediately free the page. It will also need to flag the page so the the user is warned if the page _is_ reused - the most logical way is by returning SIGBUS or SIGSEGV the first time the page is referenced after it has been released. The user process then just reads data normally. If it gets a signal, it knows it needs to re-create that page - this is presumably relatively trivial to do on a page-by-page basis (since you have no control over which page gets reused). The code to handle the `throw away' bit should be fairly trivial - it's just the existing MADV_FREE code, without the actual `free' bit. I'm less certain about being able to mark a page mapping so that a a page fault gets passed back to userland as well as mapping a zero- filled page to that location. Peter To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: A new package fetching utility, pkg_get
On 23 Sep 1999, Satoshi - Ports Wraith - Asami wrote: * % ** ./pkg_get -i gtk ** * Current server is the main distribution site, ftp.freebsd.org. * Do you want to use it? (Y/N) ** n ** : * Receiving glib-1.2.3.tgz (166214 bytes): 100% * 166214 bytes transferred in 21.9 seconds (7.40 Kbytes/s) * /usr/sbin/pkg_add gtk-1.2.3.tgz This looks great! Make it a port please! :) Sure, as soon as I get to write the manual page, and get the ftp upgrade code tested enough by unfortunate -hackers readers :-) May you live long and prosper. Jaakko Salomaa To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Out of swap handling and X lockups in 3.2R
Matthew Dillon wrote: What it all comes down to is a juxtaposition of what people believe is appropriate verses what people are actually willing to code up. I'm willing to code up my importance mechanism idea. The question is whether it's a good enough idea to throw into the tree. I think it's a good idea. It lets the admin introduce bias in the system to protect people/processes who are more likely to use huge amount of memory. Alas, taking the swap space into account in addition to RSS seems more important to me. But then, I'm happy with the way things are right now. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Thus, over the years my wife and I have physically diverged. While I have zoomed toward a crusty middle-age, she has instead clung doggedly to the sweet bloom of youth. Naturally I think this unfair. Yet, if it was the other way around, I confess I wouldn't be happy either." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: A new package fetching utility, pkg_get
I had the idea from Debian Linux's atp-get utility, which my friend praised a lot. The source tarball can be fetched from the following URL: http://www.saunalahti.fi/~jsalomaa/pkg_get.tar.gz This is quite interesting and I'm looking at it now. Just one quick question though - why did you "roll your own" ftp I/O handling instead of simply using fetch(3) or ftpio(3)? - Jordan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
[Fwd: Bsd Problem]
I was wondering if I could get help. Doug Madderom is a developer at
AGCS and has asked me to forward this to the FreeBSD newsgroup. Any
help is appreciated. Thanks.
--
Juan Lorenzana
AG Communication Systems
Phoenix, AZ
602-582-7442
[EMAIL PROTECTED]
Juan,
I have a driver problem with free bsd. Could you forward this to the
appropriate person and get the results back to me.
Doug Madderom
---
I wrote a character device driver using ioct as the method to pass data in a
structure to and from the device driver. If I do not include either a printf or
scanf in the application program that uses the driver the pointer the OS passes
to the ioctl in the driver is not set up right and the driver panics. What am I
doing wrong ? Do I need to do something special in the driver to get the ioctl
to link up properly and this is what the printf/scanf is doing in the
application? This is my first Bsd device driver I wrote several in Solaris 86
and Unixware and never seen this behavior.
Any help would be appreciated. Included is copy of source and test code that
fails.
Doug Madderom
[EMAIL PROTECTED]
623-582-7245
Here is the definition for the ioctl commands:
struct Ioctl_args {
int status;
intdevice;
unsigned int address;
unsigned char value;} ;
#define ALARMIO_READ_IOWR('a', 1, struct Ioctl_args)
#define ALARMIO_WRITE _IOWR('a', 2, struct Ioctl_args)
#define ALARMIO_TEST_IOWR('a', 3, int)
Attached is copy of progream that fails and device driver.
Program that fails ( works if first line in program is a printf)
--
#include stdio.h
#include errno.h
#include unistd.h
#include sys/wait.h
#include sys/types.h
#include sys/stat.h
#include sys/ioctl.h
#include fcntl.h
#include sys/alarmio.h
struct Ioctl_args requestData;
int main (void)
{
int result;
int fd;
requestData.address = 0x8152;
requestData.device = 0;
requestData.value = 0;
if ((fd = open("/dev/alarmio0",O_RDWR)) == -1)
{
perror("Can't open device /dev/alarmio0");
exit(1);
}
if ((result = ioctl(fd, ALARMIO_READ, requestData)) == -1)
{
if (errno == EIO)
{
printf ("Driver read access error EIO on fd = %d, error value
returned = %d\n",
fd, requestData.status);
}
else
{
printf ("Driver read access error\n");
}
}
else
{
printf ("Data read device %d address %x data %x \n",
requestData.device, requestData.address, requestData.value);
}
return 0;
}
Device driver:
--
/* alarmio driver for free BDS */
#include sys/param.h
#include sys/systm.h
#include sys/kernel.h /* SYSINIT stuff */
#include sys/conf.h /* cdevsw stuff */
#include sys/malloc.h /* malloc region definitions */
#include sys/signal.h
#include sys/signalvar.h
#include sys/filio.h /* FIOXXX */
#include machine/clock.h /* DELAY() */
#include machine/cpufunc.h /* inb and outb commands*/
#include sys/ioctl.h
#include sys/errno.h
#include /usr/src/sys/i386/isa/isa_device.h
/* Function prototypes (these should all be static ) */
static d_open_t alarmioopen;
static d_close_t alarmioclose;
static d_ioctl_t alarmioioctl;
static int alarmioprobe (struct isa_device *);
static int alarmioattach (struct isa_device *);
struct isa_driver alarmiodriver = {alarmioprobe, alarmioattach, "alarmio"};
#define CDEV_MAJOR 120
/*static struct cdevsw alarmio_cdevsw = {
alarmioopen,
alarmioclose,
NULL,
NULL,
alarmioioctl,
NULL,
NULL,
NULL,
NULL,
NULL,
NULL,
"alarmio",
NULL,
-1 }; */
static struct cdevsw alarmio_cdevsw =
{ alarmioopen, alarmioclose, noread, nowrite, /*31*/
alarmioioctl, nostop, nullreset, nodevtotty,/*alarmio */
noselect, nommap, NULL, "alarmio", NULL, -1 };
/* alarmio.c */
/*
alarmio driver to read and set alarms on radisys board
*/
#include sys/alarmio.h
static int debug=1;/* debug flag */
static unsigned char dallas1[0x4f]; /* temp arrary for dallas part*/
static unsigned char dallas2[0x4f]; /* temp arrary for dallas part*/
static int
alarmioprobe (struct isa_device *dev)
{
return 1;
}
static int
alarmioattach (struct isa_device *dev)
{
return 1;
}
static void
alarmio_drvinit(void *unused)
{
dev_t dev;
int unit;
if(debug) printf("alarmio_drvinit called \n");
dev = makedev(CDEV_MAJOR, 0);
cdevsw_add(dev, alarmio_cdevsw, NULL);
}
/*
* alarmio_open is called in response to the open(2) system call
*/
/*ARGSUSED*/
static int
alarmioopen(dev_t dev, int oflags, int devtype, struct proc *p)
{
int retval = 0;
if(debug) printf("OPEN:CALLED \n");
return (retval);
}
/*
close device
*/
static int
alarmioclose(dev_t dev, int fflag, int devtype, struct proc *p)
{
if(debug)
Re: buildfailure -current on Alpha?
As John Polstra wrote ... In article [EMAIL PROTECTED], Kenneth D. Merry [EMAIL PROTECTED] wrote: I take it you haven't rebuilt world in the past few months? There was a change that went in in June, I think, that requires that you reinstall ld-elf.so.1 manually before you can buildworld on an Alpha. Kinda annoying, I think. Annoying but unavoidable. There's no way to override the location of the dynamic linker at make world time, for security-related reasons. An important bugfix to the Alpha binutils required an updated dynamic linker to handle the new relocation type. The new dynamic linker was committed well before the binutils changes that required it. But still it can bite people who aren't tracking -current very closely. That's life in currentland. The ld-elf.so.1 trick was indeed the problem. My previous build was indeed somewhere in June. Thanks for your suggestion. -- | / o / / _ Arnhem, The Netherlands- Powered by FreeBSD - |/|/ / / /( (_) BulteWWW : http://www.tcja.nl http://www.freebsd.org To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
