Re: Shared-Memory for the Hurd?
On Fri, Dec 01, 2000 at 04:01:47AM +0100, Farid Hajji wrote: > 5.) fullsized translator, providing pseudo-filesystem: > > * each shm-segment is represented by a pseudo-file managed by a > fullsized translator settrans on e.g. /shm. The translator manages > vm_allocate()ed pages just like in 3.) and 4.), mapping them to > the pseudo-files [that is: shm-segment 34343 is mapped to /shm/34343]. Yes, this is a way to go. This could be a derivative of tmpfs, providing only a flat directory (Roland suggested /var/run/sysvshm) and only accepts certain file names coresponding to a key_t != 0 "with a funny behaviour about unlinking open files so the SHM_RMID behaviour is right." (Roland). Farid, this is an excellent medium sized project to dive into the Hurd inners. Are you still interested in implementing this? > Notes: > * shm-segments are visible as plain [pseudo-] files and could also > be mmap()-ed, or read()/write()... from other processes that have > enough permissions. Yes, mmap() would be ideal. > 5. same as in 4, but much more harder to implement, since there >is no good template/hello for a full filesystem translator in >the Hurd right now (besides full-blown ext2, ufs). >-> What about /procfs translator? Someone wrote such a beast, > but it is not in CVS. Where to get it please? Well, tmpfs should be a good start. It isn't functional currently and not tested, but there are only small changes necessary to make it work (coming soon). tmpfs uses virtual memory to store the files. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
On Mon, Dec 04, 2000 at 04:30:30AM +0100, Farid Hajji wrote: > Wow, libtreefs is being phased out? Good to know! I didn't say that. > libnetfs is exactly the library I was looking for. Then simply don't worry about libtreefs :) > It seems like I have now all I need to write that shm-translator > and a simple shm*() library that uses it. Instead writing a library, add them to libc, where they belong. The programs that use shm* are compiled against it, and filling out the stubs will them magically use the implementation without recompilation. Of course, if you think a libshm is useful for extra hurdish features ala libps or something... Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
> I don't know why Neal says libtrivfs doesn't fit the job, but I never tried > to provide a treefs with libtrivfs. There is also libnetfs, which is also > used by non-networked filesystems like usermux and hostmux, so maybe there > is some value in it. [snip] > Sounds good. As I said, with libtreefs, you are reviving a code four or five > years old. With libdiskfs, you are facing the omplexity of libstore and the > paging interface... Wow, libtreefs is being phased out? Good to know! libnetfs is exactly the library I was looking for. It provides the callbacks that are needed to write a regular fs-translator. nfs is also a good template to get started... Good example code for handling descriptor/port stuff is in hurd/init/init.c (at least that is what I needed to understand it) and file_set_size() is the way to get the shm-size request to shm-translator. It seems like I have now all I need to write that shm-translator and a simple shm*() library that uses it. A related task would be to write a memfs-translator which uses virtual memory for a memory- based filesystem a la tmpfs, but that is for later. Thanks for your help! -Farid. -- Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555 Broicherdorfstr. 83, D-41564 Kaarst, Germany | [EMAIL PROTECTED] - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - Murphy's Law fails only when you try to demonstrate it, and thus succeeds. ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
From: Marcus Brinkmann <[EMAIL PROTECTED]> Subject: Re: Shared-Memory for the Hurd? Date: Fri, 1 Dec 2000 13:06:11 +0100 > I don't know why Neal says libtrivfs doesn't fit the job, but I never tried > to provide a treefs with libtrivfs. There is also libnetfs, which is also > used by non-networked filesystems like usermux and hostmux, so maybe there > is some value in it. Yep, if you want to write a non-trivial translator which doesn't use stores, libnetfs is a good choice. I even suspect that libtreefs will be removed, because libnetfs does the equivallent. Okuji ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
From: Neal H Walfield <[EMAIL PROTECTED]> Subject: Re: Shared-Memory for the Hurd? Date: Thu, 30 Nov 2000 23:56:49 -0500 > * Getting a task's task port would likely have to be done by > requesting it from the task itself (Correct me if I am wrong, > however, I do not think that it can be requested from anyone except > the task itself; check /include/hurd/process_request.defs). The parent task is the exception. A task has send rights to its children. Okuji ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
On Fri, Dec 01, 2000 at 06:42:15AM +0100, Farid Hajji wrote: > > To conclude, the shm server would live in, e.g. /servers/shm. > > Inheriting from libtreefs or libdiskfs seems like a good idea > > (enabling you to expose the maps in that directory). Using trivfs > > is a bad idea (I did it in my procfs: > > ftp://walfield.org/pub/people/neal/hurd/procfs; don't worry, it > > will not be going in CVS). > libtreefs was what I had in mind for the "fullsized" shm-translator. > About libdiskfs, I'm not sure yet (read: I must read more sources yet). > Basically, I don't need a filesystem-backend (say ext2, ufs, ...), > just a tree (directory) structure to hold, say /servers/shm/0, > /servers/shm/1, ... libdiskfs would be useful for a memory-based > filesystem, which is not exactly what I have in mind. But again > I'll have to read more about libdiskfs before. libdiskfs assumes a backing store (provided by storeio). If you don't have this, libdiskfs probably isn't so good. libtreefs is designed to fill the gap between libtrivfs and libdiskfs, but it is probably not finished. At least it is not build currently. There isn't even a ChangeLog. I don't know why Neal says libtrivfs doesn't fit the job, but I never tried to provide a treefs with libtrivfs. There is also libnetfs, which is also used by non-networked filesystems like usermux and hostmux, so maybe there is some value in it. For a very simple directory serving translator, look at /dev/fd/, which is served by trans/magic.c > Now I must first figure out how to use libtreefs/libdiskfs by > writing a translator with dummy callbacks. If I get there, the > rest will be much more easier. Sounds good. As I said, with libtreefs, you are reviving a code four or five years old. With libdiskfs, you are facing the omplexity of libstore and the paging interface... Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
> libtreefs was what I had in mind for the "fullsized" shm-translator. > About libdiskfs, I'm not sure yet (read: I must read more sources yet). > Basically, I don't need a filesystem-backend (say ext2, ufs, ...), > just a tree (directory) structure to hold, say /servers/shm/0, > /servers/shm/1, ... libdiskfs would be useful for a memory-based > filesystem, which is not exactly what I have in mind. But again > I'll have to read more about libdiskfs before. I had the same thing in mind for each: libtreefs for the memory based implementation and libdiskfs for your disk based implementation. I do not think that libdiskfs would fit a memory based implementation at all. > Now I must first figure out how to use libtreefs/libdiskfs by > writing a translator with dummy callbacks. If I get there, the > rest will be much more easier. It's always easier when you know how ;) Best regards, -Neal -- Neal H Walfield University of Massachusetts at Lowell [EMAIL PROTECTED] or [EMAIL PROTECTED] ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
Thanks for your commments. They helped greatly clarify some ideas! > So, how does one do non-io operations? The function > file_name_lookup (/include/hurd.h) returns a file_t which happens > to be a perfectly valid port that any type of message can be sent > to (not just io messages; of course, unknown messages will fail, > however, you get the point). This is what I had in mind. So I'll try to send the task control port directly to this file_t and see if the translator task gets it. I don't know yet if the message will be grokked by mach itself (mach inserts the port right transparently in the receiver's task) or if some special handling needs to be done. > To conclude, the shm server would live in, e.g. /servers/shm. > Inheriting from libtreefs or libdiskfs seems like a good idea > (enabling you to expose the maps in that directory). Using trivfs > is a bad idea (I did it in my procfs: > ftp://walfield.org/pub/people/neal/hurd/procfs; don't worry, it > will not be going in CVS). libtreefs was what I had in mind for the "fullsized" shm-translator. About libdiskfs, I'm not sure yet (read: I must read more sources yet). Basically, I don't need a filesystem-backend (say ext2, ufs, ...), just a tree (directory) structure to hold, say /servers/shm/0, /servers/shm/1, ... libdiskfs would be useful for a memory-based filesystem, which is not exactly what I have in mind. But again I'll have to read more about libdiskfs before. > * Getting a task's task port would likely have to be done by > requesting it from the task itself (Correct me if I am wrong, > however, I do not think that it can be requested from anyone except > the task itself; check /include/hurd/process_request.defs). > However, shmat, for instance, is wrapped in a libc function, then > this would be quite transparent. Sure, the shmat()/shmget() caller will have to provide this port itself one way or the other. The easiest way will be for shmget() to send the result of mach_task_port() to the file_t of the shm-translator. Having the translator ask for the task port of its clients (e.g. pid2task()) is not so good. > * Servers normally use many demuxers. That is to say that they may > implement their own interface and use others from libraries are the > same time. For instance: [snip] Good examples! That will help too. Now I must first figure out how to use libtreefs/libdiskfs by writing a translator with dummy callbacks. If I get there, the rest will be much more easier. Thanks, -Farid. -- Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555 Broicherdorfstr. 83, D-41564 Kaarst, Germany | [EMAIL PROTECTED] - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - Murphy's Law fails only when you try to demonstrate it, and thus succeeds. ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
Al right, a lot this is stuff that I am not going to comment on as there
are much better people do do this (e.g. Roland). However, I feel can
make the following comments:
* We do not have a netname server. The hurd's equivalent is the file
system: this is where all ports are advertised. In fact, thinking
about the file system as a file system is inherently flawed: it is
much better to consider it a ``port system.''
Having said that, what's a file? Nothing more than a port managed
by a server waiting for specific messages such as open, seek etc.
(see /include/hurd/io.defs).
So, how does one do non-io operations? The function
file_name_lookup (/include/hurd.h) returns a file_t which happens
to be a perfectly valid port that any type of message can be sent
to (not just io messages; of course, unknown messages will fail,
however, you get the point).
Thus, one could write a, e.g. ``bank server,'' that uses a custom
interface (i.e. a MiG defs file) and settrans it into the file
system. Anyone who has a client that can talk the protocol also
has the ability to use the server.
To conclude, the shm server would live in, e.g. /servers/shm.
Inheriting from libtreefs or libdiskfs seems like a good idea
(enabling you to expose the maps in that directory). Using trivfs
is a bad idea (I did it in my procfs:
ftp://walfield.org/pub/people/neal/hurd/procfs; don't worry, it
will not be going in CVS).
* Getting a task's task port would likely have to be done by
requesting it from the task itself (Correct me if I am wrong,
however, I do not think that it can be requested from anyone except
the task itself; check /include/hurd/process_request.defs).
However, shmat, for instance, is wrapped in a libc function, then
this would be quite transparent.
* Servers normally use many demuxers. That is to say that they may
implement their own interface and use others from libraries are the
same time. For instance:
int
demuxer (mach_msg_header_t *inp,
mach_msg_header_t *outp)
{
int my_server (mach_msg_header_t *, mach_msg_header_t *);
int my_fsys_server (mach_msg_header_t *, mach_msg_header_t *);
int my_io_server (mach_msg_header_t *, mach_msg_header_t *);
return (my_server (inp, outp)
|| my_fsys_demuxer (inp, outp)
|| my_io_demuxer (inp, outp)
|| ports_notify_server (inp, outp)
|| ports_interrupt_server (inp, outp))
}
or using trivfs to handle all of the fsys and io details:
int
demuxer (mach_msg_header_t *inp,
mach_msg_header_t *outp)
{
int trivfs_demuxer (mach_msg_header_t *, mach_msg_header_t *);
int my_server (mach_msg_header_t *, mach_msg_header_t *);
return (my_server (inp, outp)
|| trivfs_demuxer (inp, outp))
}
Here, you are using trivfs and your own interface.
Hope this helps.
Good luck!
-Neal
--
Neal H Walfield
University of Massachusetts at Lowell
[EMAIL PROTECTED] or [EMAIL PROTECTED]
___
Help-hurd mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
[Sorry for excessive verbosity here...]
I'm trying to figure out how to implement shared memory for the Hurd.
I already have experience with Lites/Mach, but I'm still very new to
Hurd-programming, so please be patient with silly questions. One motivation
for doing this is to get familiar with Hurd internals.
Many approaches come to mind (ordered in increasing hurd-knowledge level):
1. mmap()-based shm*() library implementation.
2. using libpager and a special external pager.
3. using a shm-server that vm_map()s own memory objects to shm-clients.
4. using a trivfs-translator that will vm_map() own memory on requests
of shm-clients.
5. using a fullsized translator that will vm_map() own memory
to shm-clients, as well as providing pseudo-files mapping
shm-segment contents.
1.) mmap()-based shm*() library:
* Each memory segment with key KEY will be mmap()ped to a file
/sysv/shm/{KEY} of specified length and permissions.
shmget() would create and zero-fill the file if needed, open
it and return a descriptor to the caller. Flags and permissions
will map nearly 1:1 to file permissions and open() O_* flags.
* shmat() will simply mmap() to the file(-descriptor) returned
by shmget(). Since mmap() uses vm_map() internally, this will
have the file contents appear in the address-space of the
caller in obvious ways.
* shmdt() will simply munmap() the address mmap()-ped before,
again in obvious ways.
* shmctl() will e.g. unlink the file or do other stuff.
Notes:
* This approach is so generic, that it was almost certainly
being written before on plain old Unix systems that provide
mmap() syscalls but no shm*() interface. It can be reimplemented
on the Hurd as well, if nobody knows of an existing free library.
* The implementation will provide shm memory that is persistent
across reboots (!), which is even more than what classical
SysV shm requires.
* Accessing the files directly (read, seek(), write()) is yet
another way to change the shm segments and can't be that bad.
* ipcs and ipcrm shm functionality is trivial to implement.
* shm-segments being mmap()ed to files means that they can
even persist across reboots, if the files are backed by a
disk-based store (e.g. in the ext2fs filesystem).
* it can be costly to create files through FSYS translator, both
in time and space.
2.) libpager with special external pager.
* Is this really feasible? What happens when the reference count
of a memory object drops to zero? Mach notifies the pager that
the MO is no longer valid and the pager is free to reuse the
swap-space for other purposes. The MO was already dropped by
Mach, so there is no chance in keeping it, right?
-> special pagers wouldn't be of much help here, unless there
is a way to tell mach to keep memory objects active (e.g.
by transferring their port-rights to the special pager?),
vm_wire() and so on...
Any ideas?
3.) shm-server vm_map()ping own memory to requesting clients
* a shm-server holds vm_allocate()ed pages and vm_map()s them
to requesting shm-clients. This way, the memory object won't
be reclaimed by mach if no shm-client is present. They (the
memory objects) are still being held by the shm-server, so
they persist as sysv shm semantics require.
* a shm-client calls shmget() to open/create a shm-segment.
shmget() would send the shm-client a send-right to its
task control port to the shm-server. The shm-server vm_allocates()
a memory object big enough to satisfy the client, saves a
pointer/port to it as well as the task control port of the
client in an internal table and returns a reply [code | port?]
back to shm-client (back to shmget()).
* the shm-client calls shmat(), specifying an address where to
map the segment. shmat() sends a message to shm-server,
specifying the requested address (as well as the number/port
of the returned shmget() value). shm-server vm_map()s the
page into the address space of the shm-client, using shm-client's
previously sent-in task control port for this.
* other shm-clients could also use shmget()/shmat() to access the
same memory object in the shm-server in the same way.
* shm-client calls shmdt() sending the shm-server the address
of the shm-segment. shm-server vm_unmap()s the pages from
the shm-client (using their task control port) but keeps
the pages in its own task!
[I use here vm_unmap() for clarity. In mach, it is of course
a call to vm_deallocate(target_task, address, size), which
will affect only one task].
* some shm-client calls shmctl() to remove the shm-segment.
shmctl() contacts shm-server and sends a message specifying
the request. shm-server vm_deallocate()s the segment on its
side, freeing memory.
Notes:
* Since the Hurd doesn't provi
Re: Shared-Memory for the Hurd?
On Thu, Nov 30, 2000 at 03:37:31PM -0500, Neal H Walfield wrote: > > > 2. how to use stores > > The source (and it is not easy). If you write a filesystem, and use ext2fs as a base, you don't need to care a lot about stores. The main difficulty is the external pager interface, and the buffer return semantics, which are not described in the hurd but in Mach manuals. This is my experience at least. Thanks, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
> > 1. how to write (non-trivfs) translators The soruce (look at say ext2 and use that as a template for new ones). > > 2. how to use stores The source (and it is not easy). > > 3. how the port rights are passed from task to task The CMU documentation. But this is actually quite easy, there is two ways: a task inherits ports from its parent (the bootstrap port, the root, etc). Second, port rights are passed along inside of messages: I send you a message. It contains data and a send right to a port someone is listening on (i.e. not necessarily me). Then you reply to the indicated port. > > 4. how to program against libps/proc-server This is actually _very_ well documented in /include/ps.h. You could also take a look at procfs. -Neal -- Neal H Walfield University of Massachusetts at Lowell [EMAIL PROTECTED] or [EMAIL PROTECTED] ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
Re: Shared-Memory for the Hurd?
On Thu, Nov 30, 2000 at 05:14:13AM +0100, Farid Hajji wrote: > Who is working on System V IPC (shm*(), sem*(), msg*()) for the Hurd > right now? Nobody, you'd be the first :) > P.S.: Could you _please_ update/finish hurd/doc/hurd.texi? Please send a patch when you find inaccuracies. > It is horribly uncomplete Writing new documentation is not exactly the same as updating :) Nobody writes docs currently. in many critical places. Has someone > written hurd programming howtos/tutorials about > 1. how to write (non-trivfs) translators > 2. how to use stores > 3. how the port rights are passed from task to task > 4. how to program against libps/proc-server > in addition to hurd.texi? Even if it's not official... I don't know of such a thing. Sorry, Marcus -- `Rhubarb is no Egyptian god.' Debian http://www.debian.org [EMAIL PROTECTED] Marcus Brinkmann GNUhttp://www.gnu.org[EMAIL PROTECTED] [EMAIL PROTECTED] http://www.marcus-brinkmann.de ___ Help-hurd mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/help-hurd
