Re: [PATCH] libdiskfs: fix reference counting of peropen objects

[Neal H. Walfield]

At Fri, 29 Aug 2014 12:50:28 +0200,
Justus Winter wrote:
> 
> Quoting Neal H. Walfield (2014-08-29 11:55:07)
> > At Fri, 29 Aug 2014 11:33:44 +0200,
> > Justus Winter wrote:
> > > Previously, peropen objects were created with a reference count of
> > > zero.  Therefore, if diskfs_create_protid fails, passing such an
> > > object to diskfs_release_peropen would lead to a reference count
> > > underflow.
> > > 
> > > * libdiskfs/peropen-make.c (diskfs_peropen_make): Initialize reference
> > > count to one.
> > > * libdiskfs/protid-make.c (diskfs_start_protid): And consume this
> > > reference on success.  Update comment.
> > 
> > Are references really consumed?  (I think 'released' is the right
> > word.)
> 
> Yes, they are.  If diskfs_create_protid succeeds, the reference is in
> the belly of the newly created protid object.
> 
> Also, this terminology is used elsewhere in the Hurd.  Functions that
> contain the term 'release' in their name decrement a reference count
> (e.g. diskfs_release_peropen).  On the other hand, 'consume' indicates
> that a reference is absorbed, e.g. in exec/exec.c (use1),
> pfinet/socket.c (make_sock_user), and trans/fakeroot.c (new_node).

Ok.  I wasn't clear on the semantics.  I find "assume ownership of the
reference" clearer than "consume the reference", but if the Hurd is
already using this terminology, then I won't further dispute it.

Neal

Re: [PATCH] libdiskfs: fix reference counting of peropen objects

[Neal H. Walfield]

At Fri, 29 Aug 2014 11:33:44 +0200,
Justus Winter wrote:
> Previously, peropen objects were created with a reference count of
> zero.  Therefore, if diskfs_create_protid fails, passing such an
> object to diskfs_release_peropen would lead to a reference count
> underflow.
> 
> * libdiskfs/peropen-make.c (diskfs_peropen_make): Initialize reference
> count to one.
> * libdiskfs/protid-make.c (diskfs_start_protid): And consume this
> reference on success.  Update comment.

Are references really consumed?  (I think 'released' is the right
word.)

Neal

Re: [PATCH] Support invoking the debugger over the serial console

[Neal H. Walfield]

At Thu, 21 Aug 2014 13:16:01 +0200,
Samuel Thibault wrote:
> 
> Hello,
> 
> Thanks for sharing, I've had this kind of patch in my tree for a long
> with another, easier to parse, keypress, but this implementation seems
> nice as it keeps it ctrl-alt-d.
> 
> Justus Winter, le Thu 21 Aug 2014 12:14:50 +0200, a écrit :
> > if (tp->t_state&TS_ISOPEN) {
> > +   int escape = 0;
> > while ((line = inb(LINE_STAT(addr))) & iDR) {
> > c = inb(TXRX(addr));
> > -   ttyinput(c, tp);
> > +
> > +   if (c == 0x1b) {
> > +   escape = 1;
> > +   continue;
> > +   }
> 
> Mmm, but this means that an escape alone will be lost, which is a real
> problem for e.g. using vim :) Can't escape rather be an static array
> indexed by unit number?
> 
> > +#if MACH_KDB
> > +   if (escape && c == 4)
> 
> I like using 'C'-'@' instead of the magic-looking 4 :)

Even better:

  'D' - ('A' - 1)

Neal

Re: [PATCH 5/7] include: add lock-less reference counting primitives

[Neal H. Walfield]

At Tue, 13 May 2014 21:02:54 +0200,
Justus Winter wrote:
> +  assert (r != UINT_MAX);

How about 'assert(r != UINT_MAX || !"refcount underflowed!")'.  Since
assert (r != UINT_MAX) requires understanding the use of an unsigned
int.

Re: [PATCH 4/7] libihash: use fast binary scaling to determine the load

[Neal H. Walfield]

At Tue, 13 May 2014 21:02:53 +0200,
Justus Winter wrote:
> diff --git a/libihash/ihash.c b/libihash/ihash.c
> index d628d75..f529a17 100644
> --- a/libihash/ihash.c
> +++ b/libihash/ihash.c
>if (ht->size)
>  {
> -  /* Only fill the hash table up to its maximum load factor.  */
> -  if ((ht->nr_items * 100) >> __builtin_ctz (ht->size) <= ht->max_load)
> +  /* Only fill the hash table up to its maximum load factor given
> + as "binary percent", where 128b% corresponds to 100%.  As the
> + size is always a power of two, and 128 is also, the quotient
> + of both is also a power of two.  Therefore, we can use bit
> + shifts to scale the number of items.

This comment describing binary percent needs to be in the interface.

> diff --git a/libihash/ihash.h b/libihash/ihash.h
> index 8829e51..809166f 100644
> --- a/libihash/ihash.h
> +++ b/libihash/ihash.h
> @@ -79,7 +79,7 @@ struct hurd_ihash
>/* The offset of the location pointer from the hash value.  */
>intptr_t locp_offset;
>  
> -  /* The maximum load factor in percent.  */
> +  /* The maximum load factor in binary percent.  */
>unsigned int max_load;

Either here.

>  
> -/* Set the maximum load factor in percent to MAX_LOAD, which should be
> -   between 50 and 100.  The default is HURD_IHASH_MAX_LOAD_DEFAULT.
> -   New elements are only added to the hash table while the number of
> -   hashed elements is that much percent of the total size of the hash
> -   table.  If more elements are added, the hash table is first
> -   expanded and reorganized.  A MAX_LOAD of 100 will always fill the
> -   whole table before enlarging it, but note that this will increase
> -   the cost of operations significantly when the table is almost full.
> +/* Set the maximum load factor in binary percent to MAX_LOAD, which
> +   should be between 64 and 128.  The default is
> +   HURD_IHASH_MAX_LOAD_DEFAULT.  New elements are only added to the
> +   hash table while the number of hashed elements is that much binary
> +   percent of the total size of the hash table.  If more elements are
> +   added, the hash table is first expanded and reorganized.  A
> +   MAX_LOAD of 128 will always fill the whole table before enlarging
> +   it, but note that this will increase the cost of operations
> +   significantly when the table is almost full.
>  
> If the value is set to a smaller value than the current load
> factor, the next reorganization will happen when a new item is

Or here.

Neal

Re: [PATCH 03/11] include: add lock-less reference counting primitives

[Neal H. Walfield]

At Tue, 13 May 2014 13:47:51 +0200,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Tue 13 May 2014 13:44:37 +0200, a écrit :
> > At Tue, 13 May 2014 12:52:03 +0200,
> > Justus Winter wrote:
> > > Quoting Neal H. Walfield (2014-05-13 09:44:21)
> > > > At Mon, 12 May 2014 12:05:41 +0200,
> > > > Justus Winter wrote:
> > > > > +/* Decrement REF.  Return the result of the operation.  This function
> > > > > +   uses atomic operations.  It is not required to serialize calls to
> > > > > +   this function.  */
> > > > > +static inline unsigned int
> > > > > +refcount_deref (refcount_t *ref)
> > > > > +{
> > > > > +  return __atomic_sub_fetch (ref, 1, __ATOMIC_RELAXED);
> > > > > +}
> > > > 
> > > > How about adding assert(*ref >= 0)?
> > > 
> > > It is there, you just can't see it because I optimized it away (as gcc
> > > would, as refcount_t is unsigned ;).
> > 
> > I meant assert(*ref > 0), sorry.
> 
> Well, I'd rather check that the result didn't underflow, otherwise you
> may miss it in some rare conditions.

Good point.  The assert that I proposed would introduce a TOCTTOU bug.

:) Neal
 

Re: [PATCH 03/11] include: add lock-less reference counting primitives

[Neal H. Walfield]

At Tue, 13 May 2014 12:52:03 +0200,
Justus Winter wrote:
> 
> Quoting Neal H. Walfield (2014-05-13 09:44:21)
> > At Mon, 12 May 2014 12:05:41 +0200,
> > Justus Winter wrote:
> > > +/* Decrement REF.  Return the result of the operation.  This function
> > > +   uses atomic operations.  It is not required to serialize calls to
> > > +   this function.  */
> > > +static inline unsigned int
> > > +refcount_deref (refcount_t *ref)
> > > +{
> > > +  return __atomic_sub_fetch (ref, 1, __ATOMIC_RELAXED);
> > > +}
> > 
> > How about adding assert(*ref >= 0)?
> 
> It is there, you just can't see it because I optimized it away (as gcc
> would, as refcount_t is unsigned ;).

I meant assert(*ref > 0), sorry.

Neal

Re: [PATCH 07/11] libihash: use an integer hash function on the keys

[Neal H. Walfield]

At Mon, 12 May 2014 12:05:45 +0200,
Justus Winter wrote:
> 
> Use an integer hash function to derive the index from the key.  This
> should reduce the number of collisions.
> 
> * libihash/ihash.c (hash_int32): New function.
> (find_index): Use hash_int32 on the key to derive the index.
> (add_one): Likewise.
> ---
>  libihash/ihash.c | 23 +--
>  1 file changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/libihash/ihash.c b/libihash/ihash.c
> index d670fee..1de4c35 100644
> --- a/libihash/ihash.c
> +++ b/libihash/ihash.c
> @@ -81,6 +81,25 @@ static const unsigned int ihash_nsizes = (sizeof 
> ihash_sizes
> / sizeof ihash_sizes[0]);
>  
>  
> +/* Integer hashing follows Thomas Wang's paper about his 32/64-bits
> +   mix functions :
> +   -  http://www.concentric.net/~Ttwang/tech/inthash.htm  */

This URL is not valid ("ttwang.cnc.net Not Available. The domain
ttwang.cnc.net which you are trying to access is currently
unavailable...").  Do you have a more stable URL?  At least mention
the name of the paper.

:) Neal

Re: [PATCH 03/11] include: add lock-less reference counting primitives

[Neal H. Walfield]

At Mon, 12 May 2014 12:05:41 +0200,
Justus Winter wrote:
> +/* Decrement REF.  Return the result of the operation.  This function
> +   uses atomic operations.  It is not required to serialize calls to
> +   this function.  */
> +static inline unsigned int
> +refcount_deref (refcount_t *ref)
> +{
> +  return __atomic_sub_fetch (ref, 1, __ATOMIC_RELAXED);
> +}

How about adding assert(*ref >= 0)?

:) Neal

Re: [PATCH 04/11] doc: document mach_port_set_protected_payload

[Neal H. Walfield]

At Mon, 17 Feb 2014 23:35:12 +0100,
Richard Braun wrote:
> 
> On Mon, Feb 17, 2014 at 06:20:54PM +0100, Justus Winter wrote:
> > +@deftypefun kern_return_t mach_port_set_protected_payload (@w{ipc_space_t 
> > @var{task}}, @w{mach_port_t @var{name}}, @w{unsigned long @var{payload}})
> > +The function @code{mach_port_set_protected_payload} sets the protected
> > +payload to @var{payload}.  If @var{payload} is non-zero, the
> > +@code{msgh_protected_payload} field will be set to @var{payload} if a
> > +message is delivered to @var{name}.
> 
> If I'm right, this also means switching back from the protected payload
> is done by calling this RPC with a payload of 0. It could be worth
> emphasizing that 0 is an invalid value for a protected payload.

These semantics are unfortunate.  0 should be valid.  Add another IPC
to clear the pp or add a parameter to this IPC called, say, set, which
if true sets the pp to the provided value and otherwise clears the pp.

Neal

Re: [RFC] Enable thread termination

[Neal H. Walfield]

At Mon, 18 Nov 2013 16:14:57 +0100,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Mon 18 Nov 2013 13:41:46 +0100, a écrit :
> > At Mon, 18 Nov 2013 12:31:42 +0100,
> > Richard Braun wrote:
> > > The threading library is a
> > > low level component and should act as closely as its users expect it to.
> > 
> > I don't agree.  On Linux, creating a thread is a pretty much a single
> > system call.
> 
> Err, not really. There are a *lot* of things that libpthread fixes up,
> be it TLS, robust mutex lists, reseting cancellability state, tracing,
> etc.

Ok, fair enough.  My assumptions/recollections about how NPTL work are
probably incorrect.  I haven't looked at that code in a long time.

Neal

Re: [RFC] Enable thread termination

[Neal H. Walfield]

At Mon, 18 Nov 2013 12:31:42 +0100,
Richard Braun wrote:
> 
> On Mon, Nov 18, 2013 at 11:33:55AM +0100, Neal H. Walfield wrote:
> > Richard Braun wrote:
> > > The current state is to never terminate threads, on the assumption that
> > > they can't both terminate and release their stack on their own. Such
> > > resources are recycled by the threading library.  This patch makes use
> > > of a new GNU Mach specific call (thread_terminate_release [1]) so that
> > > threads do terminate themselves and release their stack, and in addition
> > > their last self reference, and their reply port.
> > 
> > Eliminating bugs should be our first priority.  However, recycling
> > threads is a good idea, particularly when the application sees threads
> > as lightweight resources.  Ideally, there is some mechanism that
> > identifies thread churn and sizes the reserve thread pool
> > appropriately (e.g., the maximum of the maximum number of live threads
> > during each minute in the past 5 minutes).  Note: this isn't a reason
> > not to apply the patch, but should perhaps be noted as possible future
> > work.
> 
> Personally, I consider this to be the responsibility of another
> component, such as a work queue library. The threading library is a
> low level component and should act as closely as its users expect it to.

I don't agree.  On Linux, creating a thread is a pretty much a single
system call.  On a micro-kernel based system, there is a lot more work
to do.  This is a relatively straightforward optimization that doesn't
changed the semantics of the operations.

Neal

Re: [RFC] Enable thread termination

[Neal H. Walfield]

At Sun, 17 Nov 2013 17:13:23 +0100,
Richard Braun wrote:
> The current state is to never terminate threads, on the assumption that
> they can't both terminate and release their stack on their own. Such
> resources are recycled by the threading library.  This patch makes use
> of a new GNU Mach specific call (thread_terminate_release [1]) so that
> threads do terminate themselves and release their stack, and in addition
> their last self reference, and their reply port.

Eliminating bugs should be our first priority.  However, recycling
threads is a good idea, particularly when the application sees threads
as lightweight resources.  Ideally, there is some mechanism that
identifies thread churn and sizes the reserve thread pool
appropriately (e.g., the maximum of the maximum number of live threads
during each minute in the past 5 minutes).  Note: this isn't a reason
not to apply the patch, but should perhaps be noted as possible future
work.

> To avoid right leaks, the sigstate maintains its own reference on its
> associated thread. Reusing the self reference created from libpthread
> isn't possible because the sigstate of the main thread is created
> before (actually during, because of _hurd_critical_section_lock being
> called early) libpthread initialization, when there is no self reference
> for the main thread yet. This relies on GNU Mach always using the same
> name for already existing send rights (i.e. mach_task_self always
> returns the same name as long as it's not destroyed).

This assume is reasonable: it is true and used often.

> Joining/detaching relies on pthread structures maintaining a reference
> counter. Joinable threads have 2 references, detached threads only one
> (a self reference). This replaces the previous mechanism that forced
> thread to halt before checking their state.
> 
> A few static variables were added to record that initialization had been
> done, instead of merely looking at the current number of threads.
> 
> Two resources are still recycled: the internal pthread structure
> (because it makes ID allocation easy), and thread local storage (TLS)
> because that's where the reply port is stored, and it didn't seem
> convenient to call _dl_deallocate_tls in sysdeps code.

Conceptually, your approach seems fine, but I haven't reviewed the
patch in detail.

Neal

Re: [RFC] cleanup of #includes

[Neal H. Walfield]

At Thu, 14 Nov 2013 14:38:04 +0100,
Svante Signell wrote:
> 
> On Thu, 2013-11-14 at 14:26 +0100, Neal H. Walfield wrote:
> > At Thu, 14 Nov 2013 13:56:30 +0100,
> > Svante Signell wrote:
> > > > 1. Do you consider the comment describing which symbols are pulled
> > > >from the header as noise or worthwile information?
> > > 
> > > Very useful!
> > 
> > No.  This information gets out of date.
> 
> Not if it is automatically updated regularly. I thought this was the
> intention from Justus.

The churn in the comments will only be noise in the commit history.
Identifying and removing unused headers, however, is fine.

Re: [RFC] cleanup of #includes

[Neal H. Walfield]

At Thu, 14 Nov 2013 13:56:30 +0100,
Svante Signell wrote:
> > 1. Do you consider the comment describing which symbols are pulled
> >from the header as noise or worthwile information?
> 
> Very useful!

No.  This information gets out of date.

Neal

Re: [PATCH] libports: implement lockless management of threads

[Neal H. Walfield]

Yes, this is what I was thinking of.

I recall there being type defs for appropriate atomic types.  If that
is still the recommended approach, please update your patch
appropriately.

The most important thing, however, is ensuring that the semantics are
preserved.  That is, was the use of the values also protected by the
lock?  Does moving to atomic updates introduce a possible
inconsistency?  I haven't looked at the code.  Before this is checked
in, however, someone should.

Thanks,

Neal

Re: [PATCH] libports: fix the thread counts in case the thread creation fails

[Neal H. Walfield]

At Sun, 10 Nov 2013 11:54:20 +0100,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Sun 10 Nov 2013 11:38:04 +0100, a écrit :
> > At Sat, 9 Nov 2013 18:21:51 +0100,
> > Samuel Thibault wrote:
> > > > + pthread_spin_lock (&lock);
> > > > + totalthreads--;
> > > > + nreqthreads--;
> > > > + pthread_spin_unlock (&lock);
> > 
> > It might be a good idea use atomic operations instead of the spin lock
> > (which is what the spin lock is using behind the scenes anyways).
> 
> It may not be possible: further down there is a decision taken depending
> on totalthreads/nreqthreads becoming 1, and thus you need a spinlock to
> make that coherent with the ++/--.

Too bad.  Perhaps combining the quantities into a single word as
follows would work:

  atomic.add(threads, 1<<16 | 1)

This has the further advantage of reducing the number of atomic
operations.

Neal

Re: [PATCH] libports: fix the thread counts in case the thread creation fails

[Neal H. Walfield]

At Sat, 9 Nov 2013 18:21:51 +0100,
Samuel Thibault wrote:
> > + pthread_spin_lock (&lock);
> > + totalthreads--;
> > + nreqthreads--;
> > + pthread_spin_unlock (&lock);

It might be a good idea use atomic operations instead of the spin lock
(which is what the spin lock is using behind the scenes anyways).

Neal

Re: [PATCH 3/5] libports: fix error handling in _ports_create_port_internal

[Neal H. Walfield]

At Fri, 25 Oct 2013 15:54:43 +0200,
Justus Winter wrote:
> 
> Quoting Thomas Schwinge (2013-10-25 15:27:10)
> > > --- a/libports/create-internal.c
> > > +++ b/libports/create-internal.c
> > > @@ -109,10 +109,11 @@ _ports_create_port_internal (struct port_class 
> > > *class,
> > >err = EINTR;
> > >   lose:
> > >pthread_mutex_unlock (&_ports_lock);
> > > +  error_t e;
> > >   lose_unlocked:
> > > -  err = mach_port_mod_refs (mach_task_self (), port,
> > > - MACH_PORT_RIGHT_RECEIVE, -1);
> > > -  assert_perror (err);
> > > +  e = mach_port_mod_refs (mach_task_self (), port,
> > > +   MACH_PORT_RIGHT_RECEIVE, -1);
> > > +  assert_perror (e);
> > >free (pi);
> > >  
> > >return err;
> > 
> > OK, but I suggest to move the definition of e after the lose_unlocked
> > label, 
> 
> I know it looks strange, but what you're suggesting is not
> possible. Labels can only be placed in front of a statement and a
> declaration is not a statement. Well, that's what gcc told me...

You can do: lose:;

Re: RFC: [PATCH] SCM_CREDS support

[Neal H. Walfield]

At Thu, 24 Oct 2013 15:38:11 +0200,
Svante Signell wrote:
> > Well, the question is quite simple: what happens when the sender
> > provides faked ports, e.g. pointing to other proc/auth servers?  That's
> > where having to explain how the patch is working would possibly even
> > work out the security issues.
> 
> How could it point to other proc/auth servers? The receiver is using the
> ports of the same proc server. Are you considering more than one
> instance running? This is communication on a local socket, and the
> socket read/write mode is controlling the access to it. In the
> implementation only the same user and root could send. for other users
> the socket permission has to be changed from srw-r--r-- to srw-r--rw-
> Tested by sending as another user.

There is not a check of who opened the socket, but the sender.  These
may be different.

Neal

Re: [PATCH 3/5] console-client: replace function epilogue with console_exit

[Neal H. Walfield]

At Thu, 10 Oct 2013 18:08:20 +0200,
Justus Winter wrote:
> 
> * console-client/console.c (main): Replace epilogue with console_exit.
> ---
>  console-client/console.c |3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/console-client/console.c b/console-client/console.c
> index a9f8368..4046cf3 100644
> --- a/console-client/console.c
> +++ b/console-client/console.c
> @@ -745,6 +745,5 @@ main (int argc, char *argv[])
>cons_server_loop ();
>  
>/* Never reached.  */
> -  driver_fini ();
> -  return 0;
> +  console_exit ();

Do you need to remove the comment too?

Re: [PATCH 3/7] name_equal: return early from function if src null-terminates in the loop

[Neal H. Walfield]

At Fri, 13 Sep 2013 15:06:33 +0200,
Marin Ramesa wrote:
> 
> On 13.09.2013 14:42:44, Neal H. Walfield wrote:
> > At Fri, 13 Sep 2013 13:31:53 +0200,
> > Marin Ramesa wrote:
> > > diff --git a/device/dev_name.c b/device/dev_name.c
> > > index bf541df..6ce4b19 100644
> > > --- a/device/dev_name.c
> > > +++ b/device/dev_name.c
> > > @@ -69,9 +69,12 @@ name_equal(src, len, target)
> > >   int len;
> > >   char*target;
> > >  {
> > > - while (--len >= 0)
> > > + while (--len >= 0) {
> > >   if (*src++ != *target++)
> > >   return FALSE;
> > > + if (*src == '\0' && *target != '\0')
> > > + return FALSE;
> > > + }
> > 
> > Shouldn't this return TRUE?
> > 
> > Neal
> 
> I don't think so. The function tests if 'src' and 'target' are equal 
> for 'len' characters, so if 'src' null-terminates inside the loop and 
> 'target' doesn't, it means that they are not equal (btw in the comments 
> it says that 'target' is sure to be null-terminated).

This is clearer, I think:

if (*src == '\0' || *target == '\0')
  return (*src == '\0' && *target == '\0');

Re: [PATCH 3/7] name_equal: return early from function if src null-terminates in the loop

[Neal H. Walfield]

At Fri, 13 Sep 2013 13:31:53 +0200,
Marin Ramesa wrote:
> diff --git a/device/dev_name.c b/device/dev_name.c
> index bf541df..6ce4b19 100644
> --- a/device/dev_name.c
> +++ b/device/dev_name.c
> @@ -69,9 +69,12 @@ name_equal(src, len, target)
>   int len;
>   char*target;
>  {
> - while (--len >= 0)
> + while (--len >= 0) {
>   if (*src++ != *target++)
>   return FALSE;
> + if (*src == '\0' && *target != '\0')
> + return FALSE;
> + }

Shouldn't this return TRUE?

Neal

Re: Hacking gnumach to track parental relationship of tasks

[Neal H. Walfield]

At Thu,  5 Sep 2013 16:57:41 +0200,
Justus Winter wrote:
> I made two rather small and (as I thought) straight forward changes to
> gnumach to keep track of a tasks father task and to make this
> information available.

What happens when the parent task is destroyed?  Are the children
destroyed with it automatically?  Does it stick around as a zombie?
Are the children reparented to their grandparent?  The last option has
the least overhead, but then you need to be careful about children
escaping from the task hierarchy.  (This is probably relevant to a
cgroup implementation.)

> Now I know handing out task ports right and
> left is not a wise thing to do, but I wanted to do it this way as a
> first step.

A child should not necessarily be able to send to its parent (the
parent may want to contain the child in some way).  However, it seems
reasonable to allow a parent to send to its child.


Note: the parent task should probably be the parent as provided to
task_create.

Neal

Re: [PATCH 10/16] hurd: add fsys_get_children

[Neal H. Walfield]

At Tue, 30 Jul 2013 22:44:22 +0200,
Richard Braun wrote:
> 
> On Tue, Jul 30, 2013 at 12:20:32PM +0200, Neal H. Walfield wrote:
> > > fsys_get_children returns any active translators bound to nodes of the
> > > receiving filesystem as an argz vector containing file names relative
> > > to the root of the receiving translator.
> > 
> > What if the caller is chrooted?  The filenames should be relative to
> > the caller's root.
> 
> Is that really necessary ? Support for chroot has always been incomplete
> and present for compatibility only. In addition, Linux still shows the
> paths at mount time in /proc/mounts from a chroot.

I'm not concerned about the mtab implementation.  I'm concerned about
the RPC's interface.

I'm not sure why you think chroot is only for compatibility and why we
therefore shouldn't correctly support it.

Neal

Re: [PATCH 10/16] hurd: add fsys_get_children

[Neal H. Walfield]

At Tue, 30 Jul 2013 11:59:18 +0200,
Justus Winter wrote:
> 
> fsys_get_children returns any active translators bound to nodes of the
> receiving filesystem as an argz vector containing file names relative
> to the root of the receiving translator.

What if the caller is chrooted?  The filenames should be relative to
the caller's root.

Neal

Re: feature-mtab-translator (v3)

[Neal H. Walfield]

At Mon, 22 Jul 2013 22:49:03 +0200,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Mon 22 Jul 2013 21:50:02 +0200, a écrit :
> > At Fri, 19 Jul 2013 17:25:02 +0200,
> > Justus Winter wrote:
> > > * The translator used to traverse the translator tree and if it
> > >   encountered itself, it would deadlock. This is cleanly solved by
> > >   comparing the control ports of the current node and the mtab
> > >   translator.
> > 
> > I don't think you can reliably do this.  This assumes that the same
> > port is always used for accessing the the fsys facet, which is not
> > necessarily the case.
> 
> What other port could it be?

Perhaps I don't completely understand the situation.

Here's what I'm thinking: if a program somehow fetches an fsys port,
there is no guarantee that the returned send right corresponds to a
unique port (even if this is currently the de facto case).  Consider
opening a file: the port identifies the peropen; every open results in
a new receive right.  Or, consider proxing.  If this is somehow
guaranteed not to be the case here, then I guess a numeric comparison
is acceptable.

Neal

Re: feature-mtab-translator (v3)

[Neal H. Walfield]

At Fri, 19 Jul 2013 17:25:02 +0200,
Justus Winter wrote:
> * The translator used to traverse the translator tree and if it
>   encountered itself, it would deadlock. This is cleanly solved by
>   comparing the control ports of the current node and the mtab
>   translator.

I don't think you can reliably do this.  This assumes that the same
port is always used for accessing the the fsys facet, which is not
necessarily the case.

Re: feature-mtab-translator (v3)

[Neal H. Walfield]

At Fri, 19 Jul 2013 18:04:47 +0200,
Justus Winter wrote:
> My personal preference would be to run the translator on /proc/mounts
> as unprivileged user created solely for this purpose by default.


This is not how things are done on the Hurd.  You don't need an
"unpriviledged user" you just drop all of your uids and gids.

Neal

Re: [PATCH 17/17] add mtab prototype

[Neal H. Walfield]

At Fri, 19 Jul 2013 17:25:19 +0200,
Justus Winter wrote:
> +static char *path = NULL;
> +static int insecure = 0;

Don't explicitly initialize static variables to 0 or NULL.  This takes
up unnecessary space in the executable and prevents using zero-filled
memory.

Neal

Re: [PATCH 03/17] libdiskfs: track file name in struct peropen

[Neal H. Walfield]

If you take this approach, you need to update relpath on renames.  Do
you?  (I may have missed that.)

Neal

At Fri, 19 Jul 2013 17:25:05 +0200,
Justus Winter wrote:
> 
> ---
>  libdiskfs/dir-lookup.c   |   24 
>  libdiskfs/diskfs.h   |3 +++
>  libdiskfs/fsys-getroot.c |3 ++-
>  libdiskfs/peropen-make.c |8 
>  libdiskfs/peropen-rele.c |1 +
>  5 files changed, 38 insertions(+), 1 deletion(-)
> 
> diff --git a/libdiskfs/dir-lookup.c b/libdiskfs/dir-lookup.c
> index 923be03..08047cf 100644
> --- a/libdiskfs/dir-lookup.c
> +++ b/libdiskfs/dir-lookup.c
> @@ -41,6 +41,7 @@ diskfs_S_dir_lookup (struct protid *dircred,
>struct node *np;
>int nsymlink = 0;
>char *nextname;
> +  char *relpath;
>int nextnamelen;
>error_t error = 0;
>char *pathbuf = 0;
> @@ -68,6 +69,11 @@ diskfs_S_dir_lookup (struct protid *dircred,
>while (path[0] == '/')
>  path++;
>  
> +  /* Preserve the path relative to diruser->po->path.  */
> +  relpath = strdup (path);
> +  if (! relpath)
> +return ENOMEM;
> +
>*returned_port_poly = MACH_MSG_TYPE_MAKE_SEND;
>*retry = FS_RETRY_NORMAL;
>retryname[0] = '\0';
> @@ -479,6 +485,22 @@ diskfs_S_dir_lookup (struct protid *dircred,
>  
>if (! error)
>  {
> +  free (newpi->po->path);
> +  if (dircred->po->path == NULL)
> + {
> +   /* dircred is the root directory.  */
> +   newpi->po->path = relpath;
> +   relpath = NULL; /* Do not free relpath.  */
> + }
> +  else
> + {
> +   newpi->po->path = NULL;
> +   asprintf (&newpi->po->path, "%s/%s", dircred->po->path, relpath);
> + }
> +
> +  if (! newpi->po->path)
> + error = errno;
> +
>*returned_port = ports_get_right (newpi);
>ports_port_deref (newpi);
>newpi = 0;
> @@ -500,5 +522,7 @@ diskfs_S_dir_lookup (struct protid *dircred,
>if (newpo)
>  diskfs_release_peropen (newpo);
>  
> +  free (relpath);
> +
>return error;
>  }
> diff --git a/libdiskfs/diskfs.h b/libdiskfs/diskfs.h
> index 0f9c1d3..2489517 100644
> --- a/libdiskfs/diskfs.h
> +++ b/libdiskfs/diskfs.h
> @@ -69,6 +69,9 @@ struct peropen
>mach_port_t shadow_root_parent;
>/* If in a shadow tree, its root node in this translator.  */
>struct node *shadow_root;
> +
> +  /* Path relative to the root of the translator. */
> +  char *path;
>  };
>  
>  /* A unique one of these exists for each node currently in use (and
> diff --git a/libdiskfs/fsys-getroot.c b/libdiskfs/fsys-getroot.c
> index 2e11da4..39973a8 100644
> --- a/libdiskfs/fsys-getroot.c
> +++ b/libdiskfs/fsys-getroot.c
> @@ -51,7 +51,8 @@ diskfs_S_fsys_getroot (fsys_t controlport,
>{
>  root_parent: dotdot,
>  shadow_root_parent: MACH_PORT_NULL,
> -shadow_root: _diskfs_chroot_directory ? diskfs_root_node : NULL /* XXX */
> +shadow_root: _diskfs_chroot_directory ? diskfs_root_node : NULL, /* XXX 
> */
> +path: NULL,
>};
>  
>if (!pt)
> diff --git a/libdiskfs/peropen-make.c b/libdiskfs/peropen-make.c
> index d37516c..d0ac698 100644
> --- a/libdiskfs/peropen-make.c
> +++ b/libdiskfs/peropen-make.c
> @@ -34,6 +34,7 @@ diskfs_make_peropen (struct node *np, int flags, struct 
> peropen *context,
>po->refcnt = 0;
>po->openstat = flags;
>po->np = np;
> +  po->path = NULL;
>  
>if (context)
>  {
> @@ -50,6 +51,13 @@ diskfs_make_peropen (struct node *np, int flags, struct 
> peropen *context,
>if (po->shadow_root_parent != MACH_PORT_NULL)
>   mach_port_mod_refs (mach_task_self (), po->shadow_root_parent, 
>   MACH_PORT_RIGHT_SEND, 1);
> +
> +  if (context->path)
> +{
> +  po->path = strdup (context->path);
> +  if (! po->path)
> +return ENOMEM;
> +}
>  }
>else
>  {
> diff --git a/libdiskfs/peropen-rele.c b/libdiskfs/peropen-rele.c
> index 08276ec..d3f7492 100644
> --- a/libdiskfs/peropen-rele.c
> +++ b/libdiskfs/peropen-rele.c
> @@ -45,5 +45,6 @@ diskfs_release_peropen (struct peropen *po)
>  
>diskfs_nput (po->np);
>  
> +  free (po->path);
>free (po);
>  }
> -- 
> 1.7.10.4
> 
> 
> 

Re: feature-mtab-translator (v3)

[Neal H. Walfield]

At Fri, 19 Jul 2013 18:04:47 +0200,
Justus Winter wrote:
> My personal preference would be to run the translator on /proc/mounts
> as unprivileged user created solely for this purpose by default. It's
> up to the system administrator to change that if he wishes. I know
> it's not as magically as it could be if the mtab translator would
> impersonate the requesting user, but then again, this is no problem of
> the RPC procedure or the server side implementation of it.

The translator should never impersonate the user.  At most it should
mediate access.  If more authority is needed, the client should
interact directly with the translator in question.  This can be done
by having the mediator return an unauthenticated port.  Anything else
will request in the confused deputy and a huge attack surface.

Neal

Re: [PATCH 15/17] hurd: add fsys_get_source

[Neal H. Walfield]

This needs much better documentation.

At Fri, 19 Jul 2013 17:25:17 +0200,
Justus Winter wrote:
> 
> ---
>  hurd/fsys.defs |7 +++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/hurd/fsys.defs b/hurd/fsys.defs
> index 4b649d9..27ada29 100644
> --- a/hurd/fsys.defs
> +++ b/hurd/fsys.defs
> @@ -135,3 +135,10 @@ routine fsys_get_children (
>   server: fsys_t;
>   RPT
>  out children: data_t);
> +
> +/* Return information about the source of the receiving
> +   filesystem.  */
> +routine fsys_get_source (
> + server: fsys_t;
> + RPT
> +out source: string_t);
> -- 
> 1.7.10.4
> 
> 
> 

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Thu, 18 Jul 2013 11:39:14 +0200,
Justus Winter wrote:
> He has shown me how this could be implemented, but doing so requires
> attaching credentials to all fsys_* by means of
> e.g. libdiskfs/fsmutations.h and fixing all functions that are
> affected by this. I think it's doable, but my time might be better
> spent on something else. Thoughts?

If you introduce a new RPC, it should be done right.  In particular,
it should not introduce security problems.  If you just write a
translator that doesn't require a new RPC, emulating Linux is fine.

Neal

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Mon, 15 Jul 2013 12:13:52 +0200,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Mon 15 Jul 2013 11:34:38 +0200, a écrit :
> > > > So, let's think about the
> > > > fsck problem.  What we need is a registry.  I propose an mtab
> > > > translator.  Translators register with it when they start and register
> > > > a call back.
> > > 
> > > Where do they register? Is this yet another well-known translator,
> > > outside of the filesystem? Having it distributed seems much more
> > > elegant.
> > 
> > /servers/mtab and $USER/server/mtab.
> 
> How does / register itself?

Perhaps have an RPC that the mtab server can use to cause a file
system to register itself.  When it starts, it send the RPC to well
known locations that probably started before it.  They then register.

Neal

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Mon, 15 Jul 2013 10:37:37 +0200,
Samuel Thibault wrote:
> > I don't like your solution to the mtab problem.  But, as you admit,
> > it's really a solution to the fsck problem.
> 
> Not only. It's also about umount being able to know what to unmount when
> calling umount /dev/cdrom, about mount/df being able to have a list of
> mounted filesystems, etc. The fsck problem could be solved by simply
> having an exclusive lock on the underlying device node.

Then perhaps we need fsys_set_user / fsys_get_user RPCs.  When a
process begins to use a node, it can optionally register itself as a
user.  Then others can determine whether the node is being used and
who it is.  The user port would be an unauthenticated port
implementing, for instance, the fsys interface.

> > So, let's think about the
> > fsck problem.  What we need is a registry.  I propose an mtab
> > translator.  Translators register with it when they start and register
> > a call back.
> 
> Where do they register? Is this yet another well-known translator,
> outside of the filesystem? Having it distributed seems much more
> elegant.

/servers/mtab and $USER/server/mtab.

Neal

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Fri, 12 Jul 2013 15:07:59 +0200,
Justus Winter wrote:
> Quoting Samuel Thibault (2013-07-12 14:44:31)
> > Neal H. Walfield, le Fri 12 Jul 2013 13:52:17 +0200, a écrit :
> > > I apologize if you've already explained this someplace else.
> > > 
> > > If I understand correctly, you want to get all the nodes with active
> > > and passive translators.  This potentially requires scanning every
> > > inode on an ext3 file system.  This could take a very long time.  Is
> > > that really the intention?
> 
> That's an interesting point. Btw, settrans --recursive is currently
> implemented like this, I'll patch this also to use the translator
> list.

--recursive only applies to the active translators, not the passive
translators.  Also, it doesn't need to shutdown shortcut translators,
such as, symlinks, which should perhaps be returned by your proposed
RPC.

> > No, just the active ones.
> 
> Actually the code also tracks passive translators being set, but of
> course we don't know anything about any previously set passive
> translators.

I don't like this behavior.

> Richard and me agreed that for most purposes one needs an
> mtab file for (e.g. fsck) it is also interesting to know whether a
> passive translator is set and what device it would use because any
> user could trigger the translator start at any time.

I don't think this justifies the ugliness.


I don't like your solution to the mtab problem.  But, as you admit,
it's really a solution to the fsck problem.  So, let's think about the
fsck problem.  What we need is a registry.  I propose an mtab
translator.  Translators register with it when they start and register
a call back.  On boot, the mtab translator is started and calls each
call back.  It's not perfect, but it's a lot less ugly.

Neal

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Fri, 12 Jul 2013 14:44:31 +0200,
Samuel Thibault wrote:
> 
> Neal H. Walfield, le Fri 12 Jul 2013 13:52:17 +0200, a écrit :
> > I apologize if you've already explained this someplace else.
> > 
> > If I understand correctly, you want to get all the nodes with active
> > and passive translators.  This potentially requires scanning every
> > inode on an ext3 file system.  This could take a very long time.  Is
> > that really the intention?
> 
> No, just the active ones.

How do you do permission checking?

Here's a thought:

Consider accessing a file foo/bar/file.  If the user specifies the
full path, then she needs execute permission on the containing
directories.  If the path is not known, then to find the file, she
also needs read permission on the containing directories.  That is, to
'ls foo', she needs read permission on foo, but 'cat foo/bar/file'
only requires read permission on foo and bar.

Making a directory executable but not readable is a useful way to
grant permission by knowledge of a shared secret.  If foo is not
readable, then a user can only access the contents of foo/bar if they
know that foo/bar exists.  This is essentially a swiss numbers in the
capability world.

It seems like the interface that you have created should require read
permission on the containing directories.  This is okay, but then the
caller should be able specify a root to start the search under.  This
way, she can enumerate all active translators under foo/bar even if
she doesn't have read permission on foo.

Neal

Re: [PATCH 10/17] hurd: add fsys_get_children

[Neal H. Walfield]

At Thu, 11 Jul 2013 18:09:13 +0200,
Justus Winter wrote:
> diff --git a/hurd/fsys.defs b/hurd/fsys.defs
> index 979a6cf..4b649d9 100644
> --- a/hurd/fsys.defs
> +++ b/hurd/fsys.defs
> @@ -127,3 +127,11 @@ routine fsys_get_options (
>   server: fsys_t;
>   RPT
>   out options: data_t, dealloc);
> +
> +/* Return any passive and active translators bound to nodes of the
> +   receiving filesystem. children is a argz vector containing file
> +   names relative to the root of the receiving translator.  */
> +routine fsys_get_children (
> + server: fsys_t;
> + RPT
> +out children: data_t);

I apologize if you've already explained this someplace else.

If I understand correctly, you want to get all the nodes with active
and passive translators.  This potentially requires scanning every
inode on an ext3 file system.  This could take a very long time.  Is
that really the intention?

How do you handle symlinks, which are really just translators?

How do you handle infinite file systems like hostmux?

Neal

Re: Hurd stacks, some more (was: RFC: ruby1.9.1 FTBFS)

[Neal H. Walfield]

At Thu, 4 Jul 2013 17:06:09 +0200,
Thomas Schwinge wrote:
> I wonder: if MAP_STACK is set, would it even be
> reasonable for mmap to ignore the supplied length, and instead use the
> one "proper" value, 0x20?

I think that this is only acceptable if the length exceeds the
supplied length.

This would, however, creates a small problem: when calling munmap, the
length is passed.  Since the caller doesn't know that a larger area
was mapped, we would need to record these exceptions and check for
them on munmap.  That's ugly.

Neal

Re: Guile not running properly on GNU/Hurd

[Neal H. Walfield]

At Sat, 2 Feb 2013 23:24:38 +0100,
Richard Braun wrote:
> On Sat, Feb 02, 2013 at 07:51:38PM +0100, Gabriel Schnoering wrote:
> > I can't run guile in gdb as there are some error with the garbage
> > collector.
> 
> These are probably not errors. Many language interpretors rely on
> receiving SIGSEGV to implement memory. Use the gdb "handle" command to
> make it automatically pass on SIGSEGV signals so that you only catch the
> SIGILL you're interested in.

It might be an interesting project to use Mach Memory Objects to back
garbage collected memory rather than rely on the SIGSEGV machinery.
This would be more efficient and perhaps more accurate.  You'd still
have to deal with the stack separately.

Neal

Re: [PATCH 5/7] Fix double call to pthread_mutex_unlock in _treefs_s_dir_lookup.

[Neal H. Walfield]

At Mon, 17 Dec 2012 01:31:32 +0100,
Samuel Thibault wrote:
> 
> Cyril Roelandt, le Mon 17 Dec 2012 00:51:28 +0100, a écrit :
> > * libtreefs/dir-lookup.c (_treefs_s_dir_lookup): remove a redundant call to
> > pthread_mutex_unlock.
> > 
> > Signed-off-by: Cyril Roelandt 
> > ---
> >  libtreefs/dir-lookup.c |1 -
> >  1 file changed, 1 deletion(-)
> > 
> > diff --git a/libtreefs/dir-lookup.c b/libtreefs/dir-lookup.c
> > index ce2acaf..41c34ea 100644
> > --- a/libtreefs/dir-lookup.c
> > +++ b/libtreefs/dir-lookup.c
> > @@ -199,7 +199,6 @@ _treefs_s_dir_lookup (struct treefs_handle *h,
> >  in the right order. */
> >   if (strcmp (path, "..") != 0)
> > {
> > - pthread_mutex_unlock (&node->lock);
> >   pthread_mutex_lock (&dir->lock);
> >   pthread_mutex_lock (&node->lock);
> 
> At quick sight I don't think this one is spurious, see the comment: this
> code seems to be used when one wants to lock dir->lock, which we can
> not do when we already have node->lock, that's why we have to release
> node->lock before taking dir->lock again.

I would change this to do a trylock on dir->lock.  If that fails, then
unlock node->lock and relock dir->lock and node->lock.

Neal

Re: Question about your libpthread

[Neal H. Walfield]

At Mon, 13 Aug 2012 11:54:17 +0200,
Richard Braun wrote:
> In addition, here you only mention the recylcing problem, not the
> threadvar alignment problem. Or is there no such problem ?

It's been a decade since I wrote the code, I've fotten a bit.  I don't
know offhand if there is a problem and I don't have time to
investigate deeply.  Sorry.

Neal

Re: Question about your libpthread

[Neal H. Walfield]

I've add bug-hurd to the cc.

At Mon, 13 Aug 2012 11:16:26 +0200,
Richard Braun wrote:
> 
> On Sun, Aug 12, 2012 at 07:03:06PM +0200, Neal H. Walfield wrote:
> > But, if it is a generally useful function, I don't see why not to
> > expose it as a general function.  Just add _np at the end of the
> > function name.
> 
> Sorry to disturb you again, but I found something surprising when using
> libpthread. It seems the stack size can't be changed with
> pthread_attr_setstacksize. There is a comment about alignment but I'm
> not sure I understand why it prevents different stack sizes.
> 
> This is important for my current work, as cthreads use 64k stacks
> whereas pthreads use 2 MiB, which strongly limits the maximum number of
> threads, and may prevent pagers like ext2fs to service some paging
> requests under thread storms (the problem exists with cthreads of
> course, but I'd like to reduce its likeliness by at least approaching
> what was done with them).

The problem has to do with the recycling of stacks.  The problem is
that a thread cannot easily free its own stack when it exits.  If it
frees its stack, it can't free it's kernel thread and vice versa.  To
work around this, the thread does neither.  Instead, later threads
reuse the stack and kernel thread.  If the stacks had a non-standard,
we would need some additional code to find an appropriately sized
stack or implement a mechanism to clean up exited threads
periodically.

Re: libpthread: caching of data structures and kernel resources

[Neal H. Walfield]

At Fri, 10 Aug 2012 09:15:01 +0200,
Neal H. Walfield wrote:
> > Why does Viengoos in
> > sysdeps/viengoos/pt-thread-halt.c:__pthread_thread_halt have to do
> > different things depending on whether »thread == _pthread_self« or not?
> > I'm assuming that on Mach, we can just always do thread_suspend (and take
> > care to do a thread_abort before re-using it)?
> 
> My recollection is that they are semantically equivalent in this case
> (vg_suspend cause the calling thread to wait for a message;
> vg_thread_stop marks the thread as not runnable), but vg_suspend is a
> lot cheaper than vg_thread_stop.

I now remember the main motivation for this.  Modifying a thread is an
RPC, i.e., a reply is sent.  When a thread stops itself, any reply
will be dropped.  As this is normally an error, Viengoos dumped a fair
amount of debugging output when this condition occurred.  To avoid
this gratuitous output, I used vg_suspend instead.

Neal

Re: libpthread: caching of data structures and kernel resources

[Neal H. Walfield]

Hi, Thomas,

At Fri, 10 Aug 2012 01:17:30 +0200,
Thomas Schwinge wrote:
> libpthread applies some caching of data structures
> (cf. __pthread_free_threads; »state == PTHREAD_TERMINATED«) which after a
> thread has exited can be re-used for a new one.
> ...
> Is this an optimization, assuming the second sequence of operations is
> cheaper (is surely seems to be), as well as assuming that if a thread
> exits/is terminated, surely a new one is to be created soon after?  Is
> this *only* an optimization, or does the internal libpthread logic also
> require this for some other reasons?
>
> In pthread/pt-exit.c for »state == PTHREAD_JOINABLE«, the state is then
> set to PTHREAD_EXITED.  Then, in both pthread/pt-detach.c:pthread_detach
> and pthread/pt-join.c:pthread_join, in the PTHREAD_EXITED case, the
> kernel resources will explicitly be deallocated by calling
> __pthread_stack_dealloc and __pthread_thread_dealloc, but then the
> pthread structure kept for re-use by calling __pthread_dealloc.  Why are
> the kernel resources deallocated here, contrary to re-using them like in
> other cases?

This isn't an optimization.  This pattern is necessary to properly
manage the resources.  When a thread exists, it can't destroy all of
its own resources.  If a thread should clean up all its state, it
needs to free its stack and its kernel thread.  Once it frees one, it
is no longer able to free the other (it can no longer run).

> Why does Viengoos in
> sysdeps/viengoos/pt-thread-halt.c:__pthread_thread_halt have to do
> different things depending on whether »thread == _pthread_self« or not?
> I'm assuming that on Mach, we can just always do thread_suspend (and take
> care to do a thread_abort before re-using it)?

My recollection is that they are semantically equivalent in this case
(vg_suspend cause the calling thread to wait for a message;
vg_thread_stop marks the thread as not runnable), but vg_suspend is a
lot cheaper than vg_thread_stop.


Neal

Re: libpthread cleanup: L4 port, PowerPC port

[Neal H. Walfield]

At Tue, 7 Aug 2012 23:55:12 +0200,
Thomas Schwinge wrote:
> > It's generic code.  At least glibc keeps generic code around, if I
> > recall correctly.  Further, one of the very nice things about this
> > libpthread is that it is really easy to integrate into a new OS (or
> > even use in a kernel).  It would be a shame to lose that feature.
> > But, as I'm not doing the work, you are welcome to evict this code to
> > make your life easier.
> 
> I kept it in, no problem.  In the effort to build the x86/Hurd/Mach
> libpthread as part of the glibc build, we'll have to invent some sysdeps
> mechanism so that the libpthread signal implementation is not preferred
> over the real glibc implementation, which is a problem Samuel had, as I
> understand it, but that should be doable.

If it is a burden, just remove it.

Neal

Re: libpthread cleanup: L4 port, PowerPC port

[Neal H. Walfield]

At Sat, 4 Aug 2012 15:56:01 +0200,
Thomas Schwinge wrote:
> On Sat, 04 Aug 2012 15:16:22 +0200, "Neal H. Walfield"  
> wrote:
> > At Sat, 4 Aug 2012 15:02:46 +0200,
> > Thomas Schwinge wrote:
> > > > > * signal/README: Likewise.
> > > > > * signal/TODO: Likewise.
> > > > > * signal/kill.c: Likewise.
> > > > > * signal/pt-kill-siginfo-np.c: Likewise.
> > > > > * signal/sig-internal.c: Likewise.
> > > > > * signal/sig-internal.h: Likewise.
> > > > > * signal/sigaction.c: Likewise.
> > > > > * signal/sigaltstack.c: Likewise.
> > > > > * signal/signal-dispatch.c: Likewise.
> > > > > * signal/signal.h: Likewise.
> > > > > * signal/sigpending.c: Likewise.
> > > > > * signal/sigsuspend.c: Likewise.
> > > > > * signal/sigtimedwait.c: Likewise.
> > > > > * signal/sigwaiter.c: Likewise.
> > > > > * signal/sigwaitinfo.c: Likewise.
> > > > > * sysdeps/generic/killpg.c: Likewise.
> > > > > * sysdeps/generic/raise.c: Likewise.
> > > > > * sysdeps/generic/sigaddset.c: Likewise.
> > > > > * sysdeps/generic/sigdelset.c: Likewise.
> > > > > * sysdeps/generic/sigemptyset.c: Likewise.
> > > > > * sysdeps/generic/sigfillset.c: Likewise.
> > > > > * sysdeps/generic/siginterrupt.c: Likewise.
> > > > > * sysdeps/generic/sigismember.c: Likewise.
> > > > > * sysdeps/generic/signal.c: Likewise.
> > > > > * sysdeps/generic/sigwait.c: Likewise.
> > > > 
> > > > I think these are also used by the Viengoos port.  Are you suggesting
> > > > to remove both the L4 and the Viengoos ports or just the L4 port?
> > > 
> > > I only intend to clean up libpthread's master branch (that is, remove its
> > > old L4 port).
> > 
> > Ok.  I still think that the above files are generic and not L4
> > specific.
> 
> Does that mean you'd like to keep them available in the master branch,
> though unused there?  (In principle I'm fine either way, but have a
> tendency towards only having code live in the repositories that is
> actually being used.)

It's generic code.  At least glibc keeps generic code around, if I
recall correctly.  Further, one of the very nice things about this
libpthread is that it is really easy to integrate into a new OS (or
even use in a kernel).  It would be a shame to lose that feature.
But, as I'm not doing the work, you are welcome to evict this code to
make your life easier.

Neal

Re: libpthread cleanup: L4 port, PowerPC port

[Neal H. Walfield]

At Sat, 4 Aug 2012 15:02:46 +0200,
Thomas Schwinge wrote:
> > > master-viengoos and its successor, master-viengoos-on-bare-metal.
> > 
> > master-viengoos is an implementation of Viengoos that runs on L4.
> > viengoos-on-bare-metal runs directly on x86-64 (and it a bit more
> > advanced) and provides everything that master-viengoos does and more.
> > I unfortunately never got around to issuing the right git commands to
> > make turn it into the master branch.
> 
> Note that I'm talking about branches of the hurd/libpthread.git
> repository, which contains only the libpthread branches (as used by Hurd
> on Mach: master, old L4: master, Viengoos: master-viengoos and
> viengoos-on-bare-metal), and not the hurd/viengoos.git repository which
> contains all the other Viengoos code.

That's what I thought you meant.

> > > * signal/README: Likewise.
> > > * signal/TODO: Likewise.
> > > * signal/kill.c: Likewise.
> > > * signal/pt-kill-siginfo-np.c: Likewise.
> > > * signal/sig-internal.c: Likewise.
> > > * signal/sig-internal.h: Likewise.
> > > * signal/sigaction.c: Likewise.
> > > * signal/sigaltstack.c: Likewise.
> > > * signal/signal-dispatch.c: Likewise.
> > > * signal/signal.h: Likewise.
> > > * signal/sigpending.c: Likewise.
> > > * signal/sigsuspend.c: Likewise.
> > > * signal/sigtimedwait.c: Likewise.
> > > * signal/sigwaiter.c: Likewise.
> > > * signal/sigwaitinfo.c: Likewise.
> > > * sysdeps/generic/killpg.c: Likewise.
> > > * sysdeps/generic/raise.c: Likewise.
> > > * sysdeps/generic/sigaddset.c: Likewise.
> > > * sysdeps/generic/sigdelset.c: Likewise.
> > > * sysdeps/generic/sigemptyset.c: Likewise.
> > > * sysdeps/generic/sigfillset.c: Likewise.
> > > * sysdeps/generic/siginterrupt.c: Likewise.
> > > * sysdeps/generic/sigismember.c: Likewise.
> > > * sysdeps/generic/signal.c: Likewise.
> > > * sysdeps/generic/sigwait.c: Likewise.
> > 
> > I think these are also used by the Viengoos port.  Are you suggesting
> > to remove both the L4 and the Viengoos ports or just the L4 port?
> 
> I only intend to clean up libpthread's master branch (that is, remove its
> old L4 port).

Ok.  I still think that the above files are generic and not L4
specific.

> On libpthread's master-viengoos and viengoos-on-bare-metal
> branches, the Viengoos-specific code stays as-is, and can continue to
> evolve there, with the long-term goal of eventually merging it back into
> libpthread's master branch as a new Viengoos port of libpthread.  (I
> guess that you only started using Viengoos-specific branches for
> libpthread for the reason of being able to easily do changes in the
> generic libpthread code without disturbing/having to adapt the Hurd on
> Mach port.)

My recollection is that Hurd was in CVS at the time.  You(?) created
 libpthread.git later.

> Yes, but I think given the testing it is seeing via Debian GNU/Hurd, your
> libpthread not so bad actually.  In the long term, and given someone
> willing to do it, a NPTL port would of course be possible/preferable/...,
> too.  But I don't think that's a priority currently.

If you have time, I'd recommend that you also take a look at the
asserts and disable the more expensive ones.  For instance in
sysdeps/mach/hurd/pt-sysdep.h, _pthread_self checks that the
thread->kernel_thread matches _mach_thread_self().  This requires two
system calls!  _pthread_self should be fast...

Neal

Re: libpthread cleanup: L4 port, PowerPC port

[Neal H. Walfield]

At Sat, 4 Aug 2012 12:34:28 +0200,
Thomas Schwinge wrote:
> There has been a libpthread port for Hurd on L4 use, which is dead, and
> has been superseded by a Viengoos port, which has its own branches:

For what it is worth, the L4 port works directly on L4: no further OS
personality support is required.  On the other hand, no one actually
uses it so I guess it is okay to remove it.

> master-viengoos and its successor, master-viengoos-on-bare-metal.

master-viengoos is an implementation of Viengoos that runs on L4.
viengoos-on-bare-metal runs directly on x86-64 (and it a bit more
advanced) and provides everything that master-viengoos does and more.
I unfortunately never got around to issuing the right git commands to
make turn it into the master branch.

> * signal/README: Likewise.
> * signal/TODO: Likewise.
> * signal/kill.c: Likewise.
> * signal/pt-kill-siginfo-np.c: Likewise.
> * signal/sig-internal.c: Likewise.
> * signal/sig-internal.h: Likewise.
> * signal/sigaction.c: Likewise.
> * signal/sigaltstack.c: Likewise.
> * signal/signal-dispatch.c: Likewise.
> * signal/signal.h: Likewise.
> * signal/sigpending.c: Likewise.
> * signal/sigsuspend.c: Likewise.
> * signal/sigtimedwait.c: Likewise.
> * signal/sigwaiter.c: Likewise.
> * signal/sigwaitinfo.c: Likewise.
> * sysdeps/generic/killpg.c: Likewise.
> * sysdeps/generic/raise.c: Likewise.
> * sysdeps/generic/sigaddset.c: Likewise.
> * sysdeps/generic/sigdelset.c: Likewise.
> * sysdeps/generic/sigemptyset.c: Likewise.
> * sysdeps/generic/sigfillset.c: Likewise.
> * sysdeps/generic/siginterrupt.c: Likewise.
> * sysdeps/generic/sigismember.c: Likewise.
> * sysdeps/generic/signal.c: Likewise.
> * sysdeps/generic/sigwait.c: Likewise.

I think these are also used by the Viengoos port.  Are you suggesting
to remove both the L4 and the Viengoos ports or just the L4 port?

> Also, I propose to remove the incomplete and unmaintained PowerPC port.
> OK?

I have no objections to this.

> In August 2008, Neal has been porting over some generic changes from the
> Viengoos branch to master, and I'm now identifying if there are
> additional changes to be ported.

That's a good idea.  Unfortunately, I don't remember how far I got.

Note: libpthread is not without its problems, e.g., lack of testing in
the wild.  I still think the better way forward is to use Glibc's
pthread, if possible.

Neal

Re: [PATCH] Page cache accounting

[Neal H. Walfield]

At Sun, 15 Jul 2012 19:35:31 +0200,
Richard Braun wrote:
> 
> On Sun, Jul 15, 2012 at 06:39:05PM +0200, Neal H. Walfield wrote:
> > I'd encourage you to use a more self-describing data structure.  In
> > the very least, please consider a version number and a bitmask
> > indicating which fields are valid.
> 
> What about versioning the call itself, e.g. vm_cache_statistics1 etc.. ?

In this case, I guess it doesn't make a big difference.  But, in
general, you could have the caller provide a list of versions that it
understands and allow the server to choose the best version from the
intersection of that list and what it supports.  This prevents the
client from having to make O(N) calls.

Neal

Re: [PATCH] Page cache accounting

[Neal H. Walfield]

At Sat, 14 Jul 2012 17:06:37 +0200,
Richard Braun wrote:
> 
> If no major objection is raised (particularly about the new Mig
> routine), I'll commit this patch in a few days.
> ...
> +struct vm_cache_statistics {
> + integer_t   object_count;   /* # of cached objects */
> + integer_t   page_count; /* # of cached pages */
> + integer_t   _reserved[14];
> +};

I'd encourage you to use a more self-describing data structure.  In
the very least, please consider a version number and a bitmask
indicating which fields are valid.

Neal

Re: Reworking libpager.

[Neal H. Walfield]

Hi, Максим,

If I didn't post the patches, then either I never wrote them or they
have been lost.  Sorry.

Neal

At Sun, 3 Jun 2012 14:08:27 +0300,
Максим wrote:
> 
> Hello,
> 
> I'm GSoC student and I work upon improving of disk I/O in hurd [1]. As part
> of my project I have to modify libpager's interface. I know that 10 years
> ago you intended to do this. You posted patch for libpager and wrote that
> you will follow with patches for libdiskfs and ext2fs once you have some
> comments on this work [2]. But seems that nobody reviewed libpager patch,
> and hence, you didn't post other patches. Are these patches still exist and
> can you post them now?
> 
> 1. 
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/mcsim/29002
> 2. http://permalink.gmane.org/gmane.os.hurd.bugs/446
> 
> --
> Regards,
> Maksym Planeta.
> 

Re: [PATCH, libpthread] Correct logic for PTHREAD_KEY_INVALID slots.

[Neal H. Walfield]

I haven't looked at this change in detail, but it sounds sane and if
Samuel says its okay, that's good enough for me.

Neal

Re: Link to xgethostname broken at the porting guidelines page

[Neal H. Walfield]

At Mon, 11 Apr 2011 09:58:18 +0200,
Svante Signell wrote:
> When porting packages, is replacing a call to gethostname using
> MAXHOSTNAMELEN with a call to Neal Walfields xgethostname still the
> recommended way, as described at
> http://www.gnu.org/software/hurd/hurd/porting/guidelines.html
> 
> If so, maybe the link to that function should be corrected. Currently it
> is broken. In case is is not immediately found there is a copy of
> xgethostname.{c,h} at:
> http://lists.mindrot.org/pipermail/openssh-unix-dev/2002-April/012248.html

You could also get it here:

  http://walfield.org/pub/people/neal/xgethostname/

Neal

Re: Problems with signal delivery

[Neal H. Walfield]

At Thu,  2 Sep 2010 19:52:42 -0700 (PDT),
Roland McGrath wrote:
> 
> > Do you mean that there is some code relying on these Hurd semantics, and
> > that therefore we should not try to change them to match POSIX, except
> > maybe when the pthread functions are used?
> 
> I mean the semantics are the semantics and have been for 15+ years, and we
> can't assume it's OK to just go changing them and call it a "fix".  POSIX
> only specifies anything about any kind of multithreadedness when pthread
> functions are used.

What's going to break?  I suspect that the software that relies on the
Hurd's signal semantics (if any) is software that is directly related
to the Hurd and developed by Hurd people.  As such, it should be
relatively easy to fix and to have those fixes integrated; I see no
convincing reason to not adopt POSIX/Linux signal semantics and
abandon Hurd signal semantics.

Thanks,
Neal

Re: RPC user stub libraries (was: error compiling hurd after gnumach interface change)

[Neal H. Walfield]

At Mon, 19 Jul 2010 11:16:03 +0200,
Thomas Schwinge wrote:
>   * Avoid code duplication -- may have been relevant, but is it
> still?
> 
> Actually, if I understood correctly, in his Viengoos kernel, Neal
> is doing all RPC stub code generation in the pre-processor, thus
> has it as inline code at every call site.  This has one immediate
> advantage: GCC can optimize it, as there is no function-call
> boundary.  Should we be doing the same?  Someone should do some
> measurements.  Neal, any off-hand comments?

That's what I did.  The obvious disadvantage of inlining code is that
you have more code.  That's bad for the CPU caches.  Of course, if you
execute an RPC, you've definately blown your L1 in the asynchronous
case and likely blown your L2 in the synchronous case.  In which case,
the cache issue is likely moot.


I used CPP to generate RPC stubs as I didn't want to write an IDL
(although you could say that I ended up doing exactly that!).  I
encountered two main problems.  The first is that error messages for
incorrectly written prototypes are hard to decipher: the IDL compiler
can't really emit error messages.  Second, marshalling variable length
arrays is a serious pain.  What you'd like are two arguments: the
array and a count, not one as you have for other types (i.e., just the
value).  Making a decision based on a type is hard to do with CPP.

Neal

Re: libparted backend for libstore (was: cannot boot subhurd)

[Neal H. Walfield]

At Sun, 3 Jan 2010 17:15:37 +0100,
Thomas Schwinge wrote:
> 
> [1  ]
> Hello!
> 
> On Sun, Jan 03, 2010 at 04:56:54PM +0100, Samuel Thibault wrote:
> > Thomas Schwinge, le Sun 03 Jan 2010 16:48:00 +0100, a écrit :
> > > We will need it as soon as we switch from the Mach disk device
> > > handling to something more modern (DDE).
> > 
> > It can also be useful for the Xen case, as I haven't plugged the xen
> > device drivers to any partition support.
> 
> Heh, alright.  I have only ever been using this layout for all the Xen
> GNU/Hurd systems: hd0 -- root, hd1 -- swap, hd2 -- data.
> 
> 
> Issue for that:
> 
> 
> 
> Neal, anything specific that we should know /take care of, about your
> 2001, 2002 code: libparted backend for libstore?

Unfortunately, I don't really remember much about that code.  The
motivation was to be able to evict the partitioning logic from Mach.

Neal

Re: [Patch 0/2] Proxy memory objects revisited

[Neal H. Walfield]

This looks like a fine addition.  One could imagine more complex
schemes, which enable remapping (think scatter/gather type
mechanisms), however, I don't think that this is generally useful.
Further such additional functionality would significantly increase the
complexity whereas your proposal is a very simple addition, which adds
very useful functionality.

At Mon, 18 May 2009 07:44:54 +0200,
 wrote:
> I'm not entirely sure about the
> "size" one however: It works by mapping only as many pages as the proxy
> object allows, even if the client requested more. This does serve the
> purpose for me (prevent the client from getting access to regions it
> isn't allowed to access); and it is in a way equivalent to what the
> original patch does about prot: It provides a map trimmed to what the
> client is actually allowed. I wonder though whether creating a mapping
> with less pages than the client expected isn't problematic. What do you
> think?

A proxy object narrows a view (and thus access) of the object it
proxies.  If the underlying object consists of 100 pages and the proxy
object is set up to allow the user access to just 10, then a client
should only be able to access those 10.  Moreover, the client should
not be able to distinguish whether a proxy object is a normal memory
object or proxied (i.e., the proxying should be transparent).

Given this, if a client tries to map more pages than are accessible by
a proxy object, it should be equivalent to what happens when a client
tries to map more pages than are accessible by a memory object.  I'd
guess that the kernel would return KERN_INVALID_ARGUMENT but this
needs to be confirmed.


Also, I don't think that there is any need to split the offset and
length patches up.  Really, if one wants one addition, then the other
is likely also needed.

Neal

Re: Moving to git

[Neal H. Walfield]

> Here is one additional topic I want to confirm with you all before
> committing it: the duplication of ChangeLog snippets and commit log
> messages is a pain.  However, it is not mandatory to maintain ChangeLog
> files in the VCS sources -- it's fine with the GNU Coding Standards to
> only create ChangeLog files for distribution tarballs (where one doesn't
> have access to the VCS logs anymore).  GNU Coreutils is already using
> this scheme, and I got it confirmed on the GNU-internal maintainers'
> mailing list that this is indeed fine.  When needed, the ChangeLog files
> would be created automatically from the VCS commit messages.  (We can
> steal all the mechanisms from GNU Coreutils.)

> Any objections?

I support this.

Re: Moving to git

[Neal H. Walfield]

At Sun, 11 Jan 2009 12:50:58 +0100,
Thomas Schwinge wrote:
> > > Rationale: split as far as it's still making sense.  There is no
> > > reason to have an interger hashing library, a pthread
> > > implementation, an ext2 file system interpreter, libc amendments,
> > > Hurd interfaces definition files, a library for providing an
> > > uniform interface to Mach ports, etc. in the same repository.
> > 
> > Is there a reason to keep them seperate?...
> 
> Yes.  The simple rule of manageable complexity.  Or why are there
> separate repositories for GCC, Emacs, the Linux kernel, X.org, and the
> Intel math lib for IA64?  Is it, at least in theory, more easy to find a
> maintainer for libtrivfs or for the whole set of Hurd libraries and
> stuff?

The ihash library in the main Hurd repo and the ihash library in
Viengoos have significantly diverged.

Whereas e.g. libnetfs is only going to run on something resembling the
Hurd, libpthread really tries to be OS independent.

> Why not?  I see this as one main advantage: I'd be much more comfortable
> with releasing version 0.4 of libnetfs instead of releasing version 0.4
> of the GNU Hurd.  Or finding a maintainer for libnetfs.

Not doing lock-step releases means increasing the configuration space.
I'm not sure this is really helpful.


I'm not yet convinced that splitting the "Hurd" libraries into their
own repositories is the right approach.  It seems to me that these
libraries are tightly coupled and essential to core services thereby
requiring high degrees of coordination anyways.  In which case, why
not keep them all together?

Neal

resource management that involve applications

[Neal H. Walfield]

At Tue, 23 Dec 2008 12:19:26 +0100,
Michal Suchanek wrote:
> I would like to see a design of a system where resource management
> works at least in theory by involving the applications. It sounds like
> a good idea but protecting from malfunction of the involved
> applications is somewhat challenging.

This is exactly what I am working on.  You're welcome to take a look
at the recent paper I've submitted to EuroSys 2009:

  
http://walfield.org/papers/2009-walfield-viengoos-a-framework-for-stakeholder-directed-resource-allocation.pdf

This paper covers how to provide availability information to
application to allow them to more intelligently and thus more
aggressively adapt.

I'm working on a second paper on how applications can provide
information to the resource manager, in particular, relative
priorities and discardability information.  A rough draft of that
paper can be found here:

  http://plato.walfield.org/application-specific-knowledge.pdf

My thesis (still a work in progress) goes into more detail:

  http://plato.walfield.org/viengoos-thesis-draft.pdf

Neal

Re: master's thesis on hurd

[Neal H. Walfield]

At Thu, 27 Nov 2008 22:58:07 +0100,
Ruben Pollan wrote:
> I'm student of computer engineering on spain. I'm about to finish my degree (5
> years of study). I have to do a master's thesis, I'm interesting on hurd and I
> think will be a good idea do something about hurd.
> 
> I don't have much experience on hurd, I just have user knowledge using a 
> Debian 
> GNU/Hurd. I have some knowledge programming on C, and about operative systems.
> 
> Can you give me some ideas about things that have to be done on hurd?
> 
> Thanks, and sorry if this is not the appropriate list.

I suggest you start by reading these two papers:

  A Critique of the GNU Hurd Multi-server Operating System
  http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf

  Improving Usability via Access Decomposition and Policy Refinement
  
http://walfield.org/papers/20070104-walfield-access-decomposition-policy-refinement.pdf

They highlight some of the problems with the Hurd and its underlying
design.  Specifically, we identify three general issues: resource
management, security and naming.  I'm doing work on the first in the
context of Viengoos (but it is designed to form a basis for the other
issues as well).  For an overview of resource management in Viengoos,
please read this paper:

  Viengoos: A Framework for Stakeholder-Directed Resource Allocation.
  
http://walfield.org/papers/2009-walfield-viengoos-a-framework-for-stakeholder-directed-resource-allocation.pdf

There are number of interesting projects both on the current version
of the Hurd as well as on as Viengoos.  When you have a better idea of
what sort of issues you want to tackle, feel free to write again.

Neal

Re: [PATCH] ipc/mach_port.c compiler warning fix.

[Neal H. Walfield]

Thanks.  Can you please supply a GNU change log?

At Tue, 4 Nov 2008 19:06:07 +0530,
Shakthi Kannan wrote:
> 
> Fix ipc/mach_port.c compiler warning.
> 
> ---
>  ipc/mach_port.c |3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/ipc/mach_port.c b/ipc/mach_port.c
> index 63c25dc..0d78714 100644
> --- a/ipc/mach_port.c
> +++ b/ipc/mach_port.c
> @@ -1290,7 +1290,8 @@ mach_port_insert_right(
>   if (!IO_VALID((ipc_object_t)poly))
>   return KERN_INVALID_CAPABILITY;
> 
> - return ipc_object_copyout_name(space, poly, polyPoly, FALSE, name);
> + return ipc_object_copyout_name(space, (ipc_object_t)poly,
^

According to the GNU coding standards, there needs to be a space
between a case and a variable.

Re: EQP

[Neal H. Walfield]

Hi, Jason,

> I see multiprogramming as bad as any real sense of
> time is lost and all the problems of locking and synchronization arise.

How do you deal with the following scenario:

Consider a file server: it must handle multiple simultaneous
requests; it has shared meta-data needs to be updated; and, that data
may need to be paged in.

Thanks,
Neal

Re: EQP

[Neal H. Walfield]

At Thu, 30 Oct 2008 20:56:31 +,
Jason Cozens wrote:
> My main point is that processors should not be kept busy when it leads
> to a bad programming model. This bad programming model as I see it is
> multiprogramming. 

What is different about EQP that it, unlike multiprogramming, results
in a good programming model?

> I see multiprogramming as bad as any real sense of
> time is lost and all the problems of locking and synchronization arise.

A multiprogramming environment does not necessitate locking.  There
are alternatives.  For instance, data structures can be updated using
lock-free techniques, e.g., as implemented by a transactional memory.
These models avoid the big difficulties with locking--ordering and
nesting--and significantly decrease complexity.

I don't see how your model improves upon the situation; it sounds like
your system advocates an MPI / OpenMP programming style.  There is a
lock of synchronization that goes on in these types of systems.

> A processor can be programmed as a real time device and as such
> processors have been used in safety critical applications. Most car
> brake systems have at least two microprocessors for example.

> The sort of thing I'm looking at are things such as the CM-X300. These
> are quite expensive at the moment (in small quantities) about 61
> dollars but only by two orders of magnitude. See:
> 
> http://www.compulab.co.il/all-products/html/products.htm

Providing more processors with less power per unit increases
complexity: an interactive application what used to take 2 seconds to,
e.g., decompress and scale a high resolution image, will now take
significantly longer.  To decrease the time, this operation needs to
be parallelized.  This makes the code more complex and introduces
synchronization issues.


In summary, I'd like to know what you are trying to achieve and how
EQP help to reach that goal.

Thanks,
Neal

EQP

[Neal H. Walfield]

Hi Jason,

It seems like you've put a lot of thought into this and come up with
some good ideas.  Thanks for sharing them with us!

At Wed, 29 Oct 2008 21:01:33 +,
Jason Cozens wrote:
> A basic assumption of most OS design that processors have to be kept
> busy is questioned

Can you clarify what you mean by this?  Assuming there are no adaptive
applications, a good rule of thumb is, if there is work to do and
there are available resources, use them.  When there are adaptive
applications, an adaptation should be executed if its benefit exceeds
its cost.  (The primary benefits of not using resources are saving
power and, resource use costs, e.g., for portable devices, network
costs.)  In your opinion, when should processors not be kept busy?

> and instead it is assumed that there is an abundance
> of very simple processors (1000+) and communication is cheap.

Commodity systems with 1000+ processors are, I suspect, more than 10
years off.  Current thinking is that many-core systems will be
heterogeneous, not homogeneous.  Further, they will look like
distributed systems for which communication among nodes is not that
cheap.  Moreover, they will use NUMA architectures--if they have
shared memory at all.

Would you please sketch in a bit more detail what class of systems
your algorithm is targeting.

> The versatility of a general purpose operating system often comes at the
> cost of reliability and predictability largely stemming from the
> non-deterministic programming model that is a result of its design. The
> non-determinism arises from the multiprogramming model. The ordering of
> process execution is controlled by the operating system.

I'm missing a section describing how your approach enables a
deterministic programming model.  Also, I'm missing a section on how a
program is expected to interact with the scheduler.  I feel that I
need this information before I can provide useful comments on your
approach.

> The major problem is how to manage the pool of processors. This can be
> implemented by doing away with the centralised scheduler and letting
> each processor manage itself. The problem is now really one of resource
> allocation rather than scheduling.

What's the difference between scheduling and allocation?


Neal

Re: Regarding ddekit

[Neal H. Walfield]

At Fri, 24 Oct 2008 12:35:22 +0530,
Shakthi Kannan wrote:
> DDEKit + DDE26 is run in L4 with L4Env, L4IO modules in user-space.
> Moreover on top of DDE26, they use L4Env and L4 modules as server
> applications. But, as per the project idea here:
> http://www.bddebian.com/~wiki/community/gsoc/project_ideas/
> 
> we need to run DDEKit + DDE26 in GNU Mach. Then, we need to replace
> gnumach/glue with a new glue for DDEKit. But, eventually we would like
> them to be run in user-space? Appreciate any suggestions in this
> regard.

If you want to use DDEKit, you'll need to provide the interfaces that
it expects or port it to use some other interfaces.  As much as
possible, I'd suggest the former approach as it makes using futures
version of DDEKit easier.

Whether you provide the interfaces in the kernel or in user-space is
an orthogonal question.  The latter is safer.  Given the experience
using the glue code and Linux drivers in GNU Mach, this is a real
day-to-day problem.  On the other hand, providing the intefaces in the
kernel may make it easier to get a system initially up and running.

Neal

Re: current CVS libpthread makes trivial programs linked to it hang

[Neal H. Walfield]

At Sat, 27 Sep 2008 00:55:31 +0200,
Samuel Thibault wrote:
> There seems to be at least two issues:
> 
> if (_pthread_self ())
> 
> doesn't actually make sense since _pthread_self() already asserts(self),
> so the assertion will be triggered inside it.

This is true for the Hurd on Mach implementation of _pthread_self,
however, this is not the case for the Viengoos implementation.

> in the case of a normal mutex, now libpthread checks the owner when
> NDEBUG is not defined. The problem is that libc uses mutexes that have
> to be compatible with cthread mutexes, so we shouldn't use the owner
> field at all (since libc won't have allocated it).

You're right, I forgot about that.

I'll have to rethink this bit.

Neal

Re: Current state of Xorg on Hurd?

[Neal H. Walfield]

At Tue, 2 Sep 2008 10:15:41 +0200,
Arne Babenhauserheide wrote:
> * For Desktops: Are pthreads fully implemented, now? Can I get KDE or similar 
> running? :)

What do you mean?  With the exception of the real-time extensions and
a few bits that are hard to implement without tight integration with
libc (namely, at fork handlers), the pthread library has been feature
complete since 2002.

Neal

Re: What operations in store_parsed_open() need the privilege?

[Neal H. Walfield]

At Wed, 27 Aug 2008 15:05:59 -0700,
Thomas Bushnell BSG wrote:
> 
> On Wed, 2008-08-27 at 23:32 +0200, Da Zheng wrote:
> > I know boot fails and gets EPERM when it calls store_parsed_open, but I 
> > need to know what operations inside store_parsed_open() fail. Otherwise, 
> > I don't know how to fix it.
> 
> Boot assumes that it is run as root, and assumes that quite thoroughly.
> You need to have boot simply not even *try* to open such a device.

I don't understand why boot should somehow override the user in this
regard.  It is perfectly legitimate, I think, to give a non-root user
access to, e.g., /dev/hda1.  In that case, why should boot not even
try to open the device?

Re: [PATCH] Use standard constants in boot/

[Neal H. Walfield]

Why do you think this change is important?

Do you expect this to be applied?  If so, you need to include a change
log.

Re: Is it possible to replace the scheduler in a Hurd system?

[Neal H. Walfield]

At Tue, 26 Aug 2008 12:46:49 +0200,
Arne Babenhauserheide wrote:
> Since I just skimmed over scheduling, the question bit me, if itis possible 
> to 
> replace the scheduler in a Hurd system as user. 
> 
> Is it possible to replace the scheduler for my own processes? 
> 
> -> The general scheduler governing when which user may act, the user 
> schedulers governing the processes of that user? 
> 
> (or something similar)

On GNU Mach, this is not currently possible without changing the
implementation.  For ideas on how to implement such a system, check
out:

  CPU Inheritance Scheduling, Bryan Ford and Sai R. Susarla. OSDI
  1996.
  http://www.bford.info/pub/os/inherit-sched.pdf

and:

  Towards effective user-controlled scheduling for microkernel-based
  systems, Jan Stoess.  SIGOPS OSR, July 2007.
  http://portal.acm.org/citation.cfm?id=1278901.1278910

Neal

Re: "Microkernels rule!" and the Hurd

[Neal H. Walfield]

At Mon, 25 Aug 2008 07:53:05 +0200,
Arne Babenhauserheide wrote:
> Could you write a counter-story/public reply for the Hurd-Wiki? 
> 
> It needn't be long, but it should be available online. 

What are you looking for that I didn't say in my note?

Neal

Re: The patch of boot to open a virtual network interface

[Neal H. Walfield]

At Tue, 19 Aug 2008 17:37:14 +0200,
zhengda wrote:
> Since boot only uses one interface, boot probably accepts only one 
> interface name from subhurd.

This needn't be the case.  In fact, it should be able to expose any
number of objects as devices.  A good interface might be:

  -d foo=filename1 -d bar=filename2 ...

Where foo is hd0 or eth0 and filename is some file name interpreted in
the context of boot.

Neal

Re: The patch of boot to open a virtual network interface

[Neal H. Walfield]

At Tue, 19 Aug 2008 15:48:11 +0200,
zhengda wrote:
> 2008-07-29 Zheng Da <[EMAIL PROTECTED]>
> 
>   * boot/boot.c (ds_device_open): Handle the request to open the virtual 
> network device.
> 
> diff -u boot.old/boot.c boot/boot.c
> --- boot.old/boot.c   2008-08-17 18:38:02.0 +0200
> +++ boot/boot.c   2008-08-17 18:36:40.52000 +0200
> @@ -964,6 +964,22 @@
>*devicetype = MACH_MSG_TYPE_MAKE_SEND;
>return 0;
>  }
> +  else if (strncmp (name, "veth", 4) == 0)
> +{
> +  char buf[128];
> +  mach_port_t net_device;
> +
> +  snprintf (buf, sizeof(buf), "/dev/%s", name);
> +  net_device = file_name_lookup (buf, 0, 0);
> +  if (net_device == MACH_PORT_NULL)
> + {
> +   error (0, errno, "file_name_lookup");
> +   return errno;
> + }
> +
> +  *devicetype = MACH_MSG_TYPE_MOVE_SEND;
> +  return device_open (net_device, mode, "eth", device);
> +}
>  
>*devicetype = MACH_MSG_TYPE_MOVE_SEND;
>return device_open (master_device_port, mode, name, device);

I don't like this approach.  This allows the subhurd to access any
object in the parent hurd that boot can access.  Boot should take an
option indicating which device to use for the network interface and
only make that node available.  Moreover, you should not assume that
it is in /dev: a normal use could implement a proxy translator that
implements the network interface.

Neal

Re: The patch of glibc which allows the user to override the pfinet server

[Neal H. Walfield]

At Wed, 13 Aug 2008 12:41:12 +0200,
zhengda wrote:
> >> +  if (__asprintf (&name, "SOCK_SERV_%d", domain) > 0)
> >> +{
> >> +  np = getenv (name);
> >> +  __free (name);
> >> +}
> >> 
> >
> > You need to check whether asprintf() returns an error code (-1), meaning
> > memory allocation failed, and handle it accordingly -- just like you
> > handled failure with the explicit malloc() in your original code.
> >   
> as I understand from the manual of asprintf, asprintf always returns -1 
> if there is some error.
> 
> RETURN VALUE
>When  successful,  these  functions return the number of bytes 
> printed,
>just like sprintf(3).  If memory allocation wasn't  possible,  
> or  some
>other error occurs, these functions will return -1, and the 
> contents of
>strp is undefined.

Check if the result is != -1, not > 0, which has a different meaning.

> > If asprintf() doesn't fail, the getenv() should be done unconditionally.
> >
> >   
> >> +  if (__asprintf (&name, "%s/%d", sock_servs, domain) > 0)
> >> +  np = name;

Here too.

"Microkernels rule!" and the Hurd

[Neal H. Walfield]

Gernot Heiser recently wrote an article, "Microkernels rule!" for
Embedded.com about microkernels' bad reputation.  I fully agree with
the message of his article: operating systems based on microkernel
technology don't necessarily have to be slow and can be made more
robust than their monolithic counterparts.  However, Gernot mentions
the Hurd and incorrectly describes its position in history:

  Mach, an OS that was widely used as the basis of systems, ran into
  serious performance problems...  There were spectacular failures,
  none more so than IBM's Workplace OS, which cost the company a cool
  two gigabucks...

  Needless to say, the experience with Mach and others created a bit
  of an image problem for microkernels (which didn't stop the GNU Hurd
  from repeating the mistakes of the past).  However, back in 1993,
  Jochen Liedtke demonstrated that these performance problems weren't
  inherent in the microkernel concept.

The Hurd did not repeat the errors of past.  Work on the Hurd started
in 1990.  In GNU's Bulletin January, 1994 [1], you'll find an article
detailing the Hurd's architecture.  Workplace OS was conceived in
1991; it was deemed a failure around 1995.  Jochen's article
"Improving IPC by Kernel Design" was published in December 1993.

Regarding, Workplace OS, its main goals were: machine independence,
multiple personalities, and concurrent operation of personalities [3].
The last two goals, as far as I am aware, were never a priority in the
development of the Hurd.  Further, Workplace OS had already adopted
many second generation microkernel features, for instance, L3's
synchronous IPC.  In the major "Observations and Lessons" section of
[3], this is not even mentioned; management, coordination, and focus
are cited as the major problems.

Finally, the architectural problems that we have identified with Mach
[4] are not related to IPC.  The most important are the lack of
resource accounting, and the bad resource management (paging
decisions).  Regarding the implementation that we use, the major
problems are unoptimized code (e.g., when evicting a bunch of pages,
Mach always sends them one at a time to the manager), and the decades
old code base which was designed for machines with tens of megabytes
of RAM.

Neal


[0] http://www.embedded.com/columns/guest/208800243

[1] http://www.gnu.org/bulletins/bull16.html#SEC13

[2] Improving IPC by kernel design
Jochen Liedtke
SOSP December 1993
http://portal.acm.org/citation.cfm?id=173668.168633

[3] Workplace Microkernel and OS: A Case Study
Brett D. Fleisch and Mark Allan A. Co
Software--Practice and Experience
May 1998
http://portal.acm.org/citation.cfm?id=279869.279875

[4] A Critique of the GNU Hurd Multi-server Operating System
Neal H. Walfield and Marcus Brinkmann
ACM SIGOPS Operating Systems Review
July 2007
http://portal.acm.org/citation.cfm?id=1278901.1278907

Re: The patch of glibc which allows the user to override the pfinet server

[Neal H. Walfield]

At Wed, 25 Jun 2008 16:44:51 +0200,
zhengda wrote:
> I wonder if it's really necessary to override a single pf server.

When porting pppd, I did this regularly; I used an alternative pfinet
server but continued to use the default pipe server.  I don't see why
you think it would be normal to want to override all of these.  Even
in your specific case of virtualizing the network, I suspect you only
want to override pfinet.

> If more network protocols are implemented, we have to provide more 
> environment variables for the servers.

Or you use an environment variable that is based on a stem and the
protocol number.  So instead of SOCK_INET_SERV_PATH,
SOCK_SERV_%d_PATH.  Perhaps it is best to support both.  The number of
protocol is not increasing at a terribly fast rate.

> The GNU coding standard is a bit long, so I'm not sure if my code meets 
> all GNU standards.

This is better.

Re: The patch of glibc which allows the user to override the pfinet server

[Neal H. Walfield]

This works in the case where you want to override all pf servers.
This case is important.  Also important is the ability to override a
single pf server in a similar manner.

A couple comments on the code: please follow the GNU coding standards.
Second, I assume that you left the #if 0 in because you are requesting
review.  Make sure that you remove this when you really intend to
submit the patch.  Finally, when that happens, make sure to include a
change log entry.

Neal

Re: client-side memory buffers

[Neal H. Walfield]

At Tue, 1 Apr 2008 18:01:25 -0600,
Joshua Stratton wrote:
> On Tue, Apr 1, 2008 at 3:50 PM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:
> 
> > Please don't top post.
> >
> > At Tue, 1 Apr 2008 10:48:02 -0600,
> > Joshua Stratton wrote:
> > >
> > > The problem you described was the client owning the memory object,
> > sending
> > > it to the server, and the server having the ability to unmap the memory
> > > because it has ownership, if I understand correctly.
> >
> > No.  The client has the ability to DoS the server because it manages
> > the memory object.
> 
> 
> What exactly is the difference between manages and owns?

I was using ownership as a synonym for accounted to, and manage as a
synonym for being able to control (e.g., schedule).  So if the server
is accounted the memory but the client can control the memory, then
the server is susceptible to destructive interference from the client.

> Do you think the client-side
> memory model is worthwhile?  And would the server allocating the memory
> passing it to the client using the Mach semantics allow this client-side
> memory model while avoiding the ability for clients to unmap the
> data?

Yes, I think such accounting is worthwhile, it is what I am doing with
Viengoos, however, I question the ability to realize it using Mach's
interfaces.

Neal

Re: client-side memory buffers

[Neal H. Walfield]

Please don't top post.

At Tue, 1 Apr 2008 10:48:02 -0600,
Joshua Stratton wrote:
> 
> The problem you described was the client owning the memory object, sending
> it to the server, and the server having the ability to unmap the memory
> because it has ownership, if I understand correctly.

No.  The client has the ability to DoS the server because it manages
the memory object.

>  I assumed that a lock
> was built into the system to prevent this, but I was wondering if this
> weren't the case, the client could give the ownership to the server before
> the server does any operations so the client could not unmap the memory
> object.  The server would then give the ownership back to the client after
> the operation is complete such that the client couldn't unmap the memory
> while the server is using it, and in the default state the client would have
> the responsibility of the memory block (which would help the denial of
> service inside the network stack).

If the server owns the memory, that means it is account to the
server.  In which case, why not just let the server allocate it?

Neal

Re: client-side memory buffers

[Neal H. Walfield]

At Tue, 1 Apr 2008 08:11:30 -0600,
Joshua Stratton wrote:
> On Tue, Apr 1, 2008 at 2:28 AM, Neal H. Walfield <[EMAIL PROTECTED]> wrote:
> > The problem is exactly the same as that with L4's data spaces.  When
> > the server maps and accesses the memory object, the client can revoke
> > the mapping at any time (via memory_object_lock_request), causing the
> > server to fault.  If you manage to unmap the memory while the server
> > is blocked on it (waiting for it to be paged in) and has a lock,
> > you've successfully created a denial of service.
> 
> 
> Okay, so it's a bad idea, for example, to juggle ownership of the memory
> object so the client cannot unmap while the server is operating on it?

I don't understand your example.

Neal

Re: client-side memory buffers

[Neal H. Walfield]

At Mon, 31 Mar 2008 21:23:41 -0600,
Joshua Stratton wrote:
> 
> I was on the irc channel talking about the feasibility using client-side
> memory buffers for a new network stack.  Based on some feedback about
> difficulties of implementing this in the Hurd, I thought I would ask anyone
> if they thought this would be especially difficult--particularly Marcus and
> Neal.

The way to do client side allocations is to have the client pass a
memory object to the server.  There is a problem with this approach,
however, as it allows the client to interfere with the server's
operations.

The problem is exactly the same as that with L4's data spaces.  When
the server maps and accesses the memory object, the client can revoke
the mapping at any time (via memory_object_lock_request), causing the
server to fault.  If you manage to unmap the memory while the server
is blocked on it (waiting for it to be paged in) and has a lock,
you've successfully created a denial of service.

Without changing the Mach interfaces, the best approach is to have the
server allocate the memory and to transfer it to the client using the
normal Mach buffer passing semantics.

Neal

[task #7895] cthread implementation in terms of pthread interface

[Neal H. Walfield]

URL:
  

 Summary: cthread implementation in terms of pthread
interface
 Project: The GNU Hurd
Submitted by: neal
Submitted on: Thursday 03/27/2008 at 15:26
Category: The GNU Hurd
 Should Start On: Thursday 03/27/2008 at 00:00
   Should be Finished on: Thursday 03/27/2008 at 00:00
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
 Planned Release: None
  Effort: 0.00
Wiki-like text discussion box: 

___

Details:

The Hurd servers and libraries use the cthread threading interface.  The
cthread interface is rather simple and could be implemented in terms of
libpthread functions.  A port would not have to implement the entirety of the
cthread interface, only those functions that the Hurd actually uses.

The advantage to this is that it would then be possible to use programs that
depend on libpthread with Hurd libraries.  For instance, fuse.




___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/

Re: Thread model

[Neal H. Walfield]

At Wed, 19 Mar 2008 18:35:56 -0400,
Thomas Bushnell BSG wrote:
> On Wed, 2008-03-19 at 17:56 +0100, Neal H. Walfield wrote:
> > At Wed, 19 Mar 2008 09:58:57 -0400,
> > Thomas Bushnell BSG wrote:
> > > And throwing a big wrinkle into all that is that many architectures do
> > > not make it *possible* for users to handle page faults.  The processor
> > > dumps a load of crap on the stack, and the kernel must preserve it
> > > carefully and then return the fault.  It is very hard to encapsulate
> > > that so that it can be stored and restored by users without keeping the
> > > whole stack around.
> > 
> > L4 was ported to a large number of architectures; it can't be that
> > hard.
> 
> Sparc or i386?

I don't understand the question.  [1] shows the architectures that
Pistachio currently supports.

According to [2], there was at least once a Sparc port in the making.

Neal


[1] http://l4ka.org/projects/pistachio/
[2] https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2004-August/001020.html

Re: Thread model

[Neal H. Walfield]

At Wed, 19 Mar 2008 09:58:57 -0400,
Thomas Bushnell BSG wrote:
> And throwing a big wrinkle into all that is that many architectures do
> not make it *possible* for users to handle page faults.  The processor
> dumps a load of crap on the stack, and the kernel must preserve it
> carefully and then return the fault.  It is very hard to encapsulate
> that so that it can be stored and restored by users without keeping the
> whole stack around.

L4 was ported to a large number of architectures; it can't be that
hard.

Re: Thread model

[Neal H. Walfield]

At Sun, 16 Mar 2008 08:01:22 +0100,
<[EMAIL PROTECTED]> wrote:
> On Tue, Mar 11, 2008 at 12:10:17PM +0100, Neal H. Walfield wrote:
> 
> > > using some kind of continuation mechanism: Have a limited number of
> > > threads (ideally one per CPU) handle incoming requests.  Whenever
> > > some operation would require blocking for some event (in the case of
> > > diskfs, waiting for the underlying store to finish reading/writing),
> > > the state is instead saved to some list of outstanding operations,
> > > and the thread goes on handling other requests. Only when the event
> > > completes, we read the state back and continue handling the original
> > > request.
> > 
> > What you are suggesting is essentially using a user-level thread
> > package.  (Compacting a thread's state in the form of a closure is a
> > nice optimization, but the model essentially remains the same.)  The
> > main advantage to user-level thread package is that the thread memory
> > is pagable and is thus less likely to exhaust the sparser kernel
> > resources.  In the end, however, it suffers from the same problems as
> > the current approach.
> 
> I don't agree it's equivalent -- not in any meaningful sense.

You need kernel memory for the memory maps, at least one for each user
thread.

Re: Thread model

[Neal H. Walfield]

At Sun, 16 Mar 2008 08:21:19 +0100,
<[EMAIL PROTECTED]> wrote:
> On Wed, Mar 12, 2008 at 05:12:03PM +0100, Marcus Brinkmann wrote:
> 
> > As for the threading model, more than one kernel thread per real CPU
> > doesn't seem to make much sense in most cases.
> 
> Well, add a "per processing step" to make this statement more generally
> true. In some cases, it's useful to have a distinct set of worker
> threads for each processing step, working in a assemly line like fashion
> each thread picks a request from one wait queue, does it's work, and
> pushes it to the next wait queue.
> 
> This model specifically should improve performance for servers that are
> constantly busy, processing many requests in parallel; and under the
> condition that the amount of data touched in the processing is
> relatively small compared to the amount of code.
> 
> It also simplifies control flow and locking. Certain optimisations
> become obvious.

What sort of optimizations?  It seems to me that if you have a small
amount of data, then you'll be able to run almostly entirely out of
the L2 cache.  In this case, switching to another thread will
introduce the overhead of shifting the cache to the other CPU.

Re: Thread model

[Neal H. Walfield]

At Wed, 12 Mar 2008 15:32:26 -0400,
Thomas Bushnell BSG wrote:
> On Tue, 2008-03-11 at 12:10 +0100, Neal H. Walfield wrote:
> > What you are suggesting is essentially using a user-level thread
> > package.  (Compacting a thread's state in the form of a closure is a
> > nice optimization, but the model essentially remains the same.)  The
> > main advantage to user-level thread package is that the thread memory
> > is pagable and is thus less likely to exhaust the sparser kernel
> > resources.  In the end, however, it suffers from the same problems as
> > the current approach.
> 
> Cthreads does this.

Does what?  The cthread implementation found in hurd/libthreads uses
one kernel thread per user thread.

> Still, the issue is the thread stacks; even if you are using multiplexed
> user threads, each user thread still needs a user stack.

Yes, that is what I tried to say, sorry if it wasn't clear.  My
suggestion was to use one thread per CPU and to make methods
restartable so that should an RPC ever need to wait for a resource, we
just add the message buffer to the object's wait queue (perhaps using
push-back, e.g., reply to client: try message again using this
capability, which identifies the wait queue).  When the object becomes
unblocked, the message is requeued.

Neal

Re: Thread model

[Neal H. Walfield]

Olaf,

> The real solution here of course is to fix the thread model

I fully agree that given Mach's architecture, one kernel thread per
extant RPC is the wrong approach.

> using some kind of continuation mechanism: Have a limited number of
> threads (ideally one per CPU) handle incoming requests.  Whenever
> some operation would require blocking for some event (in the case of
> diskfs, waiting for the underlying store to finish reading/writing),
> the state is instead saved to some list of outstanding operations,
> and the thread goes on handling other requests. Only when the event
> completes, we read the state back and continue handling the original
> request.

What you are suggesting is essentially using a user-level thread
package.  (Compacting a thread's state in the form of a closure is a
nice optimization, but the model essentially remains the same.)  The
main advantage to user-level thread package is that the thread memory
is pagable and is thus less likely to exhaust the sparser kernel
resources.  In the end, however, it suffers from the same problems as
the current approach.

The approach that I am taking on Viengoos is to expose an interface
that is atomic and restartable.  (This is explored by Ford et al. in
[1].)  The basic design principle that I have adopted is that object
methods should be designed such that the server can obtain all
resources it needs before making any changes to visible state.  In
this way, the server knows that once the operation starts, it will
complete atomically.  When the server fails to obtain the resources,
it frees all the resources it obtained so far and queues the message
buffer on the blocked resource.  When the blocked resource becomes
free, the message buffer is placed on the incoming queue and handled
as if it just arrived.  The result is that no intermediate state is
required!

There are some cases where it is not easy to obtain all the required
resources upfront.  (In particular, I am thinking of activity
destruction in which all resources allocated to the activity must be
released.)  In such cases, I have tried to implement the method in
such a way that no intermediate state needs to be saved on interrupt.
(Considering again the case of activity destruction, I first mark the
activity as being dead thereby blocking the activity and then iterate
over each object and deallocating it.  If an object is not in memory,
e.g., on disk, I suspend.  Starting from the beginning is safe: we
just continue deallocating objects.)



An orthogonal concern is the use of locks.  An approach to reducing
their number is the use of lock-free data structures.  See Valois'
thesis for a starting point [2].

Neal

[1] "Interface and Execution Models in the Fluke Kernel" by Bryan
Ford, Mike Hibler, Jay Lepreau, Roland McGrath and Patrick Tullmann.
http://www.bford.info/pub/os/atomic-osdi99.pdf

[2] ftp://ftp.cs.rpi.edu/pub/valoisj/thesis.ps.gz

Re: invalid pthread_t value?

[Neal H. Walfield]

At Sat, 22 Dec 2007 23:15:58 +0100,
Samuel Thibault wrote:
> The NSPR needs an invalid value for pthread_t, which the Hurd doesn't
> officially have. I had proposed them to use -1 (since that would mean
> that there are something like 2^32 threads running, which just can't
> be...), but they don't seem to have accepted this (the patch seems to
> have been turned into a #error instead... No actual mail feedback).
> 
> Since pthread_t is typedefed to int, maybe we can define
> PTHREAD_THREADS_MAX to INT_MAX to actually set an official limit (that
> limit exists anyway since above INT_MAX the comparisons in libpthread
> become bogus), and then tell them they can really use -1?

The implementation never returns a TID with a value of 0.  Is this
sufficient?  Or do you need the implementation to somehow reject this
invalid value?

Neal


___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: Entropy Patch (as an attachment)

[Neal H. Walfield]

At Mon, 6 Aug 2007 20:14:25 +0200,
Samuel Thibault wrote:
> 
> Hi,
> 
> Neal H. Walfield, le Mon 06 Aug 2007 19:56:30 +0200, a écrit :
> > > +#ifdef MACH_ENTROPY
> > > + /*  Let's grab the cylinder numbers for entropy */
> > > + entropy_putdata (prev, sizeof(io_req_t));
> > > +#endif 
> > 
> > sizeof will always give the same value.  Is this really what you want
> > to do here?
> 
> Mmm, he's giving a pointer and a size, isn't he?

You are right, sorry.  I was confused and thought the size was the
entropy.

Neal


___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: Entropy Patch (as an attachment)

[Neal H. Walfield]

> Index: device/blkio.c
> ===
> RCS file: /sources/hurd/gnumach/device/Attic/blkio.c,v
> retrieving revision 1.1.1.1
> diff -u -r1.1.1.1 blkio.c
> --- device/blkio.c25 Feb 1997 21:28:13 -  1.1.1.1
> +++ device/blkio.c6 Aug 2007 15:05:19 -
> @@ -36,7 +36,9 @@
>  #include 
>  #include 
>  
> -
> +#ifdef MACH_ENTROPY
> +#include 
> +#endif
>  
>  io_return_t block_io(strat, max_count, ior)
>   void(*strat)();
> @@ -149,6 +151,10 @@
>   do {
>   prev = next;
>   next = prev->io_next;
> +#ifdef MACH_ENTROPY
> + /*  Let's grab the cylinder numbers for entropy */
> + entropy_putdata (prev, sizeof(io_req_t));
> +#endif 

sizeof will always give the same value.  Is this really what you want
to do here?


>  cngetc()
>  {
>   if (cn_tab)
> - return ((*cn_tab->cn_getc)(cn_tab->cn_dev, 1));
> +#if defined(MACH_KERNEL) && defined(MACH_ENTROPY)
> + entropy_putchar (cn_tab->cn_dev);
> +#endif /* MACH_ENTROPY and MACH_ENTROPY */
> + return ((*cn_tab->cn_getc)(cn_tab->cn_dev, 1));

Why won't this crash?  cn_tab is null...

> +#ifdef MACH_ENTROPY
> + /* Kick some mouse data to the entropy driver */
> + entropy_putchar((buttonchanges + moved.mm_deltaX
> + + moved.mm_deltaY));
> +#endif

Please end sentences with periods followed by two spaces.


> +#ifdef MACH_ENTROPY
> +  /* Grab the packet for entropy purposes */
> +  entropy_putdata(ph + 1, skb->len - sizeof(struct ether_header));
> +#endif

This subtraction is completely useless as it doesn't change the amount
of entropy.  Just use skb->len.

> +/* Current read offset */
> +static int entropy_read_offset = 0;
> +
> +/* Current write offset */
> +static int entropy_write_offset = 0;
> +
> +/* If this device is already initalized */
> +static int entropy_init_done = 0;

No need to initialize static variables to 0; they are in the bss.

> +  /* Allocate memory*/
> +  err = device_read_alloc(ior, ior->io_count);
> +  if (err != KERN_SUCCESS) {
> +return err;
> +  }

This is not GCS conforming.



___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: Fixed Entropy Patch

[Neal H. Walfield]

At Sun, 5 Aug 2007 01:41:54 -0400 (EDT),
Michael Casadevall wrote:
> Index: device/cons.c
> ===
> RCS file: /sources/hurd/gnumach/device/Attic/cons.c,v
> retrieving revision 1.2.4.6
> diff -u -r1.2.4.6 cons.c
> - --- device/cons.c   13 Nov 2006 21:30:36 -  1.2.4.6
> +++ device/cons.c 5 Aug 2007 09:39:09 -
> @@ -44,6 +44,10 @@
>   #include 
>   #endif
> 
> +#ifdef MACH_ENTROPY
> +#include 
> +#endif
> +
>   static  int cn_inited = 0;
>   static  struct consdev *cn_tab = 0; /* physical console device info 
> */
>   #ifndef MACH_KERNEL
> @@ -230,8 +234,16 @@
>   cngetc()
>   {
>   if (cn_tab)
> - -   return ((*cn_tab->cn_getc)(cn_tab->cn_dev, 1));
> +#if defined(MACH_KERNEL) && defined(MACH_ENTROPY)
> + entropy_putchar (cn_tab->cn_dev);
> +#endif /* MACH_ENTROPY and MACH_ENTROPY */
> + return ((*cn_tab->cn_getc)(cn_tab->cn_dev, 1));

Since cn_tab is NULL, won't this cause a GPF?

> Index: linux/dev/include/linux/blk.h
> ===
> RCS file: /sources/hurd/gnumach/linux/dev/include/linux/Attic/blk.h,v
> retrieving revision 1.2
> diff -u -r1.2 blk.h
> - --- linux/dev/include/linux/blk.h   5 Apr 2001 06:39:21 -   1.2
> +++ linux/dev/include/linux/blk.h 5 Aug 2007 09:39:12 -
> @@ -90,7 +90,7 @@
>   #endif /* CONFIG_BLK_DEV_MD */
> 
>   extern void set_device_ro(kdev_t dev,int flag);
> - -void add_blkdev_randomness(int major);
> +extern void add_blkdev_randomness(int major);
> 
>   extern int floppy_init(void);
>   extern void rd_load(void);
> @@ -136,7 +136,11 @@
>   #define DEVICE_NR(device) (MINOR(device))
>   #define DEVICE_ON(device)
>   #define DEVICE_OFF(device)
> +
> +/* HACK */

Hacks should be marked with XXX or FIXME and should be accompanied by
a comment explaining why the following code is a hack.


I think you forgot to include entropy.h and entropy.c.

Neal


___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Hurd/L4 History

[Neal H. Walfield]

At Wed, 1 Aug 2007 18:30:20 +0200,
<[EMAIL PROTECTED]> wrote:
> On Wed, Jul 25, 2007 at 09:17:49PM -0400, Michael Casadevall wrote:
> 
> > Hurdng - the project of porting hurd translators to another
> > microkernel beside mach such as L4.
> 
> That is not fully correct. The original port to L4 was simply named
> Hurd/L4 -- which is exactly what it was: A port of the existing Hurd
> design to a different microkernel.

The idea of using L4 was proposed even earlier by Okuji.  He created
the l4-hurd mailing list in November 2000.  I don't think he got any
further than simply suggesting it as an alternative to Mach and doing
some reading.

I started the original Hurd/L4 port while I was at Karlsruhe in 2002.
My intention was to adapt the Hurd to exploit L4's concepts and
intended design patterns; it was not to simply provide a Mach
compatibility layer on top of L4.  When I left Karlsruhe, I no longer
had access to Pistachio as I was unwilling to sign an NDA.  Although
the specification was available, the Karlsruhe group only released
their code in May 2003 [1].  Around this time, Marcus began hacking on
Pistachio.  He created a relatively complete run-time.  I didn't
really become involved again until the second half of 2004, after I
complete by Bachelors degree.

> Later, when the original Hurd/L4 architects embraced Coyotos and
> suggested a total departure from the original Hurd design (and in fact,
> also from its goals...), still calling it "Hurd" no longer seemed
> appropriate -- so someone came up with ngHurd. (Oddly, most others
> picked that up as "Hurdng", which is ugly IMHO, and doesn't make any
> sense if you spell out the "Hurd" acronym...)

Before Marcus and I considered Coyotos, we had already rejected some
parts of the Hurd's design.  The resource management problems were
what prompted me to look at L4.  Also, some of the problems with
translators were already well-known to us.  (For a more detailed
description of the problems we have identified, see our paper in this
month's SIGOPS OSR [2].  We have also written a forward-looking
position paper [3].)

We visited Jonathan Shapiro at Hopkins in January 2006.  This resulted
in a number of discussions, some quite influential, and not always in
a way which aligned our position with that of Jonathan's.  This was
particularly true of a number of security issues.

Hurd-NG, as we originally called it, was an attempt to articulate the
system that we had come to envision in terms of interfaces and
description of the system's structure.  The new name was selected, if
I recall correctly, as it clearly wasn't the Hurd nor the Hurd based
on L4.

Neal

[1] https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2003-May/000345.html
[2] http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf
[3] 
http://walfield.org/papers/20070104-walfield-access-decomposition-policy-refinement.pdf



___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: [libchannel] Control procedures

[Neal H. Walfield]

I agree with Pierre, please provide some examples.

I suspect that if an object implements a wider interface, then the
added methods should be written up as a regular mig interface file.
That's what mig is there for: the dirty work.

As the new routines extend the old interface, just call the basic
demuxer from your extended demuxer and then installer your extended
demuxer in the server loop.  Look at how, e.g.,
ports_manage_port_operations_multithread is used.

Neal



___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: passive translators

[Neal H. Walfield]

At Thu, 21 Jun 2007 20:02:35 +0800,
Wei Shen wrote:
> 
> On 6/21/07, Neal H. Walfield <[EMAIL PROTECTED]> wrote:
> >
> > At Thu, 21 Jun 2007 15:37:49 +0800,
> > > Could you please give some explanation on "PFINETSERVER=fd:3 myprog
> > > 3 >
> > From the bash manual:
> >
> > 3.6.1 Redirecting Input
> >
> > Redirection of input causes the file whose name results from the
> > expansion of word to be opened for reading on file descriptor n, or
> > the standard input (file descriptor 0) if n is not specified.
> 
> 
> Ok, I got it. I will consider to support the file descriptor reprentation,
> but will not implement complex semantics as redirection first.

Redirection is a shell feature.  If you support the file descriptor
representation, then you can use the shell's redirection feature to
set up the file descriptors in the child process.  There is nothing
that you have to add.

> > (1) Assume a chroot process invokes "settrans -cp /foo /hurd/firmlink /".
> > >
> > > (2) The fs server will associate node /*foo* (/chroot/foo) with
> > translator *
> > > firmlink*. Since the fs server knows the quest is from a chroot process,
> > > it can save a chroot attribute "/chroot" (it is a string representation
> > of
> > > the chroot path, but not a handle or other expressions of capability),
> > in
> > > addition to the translator command "/hurd/firmlink /".
> >
> > The translator does not necessarily know a symbolic representation for
> > the capability passed to file_reparent.
> 
> 
> I can not understand. Sorry for keeping bothering you :-), but I expect to
> make it clear.
> Representation for which capability do you mean? The file node, parent, or
> target?
> Two ports (*node*, *parent*) and a string path (*target*_*name*) are passed
> to *firmlink*, and port *parent* and *target* are passed to* file_reparent*.

The translator does not necessarily know the symbolic representation
for any of the capabilities and it can't invoke them to "ask nicely".

> I proposed two solutions in last mail. In solution a), the translator is
> started chrooted. So the file node, parent, and target are respectively:
> /foo, /, and / in its eyes; and /chroot/foo, /chroot, and /chroot in
> reality.

What is reality?  These are only symbolic names: you need to establish
a naming context, and this is the root of the problem.

> I am not sure if this translator can still serve non-chrooted processes.
> Even if it can not, I do not think this is a problem, so long as it can work
> in the chroot domain.

We discuss an example where this causes problem in the critique
(think: client with a different global name space looks up .. at the
translator's root).

> In solution b), the translator is started as usual, but it will find a
> chroot argument is passed to it, so it should initiatively check the target
> path argument, and add /chroot in front of / to be /chroot/. Since the
> translator acts in the normal way, I think it should not be unable to
> understand the symbolic representation.

I think this will only introduce confusion: the translator has to be
very careful when it exposes symbolic names.

Neal


___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: passive translators

[Neal H. Walfield]

At Thu, 21 Jun 2007 15:37:49 +0800,
Wei Shen wrote:
> 
> Hi,
> 
> Could you please give some explanation on "PFINETSERVER=fd:3 myprog
> 3http://www.gnu.org/software/bash/manual/bashref.html#SEC38

So, as bash starts myprog, it opens /path/to/pfinet and inserts the
resulting file descriptor into the program as fd 3.

> (1) Assume a chroot process invokes "settrans -cp /foo /hurd/firmlink /".
> 
> (2) The fs server will associate node /*foo* (/chroot/foo) with translator *
> firmlink*. Since the fs server knows the quest is from a chroot process,
> it can save a chroot attribute "/chroot" (it is a string representation of
> the chroot path, but not a handle or other expressions of capability), in
> addition to the translator command "/hurd/firmlink /".

The translator does not necessarily know a symbolic representation for
the capability passed to file_reparent.

Neal


___
Bug-hurd mailing list
Bug-hurd@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-hurd

Re: Defualt socket server overriding

[Neal H. Walfield]

At Wed, 20 Jun 2007 10:30:02 -0700,
Thomas Bushnell BSG wrote:
> On Wed, 2007-06-20 at 16:20 +0200, Neal H. Walfield wrote:
> > When we are just interested in
> > overriding a small parts of the environment and the rest represents a
> > reasonable default, this may be fine.  Such an approach is, however,
> > completely contrary to POLP.  I think the right direction is private
> > name spaces, which can be achieved by passing capabilities.  That was
> > the other part of my suggestion.
> 
> Private namespaces are another excellent idea; they are perhaps harder
> to work into the current framework unless there's a trick I haven't
> thought of.

As I don't think you specifically addressed it, I'll repeat my main
suggestion.

One of the best private name spaces that we have, I think, is the
capability name space.  My proposal was to do the following:

  PFINETSERVER=fd:3 myprog 3http://lists.gnu.org/mailman/listinfo/bug-hurd