Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 23:15:59 +, wrote:
> > Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
> > there will be more to get it into a buildable state, e.g. the actual 
> > libpthread implementation).
> 
> What I have pushed is basically only missing the libpthread
> implementation, so you already have an idea of the minimal set of
> modifications to get something building (and IIRC essentially passing
> the testsuite).

I'd still like to have the libpthread implementation there (with a view to 
seeing if I can get build-many-glibcs.py working for Hurd with this branch 
- if the branch has sources that build for Hurd, I should have some chance 
of using 
https://git.savannah.gnu.org/cgit/hurd/incubator.git/tree/cross-gnu?h=cross-gnu/master
 
to figure out how to do the Hurd-specific pieces that will need adding to 
build-many-glibcs.py).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 23:15:59 +, wrote:
> Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
> there will be more to get it into a buildable state, e.g. the actual 
> libpthread implementation).

What I have pushed is basically only missing the libpthread
implementation, so you already have an idea of the minimal set of
modifications to get something building (and IIRC essentially passing
the testsuite).

> General observations on mainly coding style issues that should be cleaned 
> up for actual submission

Thanks for the comments!

Could bug-hurd@ help us with them?  There is a lot of non-technical
stuff there :)

> * Why are some symbols added in sysdeps/mach/hurd/bits/posix_opt.h with 
> value 200112L, when 200809L corresponds to the POSIX version generally 
> supported by glibc?

I guess simply because at the time it wasn't the case :)

Samuel

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> That's why I meant work is needed to sort out what we want to show
> people. Which takes triaging time, back to square 0. I sent to the
> bug-hurd list the list of patches which was enough at some point to get
> glibc buildable, that's what should be worked on and what I plan to
> commit to a single branch on the upstream repo.

Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
there will be more to get it into a buildable state, e.g. the actual 
libpthread implementation).

General observations on mainly coding style issues that should be cleaned 
up for actual submission (there are also obvious issues with changes to 
non-Hurd-specific files needing cleaning up / justifying, which I didn't 
list individually):

* Avoid __GNU__ conditionals in generic files (as per Roland's comments on 
the TLS patch).

* Use -2018 in copyright notices.

* There should be a descriptive comment on the first line of each file 
before the copyright notice.  Where you have such comments, make sure they 
don't say something is a generic version when actually it's a 
Hurd-specific version.

* Avoid FSF postal addresses in license notices; use a URL instead, as in 
existing files in glibc (using an https URL will avoid needing to be 
included in an http -> https fix for such URLs in future).

* No "Contributed by" notices in new files.

* verify.h should be LGPLv2.1+ for glibc use, but actually I think you 
should just use _Static_assert instead of verify.h, since the minimum 
compiler version for building glibc supports _Static_assert.

* sysdeps/mach/hurd/gai_misc.h needs a copyright / license notice (and is 
missing spaces before '(' in some places).

* In a few places, missing preprocessor indentation inside #if (as per 
,
 
should be used except for a multiple include guard round a whole file).

* Why are some symbols added in sysdeps/mach/hurd/bits/posix_opt.h with 
value 200112L, when 200809L corresponds to the POSIX version generally 
supported by glibc?

-- 
Joseph S. Myers
jos...@codesourcery.com

Hurd lecture

[Brent W. Baccala]

Hi -

FYI, I gave an hour and a half Hurd lecture yesterday at Catholic
University in Washington, D.C.

I put a screencast of the lecture on youtube:

https://www.youtube.com/watch?v=JwsuAEF2FYE

The first part of the video is a general discussion of high performance
computing, leading into my own work developing Hurd as a cluster operating
system.

The Mach/Hurd specific part starts at 43 minutes.

I developed some tikz macros to illustrate Mach/Hurd concepts using LaTeX.
For example, there's an illustration of Mach IPC at 51:20, the memory
object protocol at 1:09:30, and the netmsg server is illustrated at 1:16:25.

Others might find these macros useful for constructing similar
presentations.  If so, I've got the LaTeX source for the slides in my
github hurd repository here:

https://github.com/BrentBaccala/hurd/blob/master/slides/cua.tex

I might revise it a bit; the macros are a bit clunky.

A finished PDF is available here:

https://drive.google.com/open?id=1Wv5dZpbKP17-H1uUOb9hnnBvXBf2LHiL

Could we add that to the collection of presentations that we've got on the
documentation page?

agape
brent

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 16:47:55 +, wrote:
> On Thu, 18 Jan 2018, Samuel Thibault wrote:
> > Not all of them are necessary for managing to build glibc. Some of them
> > are just hacking, others are perhaps almost ready to commit, just
> > missing changelogs & formatting. That's the triaging thing that takes
> > time, and having to do all the work including changelogs & formatting
> > makes it get lower in my global TODOlist.
> 
> I'm still arguing on bug-standards for removing the requirement for 
> ChangeLog format (given a public distributed version control system), but 
> that wouldn't remove the requirement for proper explanations of patches, 

Thomas: this is were your help should be focused. Some of your branches
only have "WIP", which is not helpful to know what you were hacking when
writing them.

Samuel

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 16:47:55 +, wrote:
> On Thu, 18 Jan 2018, Samuel Thibault wrote:
> 
> > Joseph Myers, on jeu. 18 janv. 2018 15:34:56 +, wrote:
> > > On Thu, 18 Jan 2018, Samuel Thibault wrote:
> > > > Coding standards can be worked on by anybody, this is really something
> > > > that bug-hurd people can unload us from.
> > > 
> > > Which is also something that having a branch with the patches is helpful 
> > > for -
> > 
> > Well, we already have that on
> > 
> > git clone git.savannah.gnu.org:/srv/git/hurd/glibc.git/
> > 
> > with almost a hundred branches to be looked after...
> 
> A hundred branches with many different purposes is not something at all 
> convenient for glibc people to look over!

That's why I meant work is needed to sort out what we want to show
people. Which takes triaging time, back to square 0. I sent to the
bug-hurd list the list of patches which was enough at some point to get
glibc buildable, that's what should be worked on and what I plan to
commit to a single branch on the upstream repo.

Samuel

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 15:34:56 +, wrote:
> > On Thu, 18 Jan 2018, Samuel Thibault wrote:
> > > Coding standards can be worked on by anybody, this is really something
> > > that bug-hurd people can unload us from.
> > 
> > Which is also something that having a branch with the patches is helpful 
> > for -
> 
> Well, we already have that on
> 
> git clone git.savannah.gnu.org:/srv/git/hurd/glibc.git/
> 
> with almost a hundred branches to be looked after...

A hundred branches with many different purposes is not something at all 
convenient for glibc people to look over!

> Not all of them are necessary for managing to build glibc. Some of them
> are just hacking, others are perhaps almost ready to commit, just
> missing changelogs & formatting. That's the triaging thing that takes
> time, and having to do all the work including changelogs & formatting
> makes it get lower in my global TODOlist.

I'm still arguing on bug-standards for removing the requirement for 
ChangeLog format (given a public distributed version control system), but 
that wouldn't remove the requirement for proper explanations of patches, 
just mean that explanations intended to be read together with the patches 
themselves would suffice, without needing to duplicate the low-level 
details of how the patches change each affected named entity.

> > (A branch close to current master also provides a basis for anyone working 
> > on build-many-glibcs.py support for Hurd, if you don't already have such 
> > support among your glibc patches.)
> 
> That's why I suggested just committing the required patches to a branch
> that we rebase regularly, so there's a master that does build.

Yes, that's what I'd like to see.  From the perspective of people wanting 
to do global changes to glibc I think the priorities would be (a) building 
cleanly for Hurd; (b) building for Hurd with the intended ABI exported 
from each library at each symbol version; (c) having ABI test baselines to 
verify the ABI stays as expected; (d) having all the rest of the 
compilation parts of the testsuite passing.  Even given just (a) I might 
look at setting up build-many-glibcs.py support.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 15:34:56 +, wrote:
> On Thu, 18 Jan 2018, Samuel Thibault wrote:
> > Coding standards can be worked on by anybody, this is really something
> > that bug-hurd people can unload us from.
> 
> Which is also something that having a branch with the patches is helpful 
> for -

Well, we already have that on

git clone git.savannah.gnu.org:/srv/git/hurd/glibc.git/

with almost a hundred branches to be looked after...
Not all of them are necessary for managing to build glibc. Some of them
are just hacking, others are perhaps almost ready to commit, just
missing changelogs & formatting. That's the triaging thing that takes
time, and having to do all the work including changelogs & formatting
makes it get lower in my global TODOlist.

That repo is not on top of master, but on top of the oldest glibc used
by the few GNU/Hurd distros. That's currently 2.24, but we have branches
to fast-forward that to 2.25 and 2.26.

> (A branch close to current master also provides a basis for anyone working 
> on build-many-glibcs.py support for Hurd, if you don't already have such 
> support among your glibc patches.)

That's why I suggested just committing the required patches to a branch
that we rebase regularly, so there's a master that does build.

Samuel

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Coding standards can be worked on by anybody, this is really something
> that bug-hurd people can unload us from.

Which is also something that having a branch with the patches is helpful 
for - it allows glibc people to look over them and point out coding 
standards issues to be addressed globally by people less familiar with the 
conventions used in glibc.  (Though when those issues are addressed the 
fixes should be integrated into each patch as part of a rebase, rather 
than simply adding another patch to the branch that fixes coding standards 
issues with the previous patches.)

(A branch close to current master also provides a basis for anyone working 
on build-many-glibcs.py support for Hurd, if you don't already have such 
support among your glibc patches.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> > (I'd also suggest that it *not* use a top-level directory called
> > simply "libpthread" or similar without mention of Hurd, as that would
> > be too confusing to people looking at the source tree for pthreads for
> > other platforms.)
> 
> Sure :)
> 
> kFreeBSD/GNU used fbtl (FreeBSD Thread Library), perhaps htl? (Hurd
> Thread Library)

Seems reasonable to me.

Longer-term, once the support is in, it would be good to know how the API 
exposed for pthreads on Hurd compares to the NPTL API.  Right now, we have 
very little documentation for threads, but in principle it would be good 
to have thorough documentation for the pthreads API, and for the API to be 
the same between different OSes rather than having some interfaces only 
supported on some OSes.  (Then most of pthread.h could be shared between 
the different implementations - duplicating all the function declarations 
and associated feature test macro conditionals for them is not ideal.)

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Thomas Schwinge, on jeu. 18 janv. 2018 15:22:09 +0100, wrote:
> Despite my good intentions, in the past year (or even longer...) I failed
> to allocate sufficient time to do any substantial work on the Hurd side,
> but I'll now allocate time to at least help with...
> 
> On Thu, 18 Jan 2018 14:57:58 +0100, Samuel Thibault  
> wrote:
> > Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> > > All the usual coding standards of course apply,
> > 
> > And that's were I haven't received help so far.
> 
> ... that.  (Plus the GDB stuff I promised to look into.)

Well, cool, but I'd prefer that you spend your available time on the gdb
things and the *content* of the glibc patches which we need :)

(for some of the patches in mentioned in a separate mail, there is no
real explanation why they are needed, what should be done about them,
etc.)

Coding standards can be worked on by anybody, this is really something
that bug-hurd people can unload us from.

Samuel

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> (I'd also suggest that it *not* use a top-level directory called
> simply "libpthread" or similar without mention of Hurd, as that would
> be too confusing to people looking at the source tree for pthreads for
> other platforms.)

Sure :)

kFreeBSD/GNU used fbtl (FreeBSD Thread Library), perhaps htl? (Hurd
Thread Library)

Samuel

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Thomas Schwinge, on jeu. 18 janv. 2018 15:22:09 +0100, wrote:
> > I can pile-commit what we currently have, with all the XXXs, FIXMEs,
> > etc., if that can help the glibc side.
> 
> I'm not sure if that's a useful thing to spend time on, right now.  I
> think it'll be better if we first integrate into glibc upstream the
> changes that are non-controversial on our own side, which only touch Hurd
> code, etc.  Then, for the rest of the changes, those which also touch
> generic glibc code, we should then discuss these individually.

Note that I didn't mean all patches, just the ones needed for building
glibc.

Yes, the discussion has to happen, we can't fast-forward that.  But
putting the stuff on a branch, so people can have a look if they want to
and track the progress, can be useful.

Samuel

Re: Upstreaming the glibc Hurd port

[Thomas Schwinge]

Hi!

Despite my good intentions, in the past year (or even longer...) I failed
to allocate sufficient time to do any substantial work on the Hurd side,
but I'll now allocate time to at least help with...

On Thu, 18 Jan 2018 14:57:58 +0100, Samuel Thibault  
wrote:
> Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> > All the usual coding standards of course apply,
> 
> And that's were I haven't received help so far.

... that.  (Plus the GDB stuff I promised to look into.)

> I can pile-commit what we currently have, with all the XXXs, FIXMEs,
> etc., if that can help the glibc side.

I'm not sure if that's a useful thing to spend time on, right now.  I
think it'll be better if we first integrate into glibc upstream the
changes that are non-controversial on our own side, which only touch Hurd
code, etc.  Then, for the rest of the changes, those which also touch
generic glibc code, we should then discuss these individually.

> But the actual discussion of changes is what takes time, and bug-hurd
> people have to actually help me with it (or at least unload from me the
> coding standards and ChangeLog parts) if they want it to happen fast
> enough.

ACK.


Grüße
 Thomas

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Thu, 18 Jan 2018, Samuel Thibault wrote:

> > Before Hurd support is fully integrated in glibc, I'd encourage having a 
> > branch *in the glibc repository* that contains such support, so we can 
> > more readily see what the changes yet to be merged are (and possibly 
> > comment on issues that will need addressing when integrating them in 
> > glibc).
> 
> I can pile-commit what we currently have, with all the XXXs, FIXMEs,
> etc., if that can help the glibc side.

I think having such a branch (rebased from time to time, especially as 
Hurd changes make their way into master or get cleaned up for issues 
raised, so it doesn't fall too far behind master) would be helpful so we 
can see the changes and quite likely point out global issues to be 
addressed throughout.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 13:52:32 +, wrote:
> Incidentally, Samuel did post the Hurd TLS support three years ago 
> .  Roland's 
> review comments should of course be considered,

Yes, it has been on my TODO list for as long, hoping that perhaps
somebody could help with it, at least on the coding style part. That
hasn't materialized.

Samuel

Re: Upstreaming the glibc Hurd port

[Florian Weimer]

On 01/18/2018 01:45 PM, Samuel Thibault wrote:

Florian Weimer, on jeu. 18 janv. 2018 13:39:51 +0100, wrote:

Is it just a matter of someone doing the work of upstreaming all the
existing patches?


It's mostly that, yes.


Good to know, thanks.

The question is how we can do this incrementally (so that others can 
contribute), considering that is currently not exactly easy to set up 
and maintain a development environment to validate the changes.



If we cannot resolve this is in the coming months, I think we should
seriously consider a removal of the Hurd port, I'm afraid.


Ok, then I can take the time to do it, but as usual it'll be at the
expense of other Hurd activities, e.g. the obvious Meltdown/Spectre
issues, etc.


Information leakage and microkernels are certainly an interesting 
research topic.  Message passing results in very different security 
considerations.



Unless bug-hurd people actually help me with it.


I have repeatedly suggested that the FSF uses the toolchain fund to pay 
for this work.  I do not know there was under serious consideration (or 
if the suggestion even made it to the FSF).


Thanks,
Florian

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Joseph Myers, on jeu. 18 janv. 2018 13:47:42 +, wrote:
> Please note (as a reminder from past discussions) that the Hurd libpthread 
> will need to be included as part of glibc, much like NPTL is a 
> fully-integrated part of glibc - not a separate package (support for 
> add-ons has been removed from glibc).

Sure, I had to do that work for fixing the build of the
2.26.9000+20180108.401311cf-0experimental0 Debian snapshot

> Also as per previous discussions: Hurd port maintainers can put in changes 
> to Hurd-specific files at any time

Sure!

> All the usual coding standards of course apply,

And that's were I haven't received help so far.

> In my view, build-many-glibcs.py support for Hurd would be appropriate 
> even before the Hurd port builds (and certainly before it cleanly passes 
> the compilation tests).  That support will definitely need patch review.
> 
> Before Hurd support is fully integrated in glibc, I'd encourage having a 
> branch *in the glibc repository* that contains such support, so we can 
> more readily see what the changes yet to be merged are (and possibly 
> comment on issues that will need addressing when integrating them in 
> glibc).

I can pile-commit what we currently have, with all the XXXs, FIXMEs,
etc., if that can help the glibc side.

But the actual discussion of changes is what takes time, and bug-hurd
people have to actually help me with it (or at least unload from me the
coding standards and ChangeLog parts) if they want it to happen fast
enough.

Samuel

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Incidentally, Samuel did post the Hurd TLS support three years ago 
.  Roland's 
review comments should of course be considered, though the current Hurd 
port maintainers are free to decide they disagree with particular aspects 
of those comments.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Samuel Thibault, on jeu. 18 janv. 2018 14:40:37 +0100, wrote:
> I came up with the following list of patches

They are usually more or less named like the corresponding topgit branch
in our glibc repo. The .topmsg file can be use to track what work has
been done on the patch (checking formatting, changelog, etc.). Once a
topgit branch is known to be alright, I can commit it very quickly.

Samuel

Re: Upstreaming the glibc Hurd port

[Joseph Myers]

Please note (as a reminder from past discussions) that the Hurd libpthread 
will need to be included as part of glibc, much like NPTL is a 
fully-integrated part of glibc - not a separate package (support for 
add-ons has been removed from glibc).  (I'd also suggest that it *not* use 
a top-level directory called simply "libpthread" or similar without 
mention of Hurd, as that would be too confusing to people looking at the 
source tree for pthreads for other platforms.)

Also as per previous discussions: Hurd port maintainers can put in changes 
to Hurd-specific files at any time (regardless of release freeze state) 
until the port is actually working.  All the usual coding standards of 
course apply, and changes to files that might affect non-Hurd may need 
review and not be appropriate at some times, depending on freeze state and 
the risks associated with such patches.

In my view, build-many-glibcs.py support for Hurd would be appropriate 
even before the Hurd port builds (and certainly before it cleanly passes 
the compilation tests).  That support will definitely need patch review.

Before Hurd support is fully integrated in glibc, I'd encourage having a 
branch *in the glibc repository* that contains such support, so we can 
more readily see what the changes yet to be merged are (and possibly 
comment on issues that will need addressing when integrating them in 
glibc).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Samuel Thibault, on jeu. 18 janv. 2018 13:45:37 +0100, wrote:
> Unless bug-hurd people actually help me with it.

FI, in my previous glibc upstream build attemps, I came up with the
following list of patches which are required for upstream glibc to build
at all. Most of them don't have changelog, have formatting issues, etc.
That's what needs to be fixed until we can commit them. If people can
work on it, that'll increase my commit rate :)

Samuel

tg-fcntl-internal.h
tg-verify.h
tg-pagesize-2.23
tg-_dl_random
tg-gscope
unsubmitted-NO_HIDDEN.diff
atfork_hooks
tg-grantpt
tg-extern_inline
unsubmitted-libpthread-semaphore.h.diff
unsubmitted-timer_routines.diff
tg-libpthread_depends.diff
tg-posix_thread.diff
tg-no-hp-timing.diff
libpthread-versions.diff
unsubmitted-libpthread_build
unsubmitted-libc_alloca_cutoff
unsubmitted-gnumach.defs.diff
tg-gai_misc.diff
libpthread_build
local-libpthread_build
tg-tlsdesc
tg-tls-2.23

Re: Upstreaming the glibc Hurd port

[Samuel Thibault]

Florian Weimer, on jeu. 18 janv. 2018 13:39:51 +0100, wrote:
> Is it just a matter of someone doing the work of upstreaming all the
> existing patches?

It's mostly that, yes.

> If we cannot resolve this is in the coming months, I think we should
> seriously consider a removal of the Hurd port, I'm afraid.

Ok, then I can take the time to do it, but as usual it'll be at the
expense of other Hurd activities, e.g. the obvious Meltdown/Spectre
issues, etc.

Unless bug-hurd people actually help me with it.

Samuel