Re: [24/32] module mapper

[Jeff Law via Gcc-patches]

On 11/12/20 9:24 AM, Nathan Sidwell wrote:
> On 11/3/20 4:17 PM, Nathan Sidwell wrote:
>> this is the module mapper client and server pieces.  It features a
>> default resolver that can read a text file, or generate default
>> mappings from module name to cmi name.
>
> Richard rightly suggested on IRC that the sample server for the module
> mapper shouldn't be in the gcc/cp dir.  It happened to be that way
> because it started out much more closely coupled, but then it grew legs.
>
> So this patch creates a new c++tools toplevel directory and places the
> mapper-server and its default resolver there.  That means more changes
> to the toplevel Makefile.def and Makefile.tpl (I've not included the
> regenerated Makefile.in, nor other generated files in gcc/ and
> c++tools in this diff.)
>
> We still need to build the default resolver when building cc1plus, and
> I've placed mapper-resolver.cc there, as a simple #include forwarder
> to the source in c++tools.  I also replace 'gcc/cp/mapper.h' with a
> client-specific 'gcc/cp/mapper-client.h'.  (mapper-client is only
> linked into cc1plus, so gcc/cp seems the right place for it.)
>
> The sample server relies on gcc/version.o to pick up its version
> number, and I place it in the libexecsubdir that we place cc1plus.  I
> wasn't comfortable placing it in the install location of g++ itself. 
> I call it a sample server for a reason :)
>
> I will of course provide changelog when committing.
>
> nathan
>
>
> 24a-c++-mapper.diff
>


So I think you should just own these bits.  You're going to be far more
familiar with them than anyone else involved in GCC work :-)  So, OK for
the trunk as well as any followups into the module mapper.

jeff

Re: [24/32] module mapper

[Boris Kolpackov]

Nathan Sidwell  writes:

> These are needed as they also serve to inform the mapper of a dependency 
> edge.

Ok, that makes sense (and thanks for making it optional via a flag).

One thing that is still missing in this area is the dependency on
stdc-predef.h. In other words, we now can get everything via the
mapper as we can get with -M* except for this file.

Do you think it would be possible to report it with INCLUDE-TRANSLATE
(with the only valid response being BOOL FALSE)? There could also be
a flag to indicate that it's not really "translatable".

Re: [24/32] module mapper

[Nathan Sidwell]

On 11/3/20 4:17 PM, Nathan Sidwell wrote:
this is the module mapper client and server pieces.  It features a 
default resolver that can read a text file, or generate default mappings 
from module name to cmi name.


Richard rightly suggested on IRC that the sample server for the module 
mapper shouldn't be in the gcc/cp dir.  It happened to be that way 
because it started out much more closely coupled, but then it grew legs.


So this patch creates a new c++tools toplevel directory and places the 
mapper-server and its default resolver there.  That means more changes 
to the toplevel Makefile.def and Makefile.tpl (I've not included the 
regenerated Makefile.in, nor other generated files in gcc/ and c++tools 
in this diff.)


We still need to build the default resolver when building cc1plus, and 
I've placed mapper-resolver.cc there, as a simple #include forwarder to 
the source in c++tools.  I also replace 'gcc/cp/mapper.h' with a 
client-specific 'gcc/cp/mapper-client.h'.  (mapper-client is only linked 
into cc1plus, so gcc/cp seems the right place for it.)


The sample server relies on gcc/version.o to pick up its version number, 
and I place it in the libexecsubdir that we place cc1plus.  I wasn't 
comfortable placing it in the install location of g++ itself.  I call it 
a sample server for a reason :)


I will of course provide changelog when committing.

nathan

--
Nathan Sidwell
diff --git c/Makefile.def w/Makefile.def
index 36fd26b0367..6e98d2d3340 100644
--- c/Makefile.def
+++ w/Makefile.def
@@ -125,12 +134,13 @@ host_modules= { module= libtermcap; no_check=true;
 missing=distclean;
 missing=maintainer-clean; };
 host_modules= { module= utils; no_check=true; };
+host_modules= { module= c++tools; };
 host_modules= { module= gnattools; };
+host_modules= { module= gotools; };
 host_modules= { module= lto-plugin; bootstrap=true;
 		extra_configure_flags='--enable-shared @extra_linker_plugin_flags@ @extra_linker_plugin_configure_flags@';
 		extra_make_flags='@extra_linker_plugin_flags@'; };
 host_modules= { module= libcc1; extra_configure_flags=--enable-shared; };
-host_modules= { module= gotools; };
 host_modules= { module= libctf; no_install=true; no_check=true;
 		bootstrap=true; };
 
@@ -381,6 +392,8 @@ dependencies = { module=all-lto-plugin; on=all-libiberty-linker-plugin; };
 dependencies = { module=configure-libcc1; on=configure-gcc; };
 dependencies = { module=all-libcc1; on=all-gcc; };
 
+// we want version.o from gcc, and implicitly depend on libcody
+dependencies = { module=all-c++tools; on=all-gcc; };
 dependencies = { module=all-gotools; on=all-target-libgo; };
 
 dependencies = { module=all-utils; on=all-libiberty; };
diff --git c/Makefile.tpl w/Makefile.tpl
index efed1511750..3b88f351d5b 100644
--- c/Makefile.tpl
+++ w/Makefile.tpl
@@ -864,8 +864,8 @@ local-distclean:
 	-rm -f texinfo/doc/Makefile texinfo/po/POTFILES
 	-rmdir texinfo/doc texinfo/info texinfo/intl texinfo/lib 2>/dev/null
 	-rmdir texinfo/makeinfo texinfo/po texinfo/util 2>/dev/null
-	-rmdir fastjar gcc gnattools gotools libcc1 libiberty 2>/dev/null
-	-rmdir texinfo zlib 2>/dev/null
+	-rmdir c++tools fastjar gcc gnattools gotools 2>/dev/null
+	-rmdir libcc1 libiberty texinfo zlib 2>/dev/null
 	-find . -name config.cache -exec rm -f {} \; \; 2>/dev/null
 
 local-maintainer-clean:
diff --git c/c++tools/configure.ac w/c++tools/configure.ac
new file mode 100644
index 000..8d882e541df
--- /dev/null
+++ w/c++tools/configure.ac
@@ -0,0 +1,210 @@
+# Configure script for c++tools
+#   Copyright (C) 2020 Free Software Foundation, Inc.
+#   Written by Nathan Sidwell  while at FaceBook
+#
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; see the file COPYING3.  If not see
+# .
+
+# C++ has grown a C++20 mapper server.  This may be used to provide
+# and/or learn and/or build required modules.  This sample server
+# shows how the protocol introduced by wg21.link/p1184 may be used.
+# By default g++ uses an in-process mapper.
+
+sinclude(../config/acx.m4)
+
+AC_INIT(c++tools)
+
+AC_CONFIG_SRCDIR([server.cc])
+
+# Determine the noncanonical names used for directories.
+ACX_NONCANONICAL_HOST
+
+AC_CANONICAL_SYSTEM
+AC_PROG_INSTALL
+
+AC_PROG_CXX
+MISSING=`cd $ac_aux_dir && ${PWDCMD-pwd}`/missing
+AC_CHECK_PROGS([AUTOCONF], [autoconf], [$MISSING autoconf])
+AC_CHECK_PROGS([AUTOHEADER], [autoheader], [$MISSING autoheader])
+
+dnl Enab

Re: [24/32] module mapper

[Nathan Sidwell]

On 11/9/20 1:42 AM, Boris Kolpackov wrote:

I've noticed the following issues with the module mapper in the
-fdirectives-only mode:


1. When partially preprocessing the module interface unit, the mapper
receives the MODULE-EXPORT request that's unnecessary (BMI is not
written):

g++ ... -x c++ -E -fdirectives-only -o hello.gcm.ii hello.mxx

Similarly, in this mode, the mapper receives MODULE-IMPORT for
(non-header) module imports. Again, this is not necessary and
replying with a non-existent BMI works.


2. When doing full preprocessing of a partially preprocessed unit,
the mapper again receives MODULE-EXPORT and MODULE-IMPORT for
non-header modules:

g++-m ... -E -x c++ -fpreprocessed -fdirectives-only hello.gcm.ii

These are also unnecessary.


These are needed as they also serve to inform the mapper of a dependency 
edge.


nathan

--
Nathan Sidwell

Re: [24/32] module mapper

[Boris Kolpackov]

I've noticed the following issues with the module mapper in the
-fdirectives-only mode:


1. When partially preprocessing the module interface unit, the mapper
   receives the MODULE-EXPORT request that's unnecessary (BMI is not
   written):

   g++ ... -x c++ -E -fdirectives-only -o hello.gcm.ii hello.mxx

   Similarly, in this mode, the mapper receives MODULE-IMPORT for
   (non-header) module imports. Again, this is not necessary and
   replying with a non-existent BMI works.


2. When doing full preprocessing of a partially preprocessed unit,
   the mapper again receives MODULE-EXPORT and MODULE-IMPORT for
   non-header modules:

   g++-m ... -E -x c++ -fpreprocessed -fdirectives-only hello.gcm.ii

   These are also unnecessary.

[24/32] module mapper

[Nathan Sidwell]

this is the module mapper client and server pieces.  It features a 
default resolver that can read a text file, or generate default mappings 
from module name to cmi name.


By default the compiler will use an in-process resolved, but with 
suitable options can be instructed to communicate with a server over

a) a pipe to a spawned process's stdin/stdout
b) a pair of specified filenos
c) a unix-domain socket (if supported)
d) an ipv6 host/port (if supported)

the server can operat in modes a, c or d.


--
Nathan Sidwell

diff --git c/gcc/cp/mapper-client.cc w/gcc/cp/mapper-client.cc
new file mode 100644
index 000..de259b0564c
--- /dev/null
+++ w/gcc/cp/mapper-client.cc
@@ -0,0 +1,315 @@
+/* C++ modules.  Experimental!
+   Copyright (C) 2017-2020 Free Software Foundation, Inc.
+   Written by Nathan Sidwell  while at FaceBook
+
+   This file is part of GCC.
+
+   GCC is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3, or (at your option)
+   any later version.
+
+   GCC is distributed in the hope that it will be useful, but
+   WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+#include "config.h"
+#include "system.h"
+
+#include "line-map.h"
+#include "diagnostic-core.h"
+#define MAPPER_FOR_GCC 1
+#include "mapper.h"
+#include "intl.h"
+
+module_client::module_client (pex_obj *p, int fd_from, int fd_to)
+  : Client (fd_from, fd_to), pex (p)
+{
+}
+
+static module_client *
+spawn_mapper_program (char const **errmsg, std::string &name,
+		  char const *full_program_name)
+{
+  /* Split writable at white-space.  No space-containing args for
+ you!  */
+  // At most every other char could be an argument
+  char **argv = new char *[name.size () / 2 + 2];
+  unsigned arg_no = 0;
+  char *str = new char[name.size ()];
+  memcpy (str, name.c_str () + 1, name.size ());
+
+  for (auto ptr = str; ; ++ptr)
+{
+  while (*ptr == ' ')
+	ptr++;
+  if (!*ptr)
+	break;
+  argv[arg_no++] = ptr;
+  while (*ptr && *ptr != ' ')
+	ptr++;
+  if (!*ptr)
+	break;
+  *ptr = 0;
+}
+  argv[arg_no] = nullptr;
+
+  auto *pex = pex_init (PEX_USE_PIPES, progname, NULL);
+  FILE *to = pex_input_pipe (pex, false);
+  name = argv[0];
+  if (!to)
+*errmsg = "connecting input";
+  else
+{
+  int flags = PEX_SEARCH;
+	  
+  if (full_program_name)
+	{
+	  /* Prepend the invoking path.  */
+	  size_t dir_len = progname - full_program_name;
+	  std::string argv0;
+	  argv0.reserve (dir_len + name.size ());
+	  argv0.append (full_program_name, dir_len).append (name);
+	  name = std::move (argv0);
+	  argv[0] = const_cast  (name.c_str ());
+	  flags = 0;
+	}
+  int err;
+  *errmsg = pex_run (pex, flags, argv[0], argv, NULL, NULL, &err);
+}
+  delete[] str;
+  delete[] argv;
+
+  int fd_from = -1, fd_to = -1;
+  if (!*errmsg)
+{
+  FILE *from = pex_read_output (pex, false);
+  if (from && (fd_to = dup (fileno (to))) >= 0)
+	fd_from = fileno (from);
+  else
+	*errmsg = "connecting output";
+  fclose (to);
+}
+
+  if (*errmsg)
+{
+  pex_free (pex);
+  return nullptr;
+}
+
+  return new module_client (pex, fd_from, fd_to);
+}
+
+module_client *
+module_client::open_module_client (location_t loc, const char *o,
+   void (*set_repo) (const char *),
+   char const *full_program_name)
+{
+  module_client *c = nullptr;
+  std::string ident;
+  std::string name;
+  char const *errmsg = nullptr;
+  unsigned line = 0;
+
+  if (o && o[0])
+{
+  /* Maybe a local or ipv6 address.  */
+  name = o;
+  auto last = name.find_last_of ('?');
+  if (last != name.npos)
+	{
+	  ident = name.substr (last + 1);
+	  name.erase (last);
+	}
+
+  if (name.size ())
+	{
+	  switch (name[0])
+	{
+	case '<':
+	  // to or <>fromto, or <>
+	  {
+		int fd_from = -1, fd_to = -1;
+		char const *ptr = name.c_str ();
+		char *eptr;
+
+		fd_from = strtoul (++ptr, &eptr, 0);
+		if (*eptr == '>')
+		  {
+		ptr = eptr;
+		fd_to = strtoul (++ptr, &eptr, 0);
+		if (eptr != ptr && ptr == name.c_str () + 1)
+		  fd_from = fd_to;
+		  }
+
+		if (*eptr)
+		  errmsg = "parsing";
+		else
+		  {
+		if (name.size () == 2)
+		  {
+			fd_from = fileno (stdin);
+			fd_to = fileno (stdout);
+		  }
+		c = new module_client (fd_from, fd_to);
+		  }
+	  }
+	  break;
+
+	case '=':
+	  // =localsocket
+	  {
+		int fd = -1;
+#if CODY_NETWORKING
+		fd = Cody::OpenLocal (&errmsg, name.c_str () + 1);
+#endif
+		if (fd >= 0)
+		  c = new module_client (fd, fd);
+	  }
+	  break;
+
+	case '|