On Tuesday, 18 December 2012 at 00:15:04 UTC, H. S. Teoh wrote:
On Tue, Dec 18, 2012 at 02:08:55AM +0400, Dmitry Olshansky
wrote:
[...]
I suspect it's one of prime examples where UNIX philosophy of
combining a bunch of simple (~ dumb) programs together in
place of
one more complex program was taken *far* beyond reasonable
lengths.
Having a pipe-line:
preprocessor -> compiler -> (still?) assembler -> linker
where every program tries hard to know nothing about the
previous
ones (and be as simple as possibly can be) is bound to get
inadequate results on many fronts:
- efficiency & scalability
- cross-border error reporting and detection (linker errors?
errors
for expanded macro magic?)
- cross-file manipulations (e.g. optimization, see _how_ LTO
is done in GCC)
- multiple problems from a loss of information across pipeline*
The problem is not so much the structure preprocessor ->
compiler ->
assembler -> linker; the problem is that these logical stages
have been
arbitrarily assigned to individual processes residing in their
own
address space, communicating via files (or pipes, whatever it
may be).
The fact that they are separate processes is in itself not that
big of a
problem, but the fact that they reside in their own address
space is a
big problem, because you cannot pass any information down the
chain
except through rudimentary OS interfaces like files and pipes.
Even that
wouldn't have been so bad, if it weren't for the fact that user
interface (in the form of text input / object file format) has
also been
conflated with program interface (the compiler has to produce
the input
to the assembler, in *text*, and the assembler has to produce
object
files that do not encode any direct dependency information
because
that's the standard file format the linker expects).
Now consider if we keep the same stages, but each stage is not a
separate program but a *library*. The code then might look, in
greatly
simplified form, something like this:
import libdmd.compiler;
import libdmd.assembler;
import libdmd.linker;
void main(string[] args) {
// typeof(asmCode) is some arbitrarily complex data
// structure encoding assembly code, inter-module
// dependencies, etc.
auto asmCode = compiler.lex(args)
.parse()
.optimize()
.codegen();
// Note: no stupid redundant convert to string, parse,
// convert back to internal representation.
auto objectCode = assembler.assemble(asmCode);
// Note: linker has direct access to dependency info,
// etc., carried over from asmCode -> objectCode.
auto executable = linker.link(objectCode);
File output(outfile, "w");
executable.generate(output);
}
Note that the types asmCode, objectCode, executable, are
arbitrarily
complex, and may contain lazy-evaluated data structure,
references to
on-disk temporary storage (for large projects you can't hold
everything
in RAM), etc.. Dependency information in asmCode is propagated
to
objectCode, as necessary. The linker has full access to all
info the
compiler has access to, and can perform inter-module
optimization, etc.,
by accessing information available to the *compiler* front-end,
not just
some crippled object file format.
The root of the current nonsense is that perfectly-fine data
structures
are arbitrarily required to be flattened into some kind of
intermediate
form, written to some file (or sent down some pipe), often with
loss of
information, then read from the other end, interpreted, and
reconstituted into other data structures (with incomplete
info), then
processed. In many cases, information that didn't make it
through the
channel has to be reconstructed (often imperfectly), and then
used. Most
of these steps are redundant. If the compiler data structures
were
already directly available in the first place, none of this
baroque
dance is necessary.
*Semantic info on interdependency of symbols in a source file
is
destroyed right before the linker and thus each .obj file is
included as a whole or not at all. Thus all C run-times I've
seen
_sidestep_ this by writing each function in its own file(!).
Even
this alone should have been a clear indication.
While simplicity (and correspondingly size in memory) of
programs
was the king in 70's it's well past due. Nowadays I think is
all
about getting highest throughput and more powerful features.
[...]
Simplicity is good. Simplicity lets you modularize a very
complex piece
of software (a compiler that converts D source code into
executables)
into manageable chunks. Simplicity does not require
shoe-horning modules
into separate programs with separate address spaces with
separate (and
deficient) input/output formats.
The problem isn't with simplicity, the problem is with carrying
over the
archaic mapping of compilation stage -> separate program. I
mean,
imagine if std.regex was written so that regex compilation runs
in a
separate program with a separate address space, and the regex
matcher
that executes the match runs in another separate program with a
separate
address space, and the two talk to each other via pipes, or
worse,
intermediate files.
I've mentioned a few times before a horrendous C++ project that
I had to
work with once, where to make a single function call to a
particular
subsystem, it had to go through 6 layers of abstraction, one of
which
was IPC through a local UNIX socket, *and* another of which
involved
fwrite()ing function parameters into a file and fread()ing said
parameters from the file in another process, with the 6 layers
repeating
in reverse to propagate the return value of the function back
to the
caller.
In the new version of said project, that subsystem exposes a
library API
where to make a function call, you, um, just call the function
(gee,
what a concept). Needless to say, it didn't take a lot of
effort to
convince customers to upgrade, upon which we proceeded with
great relish
to delete every single source file having to do with that
6-layered
monstrosity, and had a celebration afterwards.
From the design POV, though, the layout of the old version of
the
project utterly made sense. It was superbly (over)engineered,
and if you
made UML diagrams of it, they would be works of art fit for the
British
Museum. The implementation, however, was "somewhat"
disappointing.
T
IMO, it's not even an issue of the separate address spaces. The
core problem is the direct result of relying on *archaic file
formats*.
Simply using serialization of the intermediate data structure
already solves the data loss problems and all that remains are
aspects of efficiency which are much less important given current
compilation speeds. Separate address spaces can be useful if we
add distributed and concurrent aspects into the mix.
