MY BUG
=======
I had a serious problem with the dependency checking appearing to
fail when compiling the build system tool flx_build_rtl.
This was a nasty problem because the make file says
build/release/host/bin/flx --test=build/release \
build/release/src/tools/flx_build_rtl --target=build/release
--build-flx
which has the effect of building both flx and flx_build_rtl, both of which are
active ("busy")
during the build. Actually I don't know how OSX handles writing to an active
executable,
and I know even less how it works if the target is an active shared library!
However, because I'm building a static link executable but I'm RUNNING
a Felix program, the executable being run to create itself (flx_build_rtl)
is in the cache, not the host bin directory. It's not clear why, given this
method, there's any point at all actually building it!
In any case, I got confused for a long time over stuff rebuilding when it
shouldn't, or failing to build when it should.
THE CAUSE
==========
What it turned out to be was that there are three flx programs!
The program "bootflx" is generated by Python. The reason for this
is so that you can run bootflx to generate flx (without overwriting
the executable). bootflx is compiled with the standard toolchains
included as source.
There are also two versions of flx: the one called 'dflx' is a standard
executable with no preloaded toolchains, so it requires the plugins
to be already built. But we need flx to build these plugins. When bootstrapping
we can use bootflx, but this only works on the host with the standard
toolchains.
Just to confuse you: two platforms characterise a build tool, the host
environment in which it is run, and the target for which it generates
output. In Felix, the target "host" has "host" as both the host environment
and the target. However consider this: suppose the host environment
is 32 bit Windows, then we get everything up and running.
But now, we want to run under 64 bit windows. So we need to build
32 bit plugins that generate 64 bit output. We can build a Felix target
that does this from the 32 bit hosted 32 bit target environment.
Then, we switch to the 64 bit environment, and build another set
of tools, this time hosted in 64 bit environment generating code
for a 64 bit environment.
Confused? Yep! The point is we need THREE toolchains:
32->32
32->64
64->64
32 bit flx can run the first two toolchains and build all three,
which allows generating a 64 bit flx which we can then be used
to rebuild the 64->64 system from then on.
Ok, so all of this is to explain why I'm confused because although I am
NOT changing the environment, I have two versions of everything:
the old stuff and the new. The new stuff tends to scrub the old stuff
so the build order is rather critical, and afterwards there's some stuff
that needs to be built again (even though it's probably right, it was built
with the old tools). Very confusing!
Ok, so we also have flx, which is the same as dflx except it preloads
the standard toolchain plugins. Note this is NOT the same as bootflx!!!
Bootflx includes the standard toolchains at source level, whereas flx
requires compiling the toolchains as object files and then linking
them with the dflx object file and a stub loader that sets up the
preload dictionary then calls dflx "as if" Felix had done it directly.
bootflx stores symbols in the preload dictionary too, but no linking
is required because they're compiled in as source.
BUT .. the ACTUAL cause of the problem was much simpler
than all the confusion, and the hint is above.
I was generating "dflx" twice. First I was making an executable.
Later when building flx, I generated an object file. The first step
actually generates an object file too, but it is hidden in the cache.
The second step explicitly names the object file so it can be linked
into flx.
BUT the dependency file *.dep is based on the Felix source file
location NOT the output location. So the (same) dependency file
got generated twice. When I tried to repeat the process, the dependency
file was newer than the dflx executable so it got rebuilt, also re-running
flxg which generated the dependency file again. So now it is newer than
the second object file as well and it also gets built again (regenerating
the dependency file yet again).
THE FIXES
=========
Two fixes were required. First, I removed "dflx". It's only purpose is to
force the dynamic loading of a toolchain plugin, so you cannot
accidentally use the wrong one. flx will use a preloaded version
if it can, which has the nasty side effect that if you modify the
plugin, flx will keep using the old toolchain (because it's statically
linked in). Sometimes that's useful (if you stuff up, there's some chance
you can still use flx to fix the bad plugin since it doesn't actually load it).
Removing dflx eliminated the duplicate dependency file generation,
but it wasn't enough!
The second fix, just committed, splits the dependency checking
into two independent parts. The first part checks if flxg needs
to be run, by checking if the dependency file itself is older than
any of the files it lists. This always includes the Felix base source file,
so if you change that, or any of the OLD dependencies, flxg will be called.
Note this will NOT work if you change the search path, you have to use
the --force switch for that at the moment.
The second check compares the binary with the dependency file.
If the binary is older, we have to recompile the C++ sources.
So now, I hope, the build is semi-workable. It doesn't cope with building
the same sources in two distinct ways entirely properly, but it should work
reasonably well even in that case, because the output from flxg is more
or less independent of almost everything. Note changes to the compiler
or grammar should force the whole cache to be scrubbed so we shouldn't
have to worry about an improper dependency file (it won't exist).
Exceptions: changes to the include search path, imports, inlining level,
debug comments in the sources, (and library used ?) will not be detected
at present. You will have to use the --force (or --force-compiler) switch.
These exceptions are rare enough not to bother about them yet.
Normally Felix programming doesn't change any of this stuff.
It is common (for me) to change the install directory, however this should
trigger a wipe out of the cache (even, it seems, if the testing directory
is installed, and everything in both is identical .. not sure why! However
install deliberately wipes the cache).
SImilarly, and perhaps more critically, changes to C++ compilation options
will not be detected. If you put -O2 to override the default optimisation,
the C++ sources won't be regenerated and neither will the dependency
file. So you won't get a C++ recompile. You can use --force --nofelix
to fix this (avoiding running flxg but still forcing C++ compilation).
COMING
=======
Now the Felix compilation is working reasonably well, in the sense
I can rebuild the whole system and only things that changed get
rebuilt, its time to fix the problem that actually generating the RTL
requires a very LONG set of C++ compilations which are forced
every time.
To fix this, I'm using gcc -M -MFfile.d kind of thing to generate
an *.d file which contains a "make" readable dependency specification
generated by the C/C++ pre-processor.
We will then parse this file, and then find the time of the newest file
in the dependency list and compare it with the compiler target.
If it is older, we can skip the compilation.
That's ROUGHLY! A new post for the messy details!
However, the point to note is that I intend to build this in at a low
enough level eventually that the C/C++ compilation toolchains
automatically do dependency checking and avoid running
the C++ compiler if it isn't necessary. We will still need a force
option because command line arguments can screw the logic up.
When this is working, you will be able to compile arbitrary C++ with
flx, using stub Felix program. The only caveat here will be you cannot
have a main() function if statically lining (because Felix provides that!).
So you'll have to rename it c_main or cxx_main or something.
Can be done by a stub file:
// new program suitable for Felix
#define main cxx_main
#include "oldprogram.cpp"
[provided you don't have any methods named main, or any main functions
in non global namespace .. ]
The only reason for requiring a Felix stub is that flx command line parsing
splits off the program name from any arguments to pass to it by detecting:
a filename without an extension
a filename with .flx extension
So the issue is entirely related to the command line parser: you have
to say:
flx myprog.cpp dummy.flx argument
because without the dummy.flx, argument is taken as the program!
Actually Felix convention is:
*.cpp, *.c : a library file
*.cxx, *.cc : a mainline (containing main function)
however this isn't universally accepted and a lot of C++ programmers use
*.cc to mean a C++ file (not a c mainline).
I will sort this out somehow, but first to get the stub variation working.
Note that Felix ALREADY checks the time stamps of C/C++ files
on the flx command line (it just doesn't check any includes).
--
john skaller
skal...@users.sourceforge.net
http://felix-lang.org
------------------------------------------------------------------------------
DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
_______________________________________________
Felix-language mailing list
Felix-language@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/felix-language
