here is the new way to "freeze" plugins: i will use the Felix
webserver (src/tools/webserver.flx) as an example.
First you write a loader:
//////////////////////////////////////////////////////
// webserver plugin linker
class WebserverPluginSymbols
{
// We have to do this dummy requirements because static
// linking removes the information in Felix.
requires package "re2";
requires package "faio";
requires package "flx_arun";
open Dynlink;
// Now add all the symbols.
proc addsymbols ()
{
static-link-plugin
fdoc2html,
flx2html,
fpc2html,
py2html,
ocaml2html,
cpp2html,
fdoc_scanner,
fdoc_slideshow,
fdoc_heading,
fdoc_fileseq,
fdoc_paragraph,
fdoc_button
;
// webserver
static-link-symbol webserver_create_thread_frame in "webserver";
static-link-symbol webserver_flx_start in "webserver";
}
}
// Add the symbols
WebserverPluginSymbols::addsymbols;
// Now invoke the webserver!
println$ "Running webserver";
val linstance = Dynlink::prepare_lib("webserver");
println$ "Webserver prepared";
var init: cont = Dynlink::get_init linstance;
Fibres::chain init;
////////////////////////////////////
Notes 1. When you generate an object file (.o, .os, .obj extensions)
and later link it, information about the libraries it is required to link
to is lost. In C/C++ it was never there, you have to use a command line
to provide the libraries containing required symbols to satisfy external
references.
With Felix, a whole program compilation retains that information
all the way through to the linkage step. However when you flddle
the "flx" command to produce object files, the information is again
lost (the dependencies are generated but not tracked, at least at
the moment).
The first half of the workaround is to re-introduce the dependencies,
which is done with the requires clauses you see. These are requirements
of the webserver. This forces linkage of the appropriate libraries.
Notes 2. The plugin variant of this command requires four entry
points in the plugin. If it is named PLUGIN these entry points are
PLUGIN_create_thread_frame
PLUGIN_flx_start
PLUGIN_setup
PLUGIN
These create the thread frame (global data object) and do C++ member
initialisation, do custom Felix initialisation which involves executing
dynamic initialisers and running "global" code, a special setup
function that accepts a string that can further configure a plugin,
and finally the core entry point.
In Felix code when writing a plugin, you must export the last two
of these symbols:
export fun setup of (string) as "PLUGIN_setup";
export fun main of (string * string) as "PLUGIN";
The first two are exported automatically for all libraries.
Plugins are in fact just ordinary Felix programs generated
as shared libraries (the default), except for these two extra
entry points.
Commonly, the core entry point is an object constructor that
returns an object whose methods are used to access the
services offered by the plugin.
Notes 3: The meaning of the previous Note 2 is evident
for the webserver itself. That was written as a "program" not
as a plugin. Since it lacks the setup and core entry points,
we have to create the linkage with the lower level variant:
// webserver
static-link-symbol webserver_create_thread_frame in "webserver";
static-link-symbol webserver_flx_start in "webserver";
otherwise we'll try to link "webserver_setup" and "webserver" which
don't exist. Actually we could modify the webserver to add these
entry points so it could be used both as a program and as a plugin.
note again the central theme: there's nothing special about programs
or plugins in Felix: they're all (shared) libraries. you have to link
one of:
flx_run
flx_arun
to make an executable program.
For static links this actually requires a special linker thunk similar to
the one above we wrote by hand, only in this case the compiler generates
the thunks for you (the thunks are needed to map the fixed symbols
that flx_run requires into the names used in the library, which are just
the program name followed by _create_thread_frame and _flx_start.
Note 4: As a design issue, Felix requires all plugins use distinct
entry points. With most dynamic linkers this isn't required, since the
library name identifies the symbol table. However it is always required
for static linkage (duplicate symbols can't be tolerated), at least without
weird linker and OS specific hackery.
Note 5: the invocation of the webserver is done by
val linstance = Dynlink::prepare_lib("webserver");
var init: cont = Dynlink::get_init linstance;
Fibres::chain init;
The first line prepares the library. This will usually do a dynamic load
of the DLL. However in this case, we've added "webserver" to the
Felix symbol table so it will not dlopen() the webserver: its already
loaded by static linkage. The function then executes
webserver_create_thread_frame
which create a new thread frame object on the heap and returns it.
This is the thing that holds the global variables. A pair consisting
of the library code and a thread frame object is called an instance.
Its exactly the CS and DS segment registers on Windows 3.1
programming model. This is also known as Harvard Architecture.
This separation is mandatory for re-entrant code.
Unix uses the archaic von Neumann architecture which is the
cause of all its problems. It's enshrined in C, which is the cause
of most of ITS problems.
Ok, so, the get_init call extracts from the instance another pair:
the flx_start symbol and the thread frame, represented by a value
of the type "cont", which is a continuation. This is a program
"read to run".
Finally, we run that continuation by passing it to the scheduler,
and discontinuing the currently executing continuation:
the chain function does this job. This loses the current
code position and thread frame. In functional terms its a tail
call.
========================
So now you have a detailed explanation how the code works,
here's how to link it: you say "make weblink":
weblink:
build/release/host/bin/flx --test=build/release -c --nolink --static \
-ox build/release/host/lib/rtl/webserver
src/tools/webserver.flx
build/release/host/bin/flx --test=build/release -c --static -ox
build/release/host/bin/weblink \
build/release/host/lib/rtl/fdoc_heading.o \
build/release/host/lib/rtl/fdoc_button.o \
build/release/host/lib/rtl/fdoc_fileseq.o \
build/release/host/lib/rtl/fdoc_paragraph.o \
build/release/host/lib/rtl/fdoc_scanner.o \
build/release/host/lib/rtl/fdoc_slideshow.o \
build/release/host/lib/rtl/fdoc2html.o \
build/release/host/lib/rtl/flx2html.o \
build/release/host/lib/rtl/py2html.o \
build/release/host/lib/rtl/cpp2html.o \
build/release/host/lib/rtl/ocaml2html.o \
build/release/host/lib/rtl/fpc2html.o \
build/release/host/lib/rtl/webserver.o \
src/tools/weblink.flx
The first step creates webserver.o object file. The usual webserver is built
from a webserver.os file. This is an object file with position independent code,
set with gcc and clang using -fPIC switch. We don't need that for static link
so we build an ordinary .o file. [Felix uses .os for position independent object
files, the "s" standards for "shared" because these files are used to build
shared libraries]. The distinction only exists on some platforms, such as
x86_64 and PPC. Its more crappy technology inherited from von Neumann
architectures, Unix, and C we should have got rid of decades ago.
The second step compiles our linkage thunk, and links it to the webserver
object file we just made and all the required plugins. Felix build system
(the NEW one written in Felix ONLY) builds the plugins as both
PIC and non-PIC code. Here we use the non-PIC objects.
Again note: Felix loses the information required to auto-link the plugins:
in fact Felix doesn't know what plugins the webserver uses since they're
dynamically loaded by string name. That's why we had to make the
weblink loader thunk.
The final product is a fully statically linked webserver called "weblink"
which does not do any dynamic loading with dlopen() and so doesn't
need any LD_LIBRARY_PATH to operate. In other words, we've built
a stand-alone executable which can be deployed or delivered to
clients without worrying about its execution environment.
==========================================================
In the future, I may look at simplifying this again. Its a serious pain having
to manually link the plugins. It is trivial to use an ordinary static link
library of plugins instead, however this only works if the complete
set of plugins is known. By their nature, users can write their own plugins.
You should also note: you can delete any of the plugins in the
static-link-plugin statement. Just delete the lines. The webserver
produced WILL STILL WORK .. it just needs to dlopen() the library.
This is transparent, modulo linking the library and setting up
the LD_LIBRARY_PATH.
In particular PARTIAL static linkage is quite possible.
I have done all this work for two reasons:
(1) I want the webserver in particular to be stand-alone again.
It used to be, then I invented plugins and created a deployment problem.
I could never tell *which* plugins got loaded because they were installed
in /usr/local/lib. Felix DOES NOT INSTALL ANYTHING in /usr/local/lib
any more.
In fact Felix now only installs ONE public entity: it installs "flx"
in /usr/local/bin. No other tools are installed. Everything else goes
in /usr/local/lib/felix/felix-latest.
(2) With the new build system (written in Felix) and new file layout,
shared sources and platform dependent stuff including libraries
and executables are cleanly separated into two directories.
This allows us to take the shared sources and build multiple
target binary directories with different configurations.
For example on the Mac the default configuration "host"
uses clang as a compiler, on Linux, gcc is used instead.
But I want to test gcc on the Mac. And clang on Linux.
Now, with the new setup, I can modify "flx" to accept a target
option, and with some fiddling, use the Felix build system
to generate additional targets.
In fact we already have, and the build system uses, plugins
for various toolchains, only "flx" does not use these yet.
Why? Because THEY WERE ONLY AVAILABLE AS DYNAMIC
LOAD PLUGINS.
Which would mean "flx" was no longer a stand-alone executable,
but depended on installed plugins and LD_LIBRARY_PATH.
Which would make it very tricky to use "flx" to build Felix.
And it would make it impossible to just install "flx"
in /usr/local/bin and have it "just work".
BUT NOW .. we can statically link plugins. So I am now free
to rewrite "flx" to use plugins knowing that I will not be losing
the ability to produce a stand alone executable.
====================================================
So, for the last 3 months I have struggled with several apparently
discrete issue: plugins, static linkage, build systems, compiler
toolchains, file system layouts, cross-compilation, and installation.
All these issues are related as you can now see. Progress in one
requires progress in the others. We're not there yet, but we're
well on the way to an enterprise level Felix installation.
The next BIG issue, which is just coming into scope, is again
the build and installation system, but this time with a greatly
expanded view: to split up the code base into separate pieces.
In other words, package management. We have Mike Maul's
package management tool scoop as an entry point. Like "flx",
scoop will have to use plugins, but now it can take advantage
of the static linking ability. This is essential for bootstrapping.
Ultimately, the core build, flx, and scoop, need to be integrated.
In an ideal world, you can run stuff straight off the internet.
Downloading and installation are transparent, just as
compilation is made more or less transparent by the current
flx tool.
There are important issues here to deal with like security
and stability. So as usual we start off with pieces of automation
manually driven and gradually integrate.
Although automatic execution of code off the internet seems
scary, i would like to point out ...
you're already doing it every day!!
In fact one of the PRIMARY advantages of Javascript in building
web pages is that you can just "source" whole libraries off
the internet, such as JQuery. Its a plugin and package, you
can provide it on your own server, but most web developers
don't.
Javascript is rewriting the whole notion of computing.
Using it you can program platform independent GUI's.
You're leveraging the huge amount of work developing
browsers and the domain object model to provide a more
or less platform independent GUI.
Of course there's a whole lot of CRAP behind this like AJAX.
What horrible crap to do http exchanges "per character"
to get interactivity. HTTP just wan't designed for that.
Never-the-less .. it has been made to work and it demonstrates
very clearly a level or re-usability never achieved at such a huge
scale before. The best we had in prior lives was CPAN for Perl,
and a very poor approximation in the form of Debian and apt.
--
john skaller
skal...@users.sourceforge.net
http://felix-lang.org
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
_______________________________________________
Felix-language mailing list
Felix-language@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/felix-language
