Re: Defining a common plugin machinery
Aye up,
I like the idea of signals. The chaining code can be automatically
generated by FFI - the code to do it is pretty trivial.
Also, if instead of having a single struct with all the signals in,
you could have a hashtable of signals referenced by a string id, then
plugins could define their own and be able to cooperate.
Cheers,
Hugh.
Brendon Costa wrote:
Sounds like you're almost in need of a generic data marshalling interface
here.
Why do we need the complication of data marshaling?
I don't see why we need to define that all plugin hooks have the same
function interface as currently proposed. I.e. a single void*. This
makes a lot of work marshaling data both as parameters and from return
values. This is already done for us by the language (Though i may have
mis-understood the train of thought here).
I will propose the start of a new idea. This needs to be fleshed out a
lot but it would be good to get some feedback.
I will use the following terminology borrowed from QT:
signal: Is a uniquely identified hook to which zero or more slots are
added. (I.e. Caller)
slot: Is a function implementation say in a plugin. This is added to a
linked list for the specified signal. (I.e. Callee)
The main concepts in this plugin hook definition are:
* Signals can define any type of function pointer so can return values
and accept any parameters without special data marshaling
* Each signal is uniquely identified as a member variable in a struct
called Hooks
* A signal is implemented as a linked list where each node has a
reference to a slot that has been connected to the signal
* A slot is a function pointer and a unique string identifier
This differs a bit from the QT definition but i find it helpful to
describe the entities.
Important things to note:
Multiple plugins are chained one after the other. I.e. It is the
responsibility of the plugin author to call any plugins that follow it
in the list. This gives the plugin authors a bit more control over how
their plugins inter-operate with other plugins, however it would be
STRONGLY recommended that they follow a standard procedure and just call
the next plugin after they have done their work.
Basically, the idea is to provide the following structure and then most
of the work will involve manipulation of the linked lists. I.e. Querying
existing items in the LL, inserting new items before/after existing
items, removing items from the LL.
This is not a proposed end product. It is just to propose an idea. There
are a few disadvantages with the way it is implemented right now:
* Too much boilerplate code for each signal definition
* The idea of chaining calls means the responsibility of calling the
next plugin ends up with the plugin developer which could be bad if a
plugin developer does not take due care, however it also provides them
with more flexibility (not sure if that is necessary).
Now, i have NO experience with the current pass manager in GCC, but
would the passes be able to be managed using this same framework
assuming that each pass is given a unique identifier?
Thanks,
Brendon.
#include stdio.h
#include stdlib.h
/* GCC : Code */
struct Hooks
{
/* Define the blah signal. */
struct BlahFPWrap
{
const char* name;
int (*fp)(struct BlahFPWrap* self, int i, char c, float f);
void* data;
struct BlahFPWrap* next;
struct BlahFPWrap* prev;
}* blah;
struct FooFPWrap
{
const char* name;
void (*fp)(struct FooFPWrap* self);
void* data;
struct FooFPWrap* next;
struct FooFPWrap* prev;
}* foo;
};
/* Initialised by main */
struct Hooks hooks;
void SomeFunc1(void)
{
/* Call plugin hook: blah */
int result = (!hooks.blah ? 0 : hooks.blah-fp(hooks.blah, 3, 'c',
2.0f));
/* ... do stuff with result ... */
(void)result;
}
void SomeFunc2(void)
{
/* Call plugin hook: foo */
if (hooks.foo) hooks.foo-fp(hooks.foo);
}
void PlgInit(struct Hooks* h);
int main()
{
hooks.blah = NULL;
hooks.foo = NULL;
PlgInit(hooks);
return 0;
}
/* Writeme... */
#define PLUGIN_INSERT_BEFORE(Hooks, Struct, Hook, FuncPtr, Before, SlotName)
/* In plugin */
#define PLUGIN_NAME myplg
static void MyFoo(struct FooFPWrap* self)
{
printf(MyFoo\n);
if (self-next) self-next-fp(self-next);
}
static void MyBlah(struct BlahFPWrap* self, int i, char c, float f)
{
printf(MyBlah\n);
if (self-next) self-next-fp(self-next, i, c, f);
}
void PlgInit(struct Hooks* h)
{
PLUGIN_INSERT_BEFORE(h, struct BlahFPWrap, blah, MyBlah, NULL,
PLUGIN_NAME _MyBlah);
PLUGIN_INSERT_BEFORE(h, struct FooFPWrap, foo, MyFoo, NULL,
PLUGIN_NAME _MyFoo);
}
void PlgShut(struct Hooks* h)
{
PLUGIN_REMOVE(h, PLUGIN_NAME _MyBlah);
PLUGIN_REMOVE(h, PLUGIN_NAME _MyFoo);
}
RE: Defining a common plugin machinery
I currently don't have any preference for a specific way to deal with data
marshaling
since currently it's enough for the MILEPOST project just to return void*
pointer,
but just wanted to mention that last year Cupertino Miranda tried to introduce
an intermediate data layer to ICI to separate program analysis from
transformations
and potentially simplify dealing with external optimization plugins. I think the
project has been frozen or Cupertino can comment on that if he follows this
thread ;),
but I thought to give a link to the tech. report Cupertino presented last year,
if anyone is interested:
http://gcc-ici.sourceforge.net/papers/mfpp2007.pdf
By the way, if I am correct, GCC MELT (developed by Basile)
also attempts to address some of these issues with data marshaling
to modularize GCC ...
Cheers,
Grigori
-Original Message-
From: Brendon Costa [mailto:[EMAIL PROTECTED]
Sent: Friday, October 10, 2008 2:33 AM
To: Dave Korn
Cc: 'Taras Glek'; 'Grigori Fursin'; 'Hugh Leather'; 'Basile STARYNKEVITCH';
gcc@gcc.gnu.org;
'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED];
'Taras Glek';
'Diego Novillo'; 'Mike O'Boyle'
Subject: Re: Defining a common plugin machinery
Sounds like you're almost in need of a generic data marshalling interface
here.
Why do we need the complication of data marshaling?
I don't see why we need to define that all plugin hooks have the same
function interface as currently proposed. I.e. a single void*. This
makes a lot of work marshaling data both as parameters and from return
values. This is already done for us by the language (Though i may have
mis-understood the train of thought here).
I will propose the start of a new idea. This needs to be fleshed out a
lot but it would be good to get some feedback.
I will use the following terminology borrowed from QT:
signal: Is a uniquely identified hook to which zero or more slots are
added. (I.e. Caller)
slot: Is a function implementation say in a plugin. This is added to a
linked list for the specified signal. (I.e. Callee)
The main concepts in this plugin hook definition are:
* Signals can define any type of function pointer so can return values
and accept any parameters without special data marshaling
* Each signal is uniquely identified as a member variable in a struct
called Hooks
* A signal is implemented as a linked list where each node has a
reference to a slot that has been connected to the signal
* A slot is a function pointer and a unique string identifier
This differs a bit from the QT definition but i find it helpful to
describe the entities.
Important things to note:
Multiple plugins are chained one after the other. I.e. It is the
responsibility of the plugin author to call any plugins that follow it
in the list. This gives the plugin authors a bit more control over how
their plugins inter-operate with other plugins, however it would be
STRONGLY recommended that they follow a standard procedure and just call
the next plugin after they have done their work.
Basically, the idea is to provide the following structure and then most
of the work will involve manipulation of the linked lists. I.e. Querying
existing items in the LL, inserting new items before/after existing
items, removing items from the LL.
This is not a proposed end product. It is just to propose an idea. There
are a few disadvantages with the way it is implemented right now:
* Too much boilerplate code for each signal definition
* The idea of chaining calls means the responsibility of calling the
next plugin ends up with the plugin developer which could be bad if a
plugin developer does not take due care, however it also provides them
with more flexibility (not sure if that is necessary).
Now, i have NO experience with the current pass manager in GCC, but
would the passes be able to be managed using this same framework
assuming that each pass is given a unique identifier?
Thanks,
Brendon.
#include stdio.h
#include stdlib.h
/* GCC : Code */
struct Hooks
{
/* Define the blah signal. */
struct BlahFPWrap
{
const char* name;
int (*fp)(struct BlahFPWrap* self, int i, char c, float f);
void* data;
struct BlahFPWrap* next;
struct BlahFPWrap* prev;
}* blah;
struct FooFPWrap
{
const char* name;
void (*fp)(struct FooFPWrap* self);
void* data;
struct FooFPWrap* next;
struct FooFPWrap* prev;
}* foo;
};
/* Initialised by main */
struct Hooks hooks;
void SomeFunc1(void)
{
/* Call plugin hook: blah */
int result = (!hooks.blah ? 0 : hooks.blah-fp(hooks.blah, 3, 'c',
2.0f));
/* ... do stuff with result ... */
(void)result;
}
void SomeFunc2(void)
{
/* Call plugin hook: foo */
if (hooks.foo) hooks.foo-fp(hooks.foo);
}
void PlgInit(struct Hooks* h);
int main()
{
hooks.blah = NULL;
hooks.foo
Re: Defining a common plugin machinery
Hello All,
(I'm replying to Grigori CC to gcc@ - I suppose every person
interested in plugins is reading the gcc@ list)
Grigori Fursin wrote:
http://gcc-ici.sourceforge.net/papers/mfpp2007.pdf
Thanks for the reference.
By the way, if I am correct, GCC MELT (developed by Basile)
also attempts to address some of these issues with data marshaling
to modularize GCC ...
I won't call it data marshaling (which for me is a synonym of
serialization, pickling, , ie machinery to encode/decode data into a
byte stream, e.g. on disk or on the network) in MELT, because there is
no transmission (at the lowest level at least).
MELT is only (often, but not always) boxing raw GCC data, into a box or
container. This box is a MELT value containing GCC stuff (like gimple or
tree) which are not values. And the issue is of course to handle unboxed
stuff (like tree-s, gimple-s etc in the sense of gcc/tree.h
gcc/gimple.h etc...) and sometimes be able to box them in MELT values. I
won't call that marshalling. Most of MELT dirty tricks are precisely
related to the ability to handle raw GCC stuff which is not boxed in
MELT values.
There is some documentation about that in the gcc/doc/melt.texi file on
the MELT branch.
Marshalling would mean, to me at last, being able to write some internal
GCC representation on a disk and later on being able to read it again
and have GCC handling the data as if it was built by the same process.
Precompiled headers are providing a very limited way of marshalling.
MELT is not (and I am trying hard to avoid to have to write gimple stuff
and to have to read it back; this is a major task).
I would suppose that the LTO branch is marshalling some stuff in DWARF
format, but I understood that for perfomance reasons it is not supposed
to be extensible (so there is no way to add additional data into LTO).
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Hi all,
As Grigori says, and well, the project is frozen (for around one year)
since there was no expectations on it, specially since future of
plugins by that time was really unclear.
In fact the idea by the time I stopped working in it was to extend
gengtype (Garbage Collector's structure parser) to store meta-
information of the structures so that having a root node to a
structure with all the important data would allow to externally
traverse / modify GCC data without having to include all the header
files from GCC.
The reason for that wasn't to avoid the usage of GCC header files but
instead to allow data serialization and perform plugins in other
languages.
Cheers,
Cupertino
On Oct 10, 2008, at 4:14 PM, Grigori Fursin wrote:
I currently don't have any preference for a specific way to deal
with data marshaling
since currently it's enough for the MILEPOST project just to return
void* pointer,
but just wanted to mention that last year Cupertino Miranda tried to
introduce
an intermediate data layer to ICI to separate program analysis from
transformations
and potentially simplify dealing with external optimization plugins.
I think the
project has been frozen or Cupertino can comment on that if he
follows this thread ;),
but I thought to give a link to the tech. report Cupertino presented
last year,
if anyone is interested:
http://gcc-ici.sourceforge.net/papers/mfpp2007.pdf
By the way, if I am correct, GCC MELT (developed by Basile)
also attempts to address some of these issues with data marshaling
to modularize GCC ...
Cheers,
Grigori
-Original Message-
From: Brendon Costa [mailto:[EMAIL PROTECTED]
Sent: Friday, October 10, 2008 2:33 AM
To: Dave Korn
Cc: 'Taras Glek'; 'Grigori Fursin'; 'Hugh Leather'; 'Basile
STARYNKEVITCH'; gcc@gcc.gnu.org;
'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]
; 'Taras Glek';
'Diego Novillo'; 'Mike O'Boyle'
Subject: Re: Defining a common plugin machinery
Sounds like you're almost in need of a generic data marshalling
interface
here.
Why do we need the complication of data marshaling?
I don't see why we need to define that all plugin hooks have the same
function interface as currently proposed. I.e. a single void*. This
makes a lot of work marshaling data both as parameters and from
return
values. This is already done for us by the language (Though i may
have
mis-understood the train of thought here).
I will propose the start of a new idea. This needs to be fleshed
out a
lot but it would be good to get some feedback.
I will use the following terminology borrowed from QT:
signal: Is a uniquely identified hook to which zero or more slots
are
added. (I.e. Caller)
slot: Is a function implementation say in a plugin. This is added
to a
linked list for the specified signal. (I.e. Callee)
The main concepts in this plugin hook definition are:
* Signals can define any type of function pointer so can return
values
and accept any parameters without special data marshaling
* Each signal is uniquely identified as a member variable in a struct
called Hooks
* A signal is implemented as a linked list where each node has a
reference to a slot that has been connected to the signal
* A slot is a function pointer and a unique string identifier
This differs a bit from the QT definition but i find it helpful to
describe the entities.
Important things to note:
Multiple plugins are chained one after the other. I.e. It is the
responsibility of the plugin author to call any plugins that follow
it
in the list. This gives the plugin authors a bit more control over
how
their plugins inter-operate with other plugins, however it would be
STRONGLY recommended that they follow a standard procedure and just
call
the next plugin after they have done their work.
Basically, the idea is to provide the following structure and then
most
of the work will involve manipulation of the linked lists. I.e.
Querying
existing items in the LL, inserting new items before/after existing
items, removing items from the LL.
This is not a proposed end product. It is just to propose an idea.
There
are a few disadvantages with the way it is implemented right now:
* Too much boilerplate code for each signal definition
* The idea of chaining calls means the responsibility of calling the
next plugin ends up with the plugin developer which could be bad if a
plugin developer does not take due care, however it also provides
them
with more flexibility (not sure if that is necessary).
Now, i have NO experience with the current pass manager in GCC, but
would the passes be able to be managed using this same framework
assuming that each pass is given a unique identifier?
Thanks,
Brendon.
#include stdio.h
#include stdlib.h
/* GCC : Code */
struct Hooks
{
/* Define the blah signal. */
struct BlahFPWrap
{
const char* name;
int (*fp)(struct BlahFPWrap* self, int i, char c, float f);
void
Re: Defining a common plugin machinery
Cupertino Miranda wrote: Hi all, As Grigori says, and well, the project is frozen (for around one year) since there was no expectations on it, specially since future of plugins by that time was really unclear. In fact the idea by the time I stopped working in it was to extend gengtype (Garbage Collector's structure parser) to store meta-information of the structures so that having a root node to a structure with all the important data would allow to externally traverse / modify GCC data without having to include all the header files from GCC. The reason for that wasn't to avoid the usage of GCC header files but instead to allow data serialization and perform plugins in other languages. My GTY patch in http://gcc.gnu.org/ml/gcc-patches/2008-07/msg01607.html accomplishes this so treehydra JavaScript scripts can access GIMPLE. http://developer.mozilla.org/en/docs/Treehydra Great to see that others independantly identified need for the same functionality. Hopefully the patch will go into the next development cycle. Taras Cheers, Cupertino On Oct 10, 2008, at 4:14 PM, Grigori Fursin wrote: I currently don't have any preference for a specific way to deal with data marshaling since currently it's enough for the MILEPOST project just to return void* pointer, but just wanted to mention that last year Cupertino Miranda tried to introduce an intermediate data layer to ICI to separate program analysis from transformations and potentially simplify dealing with external optimization plugins. I think the project has been frozen or Cupertino can comment on that if he follows this thread ;), but I thought to give a link to the tech. report Cupertino presented last year, if anyone is interested: http://gcc-ici.sourceforge.net/papers/mfpp2007.pdf By the way, if I am correct, GCC MELT (developed by Basile) also attempts to address some of these issues with data marshaling to modularize GCC ... Cheers, Grigori -Original Message- From: Brendon Costa [mailto:[EMAIL PROTECTED] Sent: Friday, October 10, 2008 2:33 AM To: Dave Korn Cc: 'Taras Glek'; 'Grigori Fursin'; 'Hugh Leather'; 'Basile STARYNKEVITCH'; gcc@gcc.gnu.org; 'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek'; 'Diego Novillo'; 'Mike O'Boyle' Subject: Re: Defining a common plugin machinery Sounds like you're almost in need of a generic data marshalling interface here. Why do we need the complication of data marshaling? I don't see why we need to define that all plugin hooks have the same function interface as currently proposed. I.e. a single void*. This makes a lot of work marshaling data both as parameters and from return values. This is already done for us by the language (Though i may have mis-understood the train of thought here). I will propose the start of a new idea. This needs to be fleshed out a lot but it would be good to get some feedback. I will use the following terminology borrowed from QT: signal: Is a uniquely identified hook to which zero or more slots are added. (I.e. Caller) slot: Is a function implementation say in a plugin. This is added to a linked list for the specified signal. (I.e. Callee) The main concepts in this plugin hook definition are: * Signals can define any type of function pointer so can return values and accept any parameters without special data marshaling * Each signal is uniquely identified as a member variable in a struct called Hooks * A signal is implemented as a linked list where each node has a reference to a slot that has been connected to the signal * A slot is a function pointer and a unique string identifier This differs a bit from the QT definition but i find it helpful to describe the entities. Important things to note: Multiple plugins are chained one after the other. I.e. It is the responsibility of the plugin author to call any plugins that follow it in the list. This gives the plugin authors a bit more control over how their plugins inter-operate with other plugins, however it would be STRONGLY recommended that they follow a standard procedure and just call the next plugin after they have done their work. Basically, the idea is to provide the following structure and then most of the work will involve manipulation of the linked lists. I.e. Querying existing items in the LL, inserting new items before/after existing items, removing items from the LL. This is not a proposed end product. It is just to propose an idea. There are a few disadvantages with the way it is implemented right now: * Too much boilerplate code for each signal definition * The idea of chaining calls means the responsibility of calling the next plugin ends up with the plugin developer which could be bad if a plugin developer does not take due care, however it also provides them with more flexibility (not sure if that is necessary). Now, i have NO experience with the current pass manager in GCC, but would the passes be able to be managed
RE: Defining a common plugin machinery
Thanks, Taras! I slightly updated this page, i.e. we would like to be able to load plugins through environment variables to be able to optimize programs transparently as it is done in MILEPOST GCC (without Makefile modifications). By the way, we plan to extend the Interactive Compilation Interface by the end of this year to access most of the internal transformations, however it will be based on the event and call-back mechanisms, which is similar to your GCC API proposal so we shouldn't have lots of compatibility problems if we later agree on the same plugin system... Take care, Grigori -Original Message- From: Taras [mailto:[EMAIL PROTECTED] Sent: Monday, October 06, 2008 11:57 PM To: Basile STARYNKEVITCH Cc: Brendon Costa; Hugh Leather; gcc@gcc.gnu.org; 'Sean Callanan'; Cupertino Miranda; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek'; 'Diego Novillo'; Mike O'Boyle; Grigori Fursin Subject: Re: Defining a common plugin machinery Basile STARYNKEVITCH wrote: snip My hypothesis is that several plugin mechanisms for GCC already exist (on some branches or somewhere else). If a small plugin patch has a better chance to get accepted into the trunk, we should limit ourselves to such a small thing. If big plugin machinery could be accepted (I would prefer that) we should understand what would make them more acceptable. In both cases, plugins have probably some requirements defined by the future runtime license, which I don't know yet. http://gcc.gnu.org/wiki/GCC_PluginAPI I put up an API proposal. It's a result of the plugin API discussion at the GCC summit. Taras
Re: Defining a common plugin machinery
Grigori Fursin wrote: Thanks, Taras! I slightly updated this page, i.e. we would like to be able to load plugins through environment variables to be able to optimize programs transparently as it is done in MILEPOST GCC (without Makefile modifications). By the way, we plan to extend the Interactive Compilation Interface by the end of this year to access most of the internal transformations, however it will be based on the event and call-back mechanisms, which is similar to your GCC API proposal so we shouldn't have lots of compatibility problems if we later agree on the same plugin system... Personally I'm against the env var idea as it would make it harder to figure out what's going on. I think someone mentioned that the same effect could be achieved using spec files. Taras
Re: Defining a common plugin machinery
Personally I'm against the env var idea as it would make it harder to figure out what's going on. I think someone mentioned that the same effect could be achieved using spec files. Ian mentioned the idea of creating small wrapper scripts with the names: gcc/g++ etc which just call the real gcc/g++... adding the necessary command line args. These can then just be put earlier in the search path. I currently use the env var method in my project, but I think the wrapper script idea is a bit nicer than using env vars personally, so i will likely change to that soon. Brendon.
RE: Defining a common plugin machinery
Ok. I am fine with that. Actually, it requires minor modifications to the GCC anyway, so I can just keep them as patches for the ICI/MILEPOST GCC ;) ... Cheers, Grigori -Original Message- From: Taras Glek [mailto:[EMAIL PROTECTED] Sent: Thursday, October 09, 2008 8:29 AM To: Grigori Fursin Cc: 'Basile STARYNKEVITCH'; 'Brendon Costa'; 'Hugh Leather'; gcc@gcc.gnu.org; 'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek'; 'Diego Novillo'; 'Mike O'Boyle' Subject: Re: Defining a common plugin machinery Grigori Fursin wrote: Thanks, Taras! I slightly updated this page, i.e. we would like to be able to load plugins through environment variables to be able to optimize programs transparently as it is done in MILEPOST GCC (without Makefile modifications). By the way, we plan to extend the Interactive Compilation Interface by the end of this year to access most of the internal transformations, however it will be based on the event and call-back mechanisms, which is similar to your GCC API proposal so we shouldn't have lots of compatibility problems if we later agree on the same plugin system... Personally I'm against the env var idea as it would make it harder to figure out what's going on. I think someone mentioned that the same effect could be achieved using spec files. Taras
RE: Defining a common plugin machinery
Personally I'm against the env var idea as it would make it harder to figure out what's going on. I think someone mentioned that the same effect could be achieved using spec files. Ian mentioned the idea of creating small wrapper scripts with the names: gcc/g++ etc which just call the real gcc/g++... adding the necessary command line args. These can then just be put earlier in the search path. I currently use the env var method in my project, but I think the wrapper script idea is a bit nicer than using env vars personally, so i will likely change to that soon. That's right. It's a nicer solution. We just already have environment variables in our ICI implementation, but it can be useful if we will one day switch to the common plugin system without support for env variables ... Cheers, Grigori Brendon.
Re: Defining a common plugin machinery
On Thu, Oct 9, 2008 at 05:26, Hugh Leather [EMAIL PROTECTED] wrote: I think the env var solution is easier for people to use and immediately understand. There would be nothing to stop those people who don't like env vars from using the shell wrapper approach. Why not allow both? Environment variables have hidden side-effects that are often hard to debug. It is easy to forget that you set them. I also agree that we should not allow environment variables to control plugins. Diego.
Re: Defining a common plugin machinery
Aye up all,
I think the env var solution is easier for people to use and
immediately understand. There would be nothing to stop those people who
don't like env vars from using the shell wrapper approach. Why not
allow both?
Are you sure about this style of event/callback mechanism? It seems
to scale poorly. Isn't it likely to be a bit inefficient, too? Through
this approach, plugins can't cooperate; they can't easily define their
own events and it feels too early to prevent that.
It looks like it's trying to replicate the abstraction of calling a
function with something worse. What I mean is that I think what people
would like to write is:
GCC side:
void fireMyEvent( 10, hello );
Plugin side:
void handleMyEvent( int n, const char* msg ) {
// use args
}
But now they have to write:
GCC side:
//Add to enum
enum plugin_event {
PLUGIN_MY_EVENT
}
// Create struct for args
typedef struct my_event_args {
int n;
const char* msg;
} my_event_args;
// Call it:
my_event_args args = { 10, hello };
plugin_callback( PLUGIN_MY_EVENT, args };
Plugin side:
void handleMyEvent( enum plugin_event, void* data, void*
registration_data ) {
if( plugin_event == PLUGIN_MY_EVENT ) {
my_event_args* args = ( my_event_args* )data;
// Use args
}
}
Which seems a bit ugly to me. Although, it does have the advantage
of being easy to implement on the GCC side.
And, if they're replacing a heuristic and need a return value then
even more lines of code are needed on both sides. How would this style
work for replacing heuristics?
Cheers,
Hugh.
Grigori Fursin wrote:
Personally I'm against the env var idea as it would make it harder to
figure out what's going on. I think someone mentioned that the same
effect could be achieved using spec files.
Ian mentioned the idea of creating small wrapper scripts with the names:
gcc/g++ etc which just call the real gcc/g++... adding the necessary
command line args. These can then just be put earlier in the search path.
I currently use the env var method in my project, but I think the
wrapper script idea is a bit nicer than using env vars personally, so i
will likely change to that soon.
That's right. It's a nicer solution. We just already have environment variables
in our ICI implementation, but it can be useful if we will one day switch
to the common plugin system without support for env variables ...
Cheers,
Grigori
Brendon.
RE: Defining a common plugin machinery
Diego Novillo wrote on 09 October 2008 14:27: On Thu, Oct 9, 2008 at 05:26, Hugh Leather [EMAIL PROTECTED] wrote: I think the env var solution is easier for people to use and immediately understand. There would be nothing to stop those people who don't like env vars from using the shell wrapper approach. Why not allow both? Environment variables have hidden side-effects that are often hard to debug. It is easy to forget that you set them. And they presumably break ccache's determinism. I also agree that we should not allow environment variables to control plugins. I think it's best if the compiler's generated output is solely determined by the command line and the pre-processed source, so I'm with this side of the argument. cheers, DaveK -- Can't think of a witty .sigline today
RE: Defining a common plugin machinery
Well, I see the point and I am fine with that. And as I mentioned I can continue using some patches for my projects that currently use environment variables or later move to the GCC wrappers if the majority decides not to support this mode ;) Cheers, Grigori -Original Message- From: Dave Korn [mailto:[EMAIL PROTECTED] Sent: Thursday, October 09, 2008 5:54 PM To: 'Diego Novillo'; 'Hugh Leather' Cc: 'Grigori Fursin'; 'Brendon Costa'; 'Taras Glek'; 'Basile STARYNKEVITCH'; gcc@gcc.gnu.org; 'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek'; 'Mike O'Boyle' Subject: RE: Defining a common plugin machinery Diego Novillo wrote on 09 October 2008 14:27: On Thu, Oct 9, 2008 at 05:26, Hugh Leather [EMAIL PROTECTED] wrote: I think the env var solution is easier for people to use and immediately understand. There would be nothing to stop those people who don't like env vars from using the shell wrapper approach. Why not allow both? Environment variables have hidden side-effects that are often hard to debug. It is easy to forget that you set them. And they presumably break ccache's determinism. I also agree that we should not allow environment variables to control plugins. I think it's best if the compiler's generated output is solely determined by the command line and the pre-processed source, so I'm with this side of the argument. cheers, DaveK -- Can't think of a witty .sigline today
RE: Defining a common plugin machinery
Grigori Fursin wrote on 09 October 2008 17:03: Well, I see the point and I am fine with that. And as I mentioned I can continue using some patches for my projects that currently use environment variables or later move to the GCC wrappers if the majority decides not to support this mode ;) Wrappers could look up your preferred environment variables and turn them into the appropriate command-line options! :) cheers, DaveK -- Can't think of a witty .sigline today
Re: Defining a common plugin machinery
Hugh Leather wrote:
Aye up all,
I think the env var solution is easier for people to use and
immediately understand. There would be nothing to stop those people
who don't like env vars from using the shell wrapper approach. Why
not allow both?
I think the other replies addressed this question. I've updated the wiki
to reflect the consensus on env vars.
Are you sure about this style of event/callback mechanism? It
seems to scale poorly. Isn't it likely to be a bit inefficient, too?
Through this approach, plugins can't cooperate; they can't easily
define their own events and it feels too early to prevent that.
I'm trying to take the pragmatic approach. There are features we
absolutely need in GCC. The simplest possible solution is likely to be
accepted sooner.
The proposed API is not to become carved in stone. If it turns out 10
years down the road we chose the wrong plugin API, I'm all for scrapping
it and replacing it with something more relevant.
It looks like it's trying to replicate the abstraction of calling a
function with something worse. What I mean is that I think what
people would like to write is:
GCC side:
void fireMyEvent( 10, hello );
Plugin side:
void handleMyEvent( int n, const char* msg ) {
// use args
}
But now they have to write:
GCC side:
//Add to enum
enum plugin_event {
PLUGIN_MY_EVENT
}
// Create struct for args
typedef struct my_event_args {
int n;
const char* msg;
} my_event_args;
// Call it:
my_event_args args = { 10, hello };
plugin_callback( PLUGIN_MY_EVENT, args };
Plugin side:
void handleMyEvent( enum plugin_event, void* data, void*
registration_data ) {
if( plugin_event == PLUGIN_MY_EVENT ) {
my_event_args* args = ( my_event_args* )data;
// Use args
}
}
Which seems a bit ugly to me. Although, it does have the advantage
of being easy to implement on the GCC side.
This approach takes 5 more minutes of development time, that wont be a
noticeable difference in the long run.
And, if they're replacing a heuristic and need a return value then
even more lines of code are needed on both sides. How would this
style work for replacing heuristics?
I don't know how heuristics work in GCC work. It is possible that these
handlers might need to return an integer value, but so far in my plugin
work I have not needed that so I didn't include return values in the
proposal.
Taras
RE: Defining a common plugin machinery
Well, we need to return values or even structures using plugins for our
MILEPOST project
to tune cost models. A simple example is loop unrolling: in our current plugin
implementation
(MILEPOST GCC/ICI) we register a plugin function that will predict loop
unrolling and pass some
code features to it (code size, etc) whenever loop unrolling is performed. Then
the plugin
communicates with the predictive model server (implemented in MATLAB) and
returns a loop unrolling
factor (as an integer). However, we will need to return more complex structures
in case of polyhedral
optimizations in GCC 4.4 (GRAPHITE) or other optimizations/code generation...
Cheers,
Grigori
-Original Message-
From: Taras Glek [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 09, 2008 6:11 PM
To: Hugh Leather
Cc: Grigori Fursin; 'Brendon Costa'; 'Basile STARYNKEVITCH'; gcc@gcc.gnu.org;
'Sean Callanan';
'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek';
'Diego Novillo'; 'Mike
O'Boyle'
Subject: Re: Defining a common plugin machinery
Hugh Leather wrote:
Aye up all,
I think the env var solution is easier for people to use and
immediately understand. There would be nothing to stop those people
who don't like env vars from using the shell wrapper approach. Why
not allow both?
I think the other replies addressed this question. I've updated the wiki
to reflect the consensus on env vars.
Are you sure about this style of event/callback mechanism? It
seems to scale poorly. Isn't it likely to be a bit inefficient, too?
Through this approach, plugins can't cooperate; they can't easily
define their own events and it feels too early to prevent that.
I'm trying to take the pragmatic approach. There are features we
absolutely need in GCC. The simplest possible solution is likely to be
accepted sooner.
The proposed API is not to become carved in stone. If it turns out 10
years down the road we chose the wrong plugin API, I'm all for scrapping
it and replacing it with something more relevant.
It looks like it's trying to replicate the abstraction of calling a
function with something worse. What I mean is that I think what
people would like to write is:
GCC side:
void fireMyEvent( 10, hello );
Plugin side:
void handleMyEvent( int n, const char* msg ) {
// use args
}
But now they have to write:
GCC side:
//Add to enum
enum plugin_event {
PLUGIN_MY_EVENT
}
// Create struct for args
typedef struct my_event_args {
int n;
const char* msg;
} my_event_args;
// Call it:
my_event_args args = { 10, hello };
plugin_callback( PLUGIN_MY_EVENT, args };
Plugin side:
void handleMyEvent( enum plugin_event, void* data, void*
registration_data ) {
if( plugin_event == PLUGIN_MY_EVENT ) {
my_event_args* args = ( my_event_args* )data;
// Use args
}
}
Which seems a bit ugly to me. Although, it does have the advantage
of being easy to implement on the GCC side.
This approach takes 5 more minutes of development time, that wont be a
noticeable difference in the long run.
And, if they're replacing a heuristic and need a return value then
even more lines of code are needed on both sides. How would this
style work for replacing heuristics?
I don't know how heuristics work in GCC work. It is possible that these
handlers might need to return an integer value, but so far in my plugin
work I have not needed that so I didn't include return values in the
proposal.
Taras
RE: Defining a common plugin machinery
That's right - I just now need to fight my laziness and at some point write a few lines of code for the wrapper ;) ... Cheers, Grigori -Original Message- From: Dave Korn [mailto:[EMAIL PROTECTED] Sent: Thursday, October 09, 2008 6:09 PM To: 'Grigori Fursin'; 'Diego Novillo'; 'Hugh Leather' Cc: 'Brendon Costa'; 'Taras Glek'; 'Basile STARYNKEVITCH'; gcc@gcc.gnu.org; 'Sean Callanan'; 'Cupertino Miranda'; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 'Taras Glek'; 'Mike O'Boyle' Subject: RE: Defining a common plugin machinery Grigori Fursin wrote on 09 October 2008 17:03: Well, I see the point and I am fine with that. And as I mentioned I can continue using some patches for my projects that currently use environment variables or later move to the GCC wrappers if the majority decides not to support this mode ;) Wrappers could look up your preferred environment variables and turn them into the appropriate command-line options! :) cheers, DaveK -- Can't think of a witty .sigline today
Re: Defining a common plugin machinery
Grigori Fursin wrote: Well, we need to return values or even structures using plugins for our MILEPOST project to tune cost models. A simple example is loop unrolling: in our current plugin implementation (MILEPOST GCC/ICI) we register a plugin function that will predict loop unrolling and pass some code features to it (code size, etc) whenever loop unrolling is performed. Then the plugin communicates with the predictive model server (implemented in MATLAB) and returns a loop unrolling factor (as an integer). However, we will need to return more complex structures in case of polyhedral optimizations in GCC 4.4 (GRAPHITE) or other optimizations/code generation... So what would you propose we do about return values? Some sort of pointer, or a code suggesting that the caller look up a variable some where? Or should we just suggest using void*event_data to pass a struct with fields to be used as outparams(I like this idea as it means no changes are needed :)? Taras
RE: Defining a common plugin machinery
Taras Glek wrote on 09 October 2008 17:37: Grigori Fursin wrote: Well, we need to return values or even structures using plugins for our MILEPOST project to tune cost models. A simple example is loop unrolling: in our current plugin implementation (MILEPOST GCC/ICI) we register a plugin function that will predict loop unrolling and pass some code features to it (code size, etc) whenever loop unrolling is performed. Then the plugin communicates with the predictive model server (implemented in MATLAB) and returns a loop unrolling factor (as an integer). However, we will need to return more complex structures in case of polyhedral optimizations in GCC 4.4 (GRAPHITE) or other optimizations/code generation... So what would you propose we do about return values? Some sort of pointer, or a code suggesting that the caller look up a variable some where? Or should we just suggest using void*event_data to pass a struct with fields to be used as outparams(I like this idea as it means no changes are needed :)? Sounds like you're almost in need of a generic data marshalling interface here. It would be really neat if you could figure out a way to specify these structs and their values as RTX. Any plugin API could then be passed an RTX with all its args in a parallel, and could return an rtx likewise. cheers, DaveK -- Can't think of a witty .sigline today
Re: Defining a common plugin machinery
On Thu, Oct 09, 2008 at 06:02:40PM +0200, Grigori Fursin wrote: Well, I see the point and I am fine with that. And as I mentioned I can continue using some patches for my projects that currently use environment variables or later move to the GCC wrappers if the majority decides not to support this mode ;) Cheers, Grigori Well your gcc wrapper scripts can always use the environment variables you want, and create the appropriate command lines. I tend to be on the side of no environment variables, because it makes is one fewer thing that a user could forget to include on their bug report. With explicit command lines, it is more likely that the GCC community would be able to debug problems. -- Michael Meissner, IBM 4 Technology Place Drive, MS 2203A, Westford, MA, 01886, USA [EMAIL PROTECTED]
Re: Defining a common plugin machinery
Sounds like you're almost in need of a generic data marshalling interface
here.
Why do we need the complication of data marshaling?
I don't see why we need to define that all plugin hooks have the same
function interface as currently proposed. I.e. a single void*. This
makes a lot of work marshaling data both as parameters and from return
values. This is already done for us by the language (Though i may have
mis-understood the train of thought here).
I will propose the start of a new idea. This needs to be fleshed out a
lot but it would be good to get some feedback.
I will use the following terminology borrowed from QT:
signal: Is a uniquely identified hook to which zero or more slots are
added. (I.e. Caller)
slot: Is a function implementation say in a plugin. This is added to a
linked list for the specified signal. (I.e. Callee)
The main concepts in this plugin hook definition are:
* Signals can define any type of function pointer so can return values
and accept any parameters without special data marshaling
* Each signal is uniquely identified as a member variable in a struct
called Hooks
* A signal is implemented as a linked list where each node has a
reference to a slot that has been connected to the signal
* A slot is a function pointer and a unique string identifier
This differs a bit from the QT definition but i find it helpful to
describe the entities.
Important things to note:
Multiple plugins are chained one after the other. I.e. It is the
responsibility of the plugin author to call any plugins that follow it
in the list. This gives the plugin authors a bit more control over how
their plugins inter-operate with other plugins, however it would be
STRONGLY recommended that they follow a standard procedure and just call
the next plugin after they have done their work.
Basically, the idea is to provide the following structure and then most
of the work will involve manipulation of the linked lists. I.e. Querying
existing items in the LL, inserting new items before/after existing
items, removing items from the LL.
This is not a proposed end product. It is just to propose an idea. There
are a few disadvantages with the way it is implemented right now:
* Too much boilerplate code for each signal definition
* The idea of chaining calls means the responsibility of calling the
next plugin ends up with the plugin developer which could be bad if a
plugin developer does not take due care, however it also provides them
with more flexibility (not sure if that is necessary).
Now, i have NO experience with the current pass manager in GCC, but
would the passes be able to be managed using this same framework
assuming that each pass is given a unique identifier?
Thanks,
Brendon.
#include stdio.h
#include stdlib.h
/* GCC : Code */
struct Hooks
{
/* Define the blah signal. */
struct BlahFPWrap
{
const char* name;
int (*fp)(struct BlahFPWrap* self, int i, char c, float f);
void* data;
struct BlahFPWrap* next;
struct BlahFPWrap* prev;
}* blah;
struct FooFPWrap
{
const char* name;
void (*fp)(struct FooFPWrap* self);
void* data;
struct FooFPWrap* next;
struct FooFPWrap* prev;
}* foo;
};
/* Initialised by main */
struct Hooks hooks;
void SomeFunc1(void)
{
/* Call plugin hook: blah */
int result = (!hooks.blah ? 0 : hooks.blah-fp(hooks.blah, 3, 'c',
2.0f));
/* ... do stuff with result ... */
(void)result;
}
void SomeFunc2(void)
{
/* Call plugin hook: foo */
if (hooks.foo) hooks.foo-fp(hooks.foo);
}
void PlgInit(struct Hooks* h);
int main()
{
hooks.blah = NULL;
hooks.foo = NULL;
PlgInit(hooks);
return 0;
}
/* Writeme... */
#define PLUGIN_INSERT_BEFORE(Hooks, Struct, Hook, FuncPtr, Before, SlotName)
/* In plugin */
#define PLUGIN_NAME myplg
static void MyFoo(struct FooFPWrap* self)
{
printf(MyFoo\n);
if (self-next) self-next-fp(self-next);
}
static void MyBlah(struct BlahFPWrap* self, int i, char c, float f)
{
printf(MyBlah\n);
if (self-next) self-next-fp(self-next, i, c, f);
}
void PlgInit(struct Hooks* h)
{
PLUGIN_INSERT_BEFORE(h, struct BlahFPWrap, blah, MyBlah, NULL,
PLUGIN_NAME _MyBlah);
PLUGIN_INSERT_BEFORE(h, struct FooFPWrap, foo, MyFoo, NULL,
PLUGIN_NAME _MyFoo);
}
void PlgShut(struct Hooks* h)
{
PLUGIN_REMOVE(h, PLUGIN_NAME _MyBlah);
PLUGIN_REMOVE(h, PLUGIN_NAME _MyFoo);
}
Re: Defining a common plugin machinery
Basile STARYNKEVITCH wrote: snip My hypothesis is that several plugin mechanisms for GCC already exist (on some branches or somewhere else). If a small plugin patch has a better chance to get accepted into the trunk, we should limit ourselves to such a small thing. If big plugin machinery could be accepted (I would prefer that) we should understand what would make them more acceptable. In both cases, plugins have probably some requirements defined by the future runtime license, which I don't know yet. http://gcc.gnu.org/wiki/GCC_PluginAPI I put up an API proposal. It's a result of the plugin API discussion at the GCC summit. Taras
Re: Defining a common plugin machinery
On Mon, 6 Oct 2008, Taras wrote: http://gcc.gnu.org/wiki/GCC_PluginAPI I put up an API proposal. It's a result of the plugin API discussion at the GCC summit. I believe the API also needs interfaces for verifying compatibility (exporting the required GCC version, target triplet and anything else likely to affect compatibility) and for --version and --help information so people can use those options together with -fplugin to get the plugin's version number and bug-reporting information, as previously discussed. -- Joseph S. Myers [EMAIL PROTECTED]
Re: Defining a common plugin machinery
Joseph S. Myers wrote: On Mon, 6 Oct 2008, Taras wrote: http://gcc.gnu.org/wiki/GCC_PluginAPI I put up an API proposal. It's a result of the plugin API discussion at the GCC summit. I believe the API also needs interfaces for verifying compatibility (exporting the required GCC version, target triplet and anything else likely to affect compatibility) and for --version and --help information so people can use those options together with -fplugin to get the plugin's version number and bug-reporting information, as previously discussed. Agreed I added that to the wiki. Thanks, Taras
Re: Defining a common plugin machinery
On Wed, 1 Oct 2008, Basile STARYNKEVITCH wrote: is the goal to only permit FSF copyrighted GPLed plugins The FSF is concerned about making and keeping software free. Copyright assignment to the FSF is required to get code into the FSF version of GCC; it is not needed otherwise. Gerald
Re: Defining a common plugin machinery
Gerald Pfeifer wrote: On Wed, 1 Oct 2008, Basile STARYNKEVITCH wrote: is the goal to only permit FSF copyrighted GPLed plugins The FSF is concerned about making and keeping software free. Copyright assignment to the FSF is required to get code into the FSF version of GCC; it is not needed otherwise. Gerald And of course, as always, it is fine to do anything with your own personal copy of gcc, including fiddling with plugins. But when it comes to distribution, the FSF would like to encourage only free software plugins, and discourage or if possible prohibit propietary plugins (what can be prohibited is quite unclear for a number of reasons, which is why this is under discussion).
Re: Defining a common plugin machinery
Hello All,
Robert Dewar wrote:
Gerald Pfeifer wrote:
On Wed, 1 Oct 2008, Basile STARYNKEVITCH wrote:
is the goal to only permit FSF copyrighted GPLed plugins
The FSF is concerned about making and keeping software free.
Copyright assignment to the FSF is required to get code into the FSF
version of
GCC; it is not needed otherwise.
Gerald
Free software is a concern I do have and do share. But notice that
GPL-ed software is different from FSF copyrighted software. First, there
exist GPL-ed software which are not FSF copyrighted, and second (and
much more relevant to GCC plugins) getting authorized to contribute to
GPL-ed software is (at least for me, and probably many others) much less
difficult than getting the FSF copyright transfer form signed by his/her
employing organization (with the additional issue, that FSF copyright
transfers are not exactly public).
Just because of that, I believe (maybe wrongly) that the plugin legal
effort, once approved by FSF (ie once the yet unknown runtime license is
adopted published) might have as a side effect that GPL-ed code which
is not FSF copyrighted may be more common in GCC (as plugins). At least
in Europe, be authorized to publish GPL-ed code is much less difficult
than getting the FSF copyright transfer.
And of course, as always, it is fine to do anything with your
own personal copy of gcc, including fiddling with plugins. But
when it comes to distribution, the FSF would like to encourage
only free software plugins, and discourage or if possible
prohibit propietary plugins (what can be prohibited is quite
unclear for a number of reasons, which is why this is under
discussion).
I do think (but I am perhaps naive, and certainly not a lawyer, and I
know nothing about the US law system) that a way to make proprietary
plugins harder is to avoid stabilizing an API. By the way, my MELT
branch in practice enforces that quite well. For technical reasons, when
a core part of MELT is slightly modified (and this might probably
happens when GCC with MELT is evolving) all the MELT generated C files
(which are generated from MELT sources gcc/melt/*.bysl) have to be
generated again. This means that in practice MELT is useful only for
code for which you have the MELT source files [of course this technical
restriction does not technically require GPL on the MELT source files;
the GPL requirement on *.bysl files is a legal consequence of GPL in
GCC], so a proprietary MELT plugin is nearly impossible.
For those having compiled the MELT branch, to be more specific, if you
modify gcc/melt/warmelt-first.bysl [I agree you might not do that often]
then you have to regenerate every warmelt*.c files, their warmelt*.so
files, and every other MELT generated file. Hence, the MELT source code
should be accessible, and the installation of a MELT branch GCC does
install every *.bysl file. In practice proprietary MELT plugins would be
very impractical (assuming MELT would be inside gcc-4.y with y=5, i.e.
a *.so file produced thru MELT for gcc-4.y.3 won't work with gcc-4.y.4
unless regenerated from its MELT source). BTW, there are other languages
with similar proprieties. For example, an Ocaml file compiled by
ocaml-3.10 cannot be linked into an Ocaml program compiled by
ocaml-3.11; you need to recompile all the Ocaml code at every Ocaml release.
Notice also that I don't know about any proprietary library which does
not ship its header files *.h, but only a precompiled header file.
This seems significant to me.
In practice, I strongly believe that the main way of avoiding
proprietary plugin is just to code as usual, i.e. avoid freezing any
internal API or internal data representations inside GCC. This is
already the case (and won't change soon even if people wanted to). So I
don't think that a lot of proprietary plugins will appear. Also, while
there is a real market for services around software tools like compiler,
the market on proprietary compilers is apparently shrinking (partly
thanks to the success of GCC) - outside of Microsoft systems at least.
So I don't think that proprietary plugins to GCC will be very
successful; I don't perceive them as a significant threat to GCC (but I
understand some of the FSF concerns regarding the runtime license).
But then, I am French, politically more on the social democrat side,
working in a government owned organization, so not struggling with the
market each hour (even if I do have market related concerns, because I
have to find fundings which require market relevances.).
The compiler landscape has significantly changed thanks to GCC. Few
organizations can ignore GCC!
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Brendon Costa wrote:
I believe we should first focus (when the runtime license will permit
that) on making whatever plugin machinery available and merged into
the trunk (when it comes back to stage one). This is not an easy task.
Isn't the point of this discussion to decide what features to put into a
plugin framework? I.e. We need a whatever plugin machinery available
to exist before we can even think about merging that into the trunk and
defining what that looks like is the point of this discussion i thought.
I entirely agree. Apologies to everyone if I badly expressed myself.
Possible steps for moving forward with this:
1) Define what features we need for the first release, and think about
what we may want in the future
2) See which frameworks currently exist and how each meets the necessary
features identified
3) Either use one of the above frameworks as a base or start a new
framework on the plugin branch
4) Work on the base set of features for a first release
5) Make sure the branch is up to date/tracking the trunk
6) Look at merging into the trunk when licensing is complete
We are still at 1 (and partially identifying projects for 2) as far as i
understand.
I also agree. What I don't understand is if having a simple crude plugin
mechanism make it easier to be accepted in the trunk. I don't understand
what makes features code easy to be accepted in a stage one trunk. I
don't understand if havving a small plugin machinery (there already
exists some) make it easier to be accepted in the trunk.
I still do not understand how and when big patches get accepted in the
trunk. What are the social issues involved? What is the best way to get
code reviewers (those able to approve a patch) interested by any big
plugin patch? (And FYI I am asking myself the same question for MELT:
what should I do now to get some day in the future MELT accepted in the
trunk?).
So far, i think libplugin seems to be the most general plugin
framework for GCC i have had a chance to look at (It was easy to look at
because it has some decent documentation online).
In practice, I think that we should first try to get some code into
the trunk which make some plugin work on some common easy host system
(Linux), and only after try to generalize the work to harder hosts.
I agree, that providing working code for only simple to implement
platforms (and basic plugin features) at first is a good idea (but do so
on a branch first, then merge that to the trunk once it is operational).
However we do not want to start with a framework that will need to be
completely redesigned in the future to later support other platforms or
usages. I.e. Thinking ahead but not necessarily implementing ahead...
I fully agree. But who thinks that the libplugin patch (or any other
plugin machinery) could be accepted into the trunk?
My main concern is plugins passes.
Yes. We have not really looked at this more important aspect in much
detail, how to manage passes with plugins. It looks like libplugin has
some ideas for pass management that may help? Any thoughts?
Apparently they have.
But we need to have a more exact picture of what the GCC steering
commitee the FSF wants (and even more importantly do not wants)
regarding plugins. I could imagine that they ( perhaps us) want some
tricks that make proprietary plugins impractical, but I have no idea of
what that would technically mean (because I have no understanding of the
legal social system involved).
My hypothesis is that several plugin mechanisms for GCC already exist
(on some branches or somewhere else). If a small plugin patch has a
better chance to get accepted into the trunk, we should limit ourselves
to such a small thing. If big plugin machinery could be accepted (I
would prefer that) we should understand what would make them more
acceptable. In both cases, plugins have probably some requirements
defined by the future runtime license, which I don't know yet.
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Hi Brendon, Thanks for reading all this :-) Comments in line. Brendon Costa wrote: I have notes inline below, following is my summary of libplugin from what i understand of your posts: * It exists as a fraemwork that works with GCC now * It uses xml files to define plugins (Allows making new plugins as combinations of others without making a new shared library, i.e. just create an xml file that describes the plugin) * It handles issues with inter-dependencies between plugins * It uses a push framework, where function pointers are replaced/chained in the original application rather than explicit calls to plugins (Provides more extensibility in a application that makes heavy use of function pointers, but produces a less explicit set of entry points or hooks for plugins) * Currently it provides automatic loading of plugins without specific user request * It already has a framework for allowing plugins to interact with the pass manager If you can think of any other points to summarize the features it might be helpful as you are closer to it. The issues i see with this framework: * it seems to provide a lot of features that we may not necessarily need (That should be up for discussion) Yes, it's entirely possible that it has more features than most people will need. Some have been taken from Eclipse, since they've some years fleshing out the problems with a major plugin system, some come from AOP. If we only ever see there being 10 entry points (i.e. not very fine grained) and not having cooperating plugins, then libplugin is definitely overkill. * plugin entry points are not well defined but can be any function pointer call Achh, hoist by my own petard! Part of my design goals was to make the changes almost invisible in GCC. I thought it might make it more acceptable to people. The difference between plugin-GCC and normal GCC would be 10 lines in toplev.c. Two solutions come to mind. One would be to provide an empty define for documentation purposes. Something like #define EXTENSION? The other is to say, well, entry points are pretty well defined in the plugin XMLs. Again, this is the approach taken by Eclipse and seems to work well. Actually, I have another which I describe later when I talk about enforcing application plugins. It is also possible to have entry points that aren't events, join-points or lists, those are just the ones that the system makes easy for you. Some questions: * How does the framework interact with the compile command line arguments? I wrote a bit about that in a previous email http://gcc.gnu.org/ml/gcc/2008-10/msg00011.html. I didn't put all the details in. For example, at the moment, the plugin system gets to see the command line before any other part of GCC. This allows plugins to alter the command line themselves. Secondly, though I was thinking of making this optional, the plugin system removes all plugin command line options from the command line before GCC gets to see it. * Does this work on platforms that dont support -rdynamic or can it be modified to do so in the future? It uses ltdl. I think that can be made to statically load dlls on those systems. You'd then have all the capabilities your base libraries provide + anything you can describe in XML only. I have no idea how practical this would be. Personally, I think plugins on those systems might be more trouble than it's worth. I've never used a compiler on one though (only compiled for them on my PC), so I don't really know much about it. I've only been developing on Linux. I don't know how much work it would be to port to other machines. Oh, I also use libffi for events and join-points. The system would certainly work without events and join-points, it just wouldn't be as fun. If you leave it in then it limits libplugin to systems with libffi. Hugh Leather wrote: *Separating Plugin system from appliction* Libplugin ships as a library. Apart from a few lines of code in toplev.c, the only other changes to GCC will be refactorings and maybe calling a few functions through pointers. As i understand the difference between the pull vs push, a plugin will load, and then modify existing function pointers in GCC to insert its own code and chain the existing code to be called after it. Is this correct? Yes, although really the plugin system does it on the plugin's behalf. We also have a 'non-enforced' distinction between plugins. There are those which use symbols in the application, making application events and join-points and those which provide their own shared libraries (some do both). So, some plugins make bits of the application extensible, some provide additional services, or both. It would be relatively easy to enforce that distinction. We could insist that no plugin is able to make bits of the application extensible unless they come with GCC. A bit like sealed packages in Java, people could
RE: Defining a common plugin machinery
Dear all, I noticed a long discussion about plugins for GCC. It seems that it's currently moving toward important legal issues, however, I wanted to backtrack and just mention that we at INRIA and in the MILEPOST project are clearly interested in having a proper plugin system in the mainline of GCC which will simplify our work on automatically tuning optimization heuristics (cost models, etc) or easily adding new transformations and modules for program analysis. We currently have a simple plugin system within Interactive Compilation Interface (http://gcc-ici.sourceforge.net) that can load external DLL plugin (transparently through the environment variables to avoid changing project Makefiles or through command line), substitutes the original Pass Manager to be able to call any passes (new analysis passes for example) in any legal order and has an event mechanism to rise events in any place in GCC and pass data to the plugin (such as information about cost model dependencies) or return parameters (such as decisions about transformations for example). Since it's relatively simple, we are currently able to port it to the new versions of GCC without problems, however, naturally, we would like to have this functionality within the GCC mainline with the defined API (that what I discussed with Taras from Mozilla during GCC Summit this year). I believe it may help making GCC a modular compiler and simplify future designs (and can be in line with the idea to use C++ for GCC development if Ian moves this project forward). By the way, Hugh Leather also developed an interesting plugin system for GCC that allows to substitute internal GCC functions with the external ones within plugins to enable hooks inside GCC (he mentioned that he plans to release it soon)... Furthermore, we will then be able to use our current MILEPOST tools and Collective Compilation Framework to automatically tune default GCC optimization heuristic for performance, code size or power consumption (instead of using -O1,2,3 levels) for a particular architecture before new versions of GCC are actually released (or for additional testing of a compiler using different combinations and orders of passes). And when the compiler is released, users can further tune their particular programs interactively or automatically through the external tools and GCC plugins. By the way, we are extending current ICI for GCC 4.4 to add cost-model tuning for major optimizations (GRAPHITE, vectorization, inlining, scheduling, register allocation, unrolling, etc) and provide function cloning with different optimizations, and naturally would like to make it compatible with the potential future common GCC plugin system, so I hope we will be able to agree on a common plugin design soon and move it forward ;) ... Regards, Grigori = Grigori Fursin, INRIA, France http://fursin.net/research -Original Message- From: Taras Glek [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 16, 2008 11:43 PM To: Diego Novillo Cc: Basile STARYNKEVITCH; gcc@gcc.gnu.org; Sean Callanan; Albert Cohen; [EMAIL PROTECTED] Subject: Re: Defining a common plugin machinery Basile STARYNKEVITCH wrote: Hello Diego and all, Diego Novillo wrote: After the FSF gives final approval on the plugin feature, we will need to coordinate towards one common plugin interface for GCC. I don't think we should be adding two different and incompatible plugin harnesses. What exactly did happen on the FSF side after the last GCC summit? I heard nothing more detailed than the Steeering Committee QA BOFS and the early draft of some legal licence involving plugins. What happened on the Steering Commitee or legal side since august 2008? Is there any annoucement regarding FSF approving plugins? I am CCing everyone who I know is working on plugin features. Apologies if I missed anyone. I would like to start by understanding what the plugin API should have. What features do we want to support? What should we export? What should be the user interface to incorporate plugin code? At what point in the compilation stage should it kick in? Once we define a common API and then we can take the implementation from the existing branches. Perhaps create a common branch? I would also like to understand what the status and features of the different branches is. The MELT plugin machinery is quite specific in its details, and I don't believe it can be used -in its current form- for other plugins. It really expects the plugin to be a MELT one. From what I remember of the plugin BOFS (but I may be wrong), an easy consensus seems to be that plugins should be loadable thru the command line (probably a -fplugin=foo meaning that some foo.so should be dlopen-ed), that they could take a single string as an argument (using -fplugin-arg=bar) and that plugins add essentially new passes
Re: Defining a common plugin machinery
/types for loops. * Add performance counter code to x86 programs. Automatically record cycles spent in any function, results finally dumped to XML. No need to link to any library (except libc) or even to change your make files. Which functions to instrument controlled by the XML specification. * A few other bits and pieces for iterative compilation stuff. I also have a scripting plugin prototype letting you write python in your plugin, but it's a long way from seeing the light of day. * *I'm sorry all that was a bit long winded. I'm also sorry not to have finished the documentation for the system, it's a bit difficult to evaluate without it. I might get it finished by mid October, but I guess that will be too late? Cheers, Hugh. Grigori Fursin wrote: Dear all, I noticed a long discussion about plugins for GCC. It seems that it's currently moving toward important legal issues, however, I wanted to backtrack and just mention that we at INRIA and in the MILEPOST project are clearly interested in having a proper plugin system in the mainline of GCC which will simplify our work on automatically tuning optimization heuristics (cost models, etc) or easily adding new transformations and modules for program analysis. We currently have a simple plugin system within Interactive Compilation Interface (http://gcc-ici.sourceforge.net) that can load external DLL plugin (transparently through the environment variables to avoid changing project Makefiles or through command line), substitutes the original Pass Manager to be able to call any passes (new analysis passes for example) in any legal order and has an event mechanism to rise events in any place in GCC and pass data to the plugin (such as information about cost model dependencies) or return parameters (such as decisions about transformations for example). Since it's relatively simple, we are currently able to port it to the new versions of GCC without problems, however, naturally, we would like to have this functionality within the GCC mainline with the defined API (that what I discussed with Taras from Mozilla during GCC Summit this year). I believe it may help making GCC a modular compiler and simplify future designs (and can be in line with the idea to use C++ for GCC development if Ian moves this project forward). By the way, Hugh Leather also developed an interesting plugin system for GCC that allows to substitute internal GCC functions with the external ones within plugins to enable hooks inside GCC (he mentioned that he plans to release it soon)... Furthermore, we will then be able to use our current MILEPOST tools and Collective Compilation Framework to automatically tune default GCC optimization heuristic for performance, code size or power consumption (instead of using -O1,2,3 levels) for a particular architecture before new versions of GCC are actually released (or for additional testing of a compiler using different combinations and orders of passes). And when the compiler is released, users can further tune their particular programs interactively or automatically through the external tools and GCC plugins. By the way, we are extending current ICI for GCC 4.4 to add cost-model tuning for major optimizations (GRAPHITE, vectorization, inlining, scheduling, register allocation, unrolling, etc) and provide function cloning with different optimizations, and naturally would like to make it compatible with the potential future common GCC plugin system, so I hope we will be able to agree on a common plugin design soon and move it forward ;) ... Regards, Grigori = Grigori Fursin, INRIA, France http://fursin.net/research -Original Message- From: Taras Glek [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 16, 2008 11:43 PM To: Diego Novillo Cc: Basile STARYNKEVITCH; gcc@gcc.gnu.org; Sean Callanan; Albert Cohen; [EMAIL PROTECTED] Subject: Re: Defining a common plugin machinery Basile STARYNKEVITCH wrote: Hello Diego and all, Diego Novillo wrote: After the FSF gives final approval on the plugin feature, we will need to coordinate towards one common plugin interface for GCC. I don't think we should be adding two different and incompatible plugin harnesses. What exactly did happen on the FSF side after the last GCC summit? I heard nothing more detailed than the Steeering Committee QA BOFS and the early draft of some legal licence involving plugins. What happened on the Steering Commitee or legal side since august 2008? Is there any annoucement regarding FSF approving plugins? I am CCing everyone who I know is working on plugin features. Apologies if I missed anyone. I would like to start by understanding what the plugin API should have. What features do we want to support? What should we export? What should be the user interface to incorporate plugin code? At what point
Re: Defining a common plugin machinery
optimization heuristics (cost models, etc) or easily adding new transformations and modules for program analysis. We currently have a simple plugin system within Interactive Compilation Interface (http://gcc-ici.sourceforge.net) that can load external DLL plugin (transparently through the environment variables to avoid changing project Makefiles or through command line), substitutes the original Pass Manager to be able to call any passes (new analysis passes for example) in any legal order and has an event mechanism to rise events in any place in GCC and pass data to the plugin (such as information about cost model dependencies) or return parameters (such as decisions about transformations for example). Since it's relatively simple, we are currently able to port it to the new versions of GCC without problems, however, naturally, we would like to have this functionality within the GCC mainline with the defined API (that what I discussed with Taras from Mozilla during GCC Summit this year). I believe it may help making GCC a modular compiler and simplify future designs (and can be in line with the idea to use C++ for GCC development if Ian moves this project forward). By the way, Hugh Leather also developed an interesting plugin system for GCC that allows to substitute internal GCC functions with the external ones within plugins to enable hooks inside GCC (he mentioned that he plans to release it soon)... Furthermore, we will then be able to use our current MILEPOST tools and Collective Compilation Framework to automatically tune default GCC optimization heuristic for performance, code size or power consumption (instead of using -O1,2,3 levels) for a particular architecture before new versions of GCC are actually released (or for additional testing of a compiler using different combinations and orders of passes). And when the compiler is released, users can further tune their particular programs interactively or automatically through the external tools and GCC plugins. By the way, we are extending current ICI for GCC 4.4 to add cost-model tuning for major optimizations (GRAPHITE, vectorization, inlining, scheduling, register allocation, unrolling, etc) and provide function cloning with different optimizations, and naturally would like to make it compatible with the potential future common GCC plugin system, so I hope we will be able to agree on a common plugin design soon and move it forward ;) ... Regards, Grigori = Grigori Fursin, INRIA, France http://fursin.net/research -Original Message- From: Taras Glek [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 16, 2008 11:43 PM To: Diego Novillo Cc: Basile STARYNKEVITCH; gcc@gcc.gnu.org; Sean Callanan; Albert Cohen; [EMAIL PROTECTED] Subject: Re: Defining a common plugin machinery Basile STARYNKEVITCH wrote: Hello Diego and all, Diego Novillo wrote: After the FSF gives final approval on the plugin feature, we will need to coordinate towards one common plugin interface for GCC. I don't think we should be adding two different and incompatible plugin harnesses. What exactly did happen on the FSF side after the last GCC summit? I heard nothing more detailed than the Steeering Committee QA BOFS and the early draft of some legal licence involving plugins. What happened on the Steering Commitee or legal side since august 2008? Is there any annoucement regarding FSF approving plugins? I am CCing everyone who I know is working on plugin features. Apologies if I missed anyone. I would like to start by understanding what the plugin API should have. What features do we want to support? What should we export? What should be the user interface to incorporate plugin code? At what point in the compilation stage should it kick in? Once we define a common API and then we can take the implementation from the existing branches. Perhaps create a common branch? I would also like to understand what the status and features of the different branches is. The MELT plugin machinery is quite specific in its details, and I don't believe it can be used -in its current form- for other plugins. It really expects the plugin to be a MELT one. From what I remember of the plugin BOFS (but I may be wrong), an easy consensus seems to be that plugins should be loadable thru the command line (probably a -fplugin=foo meaning that some foo.so should be dlopen-ed), that they could take a single string as an argument (using -fplugin-arg=bar) and that plugins add essentially new passes into pass.c - I am happy with such a picture (and could fit MELT easily into it; in its current form MELT is adding a few basilys*passes into passes.c, each having a gate which try using a MELT closure somewhere so is equivalent of testing if the pass is enabled.). Albert Cohen Grigori Fursin (both in CC) from INRIA have an Interative Compiler Interface (a big patch to GCC) which could
Re: Defining a common plugin machinery
Hugh Leather wrote: Aye up all, I've now been reading through some of the list archive. Some of the posts were about how to tell GCC which plugins to load. I thought I'd tell you how libplugin does it. Thanks for the nice explanation. I'm not sure to understand exactly how libplugin deals with adding passes; apparently, the entire pass manager (ie gcc/passes.c) has been rewritten or enhanced. Also, I did not understood the exact conceptual differences between libplugin other proposals. Apparently libplugin is much more ambitious. So we now have many plugin proposals experiments. However, we do know that there are some legal/political/license issues on these points (with the GCC community rightly wanting as hard as possible to avoid proprietary plugins), that some interaction seems to happen (notably between Steering Committee FSF), that the work is going slowly (because of lack of resource labor funding? at FSF). My perception is that the issues are not mostly technical, but still political (and probably, as Ian Taylor mentioned it in http://gcc.gnu.org/ml/gcc/2008-09/msg00442.html a lack of lawyer or other human resources at FSF, which cost much more than any reasonable person could afford individually). I actually might not understand why exactly plugins are not permitted by the current GCC licenses. What I don't understand is * what exactly do we call a plugin? I feel (but I am not a lawyer) that (on linux) it is any *.so file which is fed to dlopen. I'm not able to point what parts of the GCC license prohibit that (I actually hope that nothing prohibits it right now, if the *.so is compiled from GPLv3-ed FSF copyrighted code. the MELT branch is doing exactly that right now). * will the runtime license be working for Christmas 2008. [some messages made me think that not, it is too much lawyer work; other messages made me a bit more optimistic; I really am confused]. Of course, I don't want any hard date, but I am in the absolute darkness on the actual work already done on improving the runtime license, and even more on what needs to be fixed. Also, I have no idea of the work involved in writing new licenses (I only know that the GPLv3 effort lasted much more than one year). Did I say that I am not a lawyer, and not understanding even the basic principles of US laws (or perhaps even French ones)? * what kind of intrusiveness do we want for the plugin machinery. Do we want it to be clean and hence to touch a lot of files (in particular the details of passes the pass manager), or do we first want some quick and dirty plugin trick merged into the trunk, even if it is imperfect? * what is the plugin machinery useful for? Only adding optimisation passes, or much more ambitious (adding new front ends, back ends, targets)? * what is the interaction between the plugin machinery the rest of GCC (e.g. GGC, dump files, ) * what is the granularity plugins are wanted or needed for? Only whole passes, or something smaller than that (e.g. some specific functions inside specific passes)? * who really want plugins to happen quick, and which company would invest money [not only code] on that? * what host system do we want the plugin to work with? Is libtool dyn loader enough? Could every non static symbol inside cc1 be visible to the plugin? * do we really want one single (fits all) plugin machinery inside GCC? My feeling is that a lot of various technical efforts has already being put into plugins, but that the future runtime license may (or not) impact technicalities (perhaps making some proposed technical solutions impossible). I really don't understand what is the hard limit, i.e. what the FSF or the Steering Committee wants to avoid exactly (obviously proprietary plugins implementing new machine targets are unwanted, but what else; is the goal to only permit FSF copyrighted GPLed plugins; what would be the review policy of code going into plugins?)? I've got no idea of how would it be hard to make any plugin system accepted into the GCC trunk, and when could that work begins to start (i.e. when to send plugin patches to gcc-patches@). I tend to believe that it the main issue now. Are plugin patches supposed to be welcome -on the gcc-patches@ mailing list, for trunk acceptance- when GCC goes back in stage1? Will the first plugin patches (submitted to gcc-patches@ for acceptance into trunk) be huge or tiny patches? Technically both are possible (of course with different goals features). I even don't know what legally a plugin is. For instance, in my MELT branch code is indeed dlopen-ed, but [currently] the C code of the plugin is generated (by the plugin itself) from MELT lisp-like files, which are all inside the MELT branch (GPL-ed, FSF copyrighted) Perhaps that does not even count, from a legal point of view, as a plugin? [I really hope I am not doing unknowingly illegal things on the MELT branch; to calm everyone,
Re: Defining a common plugin machinery
On Wed, Oct 01, 2008 at 06:03:21PM +0200, Basile STARYNKEVITCH wrote: So we now have many plugin proposals experiments. However, we do know that there are some legal/political/license issues on these points (with the GCC community rightly wanting as hard as possible to avoid proprietary plugins), that some interaction seems to happen (notably between Steering Committee FSF), that the work is going slowly (because of lack of resource labor funding? at FSF). That impression isn't really right; we're getting close now to a resolution. There should be some news soon.
Re: Defining a common plugin machinery
Aye up Basile,
Thanks for wading through my gibberish :-)
*Differences with other proposals.*
I'll have a stab at some differences between this system and the
others. But, this is going to be a bit difficult since I haven't seen
them all :-)
*Separating Plugin system from appliction*
Libplugin ships as a library. Apart from a few lines of code in
toplev.c, the only other changes to GCC will be refactorings and
maybe calling a few functions through pointers.
I think it's important to separate the plugin system from the
application. Doing plugins well, IMO, requires a lot of code. It
shouldn't be spread through the app. It also cleanly separates
plugin mechanism from the actual extensions the app wants.
Finally, plugins have to be extensible too. They should really be on
a nearly equal footing with the app. Otherwise plugin developers
who want the plugins to be extensible will need to reimplement there
own extensibility system.
*Pull vs push*
Libplugin has a 'push' architecture, not a 'pull' one. What I mean
is that the system pushes plugin awareness onto the application
rather than requiring the application to call out to the plugin
system all the time.
Here's an example of that. In GCC, passes have execute and gate
functions which are already function pointers. With libplugin you
can make these replaceable/extensible/event-like without changing a
single line of code in GCC.
An external plugin, the gcc-pass-manager plugin, tells the world
that it has a join point for each gate and execute function of every
pass in the system.
A quick aside on join points. Suppose you have a function
int myHeuristic( basic_block bb, rtx insn ) {
// blah, blah
return x;
}
If we redefine that function to be called myHeuristic_default
and setup a function pointer with same name:
static int myHeuristic_default( basic_block bb, rtx insn ) {
... }
int ( *myHeuristic )( basic_block bb, rtx insn ) =
myHeuristic_default;
Now we can use the heuristic unchanged in the code.
But if we tell libplugin that that is a join point with
id=my-heuristic (in the XML for some plugin) it will create
1. An event called my-heuristic.before with signature
void (basic_block, rtx)
2. A replaceble function stack called my-heuristic.around
with signature int (basic_block, rtx)
3. An event called my-heuristic.after with signature void
(int, basic_block, rtx)
If anyone extends any of those, then the function pointer,
myHeuristic, will be replaced with a dynamically built function
which does, roughly:
int myHeuristic_dynamic( basic_block bb, rtx insn ) {
// call listeners to before
foreach f in my-heuristic.before.eventHandlers {
f( bb, insn );
}
// do the behaviour of the heuristic
top = my-heuristic.around.topOfAdviceStack;
// top is initially myHeuristic_default unless someone
overrode it
// top can also access the rest of the advice stack, but
I ignore that here
int rval = top( bb, insn );
// call listeners to after
foreach f in my-heuristic.after.eventHandlers {
f( rval, bb, insn );
}
return rval
}
It then sets myHeuristic = myHeuristic_dynamic. Note that if no
one listens to the events of pushes advice on the around stack,
then the original function pointer isn't changed - no
performance cost.
Now the dynamic functions are pushed onto each passes' gate or
execute only if someone wants to extends them. Not one line of code
was changed in GCC. This is what I mean by push not pull.
Consider the alternative, which I call 'pull' because it has to pull
plugin awareness from the system. It would require each pass and
gate to check if anyone was interested, lots of changes to the
code. Or every calling site would have to do it, similarly
unpleasant for most uses.
This is great when you already have function pointers. If you don't
you have to make only minimal changes. Your code remains efficient
if no one extends it.
*Scalable and Granularity*
The system is very scalable. Really this is due to the push
architecture.
Consider if events were implemented by something like this. A
single function:
void firePluginEvent( int eventId, void* data );
Every event would be fired by calling through this one function.
Plugins would register a callback function.
This is fine when you only have a few events but look what happens
when you have very fine grained events happening millions
Re: Defining a common plugin machinery
Hugh Leather wrote: Aye up Basile, Thanks for wading through my gibberish :-) *Differences with other proposals.* I'll have a stab at some differences between this system and the others. But, this is going to be a bit difficult since I haven't seen them all :-) *Separating Plugin system from appliction* Libplugin ships as a library. Apart from a few lines of code in toplev.c, the only other changes to GCC will be refactorings and maybe calling a few functions through pointers. The point is who will do the refactoring you mention? We should not expect tons of other people to rewrite their pass for us... This won't happen (and conversely I am not able to rewrite other people's passes; GCC is really complex for me). I actually don't understand well what kind of plugin proposal will make into the trunk. Let's assume that the trunk will go in stage 1 on Christmas 2008, and that all the legal issues are solved (ie a runtime license is *defined* and accepted and explains what kind of plugins are GCC compatible) at the same time. It is late for me, and I am dreaming of Santa Claus :-) So let's suppose that Santa Claus came and give us (on end of december 2008) the runtime license and the stage 1. What happens next? What kind of patches is sent to gcc-patches@ and who will have time to review them? I think it's important to separate the plugin system from the application. Doing plugins well, IMO, requires a lot of code. It shouldn't be spread through the app. It also cleanly separates plugin mechanism from the actual extensions the app wants. Finally, plugins have to be extensible too. They should really be on a nearly equal footing with the app. Otherwise plugin developers who want the plugins to be extensible will need to reimplement there own extensibility system. The issue I see here is to get a consensus on these ideas and on your code. (I have same frightenning feeling about trying to get some day MELT into the trunk). *Pull vs push* Libplugin has a 'push' architecture, not a 'pull' one. What I mean is that the system pushes plugin awareness onto the application rather than requiring the application to call out to the plugin system all the time. I'm not sure to understand (I need to go to sleep) and I am concerned about having any plugin (be it your's or Sean's or Tarek's etc...) related code accepted into the trunk some day. I would dream of plugins to be inside GCC before 2010, but I am not sure of that! Here's an example of that. In GCC, passes have execute and gate functions which are already function pointers. With libplugin you can make these replaceable/extensible/event-like without changing a single line of code in GCC. I'll read again in detail your proposal later. But my main concern is not technical (how to do plugins - we each have our ideas, and we hopefully would be able to merge them), but more social: what are the (future) constaints (notably legal licence contraints)? what are the technical counterparts? how to make some plugin code accepted in the trunk (I don't care whose code it is; and I don't have any generic plugin machinery myself: I feel that MELT could fit in most reasonably defined well documented plugin subsystems)? Maybe some much simpler plugin mechanism code (perhaps from Mozilla Treehydra) might be easier to accept into the trunk, not because it is better, but just because it is simpler, and quicker to be accepted into the trunk? Actually, I still don't understand how exactly (socially speaking) are big patches accepted into the trunk (at stage one) and how to help a branch (or some other code somewhere else) to be more easily accepted into GCC? Should big patches absolutely be cut into small understandable parts? If in practice a succession of small patches has a much bigger chance to be accepted into the trunk than a bigger one, I don't know what will happen next. In addition, I would suppose that the runtime license could *requires* some technical behavior by the plugin machinery, and such requirements may perhaps prohibit more technically advanced solutions. I'm really impatient to understand what kind of plugins will be permitted (and when) in GCC, and also what kind of plugins will be disallowed [this could mean more than the definition of a proprietary plugin]. So far, I have no idea (because I am not a lawyer and because I don't know anything about the current work on the runtime license). Perhaps a microscopic plugin feature is better than a better designed one. A miniplugin mechanism is enough to add more advanced mechanisms (like your libplugin proposal, perhaps like my MELT branch) inside, themselves implemented in dlopen-ed *.so. Maybe we should wait till at least end of october for some input (even unofficial rumors) by the few people working on the runtime license. I think that most of the
Re: Defining a common plugin machinery
I have notes inline below, following is my summary of libplugin from what i understand of your posts: * It exists as a fraemwork that works with GCC now * It uses xml files to define plugins (Allows making new plugins as combinations of others without making a new shared library, i.e. just create an xml file that describes the plugin) * It handles issues with inter-dependencies between plugins * It uses a push framework, where function pointers are replaced/chained in the original application rather than explicit calls to plugins (Provides more extensibility in a application that makes heavy use of function pointers, but produces a less explicit set of entry points or hooks for plugins) * Currently it provides automatic loading of plugins without specific user request * It already has a framework for allowing plugins to interact with the pass manager If you can think of any other points to summarize the features it might be helpful as you are closer to it. The issues i see with this framework: * it seems to provide a lot of features that we may not necessarily need (That should be up for discussion) * plugin entry points are not well defined but can be any function pointer call Some questions: * How does the framework interact with the compile command line arguments? * Does this work on platforms that dont support -rdynamic or can it be modified to do so in the future? Hugh Leather wrote: *Separating Plugin system from appliction* Libplugin ships as a library. Apart from a few lines of code in toplev.c, the only other changes to GCC will be refactorings and maybe calling a few functions through pointers. As i understand the difference between the pull vs push, a plugin will load, and then modify existing function pointers in GCC to insert its own code and chain the existing code to be called after it. Is this correct? Doing this will be able to make use of existing function pointers as plugin hook locations, but some hooks we may want are not already called by function pointers and so would need to be changed. This means that plugin hook locations are not explicitly defined, but rather any place where a function pointer is used can be modified. Personally i prefer explicit specification of plugin hook locations. I think it's important to separate the plugin system from the application. Doing plugins well, IMO, requires a lot of code. It shouldn't be spread through the app. It also cleanly separates plugin mechanism from the actual extensions the app wants. Finally, plugins have to be extensible too. They should really be on a nearly equal footing with the app. Otherwise plugin developers who want the plugins to be extensible will need to reimplement there own extensibility system. Without the use of plugin meta-data in XML files and auto-loading and many of the things discussed, i am not so sure that plugins will be such a large body of code. It is really a matter of deciding if such features that libplugin provides are desirable for GCC. If so, then there is a lot of code required for plugins and libplugin becomes a good idea IMO. If not, then libplugin may just be more than we need. It really depends on what doing plugins well means for the specific application. *Scalable and Granularity* The system is very scalable. Really this is due to the push architecture. The granularity as i understand it is only as fine/coarse as the number of function pointers in the system that can be overwritten. This is no different from the pull method (i.e. The granularity depends on where you put the hook locations) except that function pointers may already exist. Though i may have mis-understood something... I.e. For the pull method you can: Add a pull for firePluginEvent() or add a pull inside each existing event handler. Where as the push method requires that the existing event handlers are called via function pointers and the push chains itself to that. I have used a similar method for the push plugin in python. The advantage here is that basically anything can be pushed in python so the system becomes very flexible to extend via plugins. In C/C++ the areas that can be extended need to be defined and turned into function pointers for the push method to work. Again, assuming i have understood how it works. *Mutliple cooperating plugins *I think some of the proposals don't allow multiple plugins or plugins aren't able to be extended in the same way that the application is. In libplugin you can have lots of plugins all depending on each other. Plugins can provide extension points as well as the application - this means it isn't just a matter of the application deciding what's important and everyone else having to make do. In some senses, this is the difference between a plugin system and loading a few shared libraries. A plugin system provides an holistic framework for building and using
Re: Defining a common plugin machinery
Hello All,
Brendon Costa wrote:
Some questions:
* How does the framework interact with the compile command line arguments?
* Does this work on platforms that dont support -rdynamic or can it be
modified to do so in the future?
[I'm skipping the rest of an interesting post]
I thought that for the first plugin machinery we don't care about
platforms without -rdynamic (or those without dlopen or tl_dlopen).
I believe we should first focus (when the runtime license will permit
that) on making whatever plugin machinery available and merged into the
trunk (when it comes back to stage one). This is not an easy task.
In practice, I think that we should first try to get some code into the
trunk which make some plugin work on some common easy host system
(Linux), and only after try to generalize the work to harder hosts.
At last, I believe that the plugin system will at first be something
which can be disabled at configure time, and will be disabled by default.
My main concern is plugins passes.
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
I believe we should first focus (when the runtime license will permit that) on making whatever plugin machinery available and merged into the trunk (when it comes back to stage one). This is not an easy task. Isn't the point of this discussion to decide what features to put into a plugin framework? I.e. We need a whatever plugin machinery available to exist before we can even think about merging that into the trunk and defining what that looks like is the point of this discussion i thought. Possible steps for moving forward with this: 1) Define what features we need for the first release, and think about what we may want in the future 2) See which frameworks currently exist and how each meets the necessary features identified 3) Either use one of the above frameworks as a base or start a new framework on the plugin branch 4) Work on the base set of features for a first release 5) Make sure the branch is up to date/tracking the trunk 6) Look at merging into the trunk when licensing is complete We are still at 1 (and partially identifying projects for 2) as far as i understand. I was going to start itemizing the features we have discussed, and the frameworks mentioned on the wiki. But I am not going to have time to do so for a number of weeks now. If someone else wants to do it it may get done a bit faster. So far, i think libplugin seems to be the most general plugin framework for GCC i have had a chance to look at (It was easy to look at because it has some decent documentation online). In practice, I think that we should first try to get some code into the trunk which make some plugin work on some common easy host system (Linux), and only after try to generalize the work to harder hosts. I agree, that providing working code for only simple to implement platforms (and basic plugin features) at first is a good idea (but do so on a branch first, then merge that to the trunk once it is operational). However we do not want to start with a framework that will need to be completely redesigned in the future to later support other platforms or usages. I.e. Thinking ahead but not necessarily implementing ahead... My main concern is plugins passes. Yes. We have not really looked at this more important aspect in much detail, how to manage passes with plugins. It looks like libplugin has some ideas for pass management that may help? Any thoughts?
Re: Defining a common plugin machinery
Chris Lattner wrote:
Is the plugin machinery intended to eventually allow new (GPL
compatible) backends to be used? It would be nice to make llvm-gcc be a
plugin.
From what I remember of the plugin BOFS the Steering Committee QA
sessions at last GCC summit permitting GPL-ed plugins is one of the
intent of plugins.
AFAIK, the concern is -on the opposite of your intents- how to make
GPL-ed plugins (legally) easy, and proprietary plugins (legally)
impossible, but I am not a lawyer, I know nothing about US law, and I
don't know what is happening inside the steering committee or the FSF
regarding plugins.
The technical questions about how to make plugin practicals are
precisely the scope of the current thread on the gcc@ mailing list.
Regards
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Chris Lattner [EMAIL PROTECTED] writes: On Sep 19, 2008, at 3:25 PM, Ian Lance Taylor wrote: Basile STARYNKEVITCH [EMAIL PROTECTED] writes: I am much more worried about passes and plugins (and I am very surprised to be almost the only one mentioning passes in plugin discussions). I feel it is a major issue (not a matter of coding, much more a matter of convention understanding). So far, I have (in MELT branch) a very unsatisfactory bad solution. I defined a few passes, which may (or not) be enabled by some plugins. What I would dream is some machinery to be able to have the plugin ask 'insert my pass foo_pass after the last pass which construct the IPA SSA tuple representation' and this is not achievable today. I think we also need that for general backend use, not only for plugins. E.g., bt-load.c should move to config/sh, and at startup time the SH backend should register the pass with the pass manager. Is the plugin machinery intended to eventually allow new (GPL compatible) backends to be used? It would be nice to make llvm-gcc be a plugin. As this point it seems likely that distributing a gcc plugin will effectively put the distributed code under the GPL. Of course this would not affect other versions of the code which were not distributed as a gcc plugin. I think the main focus of the plugin efforts is to permit inserting extra passes into gcc for various different types of analysis. But I don't see any reason that it would be impossible to insert an entirely new backend. The llvm-gcc plugin could more or less take the GIMPLE structures it had access to, do whatever it liked, and then exit. Ian
Re: Defining a common plugin machinery
Ian Lance Taylor wrote: Write a one-line shell script to use as your compiler (that's what I would do), or define an environment variable which tells gcc which plugins to load (e.g., GCC_PLUGINS=/a/file:/another/file). Thanks for the input. The one-liner shell script is a very good option. The GCC_PLUGINS env var method is very similar to what i use currently, though would require GCC to look for such an environment variable. It would have basically the same affect as searching a directory for such a list of plugins, but is more configurable and another way of achieving automatically loaded plugins. I think i will follow up the script idea. Thanks, Brendon.
Re: Defining a common plugin machinery
Basile STARYNKEVITCH wrote: But what about the trunk, which uses tuples? I know nothing about the tuples and the trunk and whether tuples might affect the way plugins work. Maybe others can comment... To be honest i was more worried about how plugins will work with the garbage collector and pre-compiled headers... I assume there is no problem with this though since there are already a number of plugin frameworks that exist that people report are working. Are the below hooks the name of dlsym-ed symbols inside the plugins, or the name of some (new or existing) functions? - c_common_init(), c_common_finish(), finalize(), gimplify_function_tree() They are the names of functions in GCC 4.0.4 that i currently modify to call my code (I am not using a plugin framework at the moment but distributing a patched GCC so i have statically linked my code into GCC and called it from these places).It was earlier recognised that we will need hooks into the pass management, i was just adding the areas outside the pass manager that i have used. If we have an idea of the hook locations everyone uses it may help get an idea of what may be required of the plugin framework in terms of locations in GCC where plugins can hook into. Those were the locations i used. How do we interact with the pass manager? Some plugins definitely want to add new passes... How does that happen concretely? I imagine others who have attempted this will comment. Agreed, but perhaps this autoload feature is not the first priority. Making plugin working inside the trunk seems more important to me than the autoload thing. At last, autoload plugins might slow down quick compilation of very small files (but do we care?). Well Ian just provided a different mechanism for projects like mine to achieve the result i was after without automatically loaded plugins. So i will assume that this feature is canned for the moment unless there are other needs for it. A related issue (which does not seems a priority to me) is do we want to make easy the process of moving a plugin inside the core GCC code, precisely to avoid that (ie having significant optimisations as plugins). To start with i can imagine that no core features will be moved to plugins, maybe in later versions that might happen but i cant imagine doing so would be overly difficult. And there is one significant point we did not discuss yet. Compiling a plugin should if possible be doable outside of GCC source code; this means at least that the gcc/Makefile.in should have stuff to copy all the relevant *.h files usable by plugins outside. Yes. A location where we can place the headers/libs was mentioned by Joseph ($libsubdir) along with suggesting we create an autoconf macro that external projects can use to locate and use those headers and the library. Maybe this is not the business of GCC, and only an issue for GCC packagers. In general, we should have a convention for the file path of installed plugins : do we want them only in system places like /usr/lib or also a per-user directory like $HOME/.gcc-plugin-dir/4.5.0/ ; we also need a convention for the location of all the related *.h files needed to compile a plugin outside GCC source tree. (or is this a packaging issue only, ie have not only a gcc-4.5 package in debian, but also gcc-4.5-plugins-dev package? - but that should mean additional install* targers in our Makefile.in). I would assume the following (Say we make a libgccimpl that contains the code for GCC except the stub main() that calls gcc_main() from the library): Install libgccimpl to: $(libsubdir)/libgccimpl.so.4.3.2 Install the headers to: $(includesubdir)/*.h Install plugins to: $(libsubdir)/plugins/*.so GCC would install these things every time it is installed, and it is the job of packagers for the various distributions to split it into different subsets if they so desire. So searching for plugins (using libltdl) would look first in: $(libsubdir)/plugins, followed by $LTDL_LIBRARY_PATH, and $LD_LIBRARY_PATH. The issue with searching anywhere other than $(libsubdir)/plugins, is knowing if the plugin that is found is compatible with the given version of GCC. For example: Say a system has two versions of GCC installed: 4.3.2 and 4.4.3. A user then wants to install a plugin but does not have the privileges to do so in $(libsubdir)/plugins/*.so for the specific versions of the GCC compiler. So they build the plugin say for version 4.3.2 and install it in their home directory somewhere. If they then invoke GCC 4.4.3 with -fplugin=edoc, there is a possibility that GCC 4.4.3 will find the plugin the user build for 4.3.2 and try to use it (If the user is not careful with their environment i.e. Setting LTDL_LIBRARY_PATH). This will cause undefined behaviour. We have a few options: 1) Dont allow additional search paths for plugins 2) Ignore the problem and put it down to the user needs to understand what they are doing 3) Somehow embed something in all
Re: Defining a common plugin machinery
* Brendon Costa wrote on Fri, Sep 19, 2008 at 02:42:19AM CEST: What platforms do we want to support? I can think of the following categories: * Windows (cygwin/mingw) As i understand the issue (I am not very familiar with this) you can't have unresolved references in a plugin back to the GCC executable. I.e. Building GCC with -rdynamic will not work on this platform. Do we move most of the GCC implementation into a library/DLL having a stub main() that just calls the library implementation. Then both the application AND the plugins can link with this library/DLL in a way that will work on Windows. Or are we going for the quick solution of using -rdynamic and not supporting this platform (Not my preferred option)? AFAIK you can fix w32 issues with DEF files. I would guess putting the bulk of GCC in a library, thus compiling it with PIC code on most systems, will hurt compiler performance rather significantly. Haven't tried it though. Cheers, Ralf
Re: Defining a common plugin machinery
Ralf Wildenhues wrote: * Windows (cygwin/mingw) As i understand the issue (I am not very familiar with this) you can't have unresolved references in a plugin back to the GCC executable. I.e. Building GCC with -rdynamic will not work on this platform. Do we move most of the GCC implementation into a library/DLL having a stub main() that just calls the library implementation. Then both the application AND the plugins can link with this library/DLL in a way that will work on Windows. Or are we going for the quick solution of using -rdynamic and not supporting this platform (Not my preferred option)? AFAIK you can fix w32 issues with DEF files. Well, that is another way to do it. You don't have to use a .def file or __declspec(dllexport) markup, you can also use -Wl,--export-all-symbols which doesn't require any source code modifications. But there is another more sinister problem on PE hosts with having the plugin link to symbols from the executable: this hard codes the name of the executable in the plugin, as essentially the executable is now a library to which the plugin is linked. Obviously this is a deal-breaker because the plugin needs to be loadable from cc1, cc1plus, f951, etc. unless you want to link a separate copy of the plugin for each language. An alternative is to have the plugin resolve the symbols in the executable at runtime, using the moral equivalent of dlsym. This also makes linking the plugin a lot easier because you don't have to worry about resolving the executable's symbols at link-time (which you would otherwise have to do with an import library for the executable.) However, this makes life more difficult for the writer of the plugin as they have to deal with all those function pointers. It is possible to ameliorate this with macros that set up each function pointer to a wrapper that looks up the actual function the first time it's called and then sets the pointer to that address, essentially a poor-man's PLT. But with this method you still have to maintain a list of every functions/variable used in order to instantiate a copy of the wrapper macro for it. Moving the compiler itself into a shared library of course avoids all of the above, but I'm not sure I understand how this would work w.r.t. the different language backends. Would the shared library just contain everything, and somehow just determine its behavior based on whether it was loaded by cc1 or cc1plus or f951 etc. Or would there be a separate shared library for each? (Which would really be useless and not solve anything for PE hosts because it would be back to requiring a number of copies of the plugin, one for each language backend.) Is it really that far fetched to have the plugin not directly access anything from the executable's symbol table but instead be passed a structure that contains a defined set of interfaces and callbacks? By using this strategy you also insulate the plugin from the gcc internals, such that you can define this struct in a separate header that goes in the plugin SDK, without requiring that the SDK drags in a bunch of internal crap from the build directory. With judicious use of callbacks you can even achieve some degree of plugin binary compatibility. For example you have gcc provide function pointers for getters/setters/walkers/enumerators for internal structures in this struct that is passed to the plugin. The plugin is then insulated from internal representation changes since the callback lives in gcc too and gets rebuilt when the internal representation changes. Now, I understand that maintaining a stable binary API for plugins is probably not the number one priority, i.e. past discussions have concluded that this is too much work and pain to maintain. And maybe that's true. But I'm just suggesting that you can in fact kill two birds with one stone here: insulate the plugin from internal representation details *and* overcome portability problems with non-ELF hosts. Brian
Re: Defining a common plugin machinery
On Fri, Sep 19, 2008 at 15:30, Brian Dessent [EMAIL PROTECTED] wrote:
Ralf Wildenhues wrote:
Is it really that far fetched to have the plugin not directly access
anything from the executable's symbol table but instead be passed a
structure that contains a defined set of interfaces and callbacks? By
using this strategy you also insulate the plugin from the gcc internals,
Too much so. This goes again back to needing to decide in advance what
will be interesting; that's pretty much a deal-breaker, I think.
No, it sounds to me as if linking at runtime is the way to go for PE.
However, that's not as hard as it sounds.
Essentially, *for each plugin*, you compile a list of all the gcc
symbols that *this plugin* needs. You convert this list into a
structure with a long list of pointers and names, which you hand to a
small routine that resolves all of them (via dlsym(),
GetProcAddress(), or whatever); and the plugin uses the stuff vioa the
pointers in the structure. As the list of symbols needed for that
plugin is both known and relatively stable, the burden from that isn't
too large.
In gcc, you just make all symbols resolvable.
Gcc then loads the plugin, gets a pointer to that structure (via a
well-known symbol name), resolves all symbols therein, and calls some
initialization function so the plugin can register in all relevant
hooks - or one could do even that by putting additional data into that
structure.
And one can also put versioning data therein, so that gcc can check
compatibility before calling the first plugin routine.
Just a quickdirty mock-up:
struct s_gcc_plugin_info gcc_plugin_info = {
.version = 4.4.0;
.hook = {
{ .name=hook_parse_params; .proc=my_parse_params; }
[...]
};
.link = {
{ .name=fold; .ptr = 0; }
[...]
};
};
and possibly, for convenience,
#define gcc_fold ((prototype_for_fold)(gcc_plugin_info.link[0].ptr))
... gcc_fold(...) ...
All those .name fields are presumably const char *.
And one could presumably write a small utility that, given a list of
names, goes through gcc's headers and constructs that struct and the
convenience defines automatically. If (as is the case here) the
declarations follow a fairly consistent style, this doesn't need a
full C parser.
It seems to me that the overhead, both in runtime and in maintenance,
isn't too bad here.
Re: Defining a common plugin machinery
Hello All,
(a meta question: do we need to reply-to all, or should the gcc@ list be
enough to discuss plugins, and the only destination of this interesting
discussion?).
Brendon Costa wrote:
Basile STARYNKEVITCH wrote:
But what about the trunk, which uses tuples?
I know nothing about the tuples and the trunk and whether tuples might
affect the way plugins work. Maybe others can comment...
To be honest i was more worried about how plugins will work with the
garbage collector and pre-compiled headers... I assume there is no
problem with this though since there are already a number of plugin
frameworks that exist that people report are working.
The MELT plugin is a bit specific about GGC, since in practice all the
gengtype-d code is inside gcc/basilys.c which is not a plugin itself
(but which, among other important things, deals with garbage collection
dlopen-ing). I could imagine that the other plugin experiments did
similar stuff (meaning that they did not touch to gcc/ggc-common.h or to
the gengtype generator).
I'm not sure that pre-compiled headers (pch) should even work with
plugins. The reasonable solution is to disable them when any plugin is
used (what should happen if plugins are not used when the pch is
generated, but used when it is reloaded or viceversa; I don't know if
this situation is hard to detect. Likewise, what should happen when the
compilation of the header the compilation of the source including the
pch both dlopen-ed a different variant of the same plugin?). At least,
having pch working with plugins should be postponed after the first
approved (i.e. merged into the trunk) plugin imlementation.
A generic plugin framework should certainly cooperate with GGC (the
garbage collector inside GCC). From what I understand (but I did'nt try
and might forgot important details) it should not be very hard (assuming
we do not care about dlclose-ing plugins); one should:
1. enhance the function ggc_mark_roots in gcc/ggc-common.c so that it
marks the GGC roots defined in the plugin. If the plugin is itself a
single PLUGIN.c source processed by gengtype, this means calling the
marking routines inside the gt_ggc_r_gt_PLUGIN_h array of ggc_root_tab
structures generated by gengtype implemented in the gt-PLUGIN.h
generated file. Perhaps the plugin initialization routine (inside the
dlopen-ed plugin) should explicitly register some stuff to GGC.
2. more generally, enhance the code so that the equivalent of
gtype-c.h works with the plugins.
3. enhance the gengtype generator itself to be able to process a
plugin itself. This is perhaps more tricky since gengtype currently gets
its list of files to be processed in a very special (brittle built-in)
way - it is not simple to pass the list of files as a program argument
to gengtype, currently it is generated at gengtype build time.
We should be careful about what happens when several plugins are loaded.
So there are some issues with GGC plugins, but there should not be
very hard to solve (even if I don't claim it is very easy). I find the
issue of pass management plugins more serious (because there is more
consensus to reach on what should be done).
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Brian Dessent [EMAIL PROTECTED] writes: Is it really that far fetched to have the plugin not directly access anything from the executable's symbol table but instead be passed a structure that contains a defined set of interfaces and callbacks? Yes, that is pretty far fetched. Simply writing the list of callbacks would be very painful. Keeping that list of callbacks working for more than one release would be unbearable pain. Perhaps someday, with much more experience of plugins, this will be possible, but I don't see any real possibility of starting out that way. Ian
Re: Defining a common plugin machinery
Basile STARYNKEVITCH wrote: (a meta question: do we need to reply-to all, or should the gcc@ list be enough to discuss plugins, and the only destination of this interesting discussion?). Reply-to-all is the common standard on the list. If you don't want a personal copy then set the Reply-to: header in your message to the list address and you won't be included in CC by most email clients when someone does reply-to-all. I'm not sure that pre-compiled headers (pch) should even work with plugins. The reasonable solution is to disable them when any plugin is used (what should happen if plugins are not used when the pch is generated, but used when it is reloaded or viceversa; I don't know if this situation is hard to detect. Likewise, what should happen when the compilation of the header the compilation of the source including the pch both dlopen-ed a different variant of the same plugin?). At least, having pch working with plugins should be postponed after the first approved (i.e. merged into the trunk) plugin imlementation. There's already an existing requirement that the compiler options used when precompiling the .gch must match or be compatible with those used when using the .gch, so in essence the above is already forbidden by the fact that adding or removing -fplugin= would render the .gch unusable. I seem to recall that the .gch file records the entire set of options used when it was built and gcc refuses to consider the file if it does not exactly match, however the documentation seems to imply that only certain options are automatically checked/rejected while others are the responsibility of the user to enforce: http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html. Brian
Re: Defining a common plugin machinery
Brian Dessent wrote: Basile STARYNKEVITCH wrote: I'm not sure that pre-compiled headers (pch) should even work with plugins. The reasonable solution is to disable them when any plugin is used [...] There's already an existing requirement that the compiler options used when precompiling the .gch must match or be compatible with those used when using the .gch, so in essence the above is already forbidden by the fact that adding or removing -fplugin= would render the .gch unusable. I seem to recall that the .gch file records the entire set of options used when it was built and gcc refuses to consider the file if it does not exactly match, however the documentation seems to imply that only certain options are automatically checked/rejected while others are the responsibility of the user to enforce: http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html. In my opinion we should, in the first version of plugin code inside trunk, disable pre-compiled headers when using any plugin. Notice that PCH require specific support by GGC (probably in gcc/ggc-common.c), hence additional processing for plugins, more than what I mentioned in http://gcc.gnu.org/ml/gcc/2008-09/msg00342.html Only when plugins are actually usable, and when we get real practical experience on them, should we tackle PCH. We should not bother before. And my understanding was that PCH are essentially an experimental, not very satisfactory (but definitely useful to users) features. I even thought that some people are interested in working on a way to replace it by something neater (IIRC, this was informally discussed at the 2008 GCC summit, I forgot by whom..). I am much more worried about passes and plugins (and I am very surprised to be almost the only one mentioning passes in plugin discussions). I feel it is a major issue (not a matter of coding, much more a matter of convention understanding). So far, I have (in MELT branch) a very unsatisfactory bad solution. I defined a few passes, which may (or not) be enabled by some plugins. What I would dream is some machinery to be able to have the plugin ask 'insert my pass foo_pass after the last pass which construct the IPA SSA tuple representation' and this is not achievable today. From what I understand, the ICI http://gcc-ici.sourceforge.net/ effort is tackling with related issues. Actually, I believe that passes should *much* more described, and I mean: * each pass should be documented, by a paragraph like the beginning comment of alias.c cse.c df-core.c dse.c gcse.c ipa-cp.c ipa-inline.c ipa-type-escape.c omp-low.c tree-cfg.c tree-inline.c tree-mudflap.c or tree-nrv.c or tree-optimize.c ... I was suggesting in http://gcc.gnu.org/ml/gcc/2008-09/msg00177.html that each pass *should* have such a description (less than a hundred english words each), and even dreaming of some simplistic comment marking convention (perhaps texinfo markup inside a comment) and some simple (e.g. awk) script to extract these documentary comments from source code and merge them into a generated pass-internal-doc.texi file. The issue here is not technical [writing such an awk script is simple] but social [have each pass developer document a little his/her pass in some 'standard' format]. And such a crude device (script merging comments) would help to have more documentation. * each pass should have a unique name. Note that I submitted a patch http://gcc.gnu.org/ml/gcc-patches/2008-08/msg00057.html which has been accepted committed to the trunk (on august 1st 2008) which permit passes to be named without having any dump. A possible minor issue is probably that several test suites (I have no idea which ones) depend on the pass name. This is much more a social issue (have each pass developer be concerned of naming his pass uniquely) than a technical one (giving a unique name to every passes is realistically possible only to the elite people understanding nearly all of GCC. They are very few, and they don't need any documentation anymore). Each pass developer should, without major effort, be able to document in one paragraph his pass and uniquely name it. In contrast, having a single (or a few) person doing all this work is not realistic. * each major internal GCC representation should be more or less documented (this is probably the case today: Tuples, generic trees, RTL have a documentation, even if is imperfect), and each pass should tell what representation it changes, it invalidates, it destroys, it provides (some but not all of this is already available in struct opt_pass thru flags, eg properties_required, todo_flags_start, ...). I believe that a major (more social than technical) issue about plugins (assuming almost all of them are implementing some kind of passes) is: * where should this plugin be inserted (after which pass, before which pass) * what kind of representations this plugin consumes, provides, enhances,
Re: Defining a common plugin machinery
On Fri, 19 Sep 2008, Brendon Costa wrote: 1) Dont allow additional search paths for plugins I think we do want to allow the user to specify such a path on the compiler command line, in addition to searching the standard libsubdir or libexecsubdir location. 3) Somehow embed something in all plugins that can safely be queried, which indicates the version/build of GCC the plugin belongs to and only load plugins that match. I think the required GCC version should be exported by the plugin - in addition to the plugin's own version and bug-reporting URL which should go in the --version and --help output if those options are used together with plugin-loading options. It should also export the configured target triplet since plugins built for different targets on the same host can't be expected to be compatible (they may use target macros internally). We should make it clear that e.g. 4.5.0 and 4.5.1 are not necessarily compatible for plugins; that's an inevitable consequence of exporting all the GCC-internal interfaces. Thus the full version number is what should be checked, rather than trying to keep interfaces stable within e.g. the 4.5 series. But we do want plugins to be usable from both cc1 and cc1plus, etc., unless they use front-end-internal interfaces (which some may wish to do, at the expense of only working with one language). Do we say for the first version we use -rdynamic and don't support windows (will need build machinery to disable all this on windows if we do) or do we add the export macros and library and support the windows platform from the beginning (Are there any other platforms with quirks not covered here)? There are many supported non-ELF hosts with their own peculiarities - not just Windows - Darwin (Mach-O), HP-UX on 32-bit PA (SOM), older BSD versions (a.out), Tru64, VMS, DJGPP, AIX. I think it's very likely the support will initially be optional and only supported on a subset of hosts - but with nothing in the core compiler moved to plugins this should not cause any problems with the invariant of generating the same code for a target independent of host. -- Joseph S. Myers [EMAIL PROTECTED]
Re: Defining a common plugin machinery
Brendon Costa wrote: Joseph S. Myers wrote: I think this is a bad idea on grounds of predictability. I can understand this view and was initially reluctant to suggest the auto-load feature for this same reason. However i can not see another solution that can be used instead of this to achieve simple usability for a small subset of plugin types (described below). Separately developed plugins have many uses for doing things specific to particular projects This is the case for the Dehydra, which is specific to the Mozilla projects. However the plugin i would like to develop it is NOT designed to be used with a specific project, but to be used while compiling any project. Sorry to nitpick, but there is nothing Mozilla-specific in dehydra/treehydra. There are users outside of Mozilla. However, I do think it's awesome to be able to store plugin results in the resulting binary to do LTO-style analyses. How well is that working for you? Taras
Re: Defining a common plugin machinery
Taras wrote:
However, I do think it's awesome to be able to store plugin results in
the resulting binary to do LTO-style analyses. How well is that working
for you?
I would be delighted if that would be easily possible, but I remember
having asked at some LTO related session or meeting if LTO can be
extended to add some additional data, and the answer was we don't want
to do that.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basileatstarynkevitchdotnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***
Re: Defining a common plugin machinery
Brendon Costa wrote: * Automatically loaded plugins as well as explicit user requested plugins I would like to propose that we allow automatic loading of certain plugins in addition to the explicit request for loading of plugins using the -fplugin=name command line option. I think we agree that autoloading plugins is not an immediate need and can be worked out at a later stage. -- What should we export? I think exporting everything is the way to go. We sidestepped the win32 issue by using a mingw cross-compiler. There are a couple of static functions in GCC that are useful for plugins, it would be nice to make them global symbols so each plugin doesn't have to copy them. -- What should be the user interface to incorporate plugin code? It has been proposed that we use: -fplugin=name -fplugin-arg=arg I propose that we use slightly different format: g++ -fplugin=name -fname-arg[=value] for example: g++ -fplugin=edoc -fedoc-file=blah.edc My reasons for this are from the following cases: 1) multiple plugins with a single invocation 2) key/value arguments required for plugins 3) plugins that may want to be loaded automatically 1) This is not a big issue. I just clarity with the behavior. Say we want to use two plugins in the same invocation. This might look like: Original: g++ -fplugin=edoc -fplugin-arg=embed -fplugin=optim -fplugin-arg=max New: g++ -fplugin=edoc -fedoc-embed -fplugin=optim -foptim-max If using the original method, the order of the specification matters, with the new method it does not. I don't feel that it's worth the hassle for GCC to support multiple plugins. The thought of GCC loading multiple plugins that would stepping on each other's toes leading to all kinds of unexpected ICEs is unpleasant. My proposal is to write multiplexor plugin to make it possible to load subplugins. This would help keep the plugin part of GCC minimal. Having said that, I initially came up with the idea of a multiplexor because I wasn't sure how to deal with commandline arguments cleanly. I like your proposal of -fplugin/-fplugin-arg pairs. It is also possible to introduce llaborate command-line argument extensions via plugins :). 2) With my EDoc++ project one of the arguments i look for has a format like: -fedoc-file=blah.edc What would this look like with the -fplugin-arg= format? Possibilities with Original format: g++ -fplugin=edoc -fplugin-arg=file:blah.edc g++ -fplugin=edoc -fplugin-arg-file=blah.edc g++ -fplugin=edoc -fplugin-arg=file=blah.edc g++ -fplugin=edoc -fplugin-key=file -fplugin-value=blah.edc In Mozilla we have a getopt-style parser in the javascript part of the plugin that parses the -fplugin-arg=--long bunches -of arguments getopt-style. That seems to be working well. -- At what point in the compilation stage should it kick in? I think plugins should be loaded as soon as possible. I would propose loading plugins immediately after the processing of command line arguments or maybe even at the same time as command line argument processing depending on what might be easier/cleaner. Loading plugins ASAP is the way to go. -- Some extra questions. -- What platforms do we want to support? I can think of the following categories: I think it's fine for the initial revision to only support easy platforms, ie ones with -ldl/-rdynamic. -- How do we search for plugins and make sure we don't load incompatible ones? We force users to use absolute paths. It's not ideal. -- How/where do we install headers and the library for external plugin projects to use? Currently my plugins are built by specifying the location of the GCC source tree via ./configure. This might be a little inconvenient, but if people are going to be developing plugins, they better be able to build gcc. In fact I think it would be helpful to not bother with plugin headers/etc, so plugin developers would be encouraged to expand their scope to hacking gcc as well. In our case, plugin users do no need the sources, they just need a binary for the compiler and the plugin which can be provided by their package manager. Cheers, Taras
Re: Defining a common plugin machinery
On Fri, Sep 19, 2008 at 13:16, Basile STARYNKEVITCH [EMAIL PROTECTED] wrote: I would be delighted if that would be easily possible, but I remember having asked at some LTO related session or meeting if LTO can be extended to add some additional data, and the answer was we don't want to do that. I have no memory of that. But in principle, I don't see why this couldn't be done by either storing extended statement attributes or a new section in the LTO output. Of course, the plugin would have to be responsible for producing and consuming this data. Diego.
Re: Defining a common plugin machinery
Basile STARYNKEVITCH [EMAIL PROTECTED] writes: I am much more worried about passes and plugins (and I am very surprised to be almost the only one mentioning passes in plugin discussions). I feel it is a major issue (not a matter of coding, much more a matter of convention understanding). So far, I have (in MELT branch) a very unsatisfactory bad solution. I defined a few passes, which may (or not) be enabled by some plugins. What I would dream is some machinery to be able to have the plugin ask 'insert my pass foo_pass after the last pass which construct the IPA SSA tuple representation' and this is not achievable today. I think we also need that for general backend use, not only for plugins. E.g., bt-load.c should move to config/sh, and at startup time the SH backend should register the pass with the pass manager. Ian
Re: Defining a common plugin machinery
On Sep 19, 2008, at 3:25 PM, Ian Lance Taylor wrote: Basile STARYNKEVITCH [EMAIL PROTECTED] writes: I am much more worried about passes and plugins (and I am very surprised to be almost the only one mentioning passes in plugin discussions). I feel it is a major issue (not a matter of coding, much more a matter of convention understanding). So far, I have (in MELT branch) a very unsatisfactory bad solution. I defined a few passes, which may (or not) be enabled by some plugins. What I would dream is some machinery to be able to have the plugin ask 'insert my pass foo_pass after the last pass which construct the IPA SSA tuple representation' and this is not achievable today. I think we also need that for general backend use, not only for plugins. E.g., bt-load.c should move to config/sh, and at startup time the SH backend should register the pass with the pass manager. Is the plugin machinery intended to eventually allow new (GPL compatible) backends to be used? It would be nice to make llvm-gcc be a plugin. -Chris
Re: Defining a common plugin machinery
Hi all, Firstly, do others think it would be helpful to summarise any of this information on a wiki page, or will these emails be fine? In this email I will give my opinion on the questions asked by Deigo and ask a few additional questions that may be relevant. -- What features do we want to support? * Hooks I like the sound of what Taras described in terms of the actual format of defining the hooks throughout the GCC code. The hooks i currently use for my project are (in GCC 4.0.4): - c_common_init() : I use to initialise my module. Will be whatever we decide for plugins. - c_common_finish() : I use to finalise my module. Will be whatever we decide for plugins. - finalize() : I use this hook to write my data to a file. This is before the asm output as one of my output format options is to insert my data into a new .edoc section in the resulting binary. I do this by appending to the asm file: .section .edoc and then defining a lot of .byte lines with the data. - gimplify_function_tree() : To process the AST from the C/C++ front ends before it is gimplified. * Automatically loaded plugins as well as explicit user requested plugins I would like to propose that we allow automatic loading of certain plugins in addition to the explicit request for loading of plugins using the -fplugin=name command line option. The idea is that their are two types of plugin that are differentiated only by the location in which they are installed and how they are loaded by the GCC application(s). The first type of plugin as has been discussed is loaded on explicit user request with -fplugin=xyz, the second is a plugin that is always loaded automatically on execution of the GCC application(s). A plugin found in for example an autoload directory will be loaded every time the GCC application(s) are executed. The main reason i see for such a feature is that it allows automatically loaded plugins to become transparent. I.e. There is nothing that the user needs to do to make use of the new plugin but install it in the correct location. This could be a good thing if there is a desire in the future to move core GCC features (such as existing optimization passes) into plugins without having to require the user to specify -fplugin=xyz. Also if a user finds a good optimisation plugin that they download and they want to use it for everything they compile, they could just install it in the autoload directory and not have to worry about ensuring that all projects they compile somehow provide -fplugin=myoptim to GCC. I have other reasons for wanting this with my project but it is are primarily for convenience in my case (Using environment variables for turning on/off features instead of command line options). -- What should we export? * I think that exporting as much as we can from GCC is good (say everything but the main() function), not just a limited sub-set that plugins can access. This gives the most flexibility for plugins (even if we don't foresee people using all this at any point for now). Also it is a lot simpler to do than defining a subset of interfaces to be exported and making sure we export everything that future plugins *MAY* want to use. Possible reasons against exporting everything: * If we want to define a stable API for plugins then we would not export everything. To be honest i think doing something like this would be too much work for the benefit. * It is possible that exporting everything could be a lot of work. In particular i think for the win32 platform, we will need to mark exported entities with __declspec(dllexport) and when used with __declspec(dllimport) (Who in MS thought it was a good idea to have a different markup for export and import). I assume for non-win32 platforms we will be using default visibility, i.e. Not using: -fvisibility=hidden? -- What should be the user interface to incorporate plugin code? It has been proposed that we use: -fplugin=name -fplugin-arg=arg I propose that we use slightly different format: g++ -fplugin=name -fname-arg[=value] for example: g++ -fplugin=edoc -fedoc-file=blah.edc My reasons for this are from the following cases: 1) multiple plugins with a single invocation 2) key/value arguments required for plugins 3) plugins that may want to be loaded automatically 1) This is not a big issue. I just clarity with the behavior. Say we want to use two plugins in the same invocation. This might look like: Original: g++ -fplugin=edoc -fplugin-arg=embed -fplugin=optim -fplugin-arg=max New: g++ -fplugin=edoc -fedoc-embed -fplugin=optim -foptim-max If using the original method, the order of the specification matters, with the new method it does not. 2) With my EDoc++ project one of the arguments i look for has a format like: -fedoc-file=blah.edc What would this look like with the -fplugin-arg= format? Possibilities with Original format: g++ -fplugin=edoc
Re: Defining a common plugin machinery
On Fri, 19 Sep 2008, Brendon Costa wrote: * Automatically loaded plugins as well as explicit user requested plugins I would like to propose that we allow automatic loading of certain plugins in addition to the explicit request for loading of plugins using the -fplugin=name command line option. I think this is a bad idea on grounds of predictability. We can and should define the plugin interface to require plugins to specify version numbers and bug-reporting URLs and include them in the --help and --version output, and give warnings about bug reporting unless the core and all plugins have the same bug-reporting URL, but even with that I think it would be bad for the behavior of a user's compilation command to change not because of any compiler upgrade but because an *extra* piece of software was installed, possibly for the use of another user on the same system. This could be a good thing if there is a desire in the future to move core GCC features (such as existing optimization passes) into plugins without having to require the user to specify -fplugin=xyz. Also if a If that is done, a list of plugins to load automatically could be compiled into the GCC binaries - I don't see a particular problem with that. It's the action at a distance by the installation of new packages that's the problem. user finds a good optimisation plugin that they download and they want to use it for everything they compile, they could just install it in the I think we should discourage such good optimisation plugins being developed and maintained long-term outside the mainstream of GCC development - if it's a good generally useful optimisation we want it brought into mainline GCC, updated with mainline GCC for interface changes, etc. (whether or not still loaded internally as a plugin). Separately developed plugins have many uses for doing things specific to particular projects - such as checking project-specific coding rules, and quite possibly optimizations that are only valid when such project-specific rules are followed - and for research purposes. It's quite possible that there could usefully be plugins associated with a library that check rules for use of that library, do library-specific optimizations or even add library-specific language extensions. But then just as a user would use a -l option to link with the library they can use a -fplugin option to load its plugins (and if the library uses pkg-config this may be automatic through pkg-config giving the option in its output). However, if something is generally useful for everything they compile it would be a good candidate for the GCC core. autoload directory and not have to worry about ensuring that all projects they compile somehow provide -fplugin=myoptim to GCC. I have other reasons for wanting this with my project but it is are primarily for convenience in my case (Using environment variables for turning on/off features instead of command line options). The use of environment variables to pass information to/from collect2 is a pain, as are most uses of environment variables to pass information to subprocesses when trying to debug things. So I think this sort of thing is a mistake - but if you want to use your own plugin to do it, you can. -- How do we search for plugins and make sure we don't load incompatible ones? We want to avoid loading plugins for different/incompatible builds/versions of GCC. Things i think we need to take into account * Multiple installed versions of GCC * Multiple installed builds for the same version of GCC (Is this worth catering for?) * Cross compilers * Finding the directory where to install a plugin should not be TOO difficult for an external plugin project The natural locations are within libsubdir and libexecsubdir. It's been several years since those locations changed. Use the GNU Coding Standards to determine what goes in libsubdir and what in libexecsubdir (or get them clarified if it's not clear - it may not be clear for the plugin binaries themselves). Then comes the issue of thinking about how does an external project locate that directory? Do we add an option to the command line of GCC to obtain this directory name? We can provide configure macros for external projects to use; they'd take a configure option taking the version number of GCC (e.g. --with-gcc=4.5.0) and could deduce everything else from the standard configure options such as --target. This works in the Canadian cross case, when interrogating the compiler wouldn't. The macros could default to running $target-gcc -v and extracting the version number from that if the configure option isn't passed and $host and $build are the same. How/where do we install headers and the library for external plugin projects to use? Somewhere under libsubdir. The configure macros provided for the use of plugins can take care of the plugins finding them. --
Re: Defining a common plugin machinery
Joseph S. Myers wrote: I think this is a bad idea on grounds of predictability. I can understand this view and was initially reluctant to suggest the auto-load feature for this same reason. However i can not see another solution that can be used instead of this to achieve simple usability for a small subset of plugin types (described below). Separately developed plugins have many uses for doing things specific to particular projects This is the case for the Dehydra, which is specific to the Mozilla projects. However the plugin i would like to develop it is NOT designed to be used with a specific project, but to be used while compiling any project. Just to provide an example: Say i am developing a project called foo. Foo makes use of a library that I am NOT involved in the maintainence of such as: libblah. In order to get data on the complete callgraph and possible exception propogation (this is what my plugin does), i need to build both libblah and foo with my plugin to analyse the source for both. I.e. I can't give accurate results without data for the complete callgraph. In many cases this may be as simple as: cd blah ./configure CXXFLAGS=-fplguin=edoc -edoc-embed make make install cd ../foo ./configure CXXFLAGS=-fplguin=edoc -edoc-embed make make install But this is not always be the case (Mostly projects that dont use GNU auto-tools for the build system or do so incorrectly) I have control over my project: foo, however i do not have control over project blah. The problem is with badly defined build system that do NOT allow a user to pass flags they want to to the compiler. This will likely result in having to edit the build system code for project blah just to build with the necessary compiler flags to include use the plugin. To overcome this, my plugin looks for environment variables to enable it. Note: Initially i just used command line options to GCC, but i came across this problem a number of times and decided to make it more usable, i needed to find a different solution. As unconventional and possibly nasty as the environment variable solution is, i can not think of a better one. Using environment variables the above would change to: export EDOC_EMBED=yes ...build blah however it needs to be built ... cd ../foo ./configure make make install This will work regardless of the structure of either foo or blah's build system. This problem is likely to exist not just for my project but for a small subset of source analysis plugins that can not work with partial data sets. In the end, i can understand if this is not a good enough reason to out-weight the costs associated with automatically loading plugins (I.e. Predictability). I did however think it worth bringing up as an option. I hope that describes the problem sufficiently. If you can think of any other solutions to this problem, i would love to hear them. Thanks, Brendon.
Re: Defining a common plugin machinery
Brendon Costa wrote: Hi all, Firstly, do others think it would be helpful to summarise any of this information on a wiki page, or will these emails be fine? Yes, updating the wiki is always nice. Don't forget to put reference to the mails on gcc@ using their archive URL, please! In this email I will give my opinion on the questions asked by Deigo and ask a few additional questions that may be relevant. -- What features do we want to support? * Hooks I like the sound of what Taras described in terms of the actual format of defining the hooks throughout the GCC code. The hooks i currently use for my project are (in GCC 4.0.4): But what about the trunk, which uses tuples? Are the below hooks the name of dlsym-ed symbols inside the plugins, or the name of some (new or existing) functions? - c_common_init(), c_common_finish(), finalize(), gimplify_function_tree() How do we interact with the pass manager? Some plugins definitely want to add new passes... How does that happen concretely? * Automatically loaded plugins as well as explicit user requested plugins I would like to propose that we allow automatic loading of certain plugins in addition to the explicit request for loading of plugins using the -fplugin=name command line option. The idea is that their are two types of plugin that are differentiated only by the location in which they are installed and how they are loaded by the GCC application(s). The first type of plugin as has been discussed is loaded on explicit user request with -fplugin=xyz, the second is a plugin that is always loaded automatically on execution of the GCC application(s). Agreed, but perhaps this autoload feature is not the first priority. Making plugin working inside the trunk seems more important to me than the autoload thing. At last, autoload plugins might slow down quick compilation of very small files (but do we care?). The main reason i see for such a feature is that it allows automatically loaded plugins to become transparent. Agreed. Besides, I believe that we cannot reliably predict how plugins will be used in the long run. And in contrast to the email of Joseph S. Myers [EMAIL PROTECTED] I also think that it may latter happen that some optimisations would in the future exist only as plugins. A related issue (which does not seems a priority to me) is do we want to make easy the process of moving a plugin inside the core GCC code, precisely to avoid that (ie having significant optimisations as plugins). -- What should we export? * I think that exporting as much as we can from GCC is good (say everything but the main() function), not just a limited sub-set that plugins can access. This gives the most flexibility for plugins (even if we don't foresee people using all this at any point for now). Agreed. And I should also mention that I personally am not interested by Win32 (so I don't care of declspec(dllexport)...). Possible reasons against exporting everything: * If we want to define a stable API for plugins then we would not export everything. To be honest i think doing something like this would be too much work for the benefit. We really cannot care (at least now) to define a stable API. If we did care about that, plugins would never happen. It has been agreed at the plugin BOFS at last GCC Summit (june 2008 Ottawa) that defining a stable API for GCC internals is not realistic (and perhaps even not preferable). -- What should be the user interface to incorporate plugin code? It has been proposed that we use: -fplugin=name -fplugin-arg=arg I propose that we use slightly different format: g++ -fplugin=name -fname-arg[=value] Seems nice. But this convention might hurt the current arguments. Currently, this convention implicitly prohibits (but that is ok for me) a plugin named loop because -floop-block already means something. There are some problems though with my proposal. In particular: * A plugins name can not be the same as an existing -fxyz option. Yes, this is what I mentioned above. = And there is one significant point we did not discuss yet. Compiling a plugin should if possible be doable outside of GCC source code; this means at least that the gcc/Makefile.in should have stuff to copy all the relevant *.h files usable by plugins outside. The MELT branch does that already using code like (I'm pasting some lines of my gcc/Makefile.in). I do know that it is ugly, uses some -MMD -MF flags which are gcc specific, probably a depcomp script should be better. ## this is the local build directory (which should be copied during ## installation into the installation directory above) for include-d *.h melt_build_include_dir= melt-private-build-include Maybe this is not the business of GCC, and only an issue for GCC packagers. In general, we should have a convention for the file path of
Re: Defining a common plugin machinery
Brendon Costa [EMAIL PROTECTED] writes: I have control over my project: foo, however i do not have control over project blah. The problem is with badly defined build system that do NOT allow a user to pass flags they want to to the compiler. This will likely result in having to edit the build system code for project blah just to build with the necessary compiler flags to include use the plugin. Write a one-line shell script to use as your compiler (that's what I would do), or define an environment variable which tells gcc which plugins to load (e.g., GCC_PLUGINS=/a/file:/another/file). Ian
Re: Defining a common plugin machinery
Hi all, Diego Novillo wrote: After the FSF gives final approval on the plugin feature, we will need to coordinate towards one common plugin interface for GCC. I don't think we should be adding two different and incompatible plugin harnesses. I am CCing everyone who I know is working on plugin features. Apologies if I missed anyone. I would like to start by understanding what the plugin API should have. What features do we want to support? What should we export? What should be the user interface to incorporate plugin code? At what point in the compilation stage should it kick in? Once we define a common API and then we can take the implementation from the existing branches. Perhaps create a common branch? I would also like to understand what the status and features of the different branches is. If it would help to have a conference call or IRC meeting, I can set one up for all the parties interested. But first, I would like to hash the main issues on the lists. We, at Bordeaux University (France), have been essentially using the gcc-plugins branch from the SVN to build a prototype of model extractor of C code (we didn't look at Java yet but it was planned). So, we did plug at the 'pass_plugin_gimple' pass to get the Gimple code out. Once at this stage, we used 'current_function_decl' as starting point to extract the CFG of the functions (browsing the data-structure to extract each basic block and all useful data about the function). What we expect from the plug-ins in GCC is a way to _export_ the GIMPLEd control-flow graph, the data-types of the variables and everything which can be useful to perform a symbolic execution of it (using model-checking). Note that we might also need an annotated programs in order to link it back to the original source code which would help to report errors back to the programmer. Nevertheless, as Basile asked for it, it would be nice to have an update of the recent advances about what the GCC core team decided about the plugins ! :) Regards -- Emmanuel Fleury Associate Professor, | Room: 261 LaBRI, Domaine Universitaire | Phone: +33 (0)5 40 00 69 34 351, Cours de la Libération | Email: [EMAIL PROTECTED] 33405 Talence Cedex, France | URL: http://www.labri.fr/~fleury
Re: Defining a common plugin machinery
Basile STARYNKEVITCH wrote: Hello Diego and all, Diego Novillo wrote: After the FSF gives final approval on the plugin feature, we will need to coordinate towards one common plugin interface for GCC. I don't think we should be adding two different and incompatible plugin harnesses. What exactly did happen on the FSF side after the last GCC summit? I heard nothing more detailed than the Steeering Committee QA BOFS and the early draft of some legal licence involving plugins. What happened on the Steering Commitee or legal side since august 2008? Is there any annoucement regarding FSF approving plugins? I am CCing everyone who I know is working on plugin features. Apologies if I missed anyone. I would like to start by understanding what the plugin API should have. What features do we want to support? What should we export? What should be the user interface to incorporate plugin code? At what point in the compilation stage should it kick in? Once we define a common API and then we can take the implementation from the existing branches. Perhaps create a common branch? I would also like to understand what the status and features of the different branches is. The MELT plugin machinery is quite specific in its details, and I don't believe it can be used -in its current form- for other plugins. It really expects the plugin to be a MELT one. From what I remember of the plugin BOFS (but I may be wrong), an easy consensus seems to be that plugins should be loadable thru the command line (probably a -fplugin=foo meaning that some foo.so should be dlopen-ed), that they could take a single string as an argument (using -fplugin-arg=bar) and that plugins add essentially new passes into pass.c - I am happy with such a picture (and could fit MELT easily into it; in its current form MELT is adding a few basilys*passes into passes.c, each having a gate which try using a MELT closure somewhere so is equivalent of testing if the pass is enabled.). Albert Cohen Grigori Fursin (both in CC) from INRIA have an Interative Compiler Interface (a big patch to GCC) which could be helpful to enhance this idea of extending the pass manager. AFAIK, they also have some plugin facility (or maybe I am wrong). Perhaps some ideas or code could be reused for plugins. I have no idea if some general plugin code evolved recently, i.e. if someone is actively working on it. If someone is coding on plugins, please tell the gcc@ list. Hi Guys, We are actively using and maintaining our plugin API at Mozilla. We moderated the plugin discussion at the summit. From our discussion sounds like we need to be able to hook into the middle-end(pass manager) and the frontend. As far as I know nobody has needed anything of the backend yet. We are using the -fplugin syntax that Basile described above to dlopen our plugins. As Basile mentioned, the pass manager API is extremely important. Plugins need to be able to list passes(thus all passes need unique names) and to insert a new pass at arbitrary positions. If I recall correctly, Grigori's work also needs to be able to reorder existing passes. Beyond that, there need to be general callbacks sprinkled throughout the code. At the plugin BoF we decided that something like plugin_callback(EVENT_NAME, void *data) would suit us well. Here EVENT_NAME would be an enum describing the callback. Examples: * PLUGIN_FINISH_STRUCT(data would point at tree_node for the type that gcc just finished building) * PLUGIN_FINISH_GCC(before gcc exits) * PLUGIN_CXX_CP_PRE_GENERICIZE (data is a function body tree_node) for being able to look at fe gimple. * PLUGIN_PLUGIN_SETUP for when it's time to register with the pass manager * other stuff such being notified before particular passes execute(I think Grigori needs this). For my treehydra plugin it is important to land the GTY patch that Diego reviewed. I can start submitting patches for review whenever someone is ready to review them. Our plugin code isn't yet structured exactly as described but I will happily restructure the API to something acceptable and submit stuff for review. See http://developer.mozilla.org/en/Dehydra for more details on Mozilla's plugin work. Cheers, Taras
