Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Our SVN tree is here: svn co svn://gcc.gnu.org/svn/gcc/branches/plugin gcc This is an old branch off the GCC 4.3 mainline that Eric Christopher helped create after the 2007 GCC Summit, but we can update it if necessary. It includes libtool and libltdl, which may be useful. FWIW, the libtool/libltdl setup allows us to build and use plug-ins on all GNU/Linux platforms we've tested as well as Solaris and Mac OS X. Sean
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Sean, I don't really think most people are contributing to GCC for the bragging rights (at least I am not :-)). I would consider this a concerted, collaborating effort to get a simple, robust, and accommodating plugin design and implementation out as quickly as possible so that people can start using it. We offered to start with our patch simply because we had an implementation of the originally proposed APIs and we wanted to get the ball rolling quickly. As I mentioned in my previous email, if you (or other people/groups) have an implementation of the proposed APIs and prefer to start with yours (or theirs), I am all for it. (And please send the patch out so that people can look at it. Thanks.) Based on our initial experience, it's not that huge a task to implement basic GCC infrastructure supports for plugins. In fact, all the tasks you listed have been implemented in our patch (or in any other implementations, I believe), although there are certainly some corner cases that we didn't consider and some issues that have yet to converge. I think we just need to get started with an implementation of the proposed APIs (of course after reviewing and everything), and people can start iterating on that and adding new features/enhancements. (It's actually great that many of us have experiences implementing the plugin supports so that collectively we can cover more corner cases.) (BTW, a colleague pointed out to me that with my name in the header comment of the source files in our patch, people might have an impression that I am taking all the credits. That's certainly not my intention. I simply copied it from a standard header template that I kept for my GCC development. If we decide to start with that patch, I will certainly take it out.) Regards, Le-chun On Fri, Feb 6, 2009 at 2:54 AM, Sean Callanan wrote: > As a matter of protocol, I know there are several groups that all have > implementations. I bet any one of them would love to have the credit of > having their implementation be the one that got adopted. (I know ours > would! We're academics and would love to claim bragging rights.) > > In practice, Diego said that we have "several months" before we need to > integrate. Diego has graciously agreed to review patches; how about instead > of having a single implementation that goes in, we all pool our development > resources and do different portions of a fresh implementation? We could put > up a list of tasks on the Wiki and assign them to groups, so that everybody > has a hand in it. > > A possible list of tasks (which could each have a patch to itself) is: > > - Modify the GCC link process to use libltdl and libtool -export-dynamic > - Add the custom argument handler > - Implement the callback registration code > - Add hooks at relevant locations in GCC > - Implement the plug-in loading code and plug-in initialization > > Sean > > On Feb 5, 2009, at 7:14 PM, Le-Chun Wu wrote: > >> Attached please find the patch of our initial prototype of GCC plugin >> support based on the APIs described in the (old) wiki page. I also >> attached a sample plugin program (dumb-example.c) that shows how a >> plugin uses the APIs. >> >> Sean and Taras (and others), >> >> Diego will be creating a branch for the plugin support soon. I know we >> still have some issues that have yet to converge (such as flags for >> plugin arguments). But in order to get things moving, what do you >> think if we start with this patch and move from there? Or if you have >> other patches available that implement the currently proposed APIs, >> we can start with those too. Thanks. >> >> Le-chun > >
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Fri, 2009-02-06 at 12:32 +, Joseph S. Myers wrote: > On Fri, 6 Feb 2009, Sean Callanan wrote: > > > - Modify the GCC link process to use libltdl and libtool -export-dynamic > > Although this may make theoretical sense in terms of the work for a > cross-platform implementation already being done, I doubt it's a good idea > in practice if you want plugin-enabled GCC to be widely buildable and the > binaries not to be very closely tied to the exact system on which they > were built. (For example, for a while it was impossible to build current > versions of guile on systems with current versions of libtool installed > because of incompatible changes to libltdl, IOH, the test part of the guile build was not adequate/sufficient. The plugin implementation will have a good test coverage. > and I don't have much > confidence in future compatibility of libltdl, or of it working reliably > with GCC unless GCC always builds its own GCC-specific copy and links > against it using -Wl,-rpath,(something involving $ORIGIN). I also don't > see support for $ORIGIN in libtool, but toolchain relocatability means > that absolute RPATH settings must not be used when linking GCC.) Each plugin must do minimum checking for libraries compatibilities on the libltdl library they want to used. > I'd suggest we simply put the host-dependent code in GCC, appropriately > conditional, add code for more hosts as and when people wish to support > them with plugins, and make sure that plugin-disabled builds continue to > work for all hosts. There will undoubtedly be GCC-specific build system > changes needed for such things as building almost everything into a DLL > for Windows systems so that plugins on Windows can access symbols from > GCC (I understand from the previous discussions that this will be > necessary) and I doubt libtool would make such changes any easier. Just want to add that it is supposed to be the libtool's responsibility to resolve quirks, platforms, distribution problems. Well, i means i want the linking mechanism for plugin.
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Fri, 6 Feb 2009, Sean Callanan wrote: > - Modify the GCC link process to use libltdl and libtool -export-dynamic Although this may make theoretical sense in terms of the work for a cross-platform implementation already being done, I doubt it's a good idea in practice if you want plugin-enabled GCC to be widely buildable and the binaries not to be very closely tied to the exact system on which they were built. (For example, for a while it was impossible to build current versions of guile on systems with current versions of libtool installed because of incompatible changes to libltdl, and I don't have much confidence in future compatibility of libltdl, or of it working reliably with GCC unless GCC always builds its own GCC-specific copy and links against it using -Wl,-rpath,(something involving $ORIGIN). I also don't see support for $ORIGIN in libtool, but toolchain relocatability means that absolute RPATH settings must not be used when linking GCC.) I'd suggest we simply put the host-dependent code in GCC, appropriately conditional, add code for more hosts as and when people wish to support them with plugins, and make sure that plugin-disabled builds continue to work for all hosts. There will undoubtedly be GCC-specific build system changes needed for such things as building almost everything into a DLL for Windows systems so that plugins on Windows can access symbols from GCC (I understand from the previous discussions that this will be necessary) and I doubt libtool would make such changes any easier. -- Joseph S. Myers jos...@codesourcery.com
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
As a matter of protocol, I know there are several groups that all have implementations. I bet any one of them would love to have the credit of having their implementation be the one that got adopted. (I know ours would! We're academics and would love to claim bragging rights.) In practice, Diego said that we have "several months" before we need to integrate. Diego has graciously agreed to review patches; how about instead of having a single implementation that goes in, we all pool our development resources and do different portions of a fresh implementation? We could put up a list of tasks on the Wiki and assign them to groups, so that everybody has a hand in it. A possible list of tasks (which could each have a patch to itself) is: - Modify the GCC link process to use libltdl and libtool -export-dynamic - Add the custom argument handler - Implement the callback registration code - Add hooks at relevant locations in GCC - Implement the plug-in loading code and plug-in initialization Sean On Feb 5, 2009, at 7:14 PM, Le-Chun Wu wrote: Attached please find the patch of our initial prototype of GCC plugin support based on the APIs described in the (old) wiki page. I also attached a sample plugin program (dumb-example.c) that shows how a plugin uses the APIs. Sean and Taras (and others), Diego will be creating a branch for the plugin support soon. I know we still have some issues that have yet to converge (such as flags for plugin arguments). But in order to get things moving, what do you think if we start with this patch and move from there? Or if you have other patches available that implement the currently proposed APIs, we can start with those too. Thanks. Le-chun
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
I have updated the API Wiki at http://gcc.gnu.org/wiki/GCC_PluginAPI. 1) The way to pass arguments to plug-ins has been reverted to the approach suggested by many: -- -fplugin=/path/to/NAME.so -fNAME-arg1=value -fNAME-arg2=value -- 2) The name of struct plugin_registration is now struct callback_registration. 3) There is a name field in struct callback_registration to allow a callback to have a formal name that is reported in errors. 4) There is a new function, register_callback, that registers one callback at a time. This can be implemented as a simple wrapper on top of register_callbacks, or the other way around. Sean
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Attached please find the patch of our initial prototype of GCC plugin
support based on the APIs described in the (old) wiki page. I also
attached a sample plugin program (dumb-example.c) that shows how a
plugin uses the APIs.
Sean and Taras (and others),
Diego will be creating a branch for the plugin support soon. I know we
still have some issues that have yet to converge (such as flags for
plugin arguments). But in order to get things moving, what do you
think if we start with this patch and move from there? Or if you have
other patches available that implement the currently proposed APIs,
we can start with those too. Thanks.
Le-chun
On Thu, Feb 5, 2009 at 3:27 PM, Taras Glek wrote:
> Le-Chun Wu wrote:
>>
>> Hi Sean,
>>
>> It's great that you updated the wiki page with the latest and more
>> detailed API design.
>>
>> We (at Google) also started to look at the GCC plugin support a couple
>> of weeks ago. We had a quick prototype implemented based on the
>> original APIs that Taras put together in the old wiki. (I can send out
>> the patch later for people's reference.) The updated APIs in general
>> look good to me. I do have some comments based on our experience with
>> the initial prototyping:
>>
>
> Neat! I'd love to see that.
>>
>>
void register_callbacks(int nregistrations, struct plugin_registration
*registrations);
>>
>> Instead of passing in an array of plugin_registration objects with a
>> single register_callbacks call, it's probably better to have the
>> plugin call a sequence of register_callback with the necessary
>> information, as shown in the following example:
>>
>> void plugin_register_callback (const char *plugin_name, enum
>> plugin_event event, plugin_callback_func callback, void *user_data);
>>
>> /* In plugin code */
>> void
>> plugin_init()
>> {
>> ...
>> register_callback (plugin_name, PLUGIN_FINISH_STRUCT, handle_struct,
>> NULL);
>> register_callback (plugin_name, PLUGIN_FINISH_UNIT,
>> handle_end_of_compilation_unit, NULL);
>> ...
>> }
>>
>> In the function body of register_callback, GCC can then create the
>> plugin_registration objects and chain them together based on the event
>> type. Because GCC will need to maintain a list of plugin_registration
>> objects for each event anyway, we might as well let it create (and
>> destroy if necessary) the objects instead of leaving the task to the
>> plugins.
>>
>> Note that in my example an additional parameter, plugin_name, is added
>> in register_callback. We found it useful to have the name around when
>> emitting error messages if something goes wrong during the callback
>> registration process.
>>
>
> Sean,
> I agree with Le-Chun that separate register_callback calls would be better.
> Would you change that(in the interest of not having too many people modify
> the api), or should I? By the way, reformatting made the page much more
> readable, thanks.
>
> I think having a plugin-name parameter is something we can decide on later (
> just like the version stuff). I can see it being useful for debugging a
> situation with multiple loaded plugins, but I'm not convinced that it will
> be a common problem.
>
>>
-fplugin=/path/to/plugin.so;arg1=value;arg2=value;...
>>
>> I am not sure if it is GCC's responsibility to understand key-value
>> (or any other types of) arguments to plugins. I think GCC should
>> simply take a string (which, of course, can be something like
>> "arg1=value arg2=value") and pass it (unparsed) to the plugin. It is
>> plugin's job to parse/process the given arguments (whatever way it
>> likes). So the prototype of the plugin_init would look like this:
>>
>> void plugin_init (const char *plugin_name, const char *plugin_arg);
>>
>> In our current prototype, we implemented the originally proposed flag
>> "-fplugin-arg=", which is associated with the plugin specified in the
>> most recently parsed "-fplugin" flag. If a "-fplugin-arg" flag is used
>> in the command-line without any preceding "-fplugin", an error message
>> is emitted. Having the plugin name and its arguments concatenated as
>> currently proposed is also fine, but I would prefer a simple string
>> (like below) to a series of key-value pairs.
>>
>> -fplugin=/path/to/plugin.so;"arguments"
>>
>> (Note that the double quotes may not needed if the "arguments" doesn't
>> contain characters such as spaces.)
>>
>
> I agree with Daniel Jacobowitz's comment that letting every plugin to parse
> command-lines will lead to insanity. I'd be nice to let GCC take care of as
> much as possible there.
>
> From my brief encounter with the complexity of parameter handling code in
> GCC I would be tempted to start with the simplest possible proposal: either
> use -fplugin-arg as I described or stuffing everything into -fplugin and
> using a semi-colon separator.
> Infact, I would prefer not using -fplugin-arg as that leaves more room for
> future implementations to use -fplugin-- to do plugin
> arguments.
>>
>>
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Le-Chun Wu wrote:
Hi Sean,
It's great that you updated the wiki page with the latest and more
detailed API design.
We (at Google) also started to look at the GCC plugin support a couple
of weeks ago. We had a quick prototype implemented based on the
original APIs that Taras put together in the old wiki. (I can send out
the patch later for people's reference.) The updated APIs in general
look good to me. I do have some comments based on our experience with
the initial prototyping:
Neat! I'd love to see that.
void register_callbacks(int nregistrations, struct plugin_registration
*registrations);
Instead of passing in an array of plugin_registration objects with a
single register_callbacks call, it's probably better to have the
plugin call a sequence of register_callback with the necessary
information, as shown in the following example:
void plugin_register_callback (const char *plugin_name, enum
plugin_event event, plugin_callback_func callback, void *user_data);
/* In plugin code */
void
plugin_init()
{
...
register_callback (plugin_name, PLUGIN_FINISH_STRUCT, handle_struct, NULL);
register_callback (plugin_name, PLUGIN_FINISH_UNIT,
handle_end_of_compilation_unit, NULL);
...
}
In the function body of register_callback, GCC can then create the
plugin_registration objects and chain them together based on the event
type. Because GCC will need to maintain a list of plugin_registration
objects for each event anyway, we might as well let it create (and
destroy if necessary) the objects instead of leaving the task to the
plugins.
Note that in my example an additional parameter, plugin_name, is added
in register_callback. We found it useful to have the name around when
emitting error messages if something goes wrong during the callback
registration process.
Sean,
I agree with Le-Chun that separate register_callback calls would be
better. Would you change that(in the interest of not having too many
people modify the api), or should I? By the way, reformatting made the
page much more readable, thanks.
I think having a plugin-name parameter is something we can decide on
later ( just like the version stuff). I can see it being useful for
debugging a situation with multiple loaded plugins, but I'm not
convinced that it will be a common problem.
-fplugin=/path/to/plugin.so;arg1=value;arg2=value;...
I am not sure if it is GCC's responsibility to understand key-value
(or any other types of) arguments to plugins. I think GCC should
simply take a string (which, of course, can be something like
"arg1=value arg2=value") and pass it (unparsed) to the plugin. It is
plugin's job to parse/process the given arguments (whatever way it
likes). So the prototype of the plugin_init would look like this:
void plugin_init (const char *plugin_name, const char *plugin_arg);
In our current prototype, we implemented the originally proposed flag
"-fplugin-arg=", which is associated with the plugin specified in the
most recently parsed "-fplugin" flag. If a "-fplugin-arg" flag is used
in the command-line without any preceding "-fplugin", an error message
is emitted. Having the plugin name and its arguments concatenated as
currently proposed is also fine, but I would prefer a simple string
(like below) to a series of key-value pairs.
-fplugin=/path/to/plugin.so;"arguments"
(Note that the double quotes may not needed if the "arguments" doesn't
contain characters such as spaces.)
I agree with Daniel Jacobowitz's comment that letting every plugin to
parse command-lines will lead to insanity. I'd be nice to let GCC take
care of as much as possible there.
From my brief encounter with the complexity of parameter handling code
in GCC I would be tempted to start with the simplest possible proposal:
either use -fplugin-arg as I described or stuffing everything into
-fplugin and using a semi-colon separator.
Infact, I would prefer not using -fplugin-arg as that leaves more room
for future implementations to use -fplugin-- to do
plugin arguments.
Pass Manager
We think it would be quite daunting (and probably open up a can of
worms) to allow plugins to re-order passes. So to get things moving
quickly, in our initial prototype we only support insertion of a new
pass and replacing an existing pass. When registering a new pass with
GCC, the plugin uses the normal register_callback call with the
PLUGIN_PASS_MANAGER_SETUP event. However, instead of registering a
callback function, it passes in a plugin_pass object (see below) that
specifies the opt_pass object of the new pass, the name of the
reference pass, the static instance number of the reference pass, and
whether to insert before/after or replacing the reference pass.
enum pass_positioning_ops
{
PASS_POS_INSERT_AFTER,
PASS_POS_INSERT_BEFORE,
PASS_POS_REPLACE
};
struct plugin_pass
{
struct opt_pass *pass;
const char *reference_pass_name;
/* Insert the pass at the specified instance of the reference
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Thu, Feb 5, 2009 at 5:59 AM, Ben Elliston wrote: > On Tue, 2009-02-03 at 01:59 -0500, Sean Callanan wrote: > >> Our plugins do not break when switching compiler binaries. In fact, I >> have had plug-in binaries that perform very simple tasks work fine >> when switching (minor!) compiler releases. > > Thinking about this made me realise that the plugin framework will > require special consideration for utilities like ccache and distcc. > ccache checksums (or at least stats) the cc1 binary to decide whether a > cached object file is valid. If you change a plugin, the cc1 binary > won't change, but the generated code probably will. Why not use the output of cc1 --version, and then have the loaded plugins listed there. (This also means bug reports should have the plugins listed assuming the user uses the same command line with --version tacked on)
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Tue, 2009-02-03 at 01:59 -0500, Sean Callanan wrote: > Our plugins do not break when switching compiler binaries. In fact, I > have had plug-in binaries that perform very simple tasks work fine > when switching (minor!) compiler releases. Thinking about this made me realise that the plugin framework will require special consideration for utilities like ccache and distcc. ccache checksums (or at least stats) the cc1 binary to decide whether a cached object file is valid. If you change a plugin, the cc1 binary won't change, but the generated code probably will. Ben
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
2009/2/1 Sean Callanan : > > (3) The -fplugin-arg argument is one way to do arguments. We do it as > > -ftree-plugin=/path/to/plugin.so:arg=value:arg=value:... > In the previous discussions we had on this whole thing (http://gcc.gnu.org/ml/gcc/2008-09/msg00292.html), we were aiming towards arguments like: (Style 1) -fplugin= -f-[=] so this might look like: gcc -fplugin=edoc -fedoc-file=blah.edc -fedoc-embed The idea was that GCC would then search for the plugin on some defined search path (that will be specific for the build tuple and GCC version) OR a user could specify the plugin including path on the command line directly. So the above COULD look something like: gcc -fplugin=/usr/local/lib/gcc-4.0.4/i386-unknown-netbsdelf3.0/edoc.so.1.0.0 -fedoc-file=blah.edc -fedoc-embed gcc -fplugin=/usr/local/lib/gcc-4.0.4/i386-unknown-netbsdelf3.0/edoc.so.1.0.0 -fedoc-file=blah.edc -fedoc-embed I understand that currently we are looking at a new format like: (Style 2) -fplugin=;[=];[=]... example: gcc -fplugin=/usr/local/lib/gcc-4.0.4/i386-unknown-netbsdelf3.0/edoc.so.1.0.0;file=blah.edc;embed I personally prefer Style 1 for the plugin framework. My reasons include: * It looks more like existing options and has a level of familiarity. * Using ';' to separate args can be confusing to new people especially if they forget to quote them properly (I can see many questions on gcc-help arising from this). Also is ';' safe to use on all systems or can some filesystems include the ';' character in a file/path name? * I like to have the option for the plugin to be searched for by GCC on a pre-defined search path. For those that want to specify their plugins explicitly, they can still do so. * I know that "automatic loading" of plugins is not going to be in the plugin framework, but if in the future we DID decide to add it (for some unknown reason), then the plugin arguments would not need to be changed. I.e. With the above example if the plugin: edoc was to be automatically loaded, its arguments could be provided like: gcc -fedoc-file=blah.edc -fedoc-embed which is just the same as it was before, but the -fplugin=edoc was omitted. Brendon.
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Wed, Feb 04, 2009 at 05:04:10PM -0800, Le-Chun Wu wrote: > >> -fplugin=/path/to/plugin.so;arg1=value;arg2=value;... > >> > > I am not sure if it is GCC's responsibility to understand key-value > (or any other types of) arguments to plugins. I think GCC should > simply take a string (which, of course, can be something like > "arg1=value arg2=value") and pass it (unparsed) to the plugin. It is > plugin's job to parse/process the given arguments (whatever way it > likes). Please let GCC do this, so that argument parsing can be (A) centralized, and (B) standardized. Otherwise the right way to pass arguments will end up different for every plugin. -- Daniel Jacobowitz CodeSourcery
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Hi Sean,
It's great that you updated the wiki page with the latest and more
detailed API design.
We (at Google) also started to look at the GCC plugin support a couple
of weeks ago. We had a quick prototype implemented based on the
original APIs that Taras put together in the old wiki. (I can send out
the patch later for people's reference.) The updated APIs in general
look good to me. I do have some comments based on our experience with
the initial prototyping:
>>
>> void register_callbacks(int nregistrations, struct plugin_registration
>> *registrations);
>>
Instead of passing in an array of plugin_registration objects with a
single register_callbacks call, it's probably better to have the
plugin call a sequence of register_callback with the necessary
information, as shown in the following example:
void plugin_register_callback (const char *plugin_name, enum
plugin_event event, plugin_callback_func callback, void *user_data);
/* In plugin code */
void
plugin_init()
{
...
register_callback (plugin_name, PLUGIN_FINISH_STRUCT, handle_struct, NULL);
register_callback (plugin_name, PLUGIN_FINISH_UNIT,
handle_end_of_compilation_unit, NULL);
...
}
In the function body of register_callback, GCC can then create the
plugin_registration objects and chain them together based on the event
type. Because GCC will need to maintain a list of plugin_registration
objects for each event anyway, we might as well let it create (and
destroy if necessary) the objects instead of leaving the task to the
plugins.
Note that in my example an additional parameter, plugin_name, is added
in register_callback. We found it useful to have the name around when
emitting error messages if something goes wrong during the callback
registration process.
>>
>> -fplugin=/path/to/plugin.so;arg1=value;arg2=value;...
>>
I am not sure if it is GCC's responsibility to understand key-value
(or any other types of) arguments to plugins. I think GCC should
simply take a string (which, of course, can be something like
"arg1=value arg2=value") and pass it (unparsed) to the plugin. It is
plugin's job to parse/process the given arguments (whatever way it
likes). So the prototype of the plugin_init would look like this:
void plugin_init (const char *plugin_name, const char *plugin_arg);
In our current prototype, we implemented the originally proposed flag
"-fplugin-arg=", which is associated with the plugin specified in the
most recently parsed "-fplugin" flag. If a "-fplugin-arg" flag is used
in the command-line without any preceding "-fplugin", an error message
is emitted. Having the plugin name and its arguments concatenated as
currently proposed is also fine, but I would prefer a simple string
(like below) to a series of key-value pairs.
-fplugin=/path/to/plugin.so;"arguments"
(Note that the double quotes may not needed if the "arguments" doesn't
contain characters such as spaces.)
>>
>> Pass Manager
>>
We think it would be quite daunting (and probably open up a can of
worms) to allow plugins to re-order passes. So to get things moving
quickly, in our initial prototype we only support insertion of a new
pass and replacing an existing pass. When registering a new pass with
GCC, the plugin uses the normal register_callback call with the
PLUGIN_PASS_MANAGER_SETUP event. However, instead of registering a
callback function, it passes in a plugin_pass object (see below) that
specifies the opt_pass object of the new pass, the name of the
reference pass, the static instance number of the reference pass, and
whether to insert before/after or replacing the reference pass.
enum pass_positioning_ops
{
PASS_POS_INSERT_AFTER,
PASS_POS_INSERT_BEFORE,
PASS_POS_REPLACE
};
struct plugin_pass
{
struct opt_pass *pass;
const char *reference_pass_name;
/* Insert the pass at the specified instance of the reference pass.
If it's 0, do that for every instance. */
int ref_pass_instance_number;
enum pass_positioning_ops pos_op;
};
/* In plugin code */
void
plugin_init()
{
...
register_callback (plugin_name, PLUGIN_PASS_MANAGER_SETUP, NULL, &pass_info);
...
}
When registering and positioning a new pass, GCC will search the
passes chained up in all_lowering_passes, all_ipa_passes, and
all_passes (similar to Mozilla dehydra's implementation) to find the
reference pass(es) with the matching name and instance number, and
then either insert the new pass or replace the reference pass.
One caveat with our current implementation is that because the
registration of new plugin passes happens after the command-line
options are parsed, we cannot specify single pass dumping for plugin
passes (e.g. -fdump-tree-newpass). IR dumping of plugin passes is
enabled only when the dump-all flags (e.g. -fdump-tree-all) are
specified.
What do people think about this pass registration/positioning
interface that I described? (Again, I will send out our patch later so
that people can get a better idea if my description is not clear
enough.)
Le-chun W
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Sun, Feb 1, 2009 at 04:06, Sean Callanan wrote: > We also have a magic argument called FILE that lets you load arguments from > a file. That's what @ arguments are for. Which argues for not concatenating arguments. Would it be a problem to do -plugin=myplugin -plugin-myplugin-arg1=stuff -plugin-myplugin-someflag ... (finding /some/path/myplugin.so or C:\some\path\myplugin.dll in the plugin search path)?
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Feb 2, 2009, at 9:15 PM, Mark Mitchell wrote: Unless we have a lot more stability in plugin API than I expect we're actually going to have at first, I suggest we check something like the md5sum of the GCC binary itself. (Yes, I see the recursive problem in building such a binary.) The chances that my plugin will work with your compiler seem near-zero, and the failure modes will be ugly. But, plugins for a popular compiler binary (e.g., that associated with a GNU/Linux distribution, or with a compiler distribution) would make sense. The MD5 idea would hurt us because we have cc1 binaries compiled with - O0 -g3 -gdwarf2 and with normal optimization, and swap them out on occasion. In shops that deal with large source bases, I suspect this is not unheard-of. Our plugins do not break when switching compiler binaries. In fact, I have had plug-in binaries that perform very simple tasks work fine when switching (minor!) compiler releases. Perhaps if there is an ICE or GCC issues a warning / error, GCC could simply append "The plugin /path/to/plugin.so is not compatible with this GCC" to the message. Alternatively, GCC could always complain and you could have a flag -fforce-incompatible-plugins. Since the contents of enum machine_mode change based on target, I think it might be appropriate to check against the target as well as the version number and vendor patch level. Is there anything else we need to check? Sean
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Joseph S. Myers wrote: > I agree with checking once at startup, but that should be GCC's job, not > the plugin's. Yes. > The plugin should export the exact version string and > target triplet (and maybe configure options) of the compiler it was built > to be plugged into (the configure macros / build support provided for use > of plugins should make this automatic). GCC should check these at > startup. Unless we have a lot more stability in plugin API than I expect we're actually going to have at first, I suggest we check something like the md5sum of the GCC binary itself. (Yes, I see the recursive problem in building such a binary.) The chances that my plugin will work with your compiler seem near-zero, and the failure modes will be ugly. But, plugins for a popular compiler binary (e.g., that associated with a GNU/Linux distribution, or with a compiler distribution) would make sense. -- Mark Mitchell CodeSourcery m...@codesourcery.com (650) 331-3385 x713
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Mon, 2 Feb 2009, Taras Glek wrote: > Regarding versions: I think it's crazy to be passing a version in every single > function call. The plugin should check the gcc version it is being loaded > into on startup and bail if it doesn't match. I agree with checking once at startup, but that should be GCC's job, not the plugin's. The plugin should export the exact version string and target triplet (and maybe configure options) of the compiler it was built to be plugged into (the configure macros / build support provided for use of plugins should make this automatic). GCC should check these at startup. (OK, there's not much difference between GCC checking data automatically exported from a plugin and the build support code automatically putting checks in the plugin itself, but I think having the checks in GCC is safer for ensuring they are always done.) -- Joseph S. Myers jos...@codesourcery.com
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Taras, On Feb 2, 2009, at 7:34 PM, Taras Glek wrote: Regarding versions: I think it's crazy to be passing a version in every single function call. The plugin should check the gcc version it is being loaded into on startup and bail if it doesn't match. Since you and Diego seem to be of one mind on this, we can just strip versions. It might be useful to have an API inside GCC for picking apart version_string for purposes of comparison, but we can add that to "Nice to have" since it is in no way plugin-specific. I'll go ahead and strip the hook versioning API out of the draft. Sean
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
On Mon, Feb 2, 2009 at 19:34, Taras Glek wrote: > Regarding versions: I think it's crazy to be passing a version in every > single function call. The plugin should check the gcc version it is being > loaded into on startup and bail if it doesn't match. Agreed. Let's start with simpler mechanisms and increase complexity progressively. Diego.
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Sean Callanan wrote: Benjamin, On Feb 2, 2009, at 2:15 PM, Benjamin Smedberg wrote: It's possible for the plugin to implement every possible dlsym entry point and then farm the calls out to each individual script pass internally, but since GCC is already going to have to maintain a list of callbacks, it seems better to piggyback on the list GCC already maintains. You've got a very good point. I agree that the current callback proposal seems overly generic. Perhaps registering specific hooks into the pass manager and other plugin targets makes more sense? typedef (void pluginpasshook)(tree t, void *data); /* implemented in the pass manager */ register_plugin_event_after_pass(enum tree_pass, pluginpasshook fn, void* data) Hmmm, so could we unify tree_pass with the plugin_event structure from the API? Like, we'd add a special "virtual" pass for GCC entry and shutdown, that sort of thing. I'd like to warn against mixing pass manager passes and plugin callbacks. The pass manager needs a host of API improvements to be able to enumerate/reorder/remove/etc passes. However, the task of the plugin api should be to load outside code into GCC. The rest of the API improvements are general GCC improvements that happen to mainly benefit plugins. Once a plugin is loaded, then it could setup the necessary pass-manager configuration. Regarding versions: I think it's crazy to be passing a version in every single function call. The plugin should check the gcc version it is being loaded into on startup and bail if it doesn't match. Taras
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Benjamin, On Feb 2, 2009, at 2:15 PM, Benjamin Smedberg wrote: It's possible for the plugin to implement every possible dlsym entry point and then farm the calls out to each individual script pass internally, but since GCC is already going to have to maintain a list of callbacks, it seems better to piggyback on the list GCC already maintains. You've got a very good point. I agree that the current callback proposal seems overly generic. Perhaps registering specific hooks into the pass manager and other plugin targets makes more sense? typedef (void pluginpasshook)(tree t, void *data); /* implemented in the pass manager */ register_plugin_event_after_pass(enum tree_pass, pluginpasshook fn, void* data) Hmmm, so could we unify tree_pass with the plugin_event structure from the API? Like, we'd add a special "virtual" pass for GCC entry and shutdown, that sort of thing. Then, plugins would implement only a single entry point. GCC would inform the plugin of the arguments passed to it, and the plugin would register callbacks to perform its actual work. /* implemented in the plugin */ void gcc_plugin(const char *arguments); This is nice. One thing we do in our system is separate the arguments into (non-unique) key-value pairs. This makes arguments as close to regular arguments as possible, in that they have both names and values. What kind of verisioning? I don't think that having a single compiled plugin work with multiple versions of GCC should be a goal. I mean versioning in the sense of "this hook used to provide one tree, now it provides two." I'm not talking about versioning the entire GCC API. I'm a little worried about the colon separator. Windows file paths may legally have colons. Is there some separator character (semicolon?) which is not part of a path on any platform? Perhaps we shouldn't worry about it since -rdynamic doesn't work on Windows anyway. Agreed. OS X paths can include semicolons, though. This is something we need to think about. Sean
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
2009/2/2 Sean Callanan : > I updated the page http://gcc.gnu.org/wiki/GCC_PluginAPI. > Is there a reason for not using just -plugin=? What is with the -f*? Cheers, Manuel.
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
I updated the page http://gcc.gnu.org/wiki/GCC_PluginAPI. I cleaned up the formatting and added syntax coloring where appropriate. I also changed the API to reflect the comments so far, making it easier to register many callbacks at once while preserving the ability to register callbacks dynamically as needed. I volunteer to maintain this page. (That isn't to say no-one should edit it, just that I'll keep it neat and make threads if there are problems with the API.) Sean On Feb 2, 2009, at 2:15 PM, Benjamin Smedberg wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1/31/09 10:06 PM, Sean Callanan wrote: (1) register_callback is an unnecessary API. It's already possible to use dlsym() to get a pointer to a symbol in a plug-in. A plug-in could export a function symbol corresponding to each hook it is interested in; that way a plug-in would simply have things like void HOOK_PASS_MANAGER_SETUP(...) Mozilla currently uses only dlsym, and it's limiting. We have several unrelated analysis scripts loaded throught the same binary plugin. The binary plugin doesn't know in advance which passes the plugin wants to see. To implement this properly, we'd have to have which want to use the same callback. It's possible for the plugin to implement every possible dlsym entry point and then farm the calls out to each individual script pass internally, but since GCC is already going to have to maintain a list of callbacks, it seems better to piggyback on the list GCC already maintains. I agree that the current callback proposal seems overly generic. Perhaps registering specific hooks into the pass manager and other plugin targets makes more sense? typedef (void pluginpasshook)(tree t, void *data); /* implemented in the pass manager */ register_plugin_event_after_pass(enum tree_pass, pluginpasshook fn, void* data) Then, plugins would implement only a single entry point. GCC would inform the plugin of the arguments passed to it, and the plugin would register callbacks to perform its actual work. /* implemented in the plugin */ void gcc_plugin(const char *arguments); instead of needing to register that callback. Equivalently, a global data structure in the plug-in could map hook UIDs to function pointers, giving the same effect. This latter approach would provide very elegant hook versioning. What kind of verisioning? I don't think that having a single compiled plugin work with multiple versions of GCC should be a goal. (3) The -fplugin-arg argument is one way to do arguments. We do it as -ftree-plugin=/path/to/plugin.so:arg=value:arg=value:... We also have a magic argument called FILE that lets you load arguments from a file. I like this better than the current solution: with the current proposal it's hard to tell which -fplugin-arg is supposed to go with which -fplugin. I'm a little worried about the colon separator. Windows file paths may legally have colons. Is there some separator character (semicolon?) which is not part of a path on any platform? Perhaps we shouldn't worry about it since -rdynamic doesn't work on Windows anyway. - --BDS -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFJh0ZOSSwGp5sTYNkRAkn6AJ9e4zMwlZIrpdX7XUPDCfR0+56EYACfQW9f vMxuuIrJyP6O4hjo9b+fVwo= =OxHa -END PGP SIGNATURE-
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 1/31/09 10:06 PM, Sean Callanan wrote: > (1) register_callback is an unnecessary API. It's already possible to > use dlsym() to get a pointer to a symbol in a plug-in. A plug-in could > export a function symbol corresponding to each hook it is interested in; > that way a plug-in would simply have things like > > void HOOK_PASS_MANAGER_SETUP(...) Mozilla currently uses only dlsym, and it's limiting. We have several unrelated analysis scripts loaded throught the same binary plugin. The binary plugin doesn't know in advance which passes the plugin wants to see. To implement this properly, we'd have to have which want to use the same callback. It's possible for the plugin to implement every possible dlsym entry point and then farm the calls out to each individual script pass internally, but since GCC is already going to have to maintain a list of callbacks, it seems better to piggyback on the list GCC already maintains. I agree that the current callback proposal seems overly generic. Perhaps registering specific hooks into the pass manager and other plugin targets makes more sense? typedef (void pluginpasshook)(tree t, void *data); /* implemented in the pass manager */ register_plugin_event_after_pass(enum tree_pass, pluginpasshook fn, void* data) Then, plugins would implement only a single entry point. GCC would inform the plugin of the arguments passed to it, and the plugin would register callbacks to perform its actual work. /* implemented in the plugin */ void gcc_plugin(const char *arguments); > instead of needing to register that callback. Equivalently, a global > data structure in the plug-in could map hook UIDs to function pointers, > giving the same effect. This latter approach would provide very elegant > hook versioning. What kind of verisioning? I don't think that having a single compiled plugin work with multiple versions of GCC should be a goal. > (3) The -fplugin-arg argument is one way to do arguments. We do it as > > -ftree-plugin=/path/to/plugin.so:arg=value:arg=value:... > > We also have a magic argument called FILE that lets you load arguments > from a file. I like this better than the current solution: with the current proposal it's hard to tell which -fplugin-arg is supposed to go with which -fplugin. I'm a little worried about the colon separator. Windows file paths may legally have colons. Is there some separator character (semicolon?) which is not part of a path on any platform? Perhaps we shouldn't worry about it since -rdynamic doesn't work on Windows anyway. - --BDS -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFJh0ZOSSwGp5sTYNkRAkn6AJ9e4zMwlZIrpdX7XUPDCfR0+56EYACfQW9f vMxuuIrJyP6O4hjo9b+fVwo= =OxHa -END PGP SIGNATURE-
Re: Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
Sean Callanan wrote: 1- Agree on a common API and document it in http://gcc.gnu.org/wiki/GCC_PluginAPI So to get the ball rolling, here are some comments on the API as documented: - (1) register_callback is an unnecessary API. It's already possible to use dlsym() to get a pointer to a symbol in a plug-in. A plug-in could export a function symbol corresponding to each hook it is interested in; that way a plug-in would simply have things like void HOOK_PASS_MANAGER_SETUP(...) instead of needing to register that callback. Equivalently, a global data structure in the plug-in could map hook UIDs to function pointers, giving the same effect. This latter approach would provide very elegant hook versioning. Sounds good to me. (2) It would be very nice to be able to store trees between passes; right now, our custom version of GCC has a separate garbage-collection root for plug-ins so that they can store data between passes. Cool, we'd like that too. (3) The -fplugin-arg argument is one way to do arguments. We do it as -ftree-plugin=/path/to/plugin.so:arg=value:arg=value:... We also have a magic argument called FILE that lets you load arguments from a file. I think it's a little more convenient to use a separate argument for plugin arguments. However we still end up having to split that up into multiple arguments in many case, so you are probably right here too. (4) We strongly support the user () GCC attribute. - As a second note, we have some debugging tools we could contribute as soon as an API is fixed, including a GIMPLE visualizer with GDB integration. We also have a file (called parameter.def) which formalizes the macros that are valid for each type of GIMPLE tree node. We use this .def file extensively when handling trees in a generic way (such as for pretty-printing). Interesting. I took a completely opposite way of using GTY tags to reflect GIMPLE into JavaScript, which then pretty-prints GIMPLE(among other things) by porting the said macros to JS. https://developer.mozilla.org/en/Treehydra However any formalization should make my life easier. Sean, I agree with you and I think others will too, so please go ahead and make your API modifications on the wiki Taras
Plugin API Comments (was Re: GCC Plug-in Framework ready to port)
1- Agree on a common API and document it in http://gcc.gnu.org/wiki/GCC_PluginAPI So to get the ball rolling, here are some comments on the API as documented: - (1) register_callback is an unnecessary API. It's already possible to use dlsym() to get a pointer to a symbol in a plug-in. A plug-in could export a function symbol corresponding to each hook it is interested in; that way a plug-in would simply have things like void HOOK_PASS_MANAGER_SETUP(...) instead of needing to register that callback. Equivalently, a global data structure in the plug-in could map hook UIDs to function pointers, giving the same effect. This latter approach would provide very elegant hook versioning. (2) It would be very nice to be able to store trees between passes; right now, our custom version of GCC has a separate garbage-collection root for plug-ins so that they can store data between passes. (3) The -fplugin-arg argument is one way to do arguments. We do it as -ftree-plugin=/path/to/plugin.so:arg=value:arg=value:... We also have a magic argument called FILE that lets you load arguments from a file. (4) We strongly support the user () GCC attribute. - As a second note, we have some debugging tools we could contribute as soon as an API is fixed, including a GIMPLE visualizer with GDB integration. We also have a file (called parameter.def) which formalizes the macros that are valid for each type of GIMPLE tree node. We use this .def file extensively when handling trees in a generic way (such as for pretty-printing). Sincerely, Sean Callanan
