On 30/11/06, Andrew Borley <[EMAIL PROTECTED]> wrote:
On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> On 30/11/06, Simon Laws <[EMAIL PROTECTED]> wrote:
> >
> > On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > >
> > > On 30/11/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > >
> > > >
> > > >
> > > > On 30/11/06, Simon Laws <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > > > >
> > > > > > On 30/11/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 30/11/06, Jean-Sebastien Delfino < [EMAIL PROTECTED]>
> > > wrote:
> > > > > > > >
> > > > > > > > Pete Robbins wrote:
> > > > > > > > > Our current method of packaging and loading an extension
is
> > > > > fairly
> > > > > > > > > simple:
> > > > > > > > > we load all schema and libraries in the extensions path.
> > This
> > > > > has a
> > > > > > > > > number
> > > > > > > > > of problems.
> > > > > > > > >
> > > > > > > > > 1. An extension may consist of more than one library e.g
.
> > > > > > > > > libmy_extension.so
> > > > > > > > > and libmy_extension_utils.so. Our current loading scheme
> > will
> > > > > > attempt
> > > > > > > > to
> > > > > > > > > load both of these and may fail on the one that doesn't
> > > provide
> > > > > the
> > > > > > > > > extension initialize method. On MacOS the output of our
> > build
> > > > > > produces
> > > > > > > > a
> > > > > > > > > libx.dylib and a series of symlinks to this called
> > > libx.0.dylib,
> > > > > > > > > libx.0.0.dylib etc.. ur runtime loads ALL of these which
> > > doesn't
> > > > > > cause
> > > > > > > > > problems as they are all the sam library and just
repeatedly
> > > > > > register
> > > > > > > > to
> > > > > > > > > handle the same requests. Very inefficient though!
> > > > > > > > >
> > > > > > > > > 2. Control of whether or not to load an extension
library is
> > > > > > currently
> > > > > > > > by
> > > > > > > > > renaming the library so the runtime doesn't find it. An
> > > example
> > > > > is
> > > > > > > > > that we
> > > > > > > > > ship our python extension as
> > libtuscany_sca_python.so.diabled.
> > > > > This
> > > > > > is
> > > > > > > >
> > > > > > > > > horrible and error prone.
> > > > > > > > >
> > > > > > > > > We could improve this by having a system configuration
file
> > > that
> > > > > > lists
> > > > > > > > > the
> > > > > > > > > required extensions but the I like the self contained
> > package
> > > > > > approach
> > > > > > > > > that
> > > > > > > > > we have now. I'd like to implement an improved scheme
for
> > > > > packaging
> > > > > > an
> > > > > > > > > extension by introducing a per extension configuration
file.
> > > > > > Something
> > > > > > > > > like:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > tuscany_sca_ws_binding.extension
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > <scacpp:extension name="ws binding" enabled="true">
> > > > > > > > > <library name="tuscany_sca_ws_reference"/>
> > > > > > > > > <library name="tuscany_sca_ws_service"/>
> > > > > > > > > </scacppp:extension>
> > > > > > > > >
> > > > > > > > > So the package would look like:
> > > > > > > > >
> > > > > > > > > extensions/
> > > > > > > > > ws/
> > > > > > > > > tusany_sca_ws_binding.extension
> > > > > > > > > lib/
> > > > > > > > > xsd/
> > > > > > > > > other_folder/
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > > The .extension configuration file is saying to load the
> > > library
> > > > > > which
> > > > > > > > is
> > > > > > > > > located somewhere in the package... the runtime will
find
> > > it...
> > > > > no
> > > > > > > > > need to
> > > > > > > > > specify a path.
> > > > > > > > >
> > > > > > > > > Taking this further the configuration file could list
the
> > > schema
> > > > > to
> > > > > > be
> > > > > > > > > loaded. Currently the runtime will just load any it
finds
> > but
> > > > > these
> > > > > > > > > may not
> > > > > > > > > be needed by the runtime e.g. the schema may be for some
> > > > > extension
> > > > > > > > > implementation specific purpose.
> > > > > > > > >
> > > > > > > > > I think it would also be good for the extension
> > > initialization()
> > > > > > > > > method to
> > > > > > > > > take as a parameter the root of the extension e.g.
> > > > > > > > > extension("/tuscany/extensions/ws"). This would allow
the
> > > > > extension
> > > > > > > > > package
> > > > > > > > > to contain any configuration information that it needs.
> > > > > > > > >
> > > > > > > > > I'd like to start by at least introducing the .extension
> > file
> > > > > for
> > > > > > each
> > > > > > > > > extension and loading only the specified library(ies) if
the
> > > > > > extension
> > > > > > > > is
> > > > > > > > > enabled.
> > > > > > > > >
> > > > > > > > > Any thoughts?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > Two thoughts:
> > > > > > > > - convention over configuration
> > > > > > > > - the runtime should be consumable without having to go
tweak
> > > XML
> > > > > > > > configuration files
> > > > > > > >
> > > > > > > > If I remember correctly, renaming the dlls to .disabled
was a
> > > last
> > > > >
> > > > > > > > minute change to work around DLL loading errors with our
M2
> > > > > release on
> > > > > > > > Windows.
> > > > > > > >
> > > > > > > > I agree that we should do better than renaming to
.disabled,
> > but
> > > > > I'd
> > > > > > > > like to understand better the actual issues that we faced
> > before
> > > > > > > > inventing yet another XML based runtime configuration
> > language
> > > :)
> > > > > > > >
> > > > > > > > I'm aware of the following issues:
> > > > > > > > 1. We need the runtime to load extension libs only, not
other
> > > libs
> > > > > > under
> > > > > > > >
> > > > > > > > the extension directory which are not actual extensions
> > > > > > > > 2. Same for XSDs, we need to load XSDs that contribute to
> > SCDL,
> > > > > and
> > > > > > > > leave other XSDs under the extension directory alone
> > > > > > > > 3. Extensions that cannot be loaded because some of their
> > > > > dependencies
> > > > > > > > are not there should no break the runtime
> > > > > > > > 4. The system admin / installer should be able to disable
> > > > > extensions
> > > > > > > > that won't load because their dependencies are not there
(I'm
> > > not
> > > > > yet
> > > > > > > > convinced that this is still an issue if we manage to
solve
> > > issue
> > > > > #3)
> > > > > > > >
> > > > > > > > Did we run into any other big issues?
> > > > > > >
> > > > > > >
> > > > > > > No, that's about it.
> > > > > > >
> > > > > > > 2. XSD loading can be done by convention (schemas for the
> > runtime
> > > > > are
> > > > > > > always in a folder called 'xsd'
> > > > > > > 1. Could be solved in a similar way by only loading
libraries in
> > a
> > > > > > folder
> > > > > > > called ???
> > > > > > > 3. Can be solved by just ignoring the load errors/issuing a
> > > warning
> > > > > > rather
> > > > > > > than giving up
> > > > > > > 4. Can be solved by solution to 3.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > I recall now why 3. was a big problem. Windows sometimes
throws up
> > > an
> > > > > > error
> > > > > > dialog when the load fails so it was not just a case of the
> > runtime
> > > > > > handling
> > > > > > the load failure so we had to "disable" the extension.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > --
> > > > > > Pete
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Pete
> > > > > >
> > > > > > Pete, just summarizing so that I understand. It seems there
are
> > two
> > > > > parts
> > > > > to this.
> > > > >
> > > > > A/ Building and installing an extension in order that it can be
> > > consumed
> > > > > by
> > > > > the runtime. Building and installing extensions is potentially a
> > > > > separate
> > > > > activity from building and installing the runtime itself.
Looking at
> > > the
> > > > > M2
> > > > > download they all come together at the moment.
> > > >
> > > >
> > > > Yes. It just happens that we are the only providers of extensions
at
> > the
> > > > moment.
> > > >
> > > > B/ Optionally consuming/enabling an extension in the runtime, once
the
> > > > > extension has been installed, in such a way that the runtime is
able
> > > to
> > > > > find
> > > > > and load it.
> > > >
> > > >
> > > > Yes.
> > > >
> > > > If I understand the previous posts (and the current code) A/ is
> > achieved
> > > > > by
> > > > > building the extension in the deployed folder layout
> > > > > extension type/
> > > > > bin
> > > > > lib
> > > > > xsd
> > > >
> > > >
> > > > The current runtime does not mandate any layout. It will seek and
> > > > destroy.. I mean seek and load any library it finds and any .xsd
it
> > > finds.
> > > > That layout is just how we package extensions in ou build at the
> > moment.
> > > >
> > > >
> > > > and Pete you are proposing that this is adjusted so that there are
> > > > > directories that hold just the stuff that the contains the
extension
> > > > > rather
> > > > > than the things it depends on.
> > > > > extension type/
> > > > > bin
> > > > > extensionbin?
> > > > > lib
> > > > > xsd
> > > > > extensionxsd?
> > > > >
> > > > > You are also proposing that B/ is achieved by ensuring that this
> > > > > directory
> > > > > structure be placed in a location that the the runtime can
search
> > for
> > > > > active
> > > > > extensions (as it does at the moment in the deploy directory)
> > > >
> > > >
> > > > it currently loads any extensions from under the
> > > > <sca_install_dir>/extensions folder
> > > >
> > > > and that the
> > > > > runtime ignores any badly configured extensions if possible.
> > Removing
> > > an
> > > > > extension's directory from the deploy
> > > >
> > > >
> > > > extensions not deploy
> > > >
> > > > directory structure has the effect of
> > > > > disabling it so I guess this is the fallback if the runtime
can't
> > > > > continue.
> > > >
> > > >
> > > > That is true in today's implementation.
> > > >
> > > > Anything more complex and, as Jean-Sebastien suggests, you start
> > getting
> > > > > into the full blown package and dependency management problem
that
> > > many
> > > > > other systems try to solve in different ways, e.g. rpm or pear.
> > > >
> > > >
> > > > Agreed... I'll give up on that plan ;-)
> > > >
> > > >
> > > > What we need is to enable the runtime to identify which extension
> > > > libraries to load and which schema to load. My suggestion is:
> > > >
> > > > my_extension/
> > > > bin/
> > > > extension/
> > > > library_that_will_be_loaded_by_the_runtime.so
> > > > lib/
> > > > lib_my_extension.so
> > > > include/
> > > > ... some headers maybe
> > > > xsd/
> > > > my_binding.xsd
> > > > any_other_folder/
> > > > any_stuff_I_like.xsd
> > > >
> > > > So the runtime will load any xsd that is in a folder named xsd and
> > > attempt
> > > > to load any library in a folder called extension. It would not
attempt
> > > to
> > > > load the any_stuff_I_like.xsd or the lib_my_extension.so
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > --
> > > > Pete
> > > >
> > >
> > >
> > >
> > > --
> > > Pete
> > >
> > > Sounds like the right idea to me. What goes in the bin dir?.
>
>
>
> Anything you like!
On windows, .exe's and .dll's are often found in bin, whereas the
.lib's (used for linking against at compile time) are generally found
in lib. Some projects (e.g. Axis2C) put dll's and lib's in lib (which
is more unix-like, and, IMO, nicer)
>
> Should the
> > lib be...
> >
> > lib/
> > lib_my_extension.lib
>
>
> lif you like... again... anything you like.
>
> Maybe you could stick with bin,include, lib,xsd and then have your
"other"
> > directory have everything else in it that the extension might rely on.
> > Depends what you now anticipate being in bin and lib over and above
the
> > actual library exposing the extension.
> >
> > Will you continue to mandate a specific place where extensions are
> > installed, i.e. from you post <sca_install_dir>/extensions folder?
>
>
> Yes
>
> So that
> > you can move an extension out of there if you don't want it to load.
>
>
> Well the idea here is you don't need to remove the whole package..just
the
> library from the extension dir
>
> Simon
> >
> >
> The layout of the extension package is whatever the package creater
wants to
> make it. We don't care. All we will care about is loading any library in
a
> folder anywhere in the extension package that is in a folder called
> 'extension' and expecting it to implement the extension initialize
method.
> We will also load any schema files in any folder called 'xsd'. An
extension
> only needs the xsd folder if it has schema required for the runtime
model.
>
> As an example here is the ws binding extension implemented for axis2c:
>
> <sca_install>/extensions/
> ... somewhere in this tree... could be anywhere...
> ws/
> xsd/
> ws-binding.xsd
> reference/
> lib/
> libtuscany_sca_reference.so
> service/
> lib/
> libtuscany_sca_service.so
>
>
> OK... today the runtime would load the ws-binding.xsd as it loads any
xsd it
> finds. It would also load any .so it finds so both reference aand
service
> extensions would be loaded/initialized.
>
> With my proposed change the xsd would be loaded (as it's in a folder
named
> 'xsd' ) but the libraries would not. To get the extensions loaded you
would
> have to package with an 'extension' folder ... again anywhere in the
tree...
> which "contains" the libraries to load eg.
>
> ws/
> extension/
> libload_this.so -> reference/lib/libtuscany_sca_reference.so
> xsd/
> ws-binding.xsd
> reference/
> lib/
> libtuscany_sca_reference.so
> service/
> lib/
> libtuscany_sca_service.so
>
> Here the runtime loads anything in any folder named 'extension' so the
> reference extension would be loaded/initialized.
Windows doesn't really do symlinks - I guess on windows we'd have the
tuscany_sca_reference.dll in extension/ and nothing in reference/lib/
or reference/bin/? For extensions that have dependencies on other
libraries, those dll's could go in lib/
I would actually have a copy of the dll in extension on windows and delete
it to disable. For a large extension it would be very wasy to make a tiny
dll which just implements the initialise interface expected by the runtime
and leave everything else in a separate dll in bin/lib.
I would package the cpp language extension as
>
> cpp/
> bin/
> lib/
> include/
> xsd/
> extension/
>
> The bin, lib, include are exactly what you would expect from a package
that
> you might want to build against. The extension/ folder would hold the
> library that implements the extension interface and xsd holds... well
xsds!
>
All sounds good to me. +1
Andy
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Pete
