Re: OSGi

[Peter]

Hmm, bundle wiring, not sure, will think about it.

Yep, guilty of lazyness.
So continuing on, with the deserialization process...
The stream reads in the stream class information, now Root's ClassLoader 
is on the stack,
Child2 is loaded using Root's Bundle.
Next, Child2's bundle C2's ClassLoader is pushed onto the stack.
Child2's fields are now read from the stream, in this case a reference 
number is read in from the stream,
this reference number is looked up from the stream reference table, and 
the same instance of BImpl contained
in Class1's field is returned.
Serialization doesn't send an object twice, unless it's written unshared.
Now Class2's instance is created, it's ClassLoader popped off the stack, 
root's second field is written, it's ClassLoader popped off the stack 
and its instance returned.

If Class2's field was another instance of BImpl, the class is stored in 
the ObjectStreamClass's cache and has been verified and loaded, so we 
don't need to resolve it again.  If BImpl was actually a different 
version and was loaded by Bundle BImpl2.0.0, then the annotation won't 
imply BImpl1.0.0,
so it will load a new bundle.

The Bundle wiring will depend on a combination of the local Bundle's or 
downloaded Bundle's manifest package import declarations.

The wiring may be slightly different at each end.  A PackageAdmin 
service may decide to change the loaded bundles, we need to ensure that 
we allow them to communicate accross compatible versions.

Hope this helps clarify it a little better.

Cheers & thanks,

Peter.

On 28/01/2017 10:11 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:
Ahh... You've missed the important part :) - child2.
You cannot assume BImpl class loaded in context of C1 is assignable to 
child2 - it depends on what other unrelated bundles have been loaded 
earlier and how the container chooses to resolve them.

Imagine:

BImpl has:
Require-Bundle bundle.with.exact.child2Api.version 1.0.0

Child2 has:
Import-Package packageOf.child2Api range [1.0.0, 2.0.0)

It might be perfectly fine in one container where only 
bundle.with.exact.child2Api.version 1.0.0 was installed. But might be 
wrong in another where there are two "packageOf.child2Api" exporting 
bundles installed.

So:
1. The information about BImpl is no longer on the stack and not 
available when loading Child2
2. Even if it was available (because for example you modify your 
algorithm so that you remember all previously loaded bundles) - it 
means that when loading Child2 you no longer can depend on the 
container bundle resolution.

And 2) is the point I am trying to make from the beginning: when 
transferring object graphs you MUST serialize the BundleWiring 
information and recreate it when deserializing. You need the snapshot 
of runtime state - declarative dependency information is not enough.

Thanks,
Michal

Peter wrote:
Ah yes, that's easy, assuming all classes are Serializable :)

Firstly we register a BundleListener with our OSGi framework and keep 
track of installed Bundles.

So Root, is the object graph root, Classes Child1 and Child2 are 
visible to Root.
The first thing we do is push the ClassLoader of Root onto our stack.
When the stream reads in the fields of Root, the first field child1, 
it will read the stream class information, the class Child1 will 
resolve from Bundle BR, since it is visible from there.
Now we obtain Child1's ClassLoader (which provides a BundleReference 
to Bundle C1) and push it onto our stack.
The stream reads in the fields of Child1, the first field Child1Api, 
is an instance of BImpl, it first attempts to use Bundle C1, however 
assuming that BImpl isn't visible from Bundle C1, we use the codebase 
annotation we read in from the stream (read in for each class), to 
find any installed bundles implied by the annotation's URI.
Upon finding BImpl's bundle from the BundleContext, we first ask the 
bundle to load the class, then we check that the class is assignable 
to field myImpl in Child1, before Child1 is instantiated, using 
class.isAssignableFrom.
If this fails and there are other versions of BImpl installed we 
might iterate through them until we have a match.
If there are no BImpl bundles installed, then we use the annotation 
to load a new Bundle.
At this time BImpl's ClassLoader is pushed onto the stack.
However BImpl has no fields to deserialize, so a new instance of 
BImpl is created and before it is used to set the field myImpl in 
Child1, BImpl's ClassLoader is popped off the stack.
Now that all fields of Child1 have been created, Child1 can be 
created, and bundle C1's ClassLoader is popped off the stack.
The same process is repeated for Child2.

Cheers,

Peter.




On 28/01/2017 7:41 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:
I fail to see how it could possibly work. Could you walk 
step-by-step serialize/deserialize with the following object graph:

Bundle API:
interface Api {}

Bundle BR:
class Root {
  Child1 child1;
  Child2 child2;

  Api getApi() {
    return isEven(getRandom()) ? child1.impl : child2.impl;
  }

}

Bundle C1
class Child1 {
  Child1Api myImpl;
}

Bundle C2
class Child2 {
  Child2Api myImpl;
}

Bundle Api1:
interface Child1Api extends Api {}

Bundle Api2:
interface Child2Api extends Api {}

Bundle BImpl:
class Impl implements Child1Api, Child2Api {}

Object graph:
impl = new BImpl()
root = new Root(new Child1(impl), new Child2(impl));

Serialize and deserialize root.

Thanks,
Michal

Peter wrote:
So here's how we can put it together:

Our OIS contains a stack we can use to track ClassLoader's at each 
branch in our serialized object graph.

  1. First object being read in by the OIS, check the cache see if the
     URI annotation is implied.
  2. If yes, use this Bundle to deserialize, place this Bundle's
     BundleReference [ClassLoader] on the top of the stack.
  3. If no, walk the stack, find the first BundleReference, 
dereference
     it's Bundle, obtain the BundleContext and request a new Bundle
     with the new URL.
  4. For each field in the current object:

       * Try to load it's class from the current Bundle .
       * If successful push Bundle's ClassLoader onto stack and return
         class.
       * Catch ClassNotFoundException, it's likely this is a 
dependency
         injected class the parent object's ClassLoader or Bundle
         doesn't have type visibility for.
       * Check the Bundle cache to see if the URI annotation is 
implied.
       * If yes, then iterate through each, load a class from this
         bundle, check that the returned class can be assigned to the
         current field's type.  If not, discard and continue until a
         class is found that can be assigned to the current field.
       * If not, load the new bundle.
       * Push the current Bundle's ClassLoader on the top of the 
stack.

  5. Pop the last (Object in the graph) field's ClassLoader off the 
stack.

When the process completes we have a resolved object graph with 
correct class visiblity.

Regards,

Peter.

On 28/01/2017 1:49 PM, Peter wrote:
Having implemented an ObjectInputStream (OIS), I can share the 
following insights:

   * The object at the head of the graph is externally visible and
     encapsulates all other objects and primitives.
   * The objects and primitives at the tails of the graph are fields.
   * Circular references in object graphs are unsafe with untrusted
     input...
   * A serialized object graph without circular references is a tree.
   * Fields may have been dependency injected.
   * Each branch in the tree has an object at the head of the branch.
   * The OIS knows the types of fields in the local class as well as
     the types of fields in the deserialized class before it has been
     resolved.

The information required to deserialized an OSGi graph is in the 
OIS implementation.  The OIS implementation is not visible to the 
extending MarshalInputStream implementation.

MarshalInputStream contains a ClassLoader field, defaultLoader.

So the codebase annotation is critical for the identity of the 
first object,

Regards,

Peter.

On 28/01/2017 3:39 AM, Bharath Kumar wrote:
Yes Peter. Usage of thread context class loader is discouraged in 
OSGi
environment.

http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html 


Some of the problems are hard to solve in OSGi environment. For 
example,
creating dynamic java proxy from 2 or more interfaces that are 
located in
different bundles.

http://blog.osgi.org/2008/08/classy-solutions-to-tricky-proxies.html?m=1 


This problem can be solved using composite class loader. But it is
difficult to write it correctly. Because OSGi environment is 
dynamic.

I believe that it is possible to provide enough abstraction in 
river code,
so that service developers don't even require to use context 
class loader
in their services.



Thanks&  Regards,
Bharath


On 27-Jan-2017 6:25 PM, "Peter"<j...@zeus.net.au>  wrote:

Thanks Gregg,

Thoughts inline below.

Cheers,

Peter.


On 27/01/2017 12:35 AM, Gregg Wonderly wrote:

Is there any thought here about how a single client might use 
both an
OSGi deployed service and a conventionally deployed service?

Not yet, I'm currently considering how to support OSGi by 
implementing an
RMIClassLoaderSPI, similar to how Dennis has for Maven in Rio.

I think once a good understanding of OSGi has developed, we can 
consider
how an implementation could support that, possibly by exploiting 
something
like Pax URL built into PreferredClassProvider.


   The ContextClassLoader is a good abstraction mechanism for 
finding “the”
approriate class loader.  It allows applications to deploy a 
composite
class loader in some form that would be able to resolve classes 
from many
sources and even provide things like preferred classes.

Yes, it works well for conventional frameworks and is utilised by
PreferredClassProvider, but it's use in OSGi is discouraged, I'd 
like to
consider how it's use can be avoided in an OSGi env.


In a Java desktop application, would a transition from a 
background
thread, interacting with a service to get an object from a 
service which is
not completely resolved to applicable loaders still resolve 
correctly in an
EventDispatch Thread?  That event dispatch thread can have the 
context
class loader set on it by the thread which got the object, to 
be the class
loader of the service object, to make sure that the resolution 
of classes
happens with the correct class loader such that there will not 
be a problem
with the service object having one set of definitions and 
another service
or the application classpath having a conflicting class 
definition by the
same name.

I’ve had to spend quite a bit of time to make sure that these 
scenarios
work correctly in my Swing applications.

Have you got more information?  I'm guessing this relates to 
delayed
unmarshalling into the EventDispatch thread.

It's early days yet, I'm still working it out what information 
is required
to resolve the correct ClassLoaders&  bundles, but this is an 
important
question, Bharath mentioned Entry's can be utilised for 
versioning and this
seems like a good idea.

What follows are thoughts and observations.

A bundle can be created from a URL, provided the codebase the 
URL refers
to has an OSGi bundle manifest, so this could allow any of the 
existing URL
formats to deliver a proxy codebase for an OSGi framework.  When 
OSGi loads
the bundle, the package dependencies will be wired up by the 
local env.  If
the URL doesn't reference a bundle, then we could use Bharath's 
approach
and subclass the client's ClassLoader, this does make all the 
clients
classes visible to the proxy however, but that would happen 
anyway in a
standard Jini / River environment.

OSGi gives preference to already loaded bundles when resolving 
package
dependencies, so the client should be careful not to unmarshall 
any proxy's
that might require a higher version than the bundle already 
installed when
the client bundle resolved its dependencies.

One of the proxy bundle dependencies will be the service api 
bundle.  The
proxy bundle can limit the service api package / packages to a 
specific
version or version range, which it could advertise in an Entry.  
Boolean
logic comparison of Entry's would have to be performed locally, 
after
matching on the service type (this requires delayed 
unmarshalling to
prevent the resolution of unwanted versions).

So, the OSGi bundles installed in remote JVM's communicating 
with each
other, may not be exactly the same version, but should be 
compatible.
Multiple versions of a package or bundle may be available in a 
JVM, so we
need to ensure we've selected one imported by the proxy.

We could define a simple rule, that the first URL in an 
annotation, must
be the proxy bundle, with all dependencies to be provisioned by 
the OSGi
Framework, which would allow the same codebase annotation to be 
utilised in
a non OSGi environment, but this will probably mean the loss of any
trailing URL annotations if the proxy's remarshalled.

Once the proxy bundle has been loaded, its ClassLoader needs to 
be the
default for the remaining stream classes (smart proxy fields), 
since it
knows the most about what dependencies are required, these will 
be visible
to the proxy's ClassLoader.

If a class doesn't have an existing ClassLoader referenced by 
annotatation
and cannot be resolved from the proxy's Bundle, it probably 
requres a new
bundle to be resolved.

What's next?  Determine what annotations to retrieve from OSGi 
Bundles.

How PreferredClassProvider currently obtains codebase 
annotations (Trivia:
the application/system ClassLoader is a URLClassLoader up to 
Java 8, but is
no longer an instance of URLClassLoader in Java 9):

Summary, If ClassLoader:

   1. Is local loader? return java.rmi.server.codebase property 
or null
      if not defined.
   2. Is instance of ClassAnnoation, get annotation and return.
   3. Is instance of URLClassLoader, get URL's as string and 
return.
   4. else return java.rmi.server.codebase property or null if 
not defined.


/**
      * Returns the annotation string for the specified class 
loader
      * (possibly null).  If check is true and the annotation 
would be
      * determined from an invocation of 
URLClassLoader.getURLs() on
      * the loader, only return the true annotation if the current
      * security context has permission to connect to all of the 
URLs.
      **/
     private String getLoaderAnnotation(ClassLoader loader, 
boolean check)
{

     if (isLocalLoader(loader)) {
         return getClassAnnotation(loader);
     }

         /*
      * Get the codebase URL path for the class loader, if it 
supports
      * such a notion (i.e., if it is a URLClassLoader or 
subclass).
      */
     String annotation = null;
     if (loader instanceof ClassAnnotation) {
         /*
          * If the class loader is one of our RMI class loaders, 
we have
          * already computed the class annotation string, and no
          * permissions are required to know the URLs.
          */
         annotation = ((ClassAnnotation) 
loader).getClassAnnotation();

     } else if (loader instanceof java.net.URLClassLoader) {
         try {
         URL[] urls = ((java.net.URLClassLoader) loader).getURLs();
         if (urls != null) {
             if (check) {
             SecurityManager sm = System.getSecurityManager();
             if (sm != null) {
                 Permissions perms = new Permissions();
                 for (int i = 0; i<  urls.length; i++) {
                 Permission p =
                     urls[i].openConnection().getPermission();
                 if (p != null) {
                     if (!perms.implies(p)) {
                     sm.checkPermission(p);
                     perms.add(p);
                     }
                 }
                 }
             }
             }
             annotation = PreferredClassLoader.urlsToPath(urls);
         }
         } catch (SecurityException e) {
         /*
          * If access was denied to the knowledge of the class
          * loader's URLs, fall back to the default behavior.
          */
         } catch (IOException e) {
         /*
          * This shouldn't happen, although it is declared to be
          * thrown by openConnection() and getPermission().  If it
          * does happen, forget about this class loader's URLs and
          * fall back to the default behavior.
          */
         }
     }

     if (annotation != null) {
         return annotation;
     } else {
         return getClassAnnotation(loader);
     }
     }

/**
      * Returns the annotation string for the specified class 
loader.
      *
      *<p>This method is invoked in order to determine the 
annotation
      * string for the system class loader, an ancestor of the 
system
      * class loader, any class loader that is not an instance of
      * {@link ClassAnnotation} or {@link URLClassLoader}, or 
(for an
      * invocation of {@link #getClassAnnotation(Class)
      * getClassAnnotation(Class)}) 
a<code>URLClassLoader</code>  for
      * which the current security context does not have the
      * permissions necessary to connect to all of its URLs.
      *
      *<p><code>PreferredClassProvider</code>  implements this 
method
      * as follows:
      *
      *<p>This method returns the value of the system property
      *<code>"java.rmi.server.codebase"</code>  (or possibly an 
earlier
      * cached value).
      *
      * @param loader the class loader to obtain the annotation 
string
      * for
      *
      * @return the annotation string for the class loader, or
      *<code>null</code>
      **/
     protected String getClassAnnotation(ClassLoader loader) {
     checkInitialized();
     return codebaseProperty;
     }


Gregg
On Jan 20, 2017, at 6:08 PM, Peter<j...@zeus.net.au>   wrote:
Looking at your modifications to ServiceDiscoveryManager, I 
noticed
you've made changes to set the context ClassLoader prior to 
calling the
lookup service.

This is eventually utilised by PreferredClassProvider to 
lookup the
necessary loader to utilise for deserialization of the lookup 
results.

I think if we develop a RMIClassLoader provider for OSGi, we 
can avoid
utilising the context ClassLoader.

Since all OSGi ClassLoader's are instances of BundleReference, 
it's easy
to utilise OSGi bundle url anotations (I think this needs to 
incorporate
bundle versions).  I'd also like to utilise Java 9 jrt style 
URL's.

Cheers,

Peter.

On 20/01/2017 11:09 PM, Bharath Kumar wrote:

Thanks Peter for the review.

While creating this POC, I tried to make RIO framework as set 
of OSGI.
bundles. Rio project extends LookupDiscoveryManager class in 
one of the
class 
.org.rioproject.impl.client.DiscoveryManagementPool.SharedDi
scoveryManager.
That's why I removed the final modifier.


Regarding groovy files,
I have made the org.apache.river as system fragment bundle. 
So we can't
import any packages from other bundles. But we can use system 
bundle's
packages,. That's why i removed groovy files. If we use these 
groovy
files,
we need to import packages from groovy bundle which is not 
possible
here. I
will check JGDMS to see how it is used.


Thanks&    Regards,
Bharath


On Fri, Jan 20, 2017 at 6:09 PM, Peter<j...@zeus.net.au>    
wrote:

Hi Bharath,
Re your changes (I've found so far):

LookupDiscoveryManager is non final, I'm interested why?

BasicInvocationDispatcher, you've set the context class 
loader around a
block of code, to use the ClassLoader passed in during 
construction.
I'm
currently investigating addong methods where ClassLoader can 
be passed
in
for OSGi.

Regarding bundle structure, I've restructured the layout 
here (so you
don't need to delete Groovy config):

https://github.com/pfirmstone/JGDMS/tree/Maven_build/modularize/JGDMS 


The full commit history has been retained, so u can see all 
changes.

Cheers,

Peter.

Sent from my Samsung device.

   Include original message
---- Original message ----
From: Bharath Kumar<bharathkuma...@gmail.com>
Sent: 20/01/2017 09:42:38 pm
To: dev@river.apache.org
Subject: Re: OSGi

Hello all,

I have also added a package in org.apache.river bundle to 
create the
river
service in osgi environment ( Here RIver
uses NonActivatableServiceDescriptor).

package name is  org.apache.river.start.ext


As river bundle is system fragment, i have to remove
the groovy dependency.
So i removed groovy files.

net.jini.config.Component.groovy

net.jini.config.GroovyConfig.groovy



Thanks&    Regards,

Bharath