Maven Build and OSGi Platform Support
As most are probably aware, we are progressing with the Maven build of River, all packages have been moved and now it's time to commence moving the junit tests to their maven modules. The maven build structure is based on the work we've done in JGDMS, but there's no java code coming in from JGDMS at this time, just the module structure.At this time, it seems to make sense to also make maven modules OSGi bundles. However it is important to realise that making the modules into bundles isn't sufficient in iteself to support OSGi. This is due to a number of ServiceLoader Provider services being utilised within River. It's also an option to delay making these modules bundles until OSGi can be supported fully. I've recently got to the point where JGDMS supports OSGi, so maven modules are also OSGi bundles, but in addition all service loader services are now also OSGi services and either the service loader or OSGi service registry can be used to load them. River has two custom implementations of the service loader mechanism, one is org.apache.river.resources.Service the other net.jini.config.ConfigurationProvider. org.apache.river.resources.Service was practically identical to java.util.ServiceLoader up until Java 8, and service implementations must provide a zero argument constructor. ConfigurationProvider requires implementations to have constructors that have one or two arguments. For OSGi we needed a level of indirection to get around the constructor arguments, for this I have provided another interface that has a method that accepts the constructor arguments, Configuration providers implement this, consumers continue to use ConfigurationProvider::getConfiguration, even those using OSGi. For Java 9 and later, if we decide to make our River modules jpms modules as well, we will need to convert these providers to use ServiceLoader for module visibility. Long term I think it makes sense to have org.apache.river.resources.Service use ServiceLoader internally, rather than the current implementation, also it has been very useful as a common mechanism to consume service provider or osgi based services and leave the decision which to use to the developer. Note that I am not making any attempts to register Remote Jini Services as OSGi services. Anyone willing to assist with testing OSGi or writing some tests for it would be greatly appreciated. The OSGi platform recommends avoiding the thread context ClassLoader, this can be avoided by following advice below. Jini service implementations using the OSGi platform: 1. Must be configured to use net.jini.jeri.AtomicILFactory https://pfirmstone.github.io/JGDMS/jgdms-jeri/apidocs/net/jini/jeri/AtomicILFactory.html * AtomicILFactory requires a class argument in its Configuration, the ClassLoader of this class will be used for deserialization. * Codebase strings are not annotated in the stream by default. 2. Jini Services must implement the following: * ProxyAccessor https://pfirmstone.github.io/JGDMS/jgdms-platform/apidocs/net/jini/export/ProxyAccessor.html * CodebaseAccessor https://pfirmstone.github.io/JGDMS/jgdms-platform/apidocs/net/jini/export/CodebaseAccessor.html * Service parameters or any classes that will be serialized must implement AtomicSerial. https://pfirmstone.github.io/JGDMS/jgdms-platform/apidocs/org/apache/river/api/io/AtomicSerial.html 3. The net.jini.loader.ProxyCodebaseSpi implementation must be loaded (one is provided, it will be discovered from the OSGi service registry). 4. Proxy codebase URI Strings provided by CodebaseAccessor, must be bundles, they may be resolved using a provisioning URI mechanism. * In secure environments bundles should be signed. * The client need not trust the bundle certificates, these may be self signed, instead the service should be authenticated and the client trust the server certificates. Once the server is authenticated, the ProxyCodebaseSpi implementation will dynamically grant DeSerializationPermission to codebases signed with the certificates and URL's in the codebase annotation. 5. Once the bundle has been resolved, the service proxy will be deserialized into the bundle's ClassLoader, this bundle should be identical to that at the server endpoint, so that both endpoints have identical class visibility. This is the service proxy bundle, the service implementation at the server will depend on it and all serialized object classes will be contained within the service proxy bundle or one of its dependencies. 6. Note that it is advisable to sign proxy bundles. To avoid the overhead of NP Complete dependency provisioning, ServiceDiscoveryManager should be utilised. See also https://github.com
OSGi & Configuration
Any OSGi people on this list have advice or experience with ensuring class visibility with Configuration files? AtomicILFactory accepts a class or ClassLoader argument, for a service, this is used to locate the ClassLoader of the proxy bundle for deserialization. A class can be passed in from a configuration file. Any experience with ensuring the class is visible to the configuration? We have two configuration providers, one Groovy based, the other the old legacy Java like syntax that only uses static methods and constructors. Thanks, Peter.
Re: OSGi [PREVIOUSLY]Re: Maven Build
Thanks Gregg, those are very good points.
Cheers,
Peter.
On 2/10/2017 10:35 PM, Gregg Wonderly wrote:
I like the constructor argument mechanism as it becomes thread local detail,
rather than VM level detail. Too many “platform” things in Java have ended up
being “service type platform” like, instead of multiple services platform like.
Keeping things thread of execution specific helps us support “multiple
services platform” more readily. It also affords CodeSource level security
controls more readily.
Gregg
On Oct 2, 2017, at 5:01 AM, Peter<j...@zeus.net.au> wrote:
I'm considering api that may be required for improved OSGi support.
In order for an OSGi environment to deserialize a proxy, it needs to first
install a bundle for the proxy and resolve any dependencies. For OSGi a
ProxyPreparer must first locally marshall (create a MarshalledInstance) a
java.lang.reflect.Proxy (implementing ServiceProxyAccessor,
ServiceCodebaseAccessor, ServiceAttributesAccessor) instance (returned by
SafeServiceRegistrar.lookup) and unmarshall it, passing in the bundle
ClassLoader as the default ClassLoader. This ensures the ServiceProxy's Bundle
ClassLoader becomes the default ClassLoader for the underlying JERI endopoint.
Only at this time will a call ServiceProxyAccessor.getServiceProxy() result in
a correctly unmarshalled proxy. If this step isn't performed the default
ClassLoader for the JERI Endpoint will be the SafeServiceRegistrar's proxy
ClassLoader, and a ClassNotFoundException will be thrown when calling
ServiceProxyAccessor.getServiceProxy().
Given there's some complexity in the above, it would be prudent to implement
this in say a convenience class, perhaps called OSGiProxyPreparer, so
developers don't have to (boilerplate).
But we still need something from the underlying modular framework, to install a
Bundle for the service proxy and to ensure OSGiProxyPreparer recieves a
ClassLoader, while avoiding a dependency on OSGi. The OSGiProxyPreparer could
accept a ProxyClassLoaderProvisioner (see below) as a constructor argument?
Keep in mind the ProxyPreparer is a configuration concern.
The discovery infrastructure (LookupLocator, LookupLocatorDiscovery and
LookupDiscover classes) also needs a way to receive a ClassLoader to
deserialize lookup service proxy's. The codebase URL can be provided in a
multicast response, the same interface would need to be used as in
ProxyPreparation.
Please provide feedback, thoughts or suggestions.
/*
* Copyright 2017 The Apache Software Foundation.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package net.jini.loader;
import java.io.IOException;
/**
* Allows client code to implement provisioning of a ClassLoader
for a URI path,
* where the use of codebase annotations is unsuitable.
*
* The first URI in the path, must be the proxy URI, any
additionally appended
* URI are dependants, it's possible these may not be loaded
directly by the returned
* ClassLoader, but their classes should still be visible and
resolvable from it.
* Only the proxy URI classes are guaranteed to be loaded by a
returned
* ClassLoader.
* Dependant URI are not guaranteed to be loaded if suitable
versions are
* already.
*
* Some systems, notably OSGi, manage ClassLoader visibility
differently than
* Java's typical hierarchical ClassLoader relationships,
implementors of this
* interface may implement this to provision codebases into a
ClassLoader,
* prior to deserializing a proxy into the resolved ClassLoader.
*
* A proxy may be an instance of {@link java.lang.reflect.Proxy}
and have
* no associated codebase, in this case, a new ClassLoader should
be returned
* into which a dynamic proxy class can be provisioned.
*
* The implementing class must have {@link java.lang.RuntimePermission}
* "getClassLoader" and "createClassLoader". The implementing
class must
* ensure that the caller has {@link java.lang.RuntimePermission}
* "createClassLoader" as well.
*
*/
public interface ProxyClassLoaderProvisioner {
/**
* Create a new ClassLoader, given a space separated list of URL's.
*
* The first URL must contain the proxy class.
Re: OSGi [PREVIOUSLY]Re: Maven Build
I like the constructor argument mechanism as it becomes thread local detail,
rather than VM level detail. Too many “platform” things in Java have ended up
being “service type platform” like, instead of multiple services platform like.
Keeping things thread of execution specific helps us support “multiple
services platform” more readily. It also affords CodeSource level security
controls more readily.
Gregg
> On Oct 2, 2017, at 5:01 AM, Peter <j...@zeus.net.au> wrote:
>
> I'm considering api that may be required for improved OSGi support.
>
> In order for an OSGi environment to deserialize a proxy, it needs to first
> install a bundle for the proxy and resolve any dependencies. For OSGi a
> ProxyPreparer must first locally marshall (create a MarshalledInstance) a
> java.lang.reflect.Proxy (implementing ServiceProxyAccessor,
> ServiceCodebaseAccessor, ServiceAttributesAccessor) instance (returned by
> SafeServiceRegistrar.lookup) and unmarshall it, passing in the bundle
> ClassLoader as the default ClassLoader. This ensures the ServiceProxy's
> Bundle ClassLoader becomes the default ClassLoader for the underlying JERI
> endopoint. Only at this time will a call
> ServiceProxyAccessor.getServiceProxy() result in a correctly unmarshalled
> proxy. If this step isn't performed the default ClassLoader for the JERI
> Endpoint will be the SafeServiceRegistrar's proxy ClassLoader, and a
> ClassNotFoundException will be thrown when calling
> ServiceProxyAccessor.getServiceProxy().
>
> Given there's some complexity in the above, it would be prudent to implement
> this in say a convenience class, perhaps called OSGiProxyPreparer, so
> developers don't have to (boilerplate).
>
> But we still need something from the underlying modular framework, to install
> a Bundle for the service proxy and to ensure OSGiProxyPreparer recieves a
> ClassLoader, while avoiding a dependency on OSGi. The OSGiProxyPreparer
> could accept a ProxyClassLoaderProvisioner (see below) as a constructor
> argument? Keep in mind the ProxyPreparer is a configuration concern.
>
> The discovery infrastructure (LookupLocator, LookupLocatorDiscovery and
> LookupDiscover classes) also needs a way to receive a ClassLoader to
> deserialize lookup service proxy's. The codebase URL can be provided in a
> multicast response, the same interface would need to be used as in
> ProxyPreparation.
>
> Please provide feedback, thoughts or suggestions.
>
> /*
> * Copyright 2017 The Apache Software Foundation.
> *
> * Licensed under the Apache License, Version 2.0 (the "License");
> * you may not use this file except in compliance with the License.
> * You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package net.jini.loader;
>
> import java.io.IOException;
>
> /**
> * Allows client code to implement provisioning of a ClassLoader
> for a URI path,
> * where the use of codebase annotations is unsuitable.
> *
> * The first URI in the path, must be the proxy URI, any
> additionally appended
> * URI are dependants, it's possible these may not be loaded
> directly by the returned
> * ClassLoader, but their classes should still be visible and
> resolvable from it.
> * Only the proxy URI classes are guaranteed to be loaded by a
> returned
> * ClassLoader.
> * Dependant URI are not guaranteed to be loaded if suitable
> versions are
> * already.
> *
> * Some systems, notably OSGi, manage ClassLoader visibility
> differently than
> * Java's typical hierarchical ClassLoader relationships,
> implementors of this
> * interface may implement this to provision codebases into a
> ClassLoader,
> * prior to deserializing a proxy into the resolved ClassLoader.
> *
> * A proxy may be an instance of {@link java.lang.reflect.Proxy}
> and have
> * no associated codebase, in this case, a new ClassLoader should
> be returned
> * into which a dynamic proxy class can be provisioned.
> *
> * The implementing class must have {@link java.lang.RuntimePermission}
> * "getClassLoader" and "createClassLoader". The implementing
> class must
> * en
Re: OSGi [PREVIOUSLY]Re: Maven Build
I'm considering api that may be required for improved OSGi support.
In order for an OSGi environment to deserialize a proxy, it needs to
first install a bundle for the proxy and resolve any dependencies. For
OSGi a ProxyPreparer must first locally marshall (create a
MarshalledInstance) a java.lang.reflect.Proxy (implementing
ServiceProxyAccessor, ServiceCodebaseAccessor,
ServiceAttributesAccessor) instance (returned by
SafeServiceRegistrar.lookup) and unmarshall it, passing in the bundle
ClassLoader as the default ClassLoader. This ensures the ServiceProxy's
Bundle ClassLoader becomes the default ClassLoader for the underlying
JERI endopoint. Only at this time will a call
ServiceProxyAccessor.getServiceProxy() result in a correctly
unmarshalled proxy. If this step isn't performed the default
ClassLoader for the JERI Endpoint will be the SafeServiceRegistrar's
proxy ClassLoader, and a ClassNotFoundException will be thrown when
calling ServiceProxyAccessor.getServiceProxy().
Given there's some complexity in the above, it would be prudent to
implement this in say a convenience class, perhaps called
OSGiProxyPreparer, so developers don't have to (boilerplate).
But we still need something from the underlying modular framework, to
install a Bundle for the service proxy and to ensure OSGiProxyPreparer
recieves a ClassLoader, while avoiding a dependency on OSGi. The
OSGiProxyPreparer could accept a ProxyClassLoaderProvisioner (see below)
as a constructor argument? Keep in mind the ProxyPreparer is a
configuration concern.
The discovery infrastructure (LookupLocator, LookupLocatorDiscovery and
LookupDiscover classes) also needs a way to receive a ClassLoader to
deserialize lookup service proxy's. The codebase URL can be provided in
a multicast response, the same interface would need to be used as in
ProxyPreparation.
Please provide feedback, thoughts or suggestions.
/*
* Copyright 2017 The Apache Software Foundation.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package net.jini.loader;
import java.io.IOException;
/**
* Allows client code to implement provisioning of a ClassLoader
for a URI path,
* where the use of codebase annotations is unsuitable.
*
* The first URI in the path, must be the proxy URI, any
additionally appended
* URI are dependants, it's possible these may not be loaded
directly by the returned
* ClassLoader, but their classes should still be visible and
resolvable from it.
* Only the proxy URI classes are guaranteed to be loaded by a
returned
* ClassLoader.
* Dependant URI are not guaranteed to be loaded if suitable
versions are
* already.
*
* Some systems, notably OSGi, manage ClassLoader visibility
differently than
* Java's typical hierarchical ClassLoader relationships,
implementors of this
* interface may implement this to provision codebases into a
ClassLoader,
* prior to deserializing a proxy into the resolved ClassLoader.
*
* A proxy may be an instance of {@link java.lang.reflect.Proxy}
and have
* no associated codebase, in this case, a new ClassLoader should
be returned
* into which a dynamic proxy class can be provisioned.
*
* The implementing class must have {@link java.lang.RuntimePermission}
* "getClassLoader" and "createClassLoader". The implementing
class must
* ensure that the caller has {@link java.lang.RuntimePermission}
* "createClassLoader" as well.
*
*/
public interface ProxyClassLoaderProvisioner {
/**
* Create a new ClassLoader, given a space separated list of URL's.
*
* The first URL must contain the proxy class.
*
* The parent ClassLoader may be ignored, depending on the
underlying
* infrastructure.
*
* If uriPath is null, a ClassLoader is still created to
accommodate
* any dynamically created classes, with the parent ClassLoader
as the
* parent.
*
* @param uriPath the codebase URL path as a space-separated list
* of URLs, or null
* @param parent the default parent ClassLoader.
* @return
* @throws IOException
* @throws SecurityException if the caller
Re: OSGi [PREVIOUSLY]Re: Maven Build
Yes, I've been thinking about it.
My current thoughts; ServiceUI can be accessed through
net.jini.lookup.ServiceAttributesAccessor, so you can unmarshall
ServiceUI objects, without unmarshalling the service itself. However I
haven't given much consideration to locating codebases for ServiceUI
yet, at least not for OSGi.
The reality is that lots of different service implementations might have
common ServiceUI components.
Sounds like this is an opportune time to start thinking more about it.
It also makes me wonder whether MarshalledInstance could accept a URI
codebase string, or allow it to be set and retrieved from the current
codebase annotation byte array.
Cheers,
Peter.
On 28/09/2017 2:30 AM, Gregg Wonderly wrote:
Do you have anything planned around ServiceUI? I really use ServiceUI as a
discovery mechanism to find services which export a UI that a user can interact
with. What can happen at registration time, besides Entry specification to
help with codebase where ServiceUI bits are at? Are you just relying on the
“service” setup to include all of that detail, or is there something we can do,
to wrap ServiceUI into the mechanism you are talking about here?
Gregg
On Sep 27, 2017, at 3:59 AM, Peter<j...@zeus.net.au> wrote:
Some updates on thoughts about OSGi:
1. In JGDMS, SafeServiceRegistrar (extends ServiceRegistrar),
ServiceDiscoveryManager and ProxyPreparer allow provisioning of
OSGi bundles for Jini services.
2. SafeServiceRegistrar lookup results contain only instances of
java.lang.reflect.Proxy (implementing ServiceProxyAccessor,
ServiceCodebaseAccessor, ServiceAttributesAccessor) which a user
remarshalls and unmarshalls into their OSGi bundle provisioned
ClassLoader, prior to retrieving the actual service proxy using
ServiceProxyAccessor.
3. As a result different service principals using identical proxy
codebases, needn't share a ClassLoader, addressing the trust
domain issue previously alluded to.
4. There's no current mechanism to allow provisioning of a bundle for
a Registrar.
5. Existing discovery providers accept ClassLoader arguments for
unmarshalling Registrar's.
6. Existing Multicast responses allow for additional information to
be appended; a codebase resource for example.
7. LookupLocator, LookupDiscovery and LookupLocatorDiscovery classes
don't utilise discovery providers ClassLoader arguments.
8. Need to allow bundles to be provisioned for lookup services after
multicast discovery, by exposing discovery provider ClassLoader
arguments and allowing client to manage provisioning of bundle
into a ClassLoader, then passing that in during unicast discovery.
9. Don't break backward compatiblity.
Cheers,
Peter.
On 16/11/2016 4:18 PM, Dawid Loubser wrote:
+1 for OSGi providing the best solution to the class resolution problem,
though I think some work will have to be done around trust, as you say.
On 16/11/2016 02:23, Peter wrote:
The conventional alternatives will remain; the existing ClassLoader isolation
and the complexities surrounding multiple copies of the same or different
versions of the same classes interacting within the same jvm. Maven will
present a new alternative of maximum sharing, where different service
principals will share the same identity.
Clearly, the simplest solution is to avoid code download and only use
reflection proxy's
An inter process call isn't remote, but there is a question of how a reflection
proxy should behave when a subprocess is terminated.
UndeclaredThrowableException seems appropriate.
It would plug in via the existing ClassLoading RMIClassLoader provider
mechanism, it would be a client concern, transparent to the service or server.
The existing behaviour would remain default.
So there can be multiple class resolution options:
1. Existing PrefferedClassProvider.
2. Maven class resolution, where maximum class sharing exists. This may be
preferable in situations where there is one domain of trust, eg within one
corporation or company. Max performance.
3. Process Isolation. Interoperation between trusted entities, where code
version incompatibilities may exist, because of separate development teams and
administrators. Each domain of trust has it's own process domain. Max
compatibility, but slower.
4. OSGi.
There may be occassions where simpler (because developers don't need to
understand ClassLoaders), slow, compatible and reliable wins over fast and
complex or broken.
A subprocess may host numerous proxy's and codebases from one principal trust
domain (even a later version of River could be provisioned using Maven). A
subprocess would exist for each trust domain. So if there are two companies,
code from each remains isolated and communicates only using common api. No
unintended code versioning conflicts.
This choice would not prevent or exclude other methods of communi
Re: OSGi [PREVIOUSLY]Re: Maven Build
Yes that's correct. Sun chose to use the ClassLoader to represent the service's pincipal, dynamically granting permission after proxy verification. So that's what we have available on the call stack to use when performing authorisation permission checks: 1. Client's Principal. (Principal based grant) 2. ClassLoader (ClassLoader based grant, usually dynamically) 3. CodeSource's of other modules (CodeSource based grants). Since we trust the code with Maven (because it's downloaded and verified separately), it's probably not appropriate to use a ClassLoader to represent the service's principal in the client. I guess it really depends on the information that needs protection at the client and how much risk there is of information leaking to a service from the client, and the consequences. I suspect that for a lot of applications where the users belong to one organisation, there probably nothing to be done. It may be as simple as using a DomainCombiner to add a ProtectionDomain representing the service's principal (Subject) to the stack, or it may not be necessary to do anything at all.Currently Java only relaxes permission when injecting all ProtectionDomain's on a Thread's call stack with Principals (representing a user Subject), it's effectively granting the permissions of the user to all ProtectionDomain's on the stack. Typically in the current scenario, code, although trusted, is granted less permission in order to allow a user to be granted permission. An alternative behaviour would be to append a protection domain representing a user (Subject) to an existing call stack, such that the resuting permission set is the intersection of all domains (rather than the intersection of all code based grants and the addition of user based grants). In this case code can be granted all the permissions it needs and users might actually reduce the set of permissions, instead of only granting additional permission. So summing up, I think it's something to be aware of, but depending on circumstances probably doesn't require doing anything. Smart proxy's might share ClassLoaders; making dynamic permission grants to ClassLoaders based on the service that has just been authenticated (as per current practise), may not be appropriate. Cheers, Peter. On 28/09/2017 2:54 AM, Dennis Reedy wrote: Hi Peter, In reading your missive I'm not sure I understand when you say "Maven will present a new alternative of maximum sharing, where different service principals will share the same identity.", or "Maven class resolution". Are you referring to an approach where a service may declare it's codebase as an artifact that may have transitive dependencies, and that artifact, when resolved (from perhaps a secure repository) is downloaded onto the requesting client using Maven's local repository scheme? Then remote code becomes local code right? If services publish artifacts to (public/trusted) repositories and follow the approach that published versions are immutable, then a client would download the code only once, and re-use as necessary. Or are you thinking of this or something else? Thanks Dennis On Wed, Sep 27, 2017 at 4:59 AM, Peter<j...@zeus.net.au> wrote: Some updates on thoughts about OSGi: 1. In JGDMS, SafeServiceRegistrar (extends ServiceRegistrar), ServiceDiscoveryManager and ProxyPreparer allow provisioning of OSGi bundles for Jini services. 2. SafeServiceRegistrar lookup results contain only instances of java.lang.reflect.Proxy (implementing ServiceProxyAccessor, ServiceCodebaseAccessor, ServiceAttributesAccessor) which a user remarshalls and unmarshalls into their OSGi bundle provisioned ClassLoader, prior to retrieving the actual service proxy using ServiceProxyAccessor. 3. As a result different service principals using identical proxy codebases, needn't share a ClassLoader, addressing the trust domain issue previously alluded to. 4. There's no current mechanism to allow provisioning of a bundle for a Registrar. 5. Existing discovery providers accept ClassLoader arguments for unmarshalling Registrar's. 6. Existing Multicast responses allow for additional information to be appended; a codebase resource for example. 7. LookupLocator, LookupDiscovery and LookupLocatorDiscovery classes don't utilise discovery providers ClassLoader arguments. 8. Need to allow bundles to be provisioned for lookup services after multicast discovery, by exposing discovery provider ClassLoader arguments and allowing client to manage provisioning of bundle into a ClassLoader, then passing that in during unicast discovery. 9. Don't break backward compatiblity. Cheers, Peter. On 16/11/2016 4:18 PM, Dawid Loubser wrote: +1 for OSGi providing the best solution to the class resolution problem, though I think some work will have to be don
Re: OSGi [PREVIOUSLY]Re: Maven Build
Hi Peter, In reading your missive I'm not sure I understand when you say "Maven will present a new alternative of maximum sharing, where different service principals will share the same identity.", or "Maven class resolution". Are you referring to an approach where a service may declare it's codebase as an artifact that may have transitive dependencies, and that artifact, when resolved (from perhaps a secure repository) is downloaded onto the requesting client using Maven's local repository scheme? Then remote code becomes local code right? If services publish artifacts to (public/trusted) repositories and follow the approach that published versions are immutable, then a client would download the code only once, and re-use as necessary. Or are you thinking of this or something else? Thanks Dennis On Wed, Sep 27, 2017 at 4:59 AM, Peter <j...@zeus.net.au> wrote: > Some updates on thoughts about OSGi: > > 1. In JGDMS, SafeServiceRegistrar (extends ServiceRegistrar), > ServiceDiscoveryManager and ProxyPreparer allow provisioning of > OSGi bundles for Jini services. > 2. SafeServiceRegistrar lookup results contain only instances of > java.lang.reflect.Proxy (implementing ServiceProxyAccessor, > ServiceCodebaseAccessor, ServiceAttributesAccessor) which a user > remarshalls and unmarshalls into their OSGi bundle provisioned > ClassLoader, prior to retrieving the actual service proxy using > ServiceProxyAccessor. > 3. As a result different service principals using identical proxy > codebases, needn't share a ClassLoader, addressing the trust > domain issue previously alluded to. > 4. There's no current mechanism to allow provisioning of a bundle for > a Registrar. > 5. Existing discovery providers accept ClassLoader arguments for > unmarshalling Registrar's. > 6. Existing Multicast responses allow for additional information to > be appended; a codebase resource for example. > 7. LookupLocator, LookupDiscovery and LookupLocatorDiscovery classes > don't utilise discovery providers ClassLoader arguments. > 8. Need to allow bundles to be provisioned for lookup services after > multicast discovery, by exposing discovery provider ClassLoader > arguments and allowing client to manage provisioning of bundle > into a ClassLoader, then passing that in during unicast discovery. > 9. Don't break backward compatiblity. > > Cheers, > > Peter. > > On 16/11/2016 4:18 PM, Dawid Loubser wrote: > >> +1 for OSGi providing the best solution to the class resolution problem, >> though I think some work will have to be done around trust, as you say. >> >> >> On 16/11/2016 02:23, Peter wrote: >> >>> >>> The conventional alternatives will remain; the existing ClassLoader >>> isolation and the complexities surrounding multiple copies of the same or >>> different versions of the same classes interacting within the same jvm. >>> Maven will present a new alternative of maximum sharing, where different >>> service principals will share the same identity. >>> >>> Clearly, the simplest solution is to avoid code download and only use >>> reflection proxy's >>> >>> An inter process call isn't remote, but there is a question of how a >>> reflection proxy should behave when a subprocess is terminated. >>> >>> UndeclaredThrowableException seems appropriate. >>> >>> It would plug in via the existing ClassLoading RMIClassLoader provider >>> mechanism, it would be a client concern, transparent to the service or >>> server. >>> >>> The existing behaviour would remain default. >>> >>> So there can be multiple class resolution options: >>> >>> 1. Existing PrefferedClassProvider. >>> 2. Maven class resolution, where maximum class sharing exists. This may >>> be preferable in situations where there is one domain of trust, eg within >>> one corporation or company. Max performance. >>> 3. Process Isolation. Interoperation between trusted entities, where >>> code version incompatibilities may exist, because of separate development >>> teams and administrators. Each domain of trust has it's own process >>> domain. Max compatibility, but slower. >>> 4. OSGi. >>> >>> There may be occassions where simpler (because developers don't need to >>> understand ClassLoaders), slow, compatible and reliable wins over fast and >>> complex or broken. >>> >>> A subprocess may host numerous proxy's and codebases from one principal >>> trust domain (even a later
Re: OSGi [PREVIOUSLY]Re: Maven Build
Do you have anything planned around ServiceUI? I really use ServiceUI as a discovery mechanism to find services which export a UI that a user can interact with. What can happen at registration time, besides Entry specification to help with codebase where ServiceUI bits are at? Are you just relying on the “service” setup to include all of that detail, or is there something we can do, to wrap ServiceUI into the mechanism you are talking about here? Gregg > On Sep 27, 2017, at 3:59 AM, Peter <j...@zeus.net.au> wrote: > > Some updates on thoughts about OSGi: > > 1. In JGDMS, SafeServiceRegistrar (extends ServiceRegistrar), > ServiceDiscoveryManager and ProxyPreparer allow provisioning of > OSGi bundles for Jini services. > 2. SafeServiceRegistrar lookup results contain only instances of > java.lang.reflect.Proxy (implementing ServiceProxyAccessor, > ServiceCodebaseAccessor, ServiceAttributesAccessor) which a user > remarshalls and unmarshalls into their OSGi bundle provisioned > ClassLoader, prior to retrieving the actual service proxy using > ServiceProxyAccessor. > 3. As a result different service principals using identical proxy > codebases, needn't share a ClassLoader, addressing the trust > domain issue previously alluded to. > 4. There's no current mechanism to allow provisioning of a bundle for > a Registrar. > 5. Existing discovery providers accept ClassLoader arguments for > unmarshalling Registrar's. > 6. Existing Multicast responses allow for additional information to > be appended; a codebase resource for example. > 7. LookupLocator, LookupDiscovery and LookupLocatorDiscovery classes > don't utilise discovery providers ClassLoader arguments. > 8. Need to allow bundles to be provisioned for lookup services after > multicast discovery, by exposing discovery provider ClassLoader > arguments and allowing client to manage provisioning of bundle > into a ClassLoader, then passing that in during unicast discovery. > 9. Don't break backward compatiblity. > > Cheers, > > Peter. > > On 16/11/2016 4:18 PM, Dawid Loubser wrote: >> +1 for OSGi providing the best solution to the class resolution problem, >> though I think some work will have to be done around trust, as you say. >> >> >> On 16/11/2016 02:23, Peter wrote: >>> >>> The conventional alternatives will remain; the existing ClassLoader >>> isolation and the complexities surrounding multiple copies of the same or >>> different versions of the same classes interacting within the same jvm. >>> Maven will present a new alternative of maximum sharing, where different >>> service principals will share the same identity. >>> >>> Clearly, the simplest solution is to avoid code download and only use >>> reflection proxy's >>> >>> An inter process call isn't remote, but there is a question of how a >>> reflection proxy should behave when a subprocess is terminated. >>> >>> UndeclaredThrowableException seems appropriate. >>> >>> It would plug in via the existing ClassLoading RMIClassLoader provider >>> mechanism, it would be a client concern, transparent to the service or >>> server. >>> >>> The existing behaviour would remain default. >>> >>> So there can be multiple class resolution options: >>> >>> 1. Existing PrefferedClassProvider. >>> 2. Maven class resolution, where maximum class sharing exists. This may be >>> preferable in situations where there is one domain of trust, eg within one >>> corporation or company. Max performance. >>> 3. Process Isolation. Interoperation between trusted entities, where code >>> version incompatibilities may exist, because of separate development teams >>> and administrators. Each domain of trust has it's own process domain. Max >>> compatibility, but slower. >>> 4. OSGi. >>> >>> There may be occassions where simpler (because developers don't need to >>> understand ClassLoaders), slow, compatible and reliable wins over fast and >>> complex or broken. >>> >>> A subprocess may host numerous proxy's and codebases from one principal >>> trust domain (even a later version of River could be provisioned using >>> Maven). A subprocess would exist for each trust domain. So if there are >>> two companies, code from each remains isolated and communicates only using >>> common api. No unintended code versioning conflicts. >>> >>> This choice would not prevent or exclude other
OSGi [PREVIOUSLY]Re: Maven Build
Some updates on thoughts about OSGi: 1. In JGDMS, SafeServiceRegistrar (extends ServiceRegistrar), ServiceDiscoveryManager and ProxyPreparer allow provisioning of OSGi bundles for Jini services. 2. SafeServiceRegistrar lookup results contain only instances of java.lang.reflect.Proxy (implementing ServiceProxyAccessor, ServiceCodebaseAccessor, ServiceAttributesAccessor) which a user remarshalls and unmarshalls into their OSGi bundle provisioned ClassLoader, prior to retrieving the actual service proxy using ServiceProxyAccessor. 3. As a result different service principals using identical proxy codebases, needn't share a ClassLoader, addressing the trust domain issue previously alluded to. 4. There's no current mechanism to allow provisioning of a bundle for a Registrar. 5. Existing discovery providers accept ClassLoader arguments for unmarshalling Registrar's. 6. Existing Multicast responses allow for additional information to be appended; a codebase resource for example. 7. LookupLocator, LookupDiscovery and LookupLocatorDiscovery classes don't utilise discovery providers ClassLoader arguments. 8. Need to allow bundles to be provisioned for lookup services after multicast discovery, by exposing discovery provider ClassLoader arguments and allowing client to manage provisioning of bundle into a ClassLoader, then passing that in during unicast discovery. 9. Don't break backward compatiblity. Cheers, Peter. On 16/11/2016 4:18 PM, Dawid Loubser wrote: +1 for OSGi providing the best solution to the class resolution problem, though I think some work will have to be done around trust, as you say. On 16/11/2016 02:23, Peter wrote: The conventional alternatives will remain; the existing ClassLoader isolation and the complexities surrounding multiple copies of the same or different versions of the same classes interacting within the same jvm. Maven will present a new alternative of maximum sharing, where different service principals will share the same identity. Clearly, the simplest solution is to avoid code download and only use reflection proxy's An inter process call isn't remote, but there is a question of how a reflection proxy should behave when a subprocess is terminated. UndeclaredThrowableException seems appropriate. It would plug in via the existing ClassLoading RMIClassLoader provider mechanism, it would be a client concern, transparent to the service or server. The existing behaviour would remain default. So there can be multiple class resolution options: 1. Existing PrefferedClassProvider. 2. Maven class resolution, where maximum class sharing exists. This may be preferable in situations where there is one domain of trust, eg within one corporation or company. Max performance. 3. Process Isolation. Interoperation between trusted entities, where code version incompatibilities may exist, because of separate development teams and administrators. Each domain of trust has it's own process domain. Max compatibility, but slower. 4. OSGi. There may be occassions where simpler (because developers don't need to understand ClassLoaders), slow, compatible and reliable wins over fast and complex or broken. A subprocess may host numerous proxy's and codebases from one principal trust domain (even a later version of River could be provisioned using Maven). A subprocess would exist for each trust domain. So if there are two companies, code from each remains isolated and communicates only using common api. No unintended code versioning conflicts. This choice would not prevent or exclude other methods of communication, the service, even if isolated within it's own process will still communicate remotely over the network using JERI, JSON etc. This is orthogonal to and independant of remote communication protocols. OSGi would of course be an alternative option, if one wished to execute incompatible versions of libraries etc within one process, but different trust domains will have a shared identity, again this may not matter depending on the use case. Cheers, Peter. ESent from my Samsung device.
Re: OSGi Bundles for services
True, better to start small and grow, just looking for some buy in, 's all.
Cheers,
Peter.
On 24/02/2017 9:43 AM, Niclas Hedhman wrote:
I am sure there a lot of devils in the details here, but I think that you
have something that is workable, and better to get something smaller (but
evolvable) out the door for people to try than to spend eternity debating
the better approach.
Cheers
Niclas
On Fri, Feb 24, 2017 at 7:14 AM, Peter<j...@zeus.net.au> wrote:
When you think about classes with annotations in this environment, it also
means that classes at the client
may not return to the originating ClassLoader as boomerangs, unless
they're resolved by a common imported package by the proxy bundle. When
the boomerang returns it will be loaded in a PreferredClassLoader which has
the proxy as the parent ClassLoader in teh client instead of the
originating ClassLoader.
So if the client registered a listener with a service that remotely
filtered or sent events as batches, the client would have to view the
results via a common api interface. There would of course be no codebase
annotation losses, the point is objects may not return to their originating
loader, so it would seem best to avoid using codebase annotations in OSGi
unless a developer is very familiar with ClassLoaders etc.
It might be possible to register ClassLoader's with their codebase
annotation locally and associate them with proxy ClassLoaders, via a lookup
list in PreferredClassProvider, allowing boomerangs to return to the
correct ClassLoader, while also allowing PreferredClassProvider to discover
annotations. The annotation itself doesn't need to be a reference to the
Bundle, just a suitable list of jar files that would provide the same
classes.
The thought exercise is interesting in any case. Sometimes though,
simpler is better, it may be preferable to see how developers cope without
the bells and whistles first as this is far easier to debug and less
subject to problems, for some minor sacrifices in capability.
Regards,
Peter.
On 24/02/2017 8:52 AM, Peter wrote:
In case you're wondering why I've created bundles / modules
structured like below, if the endpoint was deserialized into the
service implementation bundle instead of the proxy bundle,
it may not be able to see deserialized classes from the proxy bundle's
transitive imports.
In effect the proxy bundle becomes the equivalent of the application
ClassLoader at each endpoint.
So any annotated classes will be loaded into PreferredClassLoader's
that have the proxy bundle as the parent ClassLoader. Any classes in
these ClassLoaders would have a very limited view, they could only
utilise packages imported by the proxy bundle, allowing the proxy
to limit available classes very easily, similar to a sandbox.
For example Mercury's proxy would allow packages in the java.* namespace
plus classes from any of the following packages:
Import-Package: net.jini.admin;version="[3.0,4)",net.jini.core.constra
int;version="[3.0,4)",net.jini.core.discovery;version="[3.0,4)",net.j
ini.core.entry;version="[3.0,4)",net.jini.core.event;version="[3.0,4)
",net.jini.core.lease;version="[3.0,4)",net.jini.event;version="[3.0,
4)",net.jini.export;version="[3.0,4)",net.jini.id;version="[3.0,4)",n
et.jini.io;version="[3.0,4)",net.jini.lookup;version="[3.0,4)",net.ji
ni.security;version="[3.0,4)",net.jini.security.proxytrust;version="[
3.0,4)",org.apache.river.admin;version="[3.0,4)",org.apache.river.api
.io;version="[3.0,4)",org.apache.river.landlord;version="[3.0,4)",org
.apache.river.proxy;version="[3.0,4)"
DownloadPermission can be used to prevent marshalling of downloaded
objects.
However the service could advertise available resouces via an Entry.
Cheers,
Peter.
On 23/02/2017 4:13 PM, Peter wrote:
Hmm, ASCII keeps getting scrubbed, so here it is:
SERVER JVM
__
| |
| Service |
| API Bundle |
|__|
|
|
Imports
API
Packages
|
|
__|___
|| I
Re: OSGi Bundles for services
When you think about classes with annotations in this environment, it
also means that classes at the client
may not return to the originating ClassLoader as boomerangs, unless
they're resolved by a common imported package by the proxy bundle. When
the boomerang returns it will be loaded in a PreferredClassLoader which
has the proxy as the parent ClassLoader in teh client instead of the
originating ClassLoader.
So if the client registered a listener with a service that remotely
filtered or sent events as batches, the client would have to view the
results via a common api interface. There would of course be no
codebase annotation losses, the point is objects may not return to their
originating loader, so it would seem best to avoid using codebase
annotations in OSGi unless a developer is very familiar with
ClassLoaders etc.
It might be possible to register ClassLoader's with their codebase
annotation locally and associate them with proxy ClassLoaders, via a
lookup list in PreferredClassProvider, allowing boomerangs to return to
the correct ClassLoader, while also allowing PreferredClassProvider to
discover annotations. The annotation itself doesn't need to be a
reference to the Bundle, just a suitable list of jar files that would
provide the same classes.
The thought exercise is interesting in any case. Sometimes though,
simpler is better, it may be preferable to see how developers cope
without the bells and whistles first as this is far easier to debug and
less subject to problems, for some minor sacrifices in capability.
Regards,
Peter.
On 24/02/2017 8:52 AM, Peter wrote:
In case you're wondering why I've created bundles / modules
structured like below, if the endpoint was deserialized into the
service implementation bundle instead of the proxy bundle,
it may not be able to see deserialized classes from the proxy bundle's
transitive imports.
In effect the proxy bundle becomes the equivalent of the application
ClassLoader at each endpoint.
So any annotated classes will be loaded into PreferredClassLoader's
that have the proxy bundle as the parent ClassLoader. Any classes in
these ClassLoaders would have a very limited view, they could only
utilise packages imported by the proxy bundle, allowing the proxy
to limit available classes very easily, similar to a sandbox.
For example Mercury's proxy would allow packages in the java.* namespace
plus classes from any of the following packages:
Import-Package: net.jini.admin;version="[3.0,4)",net.jini.core.constra
int;version="[3.0,4)",net.jini.core.discovery;version="[3.0,4)",net.j
ini.core.entry;version="[3.0,4)",net.jini.core.event;version="[3.0,4)
",net.jini.core.lease;version="[3.0,4)",net.jini.event;version="[3.0,
4)",net.jini.export;version="[3.0,4)",net.jini.id;version="[3.0,4)",n
et.jini.io;version="[3.0,4)",net.jini.lookup;version="[3.0,4)",net.ji
ni.security;version="[3.0,4)",net.jini.security.proxytrust;version="[
3.0,4)",org.apache.river.admin;version="[3.0,4)",org.apache.river.api
.io;version="[3.0,4)",org.apache.river.landlord;version="[3.0,4)",org
.apache.river.proxy;version="[3.0,4)"
DownloadPermission can be used to prevent marshalling of downloaded
objects.
However the service could advertise available resouces via an Entry.
Cheers,
Peter.
On 23/02/2017 4:13 PM, Peter wrote:
Hmm, ASCII keeps getting scrubbed, so here it is:
SERVER JVM
__
| |
| Service |
| API Bundle |
|__|
|
|
Imports
API
Packages
|
|
__|___
|| Imports packages | |
| Service Bundle |<| Proxy Bundle |---EP
| Implements | from proxy |__| |
| Proxy API||
|||
|
|
|
|
Re: OSGi Bundles for services
he ClassLoader at the server is set using the invocation layer factory (a configuration item). The client ClassLoader is determined during delayed unmarshalling and smart proxy bundle provisioning. This ensures that deserialization at each endpoint has a compatible view of classes (as recently discussed on osgi-dev). It's important at this time to distinguish between remote objects and remote services, registered with a lookup service. A remote service must have a proxy bundle for OSGi. The proxy bundle manifest determines requirements and wiring to package import versions. This includes dynamic proxy based services. For arguments sake, a remote object becomes a remote service when it is registered with a lookup service. A remote object isn't registered with a lookup service. Listeners are typically remote objects. In OSGi, it is advisable for remote objects to be dynamic proxy's without a codebase. Remote Object proxy's will be deserialized into the remote endpoint's nominated ClassLoader. For example, if it is transferred via a service, the remote object's dynamic proxy stub will be loaded into the service's proxy bundle ClassLoader in the server jvm. As Nic pointed out earlier, Bundle ClassLoader's are not instances of URLClassLoader and lack codebase annotations as a result. With the above rules in place, there is no need to create a custom RMIClassLoaderSpi specific to OSGi. With the above rules in place, it does open up an opportunity to use a remote object (eg a listener) with a conventional codebase as Mic had requested. This would use the PreferredClassProvider infrastructure, so the remote object proxy PreferredClassLoader would utilise the services server jvm proxy bundle ClassLoader as it's parent. This style of remote object should not be registered as a remote service. I would probably discourage this use case, except for advanced users. I think we need to make a distinction between remote services and remote objects for OSGi. This is a very practical solution for OSGi. Regards, Peter. Sent from my Samsung device.
Re: OSGi Bundles for services
Hmm, ASCII keeps getting scrubbed, so here it is:
SERVER JVM
__
| |
| Service |
| API Bundle |
|__|
|
|
Imports
API
Packages
|
|
__|___
|| Imports packages | |
| Service Bundle |<| Proxy Bundle |---EP
| Implements | from proxy |__| |
| Proxy API||
|||
|
|
|
|
|
N
E
CLIENT JVM T
W
O
__R
| | K
|| Imports packages | Service | |
| Management Agent |<| API Bundle | |
| Discovers & | from API |__| |
| Registers service || |
||| |
Imports|
API |
Packages |
| |
| |
__|___|
| | |
| Proxy Bundle |---EP
|__|
Note other package imports omitted for clarity.
On 23/02/2017 3:26 PM, Peter wrote:
I've attached some ASCII of the relationship between server and client
jvm bundles.
The ClassLoader at the server is set using the invocation layer
factory (a configuration item).
The client ClassLoader is determined during delayed unmarshalling and
smart proxy bundle provisioning.
This ensures that deserialization at each endpoint has a compatible
view of classes (as recently discussed on osgi-dev).
It's important at this time to distinguish between remote objects and
remote services, registered with a lookup service.
A remote service must have a proxy bundle for OSGi. The proxy bundle
manifest determines requirements and wiring to package import
versions. This includes dynamic proxy based services.
For arguments sake, a remote object becomes a remote service when it
is registered with a lookup service.
A remote object isn't registered with a lookup service. Listeners are
typically remote objects. In OSGi, it is advisable for remote objects
to be dynamic proxy's without a codebase. Remote Object proxy's will
be deserialized into the remote endpoint's nominated ClassLoader. For
example, if it is transferred via a service, the remote object's
dynamic proxy stub will be loaded into the service's proxy bundle
ClassLoader in the server jvm.
As Nic pointed out earlier, Bundle ClassLoader's are not instances of
URLClassLoader and lack codebase annotations as a result.
With the above rules in place, there is no need to create a custom
RMIClassLoaderSpi specific to OSGi.
With the above rules in place, it does open up an opportunity to use a
remote object (eg a listener) with a conventional codebase as Mic had
requested. This would use the PreferredClassProvider infrastructure,
so the remote object proxy PreferredClassLoader would utilise the
services server jvm proxy bundle ClassLoader as it's parent. This
style of remote object should not be registered as a remote service.
I would probably discourag
Re: OSGi Bundles for services
Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 23/02/2017 03:26:15 pm To: dev@river.apache.org <dev@river.apache.org> Subject: OSGi Bundles for services I've attached some ASCII of the relationship between server and client jvm bundles. The ClassLoader at the server is set using the invocation layer factory (a configuration item). The client ClassLoader is determined during delayed unmarshalling and smart proxy bundle provisioning. This ensures that deserialization at each endpoint has a compatible view of classes (as recently discussed on osgi-dev). It's important at this time to distinguish between remote objects and remote services, registered with a lookup service. A remote service must have a proxy bundle for OSGi. The proxy bundle manifest determines requirements and wiring to package import versions. This includes dynamic proxy based services. For arguments sake, a remote object becomes a remote service when it is registered with a lookup service. A remote object isn't registered with a lookup service. Listeners are typically remote objects. In OSGi, it is advisable for remote objects to be dynamic proxy's without a codebase. Remote Object proxy's will be deserialized into the remote endpoint's nominated ClassLoader. For example, if it is transferred via a service, the remote object's dynamic proxy stub will be loaded into the service's proxy bundle ClassLoader in the server jvm. As Nic pointed out earlier, Bundle ClassLoader's are not instances of URLClassLoader and lack codebase annotations as a result. With the above rules in place, there is no need to create a custom RMIClassLoaderSpi specific to OSGi. With the above rules in place, it does open up an opportunity to use a remote object (eg a listener) with a conventional codebase as Mic had requested. This would use the PreferredClassProvider infrastructure, so the remote object proxy PreferredClassLoader would utilise the services server jvm proxy bundle ClassLoader as it's parent. This style of remote object should not be registered as a remote service. I would probably discourage this use case, except for advanced users. I think we need to make a distinction between remote services and remote objects for OSGi. This is a very practical solution for OSGi. Regards, Peter. Sent from my Samsung device.
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
There is a huge misunderstanding here. I am pretty confident I understand and know how River works (at least up to version 3.0 exclusive) And my questions are not about how things are implemented but what changes Peter is proposing. Especially in the context of all the discussions that are held here around security, OSGI, serialization etc. The questions are very detailed and technical - because there is a lot of things that are not clear in the proposed designs. For example - saying that "deserializing an object is secure because it is a dynamic proxy" is simply misleading. That is why I ask more questions. You might call it "demanding your time" but it shouldn't stop anyone from asking questions. Cheers, Michal Gregg Wonderly wrote: The important detail is to understand that there is nearly a decade of development and design with experiences driving most of that around what exists today. Peter, nor really anyone any longer, can not “answer” all the questions you might have in a short amount of time. I sense that you have a view of how things should work, and believe that because that is not the case, that we need to “fix it.” I am not suggesting, nor do I sense Peter is, that there is nothing to fix or improve with River. However, it’s important to understand how River was designed to work, and that will require you to study, from several angles, the details. Yes, the code is hard to read. It doesn’t just calculate numbers, or arrange data in collections. Instead, it is interacting with the various details of security, class loading and JVM reflection to provide a flexible mechanism for RPC. I know, that you know, that there are a lot of technologies that exist today, which did not exist at the time that River was created as Jini. Instead, people without knowledge of many things that already existed to solve their problems went off to make software that works and looks the way that they thought it should for RPC or messaging at some RPC like level, and now we have a diverse set of technologies which all, in the end, allow network based communications to happen. River’s way of doing it, is but one. It’s not perfect and it needs work. Please understand that Peter and others have ideas for things/changes which will improve the user experience of River. What we are trying to do, is to understand your perspective better. The questions and comments/answers here are not going to be very good if you are just demanding our time, and not spending your time to learn the details what Peter is pointing out about how River works. Gregg On Feb 15, 2017, at 1:00 AM, Michał Kłeczek<mic...@kleczek.org> wrote: They are valid questions and you haven't answered any of them. I've described _your_ way of thinking (which I do not agree with). Apache River has many problems both technical and organizational. But I find the idea interesting and was expecting openness for contributions and open discussion. This is an open source project and there are no obligations to take part in the discussion nor answer any questions. But I find your patronizing statement disincentive to contribute to this project - especially that you are one of its main contributors. Regards, Michal Peter wrote: Finding the answer to this question should assist you to discover answers to many of the other questions you've had. While I've done my best to answer as many of your questions as I can, time is limited and I haven't had time to answer all of them or rebutt or confirm all arguments / assumptions. Sometimes the right questions are more important than answers. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter<j...@zeus.net.au> Sent: 15/02/2017 12:58:55 pm To: dev@river.apache.org<dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy The PreferredClassLoader will attempt to download the jar file in order to get the preferred list. DownloadPermission should be called DefineClassPermission, I don't think it will prevent download of the jar per say. Why must the bootstrap proxy be loaded by the codebase ClassLoader? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 15/02/2017 06:20:37 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So I've given it some thought and the only explanation I can come up with is: 1. To create an instance of the bootstrap proxy you need the codebase annotation. 2. Codebase annotation is needed because you want the bootstrap proxy's class to be defined in the proper codebase ClassLoader 3. Since you do not want to allow any code downloads before placing constraints on the bootstrap proxy - it has to be a dynami
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
The important detail is to understand that there is nearly a decade of development and design with experiences driving most of that around what exists today. Peter, nor really anyone any longer, can not “answer” all the questions you might have in a short amount of time. I sense that you have a view of how things should work, and believe that because that is not the case, that we need to “fix it.” I am not suggesting, nor do I sense Peter is, that there is nothing to fix or improve with River. However, it’s important to understand how River was designed to work, and that will require you to study, from several angles, the details. Yes, the code is hard to read. It doesn’t just calculate numbers, or arrange data in collections. Instead, it is interacting with the various details of security, class loading and JVM reflection to provide a flexible mechanism for RPC. I know, that you know, that there are a lot of technologies that exist today, which did not exist at the time that River was created as Jini. Instead, people without knowledge of many things that already existed to solve their problems went off to make software that works and looks the way that they thought it should for RPC or messaging at some RPC like level, and now we have a diverse set of technologies which all, in the end, allow network based communications to happen. River’s way of doing it, is but one. It’s not perfect and it needs work. Please understand that Peter and others have ideas for things/changes which will improve the user experience of River. What we are trying to do, is to understand your perspective better. The questions and comments/answers here are not going to be very good if you are just demanding our time, and not spending your time to learn the details what Peter is pointing out about how River works. Gregg > On Feb 15, 2017, at 1:00 AM, Michał Kłeczek <mic...@kleczek.org> wrote: > > They are valid questions and you haven't answered any of them. > I've described _your_ way of thinking (which I do not agree with). > > Apache River has many problems both technical and organizational. > But I find the idea interesting and was expecting openness > for contributions and open discussion. > > This is an open source project and there are no obligations to take part in > the discussion nor answer any questions. > But I find your patronizing statement disincentive to contribute to this > project - especially that you are one of its main contributors. > > Regards, > Michal > > Peter wrote: >> Finding the answer to this question should assist you to discover answers to >> many of the other questions you've had. >> >> While I've done my best to answer as many of your questions as I can, time >> is limited and I haven't had time to answer all of them or rebutt or confirm >> all arguments / assumptions. Sometimes the right questions are more >> important than answers. >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device. >> Include original message >> Original message >> From: Peter<j...@zeus.net.au> >> Sent: 15/02/2017 12:58:55 pm >> To: dev@river.apache.org<dev@river.apache.org> >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >> strategy >> >> The PreferredClassLoader will attempt to download the jar file in order to >> get the preferred list. >> >> DownloadPermission should be called DefineClassPermission, I don't think it >> will prevent download of the jar per say. >> >> Why must the bootstrap proxy be loaded by the codebase ClassLoader? >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device. >> Include original message >> Original message >> From: Michał Kłeczek<mic...@kleczek.org> >> Sent: 15/02/2017 06:20:37 am >> To: dev@river.apache.org >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >> strategy >> >> So I've given it some thought and the only explanation I can come up with >> is: >> 1. To create an instance of the bootstrap proxy you need the codebase >> annotation. 2. Codebase annotation is needed because you want the bootstrap >> proxy's class to be defined in the proper codebase ClassLoader 3. Since you >> do not want to allow any code downloads before placing constraints on the >> bootstrap proxy - it has to be a dynamic proxy. That way its class can be >> defined by the codebase loader and yet no code is downloaded >> So the overall sequence is as follows: 1. Get the codebase annotation and >> create the codebase loader 2. Create an instance
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
A code downloading object is of course possible. If you implement it, I can review it from a security perspective if you like. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 15/02/2017 07:48:48 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Miscommunication... as usual :D Anyway - I was really interested why you find the need for the bootstrap proxy to be necesarilly a dynamic proxy (since you seemed to find it very important from the security standpoint) - wanted to find out whether there were any issues in my thinking about providing a "code downloading object". Seems like there are not. Cheers, Michal Peter wrote: > Oh I thought it was part of your SmartProxyWrapper? > > Who'd have thought you were talking about my work lol! I wouldn't agree with >me either! > > My work: > 1. new secure discovery protocols that include registrar codebase url and >signers. > 2. authenticate lookup service during disco, grant minimal permissions if >auth successful. If unsuccessful no codebase download no deserialization. > 3. lookup service proxy can't unmarshall proxy's from other services >(insufficient permission), only has auth for its smart proxy url. > 4. lookup service can only return dynamic proxy tokens that the proxy >preparer can use to contact and authenticate each service. These tokens are >loaded in lookup service proxy ClassLoader (sorry no codebase annotations, no >codebase downloads, min permission). Lookup service not granted permission to >make network connections, can't unmarshall smart proxys for other services. > 5. All communications over trusted connections after auth. > 6. Input validation for deserialization. > > The code and docs are on github for all to see. The interfaces the tokens >impliment are documented in SafeServiceRegistrar. > > The code actually does what I've described above, but don't take my word for >it, check it for youself. :) > > If you disagree, don't use it. There is no highlander principle here, you >are free to implement alternatives. In fact I encourage you to do so as this >can only serve to increase understanding. > > Cheers, > > Peter > > > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 15/02/2017 05:00:14 pm > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > They are valid questions and you haven't answered any of them. > I've described _your_ way of thinking (which I do not agree with). > > Apache River has many problems both technical and organizational > But I find the idea interesting and was expecting openness > for contributions and open discussion. > > This is an open source project and there are no obligations to take part > in the discussion nor answer any questions. > But I find your patronizing statement disincentive to contribute to this > project - especially that you are one of its main contributors. > > Regards, > Michal > > Peter wrote: >> Finding the answer to this question should assist you to discover answers >>to many of the other questions you've had. >> >> While I've done my best to answer as many of your questions as I can, time >>is limited and I haven't had time to answer all of them or rebutt or confirm >>all arguments / assumptions. Sometimes the right questions are more >>important than answers. >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device. >> >> Include original message >> Original message >> From: Peter<j...@zeus.net.au> >> Sent: 15/02/2017 12:58:55 pm >> To: dev@river.apache.org<dev@river.apache.org> >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>invocation strategy >> >> The PreferredClassLoader will attempt to download the jar file in order to >>get the preferred list. >> >> DownloadPermission should be called DefineClassPermission, I don't think >>it will prevent download of the jar per say. >> >> Why must the bootstrap proxy be loaded by the codebase ClassLoader? >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device. >> >> Include original message >> Original message >> From: Michał Kłeczek<mic...@klecz
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
N.B. Clarification on item 4. This is for cases when we really want to lock down the registrar. Once the token has been confirmed as trusted and an instance of java.lang.reflect.Proxy, it can be marshalled and unmarshalled into the Client's ClassLoader, the client uses it to authenicate the service download the smart proxy etc. As all the tokens have identical classes with the same interfaces, only one dynamic proxy class is generated in each ClassLoader where it is used. This is the proxy preparers responsibility. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 15/02/2017 07:01:06 pm To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Oh I thought it was part of your SmartProxyWrapper? Who'd have thought you were talking about my work lol! I wouldn't agree with me either! My work: 1. new secure discovery protocols that include registrar codebase url and signers. 2. authenticate lookup service during disco, grant minimal permissions if auth successful. If unsuccessful no codebase download no deserialization. 3. lookup service proxy can't unmarshall proxy's from other services (insufficient permission), only has auth for its smart proxy url. 4. lookup service can only return dynamic proxy tokens that the proxy preparer can use to contact and authenticate each service. These tokens are loaded in lookup service proxy ClassLoader (sorry no codebase annotations, no codebase downloads, min permission). Lookup service not granted permission to make network connections, can't unmarshall smart proxys for other services. 5. All communications over trusted connections after auth 6. Input validation for deserialization. The code and docs are on github for all to see. The interfaces the tokens impliment are documented in SafeServiceRegistrar. The code actually does what I've described above, but don't take my word for it, check it for youself. :) If you disagree, don't use it. There is no highlander principle here, you are free to implement alternatives. In fact I encourage you to do so as this can only serve to increase understanding. Cheers, Peter Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 15/02/2017 05:00:14 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy They are valid questions and you haven't answered any of them. I've described _your_ way of thinking (which I do not agree with). Apache River has many problems both technical and organizational But I find the idea interesting and was expecting openness for contributions and open discussion. This is an open source project and there are no obligations to take part in the discussion nor answer any questions. But I find your patronizing statement disincentive to contribute to this project - especially that you are one of its main contributors. Regards, Michal Peter wrote: > Finding the answer to this question should assist you to discover answers to >many of the other questions you've had. > > While I've done my best to answer as many of your questions as I can, time is >limited and I haven't had time to answer all of them or rebutt or confirm all >arguments / assumptions. Sometimes the right questions are more important >than answers. > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Peter<j...@zeus.net.au> > Sent: 15/02/2017 12:58:55 pm > To: dev@river.apache.org<dev@river.apache.org> > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > The PreferredClassLoader will attempt to download the jar file in order to >get the preferred list. > > DownloadPermission should be called DefineClassPermission, I don't think it >will prevent download of the jar per say. > > Why must the bootstrap proxy be loaded by the codebase ClassLoader? > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > ---- Original message ---- > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 15/02/2017 06:20:37 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > So I've given it some thought and the only explanation I can come up > with is: > > 1. To create an instance of the bootstrap proxy you need the codebase > annotation. > 2. Codebase annotation is needed because you want the bootstrap proxy's > class to be > defined in the proper
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Oh I thought it was part of your SmartProxyWrapper? Who'd have thought you were talking about my work lol! I wouldn't agree with me either! My work: 1. new secure discovery protocols that include registrar codebase url and signers. 2. authenticate lookup service during disco, grant minimal permissions if auth successful. If unsuccessful no codebase download no deserialization. 3. lookup service proxy can't unmarshall proxy's from other services (insufficient permission), only has auth for its smart proxy url. 4. lookup service can only return dynamic proxy tokens that the proxy preparer can use to contact and authenticate each service. These tokens are loaded in lookup service proxy ClassLoader (sorry no codebase annotations, no codebase downloads, min permission). Lookup service not granted permission to make network connections, can't unmarshall smart proxys for other services. 5. All communications over trusted connections after auth. 6. Input validation for deserialization. The code and docs are on github for all to see. The interfaces the tokens impliment are documented in SafeServiceRegistrar. The code actually does what I've described above, but don't take my word for it, check it for youself. :) If you disagree, don't use it. There is no highlander principle here, you are free to implement alternatives. In fact I encourage you to do so as this can only serve to increase understanding. Cheers, Peter Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 15/02/2017 05:00:14 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy They are valid questions and you haven't answered any of them. I've described _your_ way of thinking (which I do not agree with). Apache River has many problems both technical and organizational But I find the idea interesting and was expecting openness for contributions and open discussion. This is an open source project and there are no obligations to take part in the discussion nor answer any questions. But I find your patronizing statement disincentive to contribute to this project - especially that you are one of its main contributors. Regards, Michal Peter wrote: > Finding the answer to this question should assist you to discover answers to >many of the other questions you've had. > > While I've done my best to answer as many of your questions as I can, time is >limited and I haven't had time to answer all of them or rebutt or confirm all >arguments / assumptions. Sometimes the right questions are more important >than answers. > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Peter<j...@zeus.net.au> > Sent: 15/02/2017 12:58:55 pm > To: dev@river.apache.org<dev@river.apache.org> > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > The PreferredClassLoader will attempt to download the jar file in order to >get the preferred list. > > DownloadPermission should be called DefineClassPermission, I don't think it >will prevent download of the jar per say. > > Why must the bootstrap proxy be loaded by the codebase ClassLoader? > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 15/02/2017 06:20:37 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > So I've given it some thought and the only explanation I can come up > with is: > > 1. To create an instance of the bootstrap proxy you need the codebase > annotation. > 2. Codebase annotation is needed because you want the bootstrap proxy's > class to be > defined in the proper codebase ClassLoader > 3. Since you do not want to allow any code downloads before placing > constraints on the > bootstrap proxy - it has to be a dynamic proxy. That way its class can > be defined by the codebase loader > and yet no code is downloaded > > So the overall sequence is as follows: > 1. Get the codebase annotation and create the codebase loader > 2. Create an instance of a dynamic proxy of a class defined by the > codebase loader > 3. IMPORTANT - before creating the proxy instance DO NOT grant any > download permissions > - that way we are sure the proxy does not triggers any code download and > execution due > to it implementing some foreign interfaces > 4. Once the proxy is instantiated - grant its ClassLoader download > permissions
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
They are valid questions and you haven't answered any of them. I've described _your_ way of thinking (which I do not agree with). Apache River has many problems both technical and organizational. But I find the idea interesting and was expecting openness for contributions and open discussion. This is an open source project and there are no obligations to take part in the discussion nor answer any questions. But I find your patronizing statement disincentive to contribute to this project - especially that you are one of its main contributors. Regards, Michal Peter wrote: Finding the answer to this question should assist you to discover answers to many of the other questions you've had. While I've done my best to answer as many of your questions as I can, time is limited and I haven't had time to answer all of them or rebutt or confirm all arguments / assumptions. Sometimes the right questions are more important than answers. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter<j...@zeus.net.au> Sent: 15/02/2017 12:58:55 pm To: dev@river.apache.org<dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy The PreferredClassLoader will attempt to download the jar file in order to get the preferred list. DownloadPermission should be called DefineClassPermission, I don't think it will prevent download of the jar per say. Why must the bootstrap proxy be loaded by the codebase ClassLoader? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 15/02/2017 06:20:37 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So I've given it some thought and the only explanation I can come up with is: 1. To create an instance of the bootstrap proxy you need the codebase annotation. 2. Codebase annotation is needed because you want the bootstrap proxy's class to be defined in the proper codebase ClassLoader 3. Since you do not want to allow any code downloads before placing constraints on the bootstrap proxy - it has to be a dynamic proxy. That way its class can be defined by the codebase loader and yet no code is downloaded So the overall sequence is as follows: 1. Get the codebase annotation and create the codebase loader 2. Create an instance of a dynamic proxy of a class defined by the codebase loader 3. IMPORTANT - before creating the proxy instance DO NOT grant any download permissions - that way we are sure the proxy does not triggers any code download and execution due to it implementing some foreign interfaces 4. Once the proxy is instantiated - grant its ClassLoader download permissions 5. Place the constraints on the proxy 6. Invoke a remote method on the proxy I understand the whole thing is to make sure the bootstrap proxy is defined by the codebase ClassLoader - and the ClassLoader is needed to be able to dynamically grant download permissions. What I DO NOT understand is - why the download permissions are needed at all? Since the bootstrap proxy's code MUST be local code - why not simply have its class defined by the context ClassLoader? Since downloading code is done only after authentication anyway - I don't see the reason to use DownloadPermissions at all. The only thing that comes to mind is that it is to make sure the service is not able to download code from a codebase different than its own. Which is OK but redundant. The reasoning is: Since the code of the service proxy is already trusted (we have established trust before downloading it) - we can simply place the constraints on the service proxy that instructs it to only download code meeting certain criteria. Am I correct in my thinking? Thanks, Michal Michał Kłeczek wrote: Let me dig some deeper. Comments inline. Peter wrote: Yes the dynamic proxy's are 100% local code. Remember dynamic proxy's don't have codebase s. :) Of course they do - look at PreferredClassProvider - the dynamic proxy class is defined by the codebase loader! Being a dynamic proxy does not mean there is no codebase. AtomicMarshalInputStream performs a special input validation on java.lang.reflect.Proxy thus ensuring the InvocationHandler is also trusted. If the InvocationHandler doesn't pass the test the proxy's never created. Why does it only verify dynamic proxies? Doesn't it verify normal objects? The dynamic proxy's put you in direct contact with the service provider using only local code with input validation constrained over secure connections (as configured with constraints in force). I think I've given you enough info now to investigate further. Ok - so your "token" is the same thing as my Smart
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Finding the answer to this question should assist you to discover answers to many of the other questions you've had. While I've done my best to answer as many of your questions as I can, time is limited and I haven't had time to answer all of them or rebutt or confirm all arguments / assumptions. Sometimes the right questions are more important than answers. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 15/02/2017 12:58:55 pm To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy The PreferredClassLoader will attempt to download the jar file in order to get the preferred list. DownloadPermission should be called DefineClassPermission, I don't think it will prevent download of the jar per say. Why must the bootstrap proxy be loaded by the codebase ClassLoader? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 15/02/2017 06:20:37 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So I've given it some thought and the only explanation I can come up with is: 1. To create an instance of the bootstrap proxy you need the codebase annotation. 2. Codebase annotation is needed because you want the bootstrap proxy's class to be defined in the proper codebase ClassLoader 3. Since you do not want to allow any code downloads before placing constraints on the bootstrap proxy - it has to be a dynamic proxy. That way its class can be defined by the codebase loader and yet no code is downloaded So the overall sequence is as follows: 1. Get the codebase annotation and create the codebase loader 2. Create an instance of a dynamic proxy of a class defined by the codebase loader 3. IMPORTANT - before creating the proxy instance DO NOT grant any download permissions - that way we are sure the proxy does not triggers any code download and execution due to it implementing some foreign interfaces 4. Once the proxy is instantiated - grant its ClassLoader download permissions 5. Place the constraints on the proxy 6. Invoke a remote method on the proxy I understand the whole thing is to make sure the bootstrap proxy is defined by the codebase ClassLoader - and the ClassLoader is needed to be able to dynamically grant download permissions. What I DO NOT understand is - why the download permissions are needed at all? Since the bootstrap proxy's code MUST be local code - why not simply have its class defined by the context ClassLoader? Since downloading code is done only after authentication anyway - I don't see the reason to use DownloadPermissions at all. The only thing that comes to mind is that it is to make sure the service is not able to download code from a codebase different than its own. Which is OK but redundant. The reasoning is: Since the code of the service proxy is already trusted (we have established trust before downloading it) - we can simply place the constraints on the service proxy that instructs it to only download code meeting certain criteria. Am I correct in my thinking? Thanks, Michal Michał Kłeczek wrote: > Let me dig some deeper. Comments inline. > > Peter wrote: >> Yes the dynamic proxy's are 100% local code. Remember dynamic >> proxy's don't have codebase s. :) > Of course they do - look at PreferredClassProvider - the dynamic proxy > class is defined by the codebase loader! > > Being a dynamic proxy does not mean there is no codebase. >> >> AtomicMarshalInputStream performs a special input validation on >> java.lang.reflect.Proxy thus ensuring the InvocationHandler is also >> trusted. If the InvocationHandler doesn't pass the test the proxy's >> never created. > Why does it only verify dynamic proxies? Doesn't it verify normal > objects? >> >> The dynamic proxy's put you in direct contact with the service >> provider using only local code with input validation constrained over >> secure connections (as configured with constraints in force). >> >> I think I've given you enough info now to investigate further. > Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's > call it a "bootstrap proxy", ok? > > 1. What interface this bootstrap proxy implements? > 2. Why do you think it has to be a dynamic proxy (ie. an instance of a > subclass of java.lang.Proxy)? > 3. What and when are DownloadPermissions required? How do they add to > the overall security? > I understand the security of service proxies is enforced by the > constraints placed on the bootstrap proxy.
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
The PreferredClassLoader will attempt to download the jar file in order to get the preferred list. DownloadPermission should be called DefineClassPermission, I don't think it will prevent download of the jar per say. Why must the bootstrap proxy be loaded by the codebase ClassLoader? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 15/02/2017 06:20:37 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So I've given it some thought and the only explanation I can come up with is: 1. To create an instance of the bootstrap proxy you need the codebase annotation. 2. Codebase annotation is needed because you want the bootstrap proxy's class to be defined in the proper codebase ClassLoader 3. Since you do not want to allow any code downloads before placing constraints on the bootstrap proxy - it has to be a dynamic proxy. That way its class can be defined by the codebase loader and yet no code is downloaded So the overall sequence is as follows: 1. Get the codebase annotation and create the codebase loader 2. Create an instance of a dynamic proxy of a class defined by the codebase loader 3. IMPORTANT - before creating the proxy instance DO NOT grant any download permissions - that way we are sure the proxy does not triggers any code download and execution due to it implementing some foreign interfaces 4. Once the proxy is instantiated - grant its ClassLoader download permissions 5. Place the constraints on the proxy 6. Invoke a remote method on the proxy I understand the whole thing is to make sure the bootstrap proxy is defined by the codebase ClassLoader - and the ClassLoader is needed to be able to dynamically grant download permissions. What I DO NOT understand is - why the download permissions are needed at all? Since the bootstrap proxy's code MUST be local code - why not simply have its class defined by the context ClassLoader? Since downloading code is done only after authentication anyway - I don't see the reason to use DownloadPermissions at all. The only thing that comes to mind is that it is to make sure the service is not able to download code from a codebase different than its own. Which is OK but redundant. The reasoning is: Since the code of the service proxy is already trusted (we have established trust before downloading it) - we can simply place the constraints on the service proxy that instructs it to only download code meeting certain criteria. Am I correct in my thinking? Thanks, Michal Michał Kłeczek wrote: > Let me dig some deeper. Comments inline. > > Peter wrote: >> Yes the dynamic proxy's are 100% local code. Remember dynamic >> proxy's don't have codebase s. :) > Of course they do - look at PreferredClassProvider - the dynamic proxy > class is defined by the codebase loader! > > Being a dynamic proxy does not mean there is no codebase. >> >> AtomicMarshalInputStream performs a special input validation on >> java.lang.reflect.Proxy thus ensuring the InvocationHandler is also >> trusted. If the InvocationHandler doesn't pass the test the proxy's >> never created. > Why does it only verify dynamic proxies? Doesn't it verify normal > objects? >> >> The dynamic proxy's put you in direct contact with the service >> provider using only local code with input validation constrained over >> secure connections (as configured with constraints in force). >> >> I think I've given you enough info now to investigate further. > Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's > call it a "bootstrap proxy", ok? > > 1. What interface this bootstrap proxy implements? > 2. Why do you think it has to be a dynamic proxy (ie. an instance of a > subclass of java.lang.Proxy)? > 3. What and when are DownloadPermissions required? How do they add to > the overall security? > I understand the security of service proxies is enforced by the > constraints placed on the bootstrap proxy. > So where is the place for DownloadPermissions? > > 4. Finally - how is the lookup service proxy verified? Does it also > provide the bootstrap proxy? > If so - what special role does it play in the architecture? > The bootstrap proxy does not have to be provided by the lookup > service, does it? > > If it is verified differently - why and how? > > Thanks, > Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
N.B. The Certs aren't encoded in the codebase annotation. The jar file itself is signed. DownloadPermission is actually misnamed, it really should be called ClassDefiningPermission. The JarFile is actually downloaded. On 14/02/2017 1:45 AM, Michał Kłeczek wrote: Peter wrote: The codebase is signed and download permission is granted only to the signed codebase. What is "signed codebase"? How do you encode the signature in the codebase annotation? Codebase of what service? All of them? Thanks, Michal Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 01:27:09 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy See below. Peter wrote: Using one of the secure discovery providers with authentication and input validation. Download and deserialization permissions are granted dynamically just after authentication, but before download. But now you just moved trust decisions to SafeServiceRegistrar implementation. It is even worse than with "CodeDownloadingSmartProxyWrapper" because SafeServiceRegistrar implementation classes are dynamically downloaded while the CodeDownloadingSmartProxyWrapper class is local. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
So I've given it some thought and the only explanation I can come up with is: 1. To create an instance of the bootstrap proxy you need the codebase annotation. 2. Codebase annotation is needed because you want the bootstrap proxy's class to be defined in the proper codebase ClassLoader 3. Since you do not want to allow any code downloads before placing constraints on the bootstrap proxy - it has to be a dynamic proxy. That way its class can be defined by the codebase loader and yet no code is downloaded So the overall sequence is as follows: 1. Get the codebase annotation and create the codebase loader 2. Create an instance of a dynamic proxy of a class defined by the codebase loader 3. IMPORTANT - before creating the proxy instance DO NOT grant any download permissions - that way we are sure the proxy does not triggers any code download and execution due to it implementing some foreign interfaces 4. Once the proxy is instantiated - grant its ClassLoader download permissions 5. Place the constraints on the proxy 6. Invoke a remote method on the proxy I understand the whole thing is to make sure the bootstrap proxy is defined by the codebase ClassLoader - and the ClassLoader is needed to be able to dynamically grant download permissions. What I DO NOT understand is - why the download permissions are needed at all? Since the bootstrap proxy's code MUST be local code - why not simply have its class defined by the context ClassLoader? Since downloading code is done only after authentication anyway - I don't see the reason to use DownloadPermissions at all. The only thing that comes to mind is that it is to make sure the service is not able to download code from a codebase different than its own. Which is OK but redundant. The reasoning is: Since the code of the service proxy is already trusted (we have established trust before downloading it) - we can simply place the constraints on the service proxy that instructs it to only download code meeting certain criteria. Am I correct in my thinking? Thanks, Michal Michał Kłeczek wrote: Let me dig some deeper. Comments inline. Peter wrote: Yes the dynamic proxy's are 100% local code. Remember dynamic proxy's don't have codebase s. :) Of course they do - look at PreferredClassProvider - the dynamic proxy class is defined by the codebase loader! Being a dynamic proxy does not mean there is no codebase. AtomicMarshalInputStream performs a special input validation on java.lang.reflect.Proxy thus ensuring the InvocationHandler is also trusted. If the InvocationHandler doesn't pass the test the proxy's never created. Why does it only verify dynamic proxies? Doesn't it verify normal objects? The dynamic proxy's put you in direct contact with the service provider using only local code with input validation constrained over secure connections (as configured with constraints in force). I think I've given you enough info now to investigate further. Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's call it a "bootstrap proxy", ok? 1. What interface this bootstrap proxy implements? 2. Why do you think it has to be a dynamic proxy (ie. an instance of a subclass of java.lang.Proxy)? 3. What and when are DownloadPermissions required? How do they add to the overall security? I understand the security of service proxies is enforced by the constraints placed on the bootstrap proxy. So where is the place for DownloadPermissions? 4. Finally - how is the lookup service proxy verified? Does it also provide the bootstrap proxy? If so - what special role does it play in the architecture? The bootstrap proxy does not have to be provided by the lookup service, does it? If it is verified differently - why and how? Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Let me dig some deeper. Comments inline. Peter wrote: Yes the dynamic proxy's are 100% local code. Remember dynamic proxy's don't have codebase s. :) Of course they do - look at PreferredClassProvider - the dynamic proxy class is defined by the codebase loader! Being a dynamic proxy does not mean there is no codebase. AtomicMarshalInputStream performs a special input validation on java.lang.reflect.Proxy thus ensuring the InvocationHandler is also trusted. If the InvocationHandler doesn't pass the test the proxy's never created. Why does it only verify dynamic proxies? Doesn't it verify normal objects? The dynamic proxy's put you in direct contact with the service provider using only local code with input validation constrained over secure connections (as configured with constraints in force). I think I've given you enough info now to investigate further. Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's call it a "bootstrap proxy", ok? 1. What interface this bootstrap proxy implements? 2. Why do you think it has to be a dynamic proxy (ie. an instance of a subclass of java.lang.Proxy)? 3. What and when are DownloadPermissions required? How do they add to the overall security? I understand the security of service proxies is enforced by the constraints placed on the bootstrap proxy. So where is the place for DownloadPermissions? 4. Finally - how is the lookup service proxy verified? Does it also provide the bootstrap proxy? If so - what special role does it play in the architecture? The bootstrap proxy does not have to be provided by the lookup service, does it? If it is verified differently - why and how? Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Yes the dynamic proxy's are 100% local code. Remember dynamic proxy's don't have codebase s. :) AtomicMarshalInputStream performs a special input validation on java.lang.reflect.Proxy thus ensuring the InvocationHandler is also trusted. If the InvocationHandler doesn't pass the test the proxy's never created. The dynamic proxy's put you in direct contact with the service provider using only local code with input validation constrained over secure connections (as configured with constraints in force). I think I've given you enough info now to investigate further. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 14/02/2017 09:34:01 pm To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy There's a new constraint AtomicInputValidation that jeri uses to prevent standard serialization being used. Reggie has been granted DownloadPermission and DeserializationPermission but only for its own codebase. Reggie has also been granted necessary permissions to communicate with its service. No perms have been granted allowing Reggie to do anything else Reggie gives the client a dynamic proxy token (via SafeServiceRegistrar's lookup method). If your class is a client class you can give it the power to open network connections and do as you wish, but it's not a good idea to grant that power to Reggie. But it's also worth remembering it's risky to deserialize into privileged context and you're going to have to do that in this case. SafeServiceRegistrar will still allow you to use your wrapper class, it just adds an additional step of allowing you to authenticate that service before you deserialize your service wrapper, making it a little safer. You can grant permissions to services only to the codebase certificate signers if you want. Using the url in the permission grant just restricts it more, but it's not mandatory. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 07:39:52 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: > Some of your assumptions around forbidding code downloading are incorrect, >which means the other assumptions that rely upon it are also wrong. Which assumptions? I know about DownloadPermissions. And while using RequireDlPermProvider closes the security vulnerability, the problem with DownloadPermissions is that they are not flexible enough. They preclude scenarios like downloading code from BitTorrent or any other "locationless" source or any source for which the client does not have a URL handler installed. Scalability has to be understood in a broader sense. Not only in terms of runtime performance but also in terms of flexibility and openness of the system for extension. Contrary to RequireDlPermProvider the use of SmartProxyWrapper is both secure and flexible. > > In existing River Jini applications, code download can be limited by using >RequireDlPermProvider, it's not possible for a Serializable class to do so via >deserialization api. We must dynamically grant specific download permission >after authentication, but prior to loading the remote code, this limits scope. > If we using AtomicSerial we also need to dynamically grant >DeserializationPermission. I have an impression that you are only moving the problem around. Is use of AtomicSerial mandatory? If not then it does not increase security in any way. And secondly: Based on what information are DownloadPermissions and DeserializationPermissions granted? Only URL of the jar file? See above for why it does not scale. Any other information would require changing the format of codebase annotations (like somehow encoding electronic signature in it). > > Downloading smart proxy's directly from each service, instead of Reggie >significantly reduces the network burden on the lookup service. This is exactly what my SmartProxyWrapper is doing. The real proxy can even be downloaded from a third party service (neither the lookup service nor the service itself). And the bootstrap proxy can be downloaded from whatever place the client wants to treat as the lookup service - it might be as well serialized base64 encoded TXT record in DNS. The beauty of it is that it is orthogonal to the lookup service interface - the lookup service now is just another service on the network - not being special in any way. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
It seems like I am missing the high level view on the architecture of your software. Would it be possible for you to write one? Peter wrote: There's a new constraint AtomicInputValidation that jeri uses to prevent standard serialization being used. Reggie has been granted DownloadPermission and DeserializationPermission but only for its own codebase. I do not understand this. How does the client know it is "Reggie" it is talking to? How Reggie can grant itself permissions in the client environment??? Could you be more precise? Reggie has also been granted necessary permissions to communicate with its service. No perms have been granted allowing Reggie to do anything else Reggie gives the client a dynamic proxy token (via SafeServiceRegistrar's lookup method). What is this "token"? Is it a Java object implementing a well known interface? What interface is it? If your class is a client class you can give it the power to open network connections and do as you wish, but it's not a good idea to grant that power to Reggie. But it's also worth remembering it's risky to deserialize into privileged context and you're going to have to do that in this case. I do not understand this. I the SmartProxyWrapper there are no permissions granted to any downloaded code. The code can be downloaded and classes defined only if the client imposed constraints are met. SafeServiceRegistrar will still allow you to use your wrapper class, it just adds an additional step of allowing you to authenticate that service before you deserialize your service wrapper, making it a little safer. How??? By using the "token" you talk about earlier? But you have to verify the token to be secure, don't you? So you delegate "token" authentication to Reggie, but that requires verification of Reggie proxy. But how do you verify Reggie proxy? Does Reggie also provide the "token"? If so - how is it authenticated? If not - how do you authenticate reggie? You have to break the chain somewhere - and the shorter the trust chain the more secure system is. You can grant permissions to services only to the codebase certificate signers if you want. Using the url in the permission grant just restricts it more, but it's not mandatory. Fair enough. The downside is that the permission check requires downloading the jar file and reading it to verify the code signers. In my proposal the check is done even before actual code downloading. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
There's a new constraint AtomicInputValidation that jeri uses to prevent standard serialization being used. Reggie has been granted DownloadPermission and DeserializationPermission but only for its own codebase. Reggie has also been granted necessary permissions to communicate with its service. No perms have been granted allowing Reggie to do anything else Reggie gives the client a dynamic proxy token (via SafeServiceRegistrar's lookup method). If your class is a client class you can give it the power to open network connections and do as you wish, but it's not a good idea to grant that power to Reggie. But it's also worth remembering it's risky to deserialize into privileged context and you're going to have to do that in this case. SafeServiceRegistrar will still allow you to use your wrapper class, it just adds an additional step of allowing you to authenticate that service before you deserialize your service wrapper, making it a little safer. You can grant permissions to services only to the codebase certificate signers if you want. Using the url in the permission grant just restricts it more, but it's not mandatory. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 07:39:52 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: > Some of your assumptions around forbidding code downloading are incorrect, >which means the other assumptions that rely upon it are also wrong. Which assumptions? I know about DownloadPermissions. And while using RequireDlPermProvider closes the security vulnerability, the problem with DownloadPermissions is that they are not flexible enough. They preclude scenarios like downloading code from BitTorrent or any other "locationless" source or any source for which the client does not have a URL handler installed. Scalability has to be understood in a broader sense. Not only in terms of runtime performance but also in terms of flexibility and openness of the system for extension. Contrary to RequireDlPermProvider the use of SmartProxyWrapper is both secure and flexible. > > In existing River Jini applications, code download can be limited by using >RequireDlPermProvider, it's not possible for a Serializable class to do so via >deserialization api. We must dynamically grant specific download permission >after authentication, but prior to loading the remote code, this limits scope. > If we using AtomicSerial we also need to dynamically grant >DeserializationPermission. I have an impression that you are only moving the problem around. Is use of AtomicSerial mandatory? If not then it does not increase security in any way. And secondly: Based on what information are DownloadPermissions and DeserializationPermissions granted? Only URL of the jar file? See above for why it does not scale. Any other information would require changing the format of codebase annotations (like somehow encoding electronic signature in it). > > Downloading smart proxy's directly from each service, instead of Reggie >significantly reduces the network burden on the lookup service. This is exactly what my SmartProxyWrapper is doing. The real proxy can even be downloaded from a third party service (neither the lookup service nor the service itself). And the bootstrap proxy can be downloaded from whatever place the client wants to treat as the lookup service - it might be as well serialized base64 encoded TXT record in DNS. The beauty of it is that it is orthogonal to the lookup service interface - the lookup service now is just another service on the network - not being special in any way. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Comments inline. Peter wrote: Some of your assumptions around forbidding code downloading are incorrect, which means the other assumptions that rely upon it are also wrong. Which assumptions? I know about DownloadPermissions. And while using RequireDlPermProvider closes the security vulnerability, the problem with DownloadPermissions is that they are not flexible enough. They preclude scenarios like downloading code from BitTorrent or any other "locationless" source or any source for which the client does not have a URL handler installed. Scalability has to be understood in a broader sense. Not only in terms of runtime performance but also in terms of flexibility and openness of the system for extension. Contrary to RequireDlPermProvider the use of SmartProxyWrapper is both secure and flexible. In existing River Jini applications, code download can be limited by using RequireDlPermProvider, it's not possible for a Serializable class to do so via deserialization api. We must dynamically grant specific download permission after authentication, but prior to loading the remote code, this limits scope. If we using AtomicSerial we also need to dynamically grant DeserializationPermission. I have an impression that you are only moving the problem around. Is use of AtomicSerial mandatory? If not then it does not increase security in any way. And secondly: Based on what information are DownloadPermissions and DeserializationPermissions granted? Only URL of the jar file? See above for why it does not scale. Any other information would require changing the format of codebase annotations (like somehow encoding electronic signature in it). Downloading smart proxy's directly from each service, instead of Reggie significantly reduces the network burden on the lookup service. This is exactly what my SmartProxyWrapper is doing. The real proxy can even be downloaded from a third party service (neither the lookup service nor the service itself). And the bootstrap proxy can be downloaded from whatever place the client wants to treat as the lookup service - it might be as well serialized base64 encoded TXT record in DNS. The beauty of it is that it is orthogonal to the lookup service interface - the lookup service now is just another service on the network - not being special in any way. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Some of your assumptions around forbidding code downloading are incorrect, which means the other assumptions that rely upon it are also wrong. In existing River Jini applications, code download can be limited by using RequireDlPermProvider, it's not possible for a Serializable class to do so via deserialization api. We must dynamically grant specific download permission after authentication, but prior to loading the remote code, this limits scope. If we using AtomicSerial we also need to dynamically grant DeserializationPermission. Downloading smart proxy's directly from each service, instead of Reggie significantly reduces the network burden on the lookup service. Thus security scales much better than remaining vulnerable in this case. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 03:43:39 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Wouldn't it be easier if you simply forbid code downloading during unmarshalling as in SmartProxyWrapper I've shown in another message? Then u use the unmarshalled bootstrap object to securely download the real proxy using existing Jeri implementation. Then you do not need any advanced discovery packet inspections at all. What's more - you can apply the same technique to every service proxy - so your security does not depend on how secure the lookup service is. Relying on the lookup service for security does not scale. If you dream of internet scale Jini it has to work in a similar way to current web. I can contact my bank securely even though an attacker might have taken control over my home router (so both routing and naming service is not secure). That's because it is built on end-to-end principle: it has to be secure even if the intermediate services (whatever they are) are insecure. The lookup service is an analogue to DNS. It shouldn't be necessary to make it secure to make the whole system secure. Thanks, Michal Peter wrote: > The certs aren't encoded in the codebase annotation, but sent in packets as >strings and bytes that are used to reconstruct the certificates during >discovery. > > The certs are also included in the jar file. If Download permission hasn't >been granted, the classes can't be defined. DownloadPermission is incorrectly >named, it should be called DefineClassPermission. > > Regards, > > Peter. >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Wouldn't it be easier if you simply forbid code downloading during unmarshalling as in SmartProxyWrapper I've shown in another message? Then u use the unmarshalled bootstrap object to securely download the real proxy using existing Jeri implementation. Then you do not need any advanced discovery packet inspections at all. What's more - you can apply the same technique to every service proxy - so your security does not depend on how secure the lookup service is. Relying on the lookup service for security does not scale. If you dream of internet scale Jini it has to work in a similar way to current web. I can contact my bank securely even though an attacker might have taken control over my home router (so both routing and naming service is not secure). That's because it is built on end-to-end principle: it has to be secure even if the intermediate services (whatever they are) are insecure. The lookup service is an analogue to DNS. It shouldn't be necessary to make it secure to make the whole system secure. Thanks, Michal Peter wrote: The certs aren't encoded in the codebase annotation, but sent in packets as strings and bytes that are used to reconstruct the certificates during discovery. The certs are also included in the jar file. If Download permission hasn't been granted, the classes can't be defined. DownloadPermission is incorrectly named, it should be called DefineClassPermission. Regards, Peter.
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
The certs aren't encoded in the codebase annotation, but sent in packets as strings and bytes that are used to reconstruct the certificates during discovery. The certs are also included in the jar file. If Download permission hasn't been granted, the classes can't be defined. DownloadPermission is incorrectly named, it should be called DefineClassPermission. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:45:43 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: > The codebase is signed and download permission is granted only to the signed >codebase. What is "signed codebase"? How do you encode the signature in the codebase annotation? Codebase of what service? All of them? Thanks, Michal > > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 01:27:09 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > See below. > > Peter wrote: >> Using one of the secure discovery providers with authentication and input >>validation. Download and deserialization permissions are granted dynamically >>just after authentication, but before download. > But now you just moved trust decisions to SafeServiceRegistrar > implementation. > It is even worse than with "CodeDownloadingSmartProxyWrapper" because > SafeServiceRegistrar implementation classes are dynamically downloaded > while the CodeDownloadingSmartProxyWrapper class is local. > > Thanks, > Michal > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Option 2 consumes minimal memory as sharing is maximised. Option 1 may be ok in the enterprise, but no good for embedded. Providing option 2 doesn't exclude option 1 and vice versa. Option 1 ties the proxy to the client and if published outside the client, prevents the client bundle Classloader from being gc'd. Option 2 is loosely coupled ie the OSGi way and published via the service registry, so multiple client consumer bundles may utilise it without causing a ClassLoader reference to any client consumer bundle. For a smart proxy listener, the service could allow a smart proxy using option 1, with the service being provided by option 2. But how to do that in a secure manner without inadevertantly opening an insecure port on the server without proxy preparation isn't yet clear. Can the server trust the client to do the right thing or do we need some defensive programming? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:42:23 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Both ways have their pros and cons. But the decision whether there should be one proxy instance or more is independent of the decision on how to resolve the proxy class. The latter being much more difficult to design properly. So far we have: 1. Use existing PreferredClassLoader infrastructure This has the issue that you are not reusing any OSGi container class loading infrastructure 2. Use OSGi container to install proxy's class bundle - which has other problems discusses so far IMHO there is a third way - but I am still not finished with it :) Thanks, Michal Peter wrote: > My take is a little different, instead services are discovered and >registered, then the client can utilise them. > > The approach you're now suggesting is exactly what Bharath has implented, >this is a great first effort and I believe it can be improved. It's only >worked since River 3.0, I think we can do much more to support OSGi. > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<michal@kleczekorg> > Sent: 14/02/2017 01:05:54 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > There are actually two things that we are discussing at the same time: > > 1. The need to have an "installer" object and how it should be provided > to the client > 2. The algorithm of class loader resolution in OSGi > > These two things are orthogonal to each other and your question is about > point 2. > > My take on this is that in OSGi you have to delay the unmarshalling > decision until a client bundle > asks for a service. > Then you simply create a proxy ClassLoader with the delegation parent > set to the requesting bundle ClassLoader. > It means that many instances of the service proxy might be present in > the client environment - > but that is fine since it is no different than deserializing the service > proxy in two JVMs. > > Thanks, > Michal > > Peter wrote: >> service providers for each api version can still be loaded. >> >> But I don't see a way to force all clients to use a single service api >>version without a compatibility layer. Why not just reload the clients so >>they can use the latest compatible version of a service? >> >> How does your proposed solution solve this problem? >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device >> >> Include original message >> Original message >> From: Michał Kłeczek<mic...@kleczek.org> >> Sent: 14/02/2017 12:39:40 am >> To: dev@river.apache.org >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>invocation strategy >> >> But what if the client has MULTIPLE clients - each with its own exact >> API version? >> >> OSGi handles this case just fine with service trackers. >> >> Do you want to give up on this? >> >> Thanks, >> Michal >> >> Peter wrote: >>> You can however for each service api package version, it's all in the >>>smart proxy bundle manifest. >>> >>> You are bound by the dependency resolution process, the client can only >>>choose from compatible versions. The service has the power to constrain its >>>proxy bundle manifest if it wishes. >>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Input validation only works if failure is atomic. If an instance of
SmartProxyWrapper is created, when a CCE is thrown whilst trying to read in
ServiceProxyDownloader because the attacker has chosen to deserialize a gadget
chain, the attacker potentially wins.
Of course if you implement AtomicExternal and throw a CCE before the object
superclass method is called, you can prevent your object from being created at
all.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek <mic...@kleczek.org>
Sent: 14/02/2017 01:55:56 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
It is as simple as:
interface ServiceProxyDownloader extends Remote, RemoteMethodControl {
Object downloadServiceProxy() throws RemoteException;
}
//this class is local to the client - to make it more secure it
//performs full input validation in readExternal
class SmartProxyWrapper implements Externalizable {
private ServiceProxyDownloader proxyDownloader;
//the constraints are NOT read from the stream
//but configured in the client environment
private MethodConstraints downloadConstraints;
public void readExternal(ObjectInput input) {
//this is safe since no code downloading
//is done by the ObjectInputStream
ServiceProxyDownloader noConstraintsProxyDownloader =
input.readObject();
proxyDownloader =
noConstraintsProxyDownloader.setConstraints(downloadConstraints);
}
Object readResolve() throws ObjectStreamException {
try {
return proxyDownloader.downloadServiceProxy();
}
catch (Exception e) {
//handle aproprietly
}
}
}
Thanks,
Michal
Peter wrote:
> It can impliment AtomicSerial and perform input validation. How do you get
>from discovery to service without SafeServiceRegistrar lookup?
>
> How does your wrapper authenticate the service before codebase and smart
>proxy download?
>
> Sent from my Samsung device.
>
>Include original message
> Original message
> From: Michał Kłeczek<mic...@kleczek.org>
> Sent: 14/02/2017 01:21:46 am
> To: dev@river.apache.org
> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
>strategy
>
> So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then
> it is fine?
> Cool - lets do that.
>
> Thanks,
> Michal
>
> Peter wrote:
>> Any Serializable class that implements AtomicSerial can perform input
>>validation.
>>
>> Using one of the secure discovery providers with authentication and input
>>validation. Download and deserialization permissions are granted dynamically
>>just after authentication, but before download.
>>
>> Regards,
>>
>> Peter.
>>
>> Sent from my Samsung device.
>>
>> Include original message
>> Original message
>> From: Michał Kłeczek<mic...@kleczek.org>
>> Sent: 14/02/2017 01:13:04 am
>> To: dev@river.apache.org
>> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote
>>invocation strategy
>>
>> So your SaveUnmarshallingProxy can do input validation fist as well,
>> can't it?
>>
>> BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a
>> secure way?
>>
>> Thanks,
>> Michal
>>
>> Peter wrote:
>>> I did notice that.
>>>
>>> Are you comnnected to a network and performing deserialization without
>>>input validation? Does the secure endpoint allow anon clients? That is
>>>even if you are using client certificates does the endpoint allow anon?
>>>Does your endpoint allow insecure cyphers?
>>>
>>> Have a look at the changes in JGDMS.
>>>
>>> SafeServiceRegistrar authenticates and performs input validation first.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> Sent from my Samsung device.
>>>
>>>Include original message
>>> Original message
>>> From: Michał Kłeczek<mic...@kleczek.org>
>>> Sent: 14/02/2017 12:42:43 am
>>> To: dev@river.apache.org
>>> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote
>>>invocation strategy
>>>
>>> I fail to understand how you are more vulnerable because of trusted
>>> local class that securely downloads code on behalf of a service.
>>>
>>> And how in terms of
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
JGDMS gives you that ability, to grant permission to download an anonymously signed codebase, the authenticated service vouches for it. It's like this because the free CA "lets encrypt" only signs identity certs, not codebase certs. But we're diverging now. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:45:43 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: > The codebase is signed and download permission is granted only to the signed >codebase. What is "signed codebase"? How do you encode the signature in the codebase annotation? Codebase of what service? All of them? Thanks, Michal > > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 01:27:09 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > See below. > > Peter wrote: >> Using one of the secure discovery providers with authentication and input >>validation. Download and deserialization permissions are granted dynamically >>just after authentication, but before download. > But now you just moved trust decisions to SafeServiceRegistrar > implementation. > It is even worse than with "CodeDownloadingSmartProxyWrapper" because > SafeServiceRegistrar implementation classes are dynamically downloaded > while the CodeDownloadingSmartProxyWrapper class is local. > > Thanks, > Michal > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
It is as simple as:
interface ServiceProxyDownloader extends Remote, RemoteMethodControl {
Object downloadServiceProxy() throws RemoteException;
}
//this class is local to the client - to make it more secure it
//performs full input validation in readExternal
class SmartProxyWrapper implements Externalizable {
private ServiceProxyDownloader proxyDownloader;
//the constraints are NOT read from the stream
//but configured in the client environment
private MethodConstraints downloadConstraints;
public void readExternal(ObjectInput input) {
//this is safe since no code downloading
//is done by the ObjectInputStream
ServiceProxyDownloader noConstraintsProxyDownloader =
input.readObject();
proxyDownloader =
noConstraintsProxyDownloader.setConstraints(downloadConstraints);
}
Object readResolve() throws ObjectStreamException {
try {
return proxyDownloader.downloadServiceProxy();
}
catch (Exception e) {
//handle aproprietly
}
}
}
Thanks,
Michal
Peter wrote:
It can impliment AtomicSerial and perform input validation. How do you get
from discovery to service without SafeServiceRegistrar lookup?
How does your wrapper authenticate the service before codebase and smart proxy
download?
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:21:46 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then
it is fine?
Cool - lets do that.
Thanks,
Michal
Peter wrote:
Any Serializable class that implements AtomicSerial can perform input
validation.
Using one of the secure discovery providers with authentication and input
validation. Download and deserialization permissions are granted dynamically
just after authentication, but before download.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:13:04 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
So your SaveUnmarshallingProxy can do input validation fist as well,
can't it?
BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a
secure way?
Thanks,
Michal
Peter wrote:
I did notice that.
Are you comnnected to a network and performing deserialization without
input validation? Does the secure endpoint allow anon clients? That is even
if you are using client certificates does the endpoint allow anon? Does your
endpoint allow insecure cyphers?
Have a look at the changes in JGDMS.
SafeServiceRegistrar authenticates and performs input validation first.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:42:43 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
I fail to understand how you are more vulnerable because of trusted
local class that securely downloads code on behalf of a service.
And how in terms of security it is different from your
SecureServiceRegistrar.
Thanks,
Michal
Peter wrote:
Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc.
JGDMS is as secure as possible with current cyphers, no anon certs, no
known insecure cyphers (tlsv1.2), input validation during deserialization,
delayed unmarshalling with authentication.
I don't see why a compelling reason to give that up for a local class
with a readResolve method?
Sorry.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:14:41 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote
invocation strategy
Peter wrote:
In jgdms I've enabled support for https unicast lookup in LookupLocator
this establishes a connection to a Registrar only, not any service. This
functionality doesn't exist in River.
How do you propose establishing a connection using one of these
endpoints?
I am not sure I understand the question.
In exactly the same way how today the connection is established by for
example a ProxyTrust instance
Thanks,
Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Peter wrote: The codebase is signed and download permission is granted only to the signed codebase. What is "signed codebase"? How do you encode the signature in the codebase annotation? Codebase of what service? All of them? Thanks, Michal Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 01:27:09 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy See below. Peter wrote: Using one of the secure discovery providers with authentication and input validation. Download and deserialization permissions are granted dynamically just after authentication, but before download. But now you just moved trust decisions to SafeServiceRegistrar implementation. It is even worse than with "CodeDownloadingSmartProxyWrapper" because SafeServiceRegistrar implementation classes are dynamically downloaded while the CodeDownloadingSmartProxyWrapper class is local. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Both ways have their pros and cons. But the decision whether there should be one proxy instance or more is independent of the decision on how to resolve the proxy class. The latter being much more difficult to design properly. So far we have: 1. Use existing PreferredClassLoader infrastructure This has the issue that you are not reusing any OSGi container class loading infrastructure 2. Use OSGi container to install proxy's class bundle - which has other problems discusses so far IMHO there is a third way - but I am still not finished with it :) Thanks, Michal Peter wrote: My take is a little different, instead services are discovered and registered, then the client can utilise them. The approach you're now suggesting is exactly what Bharath has implented, this is a great first effort and I believe it can be improved. It's only worked since River 3.0, I think we can do much more to support OSGi. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<michal@kleczekorg> Sent: 14/02/2017 01:05:54 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy There are actually two things that we are discussing at the same time: 1. The need to have an "installer" object and how it should be provided to the client 2. The algorithm of class loader resolution in OSGi These two things are orthogonal to each other and your question is about point 2. My take on this is that in OSGi you have to delay the unmarshalling decision until a client bundle asks for a service. Then you simply create a proxy ClassLoader with the delegation parent set to the requesting bundle ClassLoader. It means that many instances of the service proxy might be present in the client environment - but that is fine since it is no different than deserializing the service proxy in two JVMs. Thanks, Michal Peter wrote: service providers for each api version can still be loaded. But I don't see a way to force all clients to use a single service api version without a compatibility layer. Why not just reload the clients so they can use the latest compatible version of a service? How does your proposed solution solve this problem? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:39:40 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy But what if the client has MULTIPLE clients - each with its own exact API version? OSGi handles this case just fine with service trackers. Do you want to give up on this? Thanks, Michal Peter wrote: You can however for each service api package version, it's all in the smart proxy bundle manifest. You are bound by the dependency resolution process, the client can only choose from compatible versions. The service has the power to constrain its proxy bundle manifest if it wishes. Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:24:58 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: There a multiple remote services, if one client cant obtain a service because there is also a later version installed then you need a service that doesn't import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
The codebase is signed and download permission is granted only to the signed codebase. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:27:09 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy See below. Peter wrote: > Using one of the secure discovery providers with authentication and input >validation. Download and deserialization permissions are granted dynamically >just after authentication, but before download. But now you just moved trust decisions to SafeServiceRegistrar implementation. It is even worse than with "CodeDownloadingSmartProxyWrapper" because SafeServiceRegistrar implementation classes are dynamically downloaded while the CodeDownloadingSmartProxyWrapper class is local. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
It can impliment AtomicSerial and perform input validation. How do you get from discovery to service without SafeServiceRegistrar lookup? How does your wrapper authenticate the service before codebase and smart proxy download? Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:21:46 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then it is fine? Cool - lets do that. Thanks, Michal Peter wrote: > Any Serializable class that implements AtomicSerial can perform input >validation. > > Using one of the secure discovery providers with authentication and input >validation. Download and deserialization permissions are granted dynamically >just after authentication, but before download. > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 01:13:04 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > So your SaveUnmarshallingProxy can do input validation fist as well, > can't it? > > BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a > secure way? > > Thanks, > Michal > > Peter wrote: >> I did notice that. >> >> Are you comnnected to a network and performing deserialization without >>input validation? Does the secure endpoint allow anon clients? That is >>even if you are using client certificates does the endpoint allow anon? Does >>your endpoint allow insecure cyphers? >> >> Have a look at the changes in JGDMS. >> >> SafeServiceRegistrar authenticates and performs input validation first. >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device. >> >> Include original message >> Original message >> From: Michał Kłeczek<mic...@kleczek.org> >> Sent: 14/02/2017 12:42:43 am >> To: dev@river.apache.org >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>invocation strategy >> >> I fail to understand how you are more vulnerable because of trusted >> local class that securely downloads code on behalf of a service. >> >> And how in terms of security it is different from your >> SecureServiceRegistrar. >> >> Thanks, >> Michal >> >> Peter wrote: >>> Then you are vulnerable to deserialization gadget attacks, insecure >>>cyphers anon certs etc. >>> >>> JGDMS is as secure as possible with current cyphers, no anon certs, no >>>known insecure cyphers (tlsv1.2), input validation during deserialization, >>>delayed unmarshalling with authentication. >>> >>> I don't see why a compelling reason to give that up for a local class >>>with a readResolve method? >>> >>> Sorry. >>> >>> Regards, >>> >>> Peter. >>> Sent from my Samsung device. >>> >>>Include original message >>> Original message >>> From: Michał Kłeczek<mic...@kleczek.org> >>> Sent: 14/02/2017 12:14:41 am >>> To: dev@river.apache.org >>> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>>invocation strategy >>> >>> >>> Peter wrote: >>>> In jgdms I've enabled support for https unicast lookup in >>>>LookupLocator this establishes a connection to a Registrar only, not any >>>>service. This functionality doesn't exist in River. >>>> >>>> How do you propose establishing a connection using one of these >>>>endpoints? >>> I am not sure I understand the question. >>> In exactly the same way how today the connection is established by for >>> example a ProxyTrust instance >>> >>> Thanks, >>> Michal >>> >>> >> > > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
My take is a little different, instead services are discovered and registered, then the client can utilise them. The approach you're now suggesting is exactly what Bharath has implented, this is a great first effort and I believe it can be improved. It's only worked since River 3.0, I think we can do much more to support OSGi. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <michal@kleczekorg> Sent: 14/02/2017 01:05:54 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy There are actually two things that we are discussing at the same time: 1. The need to have an "installer" object and how it should be provided to the client 2. The algorithm of class loader resolution in OSGi These two things are orthogonal to each other and your question is about point 2. My take on this is that in OSGi you have to delay the unmarshalling decision until a client bundle asks for a service. Then you simply create a proxy ClassLoader with the delegation parent set to the requesting bundle ClassLoader. It means that many instances of the service proxy might be present in the client environment - but that is fine since it is no different than deserializing the service proxy in two JVMs. Thanks, Michal Peter wrote: > service providers for each api version can still be loaded. > > But I don't see a way to force all clients to use a single service api >version without a compatibility layer. Why not just reload the clients so >they can use the latest compatible version of a service? > > How does your proposed solution solve this problem? > > Regards, > > Peter. > > Sent from my Samsung device > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 12:39:40 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > But what if the client has MULTIPLE clients - each with its own exact > API version? > > OSGi handles this case just fine with service trackers. > > Do you want to give up on this? > > Thanks, > Michal > > Peter wrote: >> You can however for each service api package version, it's all in the >>smart proxy bundle manifest. >> >> You are bound by the dependency resolution process, the client can only >>choose from compatible versions. The service has the power to constrain its >>proxy bundle manifest if it wishes. >> >> Regards, >> >> Peter. >> >> Sent from my Samsung device >> >> Include original message >> Original message >> From: Michał Kłeczek<mic...@kleczek.org> >> Sent: 14/02/2017 12:24:58 am >> To: dev@river.apache.org >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>invocation strategy >> >> >> Peter wrote: >>> There a multiple remote services, if one client cant obtain a service >>>because there is also a later version installed then you need a service that >>>doesn't import the later version. You can still supply another service to >>>cater. >> This does not scale because you would have to have one service per each >> service interface version any client might require. >> >> No... You have to be able to make this class resolution decision on the >> client. >> And if the client VM allows to have many class loading context at the >> same time (as is the case with OSGI) then >> the infrastructure has to take care of this resolution. >> >> But you cannot force the service provider to provide separate instance >> for each case. >> >> Thanks, >> Michal >> >> > > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
See below. Peter wrote: Using one of the secure discovery providers with authentication and input validation. Download and deserialization permissions are granted dynamically just after authentication, but before download. But now you just moved trust decisions to SafeServiceRegistrar implementation. It is even worse than with "CodeDownloadingSmartProxyWrapper" because SafeServiceRegistrar implementation classes are dynamically downloaded while the CodeDownloadingSmartProxyWrapper class is local. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then
it is fine?
Cool - lets do that.
Thanks,
Michal
Peter wrote:
Any Serializable class that implements AtomicSerial can perform input
validation.
Using one of the secure discovery providers with authentication and input
validation. Download and deserialization permissions are granted dynamically
just after authentication, but before download.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:13:04 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
So your SaveUnmarshallingProxy can do input validation fist as well,
can't it?
BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a
secure way?
Thanks,
Michal
Peter wrote:
I did notice that.
Are you comnnected to a network and performing deserialization without input
validation? Does the secure endpoint allow anon clients? That is even if you
are using client certificates does the endpoint allow anon? Does your endpoint
allow insecure cyphers?
Have a look at the changes in JGDMS.
SafeServiceRegistrar authenticates and performs input validation first.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:42:43 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
I fail to understand how you are more vulnerable because of trusted
local class that securely downloads code on behalf of a service.
And how in terms of security it is different from your
SecureServiceRegistrar.
Thanks,
Michal
Peter wrote:
Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc.
JGDMS is as secure as possible with current cyphers, no anon certs, no
known insecure cyphers (tlsv1.2), input validation during deserialization,
delayed unmarshalling with authentication.
I don't see why a compelling reason to give that up for a local class with
a readResolve method?
Sorry.
Regards,
Peter.
Sent from my Samsung device.
Include original message
Original message
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:14:41 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Peter wrote:
In jgdms I've enabled support for https unicast lookup in LookupLocator
this establishes a connection to a Registrar only, not any service. This
functionality doesn't exist in River.
How do you propose establishing a connection using one of these endpoints?
I am not sure I understand the question.
In exactly the same way how today the connection is established by for
example a ProxyTrust instance
Thanks,
Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Any Serializable class that implements AtomicSerial can perform input validation. Using one of the secure discovery providers with authentication and input validation. Download and deserialization permissions are granted dynamically just after authentication, but before download. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 01:13:04 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So your SaveUnmarshallingProxy can do input validation fist as well, can't it? BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a secure way? Thanks, Michal Peter wrote: > I did notice that. > > Are you comnnected to a network and performing deserialization without input >validation? Does the secure endpoint allow anon clients? That is even if >you are using client certificates does the endpoint allow anon? Does your >endpoint allow insecure cyphers? > > Have a look at the changes in JGDMS. > > SafeServiceRegistrar authenticates and performs input validation first. > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 12:42:43 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > I fail to understand how you are more vulnerable because of trusted > local class that securely downloads code on behalf of a service. > > And how in terms of security it is different from your > SecureServiceRegistrar. > > Thanks, > Michal > > Peter wrote: >> Then you are vulnerable to deserialization gadget attacks, insecure >>cyphers anon certs etc. >> >> JGDMS is as secure as possible with current cyphers, no anon certs, no >>known insecure cyphers (tlsv1.2), input validation during deserialization, >>delayed unmarshalling with authentication. >> >> I don't see why a compelling reason to give that up for a local class with >>a readResolve method? >> >> Sorry. >> >> Regards, >> >> Peter. >> Sent from my Samsung device. >> >> Include original message >> Original message >> From: Michał Kłeczek<mic...@kleczek.org> >> Sent: 14/02/2017 12:14:41 am >> To: dev@river.apache.org >> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote >>invocation strategy >> >> >> Peter wrote: >>> In jgdms I've enabled support for https unicast lookup in LookupLocator >>>this establishes a connection to a Registrar only, not any service. This >>>functionality doesn't exist in River. >>> >>> How do you propose establishing a connection using one of these >>>endpoints? >> I am not sure I understand the question. >> In exactly the same way how today the connection is established by for >> example a ProxyTrust instance >> >> Thanks, >> Michal >> >> > > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
There are actually two things that we are discussing at the same time: 1. The need to have an "installer" object and how it should be provided to the client 2. The algorithm of class loader resolution in OSGi These two things are orthogonal to each other and your question is about point 2. My take on this is that in OSGi you have to delay the unmarshalling decision until a client bundle asks for a service. Then you simply create a proxy ClassLoader with the delegation parent set to the requesting bundle ClassLoader. It means that many instances of the service proxy might be present in the client environment - but that is fine since it is no different than deserializing the service proxy in two JVMs. Thanks, Michal Peter wrote: service providers for each api version can still be loaded. But I don't see a way to force all clients to use a single service api version without a compatibility layer. Why not just reload the clients so they can use the latest compatible version of a service? How does your proposed solution solve this problem? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:39:40 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy But what if the client has MULTIPLE clients - each with its own exact API version? OSGi handles this case just fine with service trackers. Do you want to give up on this? Thanks, Michal Peter wrote: You can however for each service api package version, it's all in the smart proxy bundle manifest. You are bound by the dependency resolution process, the client can only choose from compatible versions. The service has the power to constrain its proxy bundle manifest if it wishes. Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:24:58 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: There a multiple remote services, if one client cant obtain a service because there is also a later version installed then you need a service that doesn't import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
service providers for each api version can still be loaded. But I don't see a way to force all clients to use a single service api version without a compatibility layer. Why not just reload the clients so they can use the latest compatible version of a service? How does your proposed solution solve this problem? Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 12:39:40 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy But what if the client has MULTIPLE clients - each with its own exact API version? OSGi handles this case just fine with service trackers. Do you want to give up on this? Thanks, Michal Peter wrote: > You can however for each service api package version, it's all in the smart >proxy bundle manifest. > > You are bound by the dependency resolution process, the client can only >choose from compatible versions. The service has the power to constrain its >proxy bundle manifest if it wishes. > > Regards, > > Peter. > > Sent from my Samsung device > >Include original message > Original message > From: Michał Kłeczek<mic...@kleczek.org> > Sent: 14/02/2017 12:24:58 am > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > > Peter wrote: >> There a multiple remote services, if one client cant obtain a service >>because there is also a later version installed then you need a service that >>doesn't import the later version. You can still supply another service to >>cater. > This does not scale because you would have to have one service per each > service interface version any client might require. > > No... You have to be able to make this class resolution decision on the > client. > And if the client VM allows to have many class loading context at the > same time (as is the case with OSGI) then > the infrastructure has to take care of this resolution. > > But you cannot force the service provider to provide separate instance > for each case. > > Thanks, > Michal > >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
I fail to understand how you are more vulnerable because of trusted local class that securely downloads code on behalf of a service. And how in terms of security it is different from your SecureServiceRegistrar. Thanks, Michal Peter wrote: Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc. JGDMS is as secure as possible with current cyphers, no anon certs, no known insecure cyphers (tlsv1.2), input validation during deserialization, delayed unmarshalling with authentication. I don't see why a compelling reason to give that up for a local class with a readResolve method? Sorry. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:14:41 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: In jgdms I've enabled support for https unicast lookup in LookupLocator this establishes a connection to a Registrar only, not any service. This functionality doesn't exist in River. How do you propose establishing a connection using one of these endpoints? I am not sure I understand the question. In exactly the same way how today the connection is established by for example a ProxyTrust instance. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
But what if the client has MULTIPLE clients - each with its own exact API version? OSGi handles this case just fine with service trackers. Do you want to give up on this? Thanks, Michal Peter wrote: You can however for each service api package version, it's all in the smart proxy bundle manifest. You are bound by the dependency resolution process, the client can only choose from compatible versions. The service has the power to constrain its proxy bundle manifest if it wishes. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 14/02/2017 12:24:58 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: There a multiple remote services, if one client cant obtain a service because there is also a later version installed then you need a service that doesn't import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
N.B. Thanks for discussing, I think all these questions are helping people understand the proposal better and clearing up confusion. If it stands up to scruitiny, then it also provides more confidence in the solution. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 14/02/2017 12:33:54 am To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy You can however for each service api package version, it's all in the smart proxy bundle manifest. You are bound by the dependency resolution process, the client can only choose from compatible versions. The service has the power to constrain its proxy bundle manifest if it wishes. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 12:24:58 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: > There a multiple remote services, if one client cant obtain a service because >there is also a later version installed then you need a service that doesn't >import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc. JGDMS is as secure as possible with current cyphers, no anon certs, no known insecure cyphers (tlsv1.2), input validation during deserialization, delayed unmarshalling with authentication. I don't see why a compelling reason to give that up for a local class with a readResolve method? Sorry. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 12:14:41 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: > In jgdms I've enabled support for https unicast lookup in LookupLocator this >establishes a connection to a Registrar only, not any service. This >functionality doesn't exist in River. > > How do you propose establishing a connection using one of these endpoints? I am not sure I understand the question. In exactly the same way how today the connection is established by for example a ProxyTrust instance. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
You can however for each service api package version, it's all in the smart proxy bundle manifest. You are bound by the dependency resolution process, the client can only choose from compatible versions. The service has the power to constrain its proxy bundle manifest if it wishes. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 14/02/2017 12:24:58 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Peter wrote: > There a multiple remote services, if one client cant obtain a service because >there is also a later version installed then you need a service that doesn't >import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Peter wrote: There a multiple remote services, if one client cant obtain a service because there is also a later version installed then you need a service that doesn't import the later version. You can still supply another service to cater. This does not scale because you would have to have one service per each service interface version any client might require. No... You have to be able to make this class resolution decision on the client. And if the client VM allows to have many class loading context at the same time (as is the case with OSGI) then the infrastructure has to take care of this resolution. But you cannot force the service provider to provide separate instance for each case. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
There a multiple remote services, if one client cant obtain a service because there is also a later version installed then you need a service that doesn't import the later version. You can still supply another service to cater. Resolution is performed after remote discovery, SafeServiceRegistrar lookup and authentication but prior to deserialization of the smart proxy. It can take as long as it needs, its a background task. The ClassLoader of the provisioned resolved and started smart proxy bundle is used to deserialize the proxy. Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 13/02/2017 11:59:34 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: N.B Can't see any chicken egg problem. If service doesn't resolve to same service api as client, then it isn't compatible. The client isn't interested in incompatible services, only those that are compatible This is just an artifact of the dependency resolution process. But when do you perform resolution? Lets say you have two client bundles: Client1 is resolved to depend on package a.b.c ver 1.1.3 from bundle a.b.c 1.1.3 Client2 is resolved to depend on package a.b.c ver 1.2.0 from bundle a.b.c 1.2.0 Your service supports both clients. How do you resolve it so that it can be linked with both clients? The answer is - you can not. You must create two different instances of the service proxy each one instantiated from a class loaded in a different context. But you cannot decide upon this context in advance!!! You have to know which client you are serving. No bundle indentifiers are necesary in the stream, the smart proxy ClassLoader decides visibility and delegation, not RMIClassLoader. But what ClassLoader is used to load the smart proxy??? You're attempting to do too many things with one class / object, there's a risk that this very powerful class could be leveraged by an attacker, best to break up the functionality. Why do you think that it is handled by one class/object??? Also ServiceDiscoveryManager handles a lot of scenarios that occur with remote services, your class wont, there are thousands of lines of code. Have a look at SDM in JGDMS. Sorry for sarcasm, but... This is one class/object doing so many things... Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Peter wrote: In jgdms I've enabled support for https unicast lookup in LookupLocator this establishes a connection to a Registrar only, not any service. This functionality doesn't exist in River. How do you propose establishing a connection using one of these endpoints? I am not sure I understand the question. In exactly the same way how today the connection is established by for example a ProxyTrust instance. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
In jgdms I've enabled support for https unicast lookup in LookupLocator this establishes a connection to a Registrar only, not any service. This functionality doesn't exist in River. How do you propose establishing a connection using one of these endpoints? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 14/02/2017 12:00:02 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy KerberosEnpoint? HttpsEndpoint? Thanks, Michal Peter wrote: > How do you establish the secure jeri connection? > > Regards, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> > Sent: 13/02/2017 11:34:45 pm > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > 1. The connection can be done using normal (secure) Jeri. > We do not have to verify the installer object since its classes were loaded >locally and (by definition) are trusted. > > 2 The attacker cannot instantiate any non-local class. That is the whole >point. > Since the "installer" classes must be local, then we can trust the installer >to > honor any invocation constraints we place on it. So any code downloads > are secure - in the sense that the client can require >authentication/integrity/confidentiality etc. > > Note that (if necessary) we can apply the same logic recursively - we can >provide an "installer of an installer" > and still be sure any code download is going to honor the security >constraints we require. > > Thanks, > Michal > > Peter wrote: > So this object that you have with a locally installed class is tasked with >authenticating the remote service, provisioning and resolving a bundle, >deserializing the smart proxy and registering it with the OSGi service >registrar in a readResolve or readObject method? > > How do you propose the connection from the client to the service established >in order to enable this to occur? > > How do you prevent an attacker from choosing a different class to >deserialize? > > Regards, > > Peter. > > Sent from my Samsung device > >Include original message > Original message > From: Michał Kłeczek<michal@kleczekorg> > Sent: 13/02/2017 10:07:28 pm > To: dev@river.apache.org > Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation >strategy > > Comments inline. > > Peter wrote: > Mic, > > I'm attempting to get my head around your proposal: > > In the case of JERI, the InvocationHandler is part of the smart > proxy's serialized state. A number of smart proxy classes will need > to be unmarshalled before the UnmarshallingInvocationHandler is > deserialized. > > The smart proxy contains a reference to a dynamic proxy (which sun > called the bootstrap proxy) and the dynamic proxy contains a reference > to your UnmarshallingInvocationHandler.This means the smart proxy > must be unmarshalled first. > > How do you get access to UnmarshallingInvocationHandler without > unmarshalling the smart proxy first? > > No no - I am saying about wrapping the smart proxy inside another > object. It can be either a dynamic proxy, or simply an object that > implements "readResolve" returning the unmarshalled smart proxy. > > More comments inline below. > > On 13/02/2017 6:11 PM, Michał Kłeczek wrote: > We are talking about the same thing. > > We are turning circles, Peter - all of this has been already discussed. > > 1. Yes - you need to resolve bundles in advance (in OSGi it is not > possible to do otherwise anyway) > Agree. > 2. You cannot decide upon the bundle chosen by the container to load > the proxy class (the container does the resolution) > Disagree, nothing in the client depends on the proxy bundle, there's > no reason to provision a different version. > 3 The runtime graph of object places additional constraints on the > bundle resolution process (to what is specified in bundles' manifests). > Since you do not have any way to pass these additional constraints to > the container - the case is lost. > Disagree. The proxy bundle contains a manifest with requirements. > The stream has no knowledge of versioning, nor does it need to, there > are no addit
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Nope - not at all. I am only trying to convince you that there is no reason to involve ServiceRegistrar or SDM for code downloading. HOW the class resolution is done - is another story. I actually tend to think in a similar way to what Niclas said: Do not use OSGi to load proxy class - create a separate ClassLoader - but make sure it delegates to the client bundle ClassLoader for non-preferred classes. Thanks, Michal Peter wrote: I changed it to highlite Nic's point that it's not feasible to resolve and provision osgi bundle transitive dependencies during deserialization because the time taken to do that can be excessive due to NP Complete nature of resolution. It is incompatible with stream codebase annotations. I think Mic's currently arguing for a solution that relies on resolution and provisioning to occur during deserialization and I'm arguing against it. I'm arguing for a background task that preceeds deserialization of the proxy. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Patricia Shanahan<p...@acm.org> Sent: 13/02/2017 11:27:27 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Sorry, I'm trying to find out the meaning of the current subject line. I'm not sure when it changed to "OSGi MP Complete". On 2/12/2017 10:50 PM, Michał Kłeczek wrote: Sorry, NP Completness of what? I have been the first to mention NP hardness of constraint satisfaction problem but I am not sure if this is what you are asking about. Thanks, Michal Patricia Shanahan wrote: Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
I changed it to highlite Nic's point that it's not feasible to resolve and provision osgi bundle transitive dependencies during deserialization because the time taken to do that can be excessive due to NP Complete nature of resolution. It is incompatible with stream codebase annotations. I think Mic's currently arguing for a solution that relies on resolution and provisioning to occur during deserialization and I'm arguing against it. I'm arguing for a background task that preceeds deserialization of the proxy. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Patricia Shanahan <p...@acm.org> Sent: 13/02/2017 11:27:27 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Sorry, I'm trying to find out the meaning of the current subject line. I'm not sure when it changed to "OSGi MP Complete". On 2/12/2017 10:50 PM, Michał Kłeczek wrote: > Sorry, NP Completness of what? > I have been the first to mention NP hardness of constraint satisfaction > problem > but I am not sure if this is what you are asking about. > > Thanks, > Michal > > Patricia Shanahan wrote: >> Are you literally claiming NP Completeness, or just using that as an >> analogy for really, really difficult? >> >
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
KerberosEnpoint? HttpsEndpoint? Thanks, Michal Peter wrote: How do you establish the secure jeri connection? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 13/02/2017 11:34:45 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy 1. The connection can be done using normal (secure) Jeri. We do not have to verify the installer object since its classes were loaded locally and (by definition) are trusted. 2 The attacker cannot instantiate any non-local class. That is the whole point. Since the "installer" classes must be local, then we can trust the installer to honor any invocation constraints we place on it. So any code downloads are secure - in the sense that the client can require authentication/integrity/confidentiality etc. Note that (if necessary) we can apply the same logic recursively - we can provide an "installer of an installer" and still be sure any code download is going to honor the security constraints we require. Thanks, Michal Peter wrote: So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek<michal@kleczekorg> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: Mic, I'm attempting to get my head around your proposal: In the case of JERI, the InvocationHandler is part of the smart proxy's serialized state. A number of smart proxy classes will need to be unmarshalled before the UnmarshallingInvocationHandler is deserialized. The smart proxy contains a reference to a dynamic proxy (which sun called the bootstrap proxy) and the dynamic proxy contains a reference to your UnmarshallingInvocationHandler.This means the smart proxy must be unmarshalled first. How do you get access to UnmarshallingInvocationHandler without unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. More comments inline below. On 13/02/2017 6:11 PM, Michał Kłeczek wrote: We are talking about the same thing. We are turning circles, Peter - all of this has been already discussed. 1. Yes - you need to resolve bundles in advance (in OSGi it is not possible to do otherwise anyway) Agree. 2. You cannot decide upon the bundle chosen by the container to load the proxy class (the container does the resolution) Disagree, nothing in the client depends on the proxy bundle, there's no reason to provision a different version. 3. The runtime graph of object places additional constraints on the bundle resolution process (to what is specified in bundles' manifests). Since you do not have any way to pass these additional constraints to the container - the case is lost. Disagree. The proxy bundle contains a manifest with requirements. The stream has no knowledge of versioning, nor does it need to, there are no additional constraints. If the service proxy dependencies cannot be resolved, or it doesn't unmarshall, then it will not be registered with the OSGi registry in the client, client code will not discover it and the client will have no knowledge of it's existance except for some logging. This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. Additionally - to explain what I've said before about wrong level of abstraction: Your general idea is very similar to mine: have a special object (let's call it installer) that will install software prior to proxy unmarshalling. 1. For some reason unclear to me you want to constrain the way how this "installer object" is passed only via the route of ServiceRegistrar (as attributes) Disagree, I'm not proposing the service have any control over installation at the client, other than
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Comments inline. Peter wrote: N.B Can't see any chicken egg problem. If service doesn't resolve to same service api as client, then it isn't compatible. The client isn't interested in incompatible services, only those that are compatible This is just an artifact of the dependency resolution process. But when do you perform resolution? Lets say you have two client bundles: Client1 is resolved to depend on package a.b.c ver 1.1.3 from bundle a.b.c 1.1.3 Client2 is resolved to depend on package a.b.c ver 1.2.0 from bundle a.b.c 1.2.0 Your service supports both clients. How do you resolve it so that it can be linked with both clients? The answer is - you can not. You must create two different instances of the service proxy each one instantiated from a class loaded in a different context. But you cannot decide upon this context in advance!!! You have to know which client you are serving. No bundle indentifiers are necesary in the stream, the smart proxy ClassLoader decides visibility and delegation, not RMIClassLoader. But what ClassLoader is used to load the smart proxy??? You're attempting to do too many things with one class / object, there's a risk that this very powerful class could be leveraged by an attacker, best to break up the functionality. Why do you think that it is handled by one class/object??? Also ServiceDiscoveryManager handles a lot of scenarios that occur with remote services, your class wont, there are thousands of lines of code. Have a look at SDM in JGDMS. Sorry for sarcasm, but... This is one class/object doing so many things... Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
N.B. No released versions of River / Jini's Jeri are secure now. Ignoring that for a moment, how you establish the connection is important also. Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 13/02/2017 11:34:45 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy 1. The connection can be done using normal (secure) Jeri. We do not have to verify the installer object since its classes were loaded locally and (by definition) are trusted. 2. The attacker cannot instantiate any non-local class. That is the whole point. Since the "installer" classes must be local, then we can trust the installer to honor any invocation constraints we place on it. So any code downloads are secure - in the sense that the client can require authentication/integrity/confidentiality etc. Note that (if necessary) we can apply the same logic recursively - we can provide an "installer of an installer" and still be sure any code download is going to honor the security constraints we require. Thanks, Michal Peter wrote: So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: Mic, I'm attempting to get my head around your proposal: In the case of JERI, the InvocationHandler is part of the smart proxy's serialized state. A number of smart proxy classes will need to be unmarshalled before the UnmarshallingInvocationHandler is deserialized. The smart proxy contains a reference to a dynamic proxy (which sun called the bootstrap proxy) and the dynamic proxy contains a reference to your UnmarshallingInvocationHandler.This means the smart proxy must be unmarshalled first. How do you get access to UnmarshallingInvocationHandler without unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. More comments inline below. On 13/02/2017 6:11 PM, Michał Kłeczek wrote: We are talking about the same thing. We are turning circles, Peter - all of this has been already discussed. 1. Yes - you need to resolve bundles in advance (in OSGi it is not possible to do otherwise anyway) Agree. 2. You cannot decide upon the bundle chosen by the container to load the proxy class (the container does the resolution) Disagree, nothing in the client depends on the proxy bundle, there's no reason to provision a different version 3. The runtime graph of object places additional constraints on the bundle resolution process (to what is specified in bundles' manifests). Since you do not have any way to pass these additional constraints to the container - the case is lost. Disagree. The proxy bundle contains a manifest with requirements The stream has no knowledge of versioning, nor does it need to, there are no additional constraints. If the service proxy dependencies cannot be resolved, or it doesn't unmarshall, then it will not be registered with the OSGi registry in the client, client code will not discover it and the client will have no knowledge of it's existance except for some logging. This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. Additionally - to explain what I've said before about wrong level of abstraction: Your general idea is very similar to mine: have a special object (let's call it installer) that will install software prior to proxy unmarshalling. 1. For some reason unclear to me you want to constrain the way how this "installer object" is passed only via the route of ServiceRegistrar (as attributes) Disagree, I'm not proposing the service have any control over in
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
How do you establish the secure jeri connection? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 13/02/2017 11:34:45 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy 1. The connection can be done using normal (secure) Jeri. We do not have to verify the installer object since its classes were loaded locally and (by definition) are trusted. 2 The attacker cannot instantiate any non-local class. That is the whole point. Since the "installer" classes must be local, then we can trust the installer to honor any invocation constraints we place on it. So any code downloads are secure - in the sense that the client can require authentication/integrity/confidentiality etc. Note that (if necessary) we can apply the same logic recursively - we can provide an "installer of an installer" and still be sure any code download is going to honor the security constraints we require. Thanks, Michal Peter wrote: So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek <michal@kleczekorg> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: Mic, I'm attempting to get my head around your proposal: In the case of JERI, the InvocationHandler is part of the smart proxy's serialized state. A number of smart proxy classes will need to be unmarshalled before the UnmarshallingInvocationHandler is deserialized. The smart proxy contains a reference to a dynamic proxy (which sun called the bootstrap proxy) and the dynamic proxy contains a reference to your UnmarshallingInvocationHandler.This means the smart proxy must be unmarshalled first. How do you get access to UnmarshallingInvocationHandler without unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. More comments inline below. On 13/02/2017 6:11 PM, Michał Kłeczek wrote: We are talking about the same thing. We are turning circles, Peter - all of this has been already discussed. 1. Yes - you need to resolve bundles in advance (in OSGi it is not possible to do otherwise anyway) Agree. 2. You cannot decide upon the bundle chosen by the container to load the proxy class (the container does the resolution) Disagree, nothing in the client depends on the proxy bundle, there's no reason to provision a different version. 3. The runtime graph of object places additional constraints on the bundle resolution process (to what is specified in bundles' manifests). Since you do not have any way to pass these additional constraints to the container - the case is lost. Disagree. The proxy bundle contains a manifest with requirements. The stream has no knowledge of versioning, nor does it need to, there are no additional constraints. If the service proxy dependencies cannot be resolved, or it doesn't unmarshall, then it will not be registered with the OSGi registry in the client, client code will not discover it and the client will have no knowledge of it's existance except for some logging. This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. Additionally - to explain what I've said before about wrong level of abstraction: Your general idea is very similar to mine: have a special object (let's call it installer) that will install software prior to proxy unmarshalling. 1. For some reason unclear to me you want to constrain the way how this "installer object" is passed only via the route of ServiceRegistrar (as attributes) Disagree, I'm not proposing the service have any control over installation at the client, other than the manifest in the proxy bundle, nor am I proposing using ser
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
N.B Can't see any chicken egg problem. If service doesn't resolve to same service api as client, then it isn't compatible. The client isn't interested in incompatible services, only those that are compatible This is just an artifact of the dependency resolution process. No bundle indentifiers are necesary in the stream, the smart proxy ClassLoader decides visibility and delegation, not RMIClassLoader. You're attempting to do too many things with one class / object, there's a risk that this very powerful class could be leveraged by an attacker, best to break up the functionality. Also ServiceDiscoveryManager handles a lot of scenarios that occur with remote services, your class wont, there are thousands of lines of code. Have a look at SDM in JGDMS. Regards, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 13/02/2017 11:15:27 pm To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: > Mic, > > I'm attempting to get my head around your proposal: > > In the case of JERI, the InvocationHandler is part of the smart > proxy's serialized state. A number of smart proxy classes will need > to be unmarshalled before the UnmarshallingInvocationHandler is > deserialized. > > The smart proxy contains a reference to a dynamic proxy (which sun > called the bootstrap proxy) and the dynamic proxy contains a reference > to your UnmarshallingInvocationHandler.This means the smart proxy > must be unmarshalled first. > > How do you get access to UnmarshallingInvocationHandler without > unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. > > More comments inline below. > > On 13/02/2017 6:11 PM, Michał Kłeczek wrote: >> We are talking about the same thing. >> >> We are turning circles, Peter - all of this has been already discussed. >> >> 1. Yes - you need to resolve bundles in advance (in OSGi it is not >> possible to do otherwise anyway) > Agree. >> 2. You cannot decide upon the bundle chosen by the container to load >> the proxy class (the container does the resolution) > Disagree, nothing in the client depends on the proxy bundle, there's > no reason to provision a different version. >> 3. The runtime graph of object places additional constraints on the >> bundle resolution process (to what is specified in bundles' manifests). >> Since you do not have any way to pass these additional constraints to >> the container - the case is lost. > Disagree. The proxy bundle contains a manifest with requirements. > The stream has no knowledge of versioning, nor does it need to, there > are no additional constraints. If the service proxy dependencies > cannot be resolved, or it doesn't unmarshall, then it will not be > registered with the OSGi registry in the client, client code will not > discover it and the client will have no knowledge of it's existance > except for some logging. > This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. >> >> Additionally - to explain what I've said before about wrong level of >> abstraction: >> >> Your general idea is very similar to mine: have a special object >> (let's call it installer) that will install software prior to proxy >> unmarshalling. >> >> 1. For some reason unclear to me you want to constrain the way how >> this "ins
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
1. The connection can be done using normal (secure) Jeri. We do not have to verify the installer object since its classes were loaded locally and (by definition) are trusted. 2. The attacker cannot instantiate any non-local class. That is the whole point. Since the "installer" classes must be local, then we can trust the installer to honor any invocation constraints we place on it. So any code downloads are secure - in the sense that the client can require authentication/integrity/confidentiality etc. Note that (if necessary) we can apply the same logic recursively - we can provide an "installer of an installer" and still be sure any code download is going to honor the security constraints we require. Thanks, Michal Peter wrote: So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek<mic...@kleczek.org> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: Mic, I'm attempting to get my head around your proposal: In the case of JERI, the InvocationHandler is part of the smart proxy's serialized state. A number of smart proxy classes will need to be unmarshalled before the UnmarshallingInvocationHandler is deserialized. The smart proxy contains a reference to a dynamic proxy (which sun called the bootstrap proxy) and the dynamic proxy contains a reference to your UnmarshallingInvocationHandler.This means the smart proxy must be unmarshalled first. How do you get access to UnmarshallingInvocationHandler without unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. More comments inline below. On 13/02/2017 6:11 PM, Michał Kłeczek wrote: We are talking about the same thing. We are turning circles, Peter - all of this has been already discussed. 1. Yes - you need to resolve bundles in advance (in OSGi it is not possible to do otherwise anyway) Agree. 2. You cannot decide upon the bundle chosen by the container to load the proxy class (the container does the resolution) Disagree, nothing in the client depends on the proxy bundle, there's no reason to provision a different version. 3. The runtime graph of object places additional constraints on the bundle resolution process (to what is specified in bundles' manifests). Since you do not have any way to pass these additional constraints to the container - the case is lost. Disagree. The proxy bundle contains a manifest with requirements. The stream has no knowledge of versioning, nor does it need to, there are no additional constraints. If the service proxy dependencies cannot be resolved, or it doesn't unmarshall, then it will not be registered with the OSGi registry in the client, client code will not discover it and the client will have no knowledge of it's existance except for some logging. This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. Additionally - to explain what I've said before about wrong level of abstraction: Your general idea is very similar to mine: have a special object (let's call it installer) that will install software prior to proxy unmarshalling. 1. For some reason unclear to me you want to constrain the way how this "installer object" is passed only via the route of ServiceRegistrar (as attributes) Disagree, I'm not proposing the service have any control over installation at the client, other than the manifest in the proxy bundle, nor am I proposing using service attributes, or the use of any existing ServiceRegistar methods (see SafeServiceRegistrar link posted earlier). If you think about it from the higher architectural view - there is no difference. It does not really matter what steps are made - important thing is that: a) you have a special object used to download code - this object is supposed t
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Sorry, I'm trying to find out the meaning of the current subject line. I'm not sure when it changed to "OSGi MP Complete". On 2/12/2017 10:50 PM, Michał Kłeczek wrote: Sorry, NP Completness of what? I have been the first to mention NP hardness of constraint satisfaction problem but I am not sure if this is what you are asking about. Thanks, Michal Patricia Shanahan wrote: Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
So this object that you have with a locally installed class is tasked with authenticating the remote service, provisioning and resolving a bundle, deserializing the smart proxy and registering it with the OSGi service registrar in a readResolve or readObject method? How do you propose the connection from the client to the service established in order to enable this to occur? How do you prevent an attacker from choosing a different class to deserialize? Regards, Peter. Sent from my Samsung device Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 13/02/2017 10:07:28 pm To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Comments inline. Peter wrote: > Mic, > > I'm attempting to get my head around your proposal: > > In the case of JERI, the InvocationHandler is part of the smart > proxy's serialized state. A number of smart proxy classes will need > to be unmarshalled before the UnmarshallingInvocationHandler is > deserialized. > > The smart proxy contains a reference to a dynamic proxy (which sun > called the bootstrap proxy) and the dynamic proxy contains a reference > to your UnmarshallingInvocationHandler.This means the smart proxy > must be unmarshalled first. > > How do you get access to UnmarshallingInvocationHandler without > unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. > > More comments inline below. > > On 13/02/2017 6:11 PM, Michał Kłeczek wrote: >> We are talking about the same thing. >> >> We are turning circles, Peter - all of this has been already discussed. >> >> 1. Yes - you need to resolve bundles in advance (in OSGi it is not >> possible to do otherwise anyway) > Agree. >> 2. You cannot decide upon the bundle chosen by the container to load >> the proxy class (the container does the resolution) > Disagree, nothing in the client depends on the proxy bundle, there's > no reason to provision a different version. >> 3. The runtime graph of object places additional constraints on the >> bundle resolution process (to what is specified in bundles' manifests). >> Since you do not have any way to pass these additional constraints to >> the container - the case is lost. > Disagree. The proxy bundle contains a manifest with requirements. > The stream has no knowledge of versioning, nor does it need to, there > are no additional constraints. If the service proxy dependencies > cannot be resolved, or it doesn't unmarshall, then it will not be > registered with the OSGi registry in the client, client code will not > discover it and the client will have no knowledge of it's existance > except for some logging. > This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. >> >> Additionally - to explain what I've said before about wrong level of >> abstraction: >> >> Your general idea is very similar to mine: have a special object >> (let's call it installer) that will install software prior to proxy >> unmarshalling. >> >> 1. For some reason unclear to me you want to constrain the way how >> this "installer object" is passed only via the route of >> ServiceRegistrar (as attributes) > > Disagree, I'm not proposing the service have any control over > installation at the client, other than the manifest in the proxy > bundle, nor am I proposing using service attributes, or the use of any > existing ServiceRegistar methods (see SafeServiceRegistrar link posted > earlier). If you think about it from the higher architectural view - there is no difference. It does not really matter what steps are made - important thing is that: a) you have a special object used to download code - this object is supposed to be of a class installed locally in advance b) the above object is used to create a ClassLoader that you will use it load the actual deserialized object's class It does not matter where the first object is taken from - be it "SafeServiceRegistrar", the stream itself, a JavaSpace or the Moon. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Comments inline. Peter wrote: Mic, I'm attempting to get my head around your proposal: In the case of JERI, the InvocationHandler is part of the smart proxy's serialized state. A number of smart proxy classes will need to be unmarshalled before the UnmarshallingInvocationHandler is deserialized. The smart proxy contains a reference to a dynamic proxy (which sun called the bootstrap proxy) and the dynamic proxy contains a reference to your UnmarshallingInvocationHandler.This means the smart proxy must be unmarshalled first. How do you get access to UnmarshallingInvocationHandler without unmarshalling the smart proxy first? No no - I am saying about wrapping the smart proxy inside another object. It can be either a dynamic proxy, or simply an object that implements "readResolve" returning the unmarshalled smart proxy. More comments inline below. On 13/02/2017 6:11 PM, Michał Kłeczek wrote: We are talking about the same thing. We are turning circles, Peter - all of this has been already discussed. 1. Yes - you need to resolve bundles in advance (in OSGi it is not possible to do otherwise anyway) Agree. 2. You cannot decide upon the bundle chosen by the container to load the proxy class (the container does the resolution) Disagree, nothing in the client depends on the proxy bundle, there's no reason to provision a different version. 3. The runtime graph of object places additional constraints on the bundle resolution process (to what is specified in bundles' manifests). Since you do not have any way to pass these additional constraints to the container - the case is lost. Disagree. The proxy bundle contains a manifest with requirements. The stream has no knowledge of versioning, nor does it need to, there are no additional constraints. If the service proxy dependencies cannot be resolved, or it doesn't unmarshall, then it will not be registered with the OSGi registry in the client, client code will not discover it and the client will have no knowledge of it's existance except for some logging. This is totally backwards. That way no client is able to find any service because there is a chicken and egg problem - we do not know the proxy interfaces until the proxy's bundle is resolved. Understand that when you place a bundle identifier in the stream - it is equivalent to specifying a Require-Bundle constraint - nothing more nothing less. Additionally - to explain what I've said before about wrong level of abstraction: Your general idea is very similar to mine: have a special object (let's call it installer) that will install software prior to proxy unmarshalling. 1. For some reason unclear to me you want to constrain the way how this "installer object" is passed only via the route of ServiceRegistrar (as attributes) Disagree, I'm not proposing the service have any control over installation at the client, other than the manifest in the proxy bundle, nor am I proposing using service attributes, or the use of any existing ServiceRegistar methods (see SafeServiceRegistrar link posted earlier). If you think about it from the higher architectural view - there is no difference. It does not really matter what steps are made - important thing is that: a) you have a special object used to download code - this object is supposed to be of a class installed locally in advance b) the above object is used to create a ClassLoader that you will use it load the actual deserialized object's class It does not matter where the first object is taken from - be it "SafeServiceRegistrar", the stream itself, a JavaSpace or the Moon. Thanks, Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
To be fair, my position changed somewhat after Nic's email and some
further research, it may of course develop further with understanding
and experimentation.
Cheers,
Peter.
On 13/02/2017 7:52 PM, Peter wrote:
Mic,
I'm attempting to get my head around your proposal:
In the case of JERI, the InvocationHandler is part of the smart
proxy's serialized state. A number of smart proxy classes will need
to be unmarshalled before the UnmarshallingInvocationHandler is
deserialized.
The smart proxy contains a reference to a dynamic proxy (which sun
called the bootstrap proxy) and the dynamic proxy contains a reference
to your UnmarshallingInvocationHandler.This means the smart proxy
must be unmarshalled first.
How do you get access to UnmarshallingInvocationHandler without
unmarshalling the smart proxy first?
More comments inline below.
On 13/02/2017 6:11 PM, Michał Kłeczek wrote:
We are talking about the same thing.
We are turning circles, Peter - all of this has been already discussed.
1. Yes - you need to resolve bundles in advance (in OSGi it is not
possible to do otherwise anyway)
Agree.
2. You cannot decide upon the bundle chosen by the container to load
the proxy class (the container does the resolution)
Disagree, nothing in the client depends on the proxy bundle, there's
no reason to provision a different version.
3. The runtime graph of object places additional constraints on the
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to
the container - the case is lost.
Disagree. The proxy bundle contains a manifest with requirements.
The stream has no knowledge of versioning, nor does it need to, there
are no additional constraints. If the service proxy dependencies
cannot be resolved, or it doesn't unmarshall, then it will not be
registered with the OSGi registry in the client, client code will not
discover it and the client will have no knowledge of it's existance
except for some logging.
Additionally - to explain what I've said before about wrong level of
abstraction:
Your general idea is very similar to mine: have a special object
(let's call it installer) that will install software prior to proxy
unmarshalling.
1. For some reason unclear to me you want to constrain the way how
this "installer object" is passed only via the route of
ServiceRegistrar (as attributes)
Disagree, I'm not proposing the service have any control over
installation at the client, other than the manifest in the proxy
bundle, nor am I proposing using service attributes, or the use of any
existing ServiceRegistar methods (see SafeServiceRegistrar link posted
earlier).
But why not:
public interface RemoteEventProducer {
void registerListener(SmartProxyInstaller installer, byte[]
remoteEventListenerBytes);
}
I guess you could have a service that installs the proxy bundle that
way, but how will you know which ClassLoader the deserialize into at
the client?
I cannot see a difference at all and this is why I say that mixing
ServiceRegistrar and ServiceDiscoveryManager
into it is really mixing levels of abstraction (and does not add
anything to the solution).
I'm proposing to use ServiceDiscoveryManager to discover
SafeServiceRegistrar's, then look up matching services, receive
dynamic proxy's from each of the matching services (using input
validating deserialization, a security measure) that allow the
ProxyPreparer to authenticate each service, get attributes, perform
local logical attribute comparisons, get the codebase URL string,
certificate signers, grant permissions, then finally provision the
codebase and finally retrieve the smart proxy, directly from the
dynamic proxy.
Now we could put your UnmarshallingInvocationHandler into the dynamic
proxy, but then we might be installing a bundle we decide we don't
want after logical comparisons of attributes. The InvocationHandler's
intent is to marshall object arguments, when methods are invocated on
the dynamic proxy. The InvocationDispatcher unmarshalls the arguments
at the remote end and invokes the methods on the Remote object.
So I'm proposing to write some code that performs service discovery
using the above and registers the successfully matched services with
the local OSGi service registry. So the client needn't be concerned
with the time taken to remotely discover and provision a service, all
that happens in the background until the services can be made
available in the OSGi service registry for the client to utilise.
2. If you allow to pass "installer" to unmarshall proxies - then the
next question is - why do you require doing it explicitly in the
application code???
That's not what I'm proposing, see above.
Cheers,
Peter.
THIS is really mixing levels of abstraction. A programmer expects a
natural programming model (and will not buy into Jini i
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Mic,
I'm attempting to get my head around your proposal:
In the case of JERI, the InvocationHandler is part of the smart proxy's
serialized state. A number of smart proxy classes will need to be
unmarshalled before the UnmarshallingInvocationHandler is deserialized.
The smart proxy contains a reference to a dynamic proxy (which sun
called the bootstrap proxy) and the dynamic proxy contains a reference
to your UnmarshallingInvocationHandler.This means the smart proxy
must be unmarshalled first.
How do you get access to UnmarshallingInvocationHandler without
unmarshalling the smart proxy first?
More comments inline below.
On 13/02/2017 6:11 PM, Michał Kłeczek wrote:
We are talking about the same thing.
We are turning circles, Peter - all of this has been already discussed.
1. Yes - you need to resolve bundles in advance (in OSGi it is not
possible to do otherwise anyway)
Agree.
2. You cannot decide upon the bundle chosen by the container to load
the proxy class (the container does the resolution)
Disagree, nothing in the client depends on the proxy bundle, there's no
reason to provision a different version.
3. The runtime graph of object places additional constraints on the
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to
the container - the case is lost.
Disagree. The proxy bundle contains a manifest with requirements. The
stream has no knowledge of versioning, nor does it need to, there are no
additional constraints. If the service proxy dependencies cannot be
resolved, or it doesn't unmarshall, then it will not be registered with
the OSGi registry in the client, client code will not discover it and
the client will have no knowledge of it's existance except for some logging.
Additionally - to explain what I've said before about wrong level of
abstraction:
Your general idea is very similar to mine: have a special object
(let's call it installer) that will install software prior to proxy
unmarshalling.
1. For some reason unclear to me you want to constrain the way how
this "installer object" is passed only via the route of
ServiceRegistrar (as attributes)
Disagree, I'm not proposing the service have any control over
installation at the client, other than the manifest in the proxy bundle,
nor am I proposing using service attributes, or the use of any existing
ServiceRegistar methods (see SafeServiceRegistrar link posted earlier).
But why not:
public interface RemoteEventProducer {
void registerListener(SmartProxyInstaller installer, byte[]
remoteEventListenerBytes);
}
I guess you could have a service that installs the proxy bundle that
way, but how will you know which ClassLoader the deserialize into at the
client?
I cannot see a difference at all and this is why I say that mixing
ServiceRegistrar and ServiceDiscoveryManager
into it is really mixing levels of abstraction (and does not add
anything to the solution).
I'm proposing to use ServiceDiscoveryManager to discover
SafeServiceRegistrar's, then look up matching services, receive dynamic
proxy's from each of the matching services (using input validating
deserialization, a security measure) that allow the ProxyPreparer to
authenticate each service, get attributes, perform local logical
attribute comparisons, get the codebase URL string, certificate signers,
grant permissions, then finally provision the codebase and finally
retrieve the smart proxy, directly from the dynamic proxy.
Now we could put your UnmarshallingInvocationHandler into the dynamic
proxy, but then we might be installing a bundle we decide we don't want
after logical comparisons of attributes. The InvocationHandler's intent
is to marshall object arguments, when methods are invocated on the
dynamic proxy. The InvocationDispatcher unmarshalls the arguments at
the remote end and invokes the methods on the Remote object.
So I'm proposing to write some code that performs service discovery
using the above and registers the successfully matched services with the
local OSGi service registry. So the client needn't be concerned with
the time taken to remotely discover and provision a service, all that
happens in the background until the services can be made available in
the OSGi service registry for the client to utilise.
2. If you allow to pass "installer" to unmarshall proxies - then the
next question is - why do you require doing it explicitly in the
application code???
That's not what I'm proposing, see above.
Cheers,
Peter.
THIS is really mixing levels of abstraction. A programmer expects a
natural programming model (and will not buy into Jini if it does not
offer such):
registerListener(RemoteEventListener listener);
without any additional complexities that should be solved by the
infrastructure code.
3. But the above is easy to solve:
class UnmarshallingInvocati
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
We are talking about the same thing.
We are turning circles, Peter - all of this has been already discussed.
1. Yes - you need to resolve bundles in advance (in OSGi it is not
possible to do otherwise anyway)
2. You cannot decide upon the bundle chosen by the container to load the
proxy class (the container does the resolution)
3. The runtime graph of object places additional constraints on the
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to
the container - the case is lost.
Additionally - to explain what I've said before about wrong level of
abstraction:
Your general idea is very similar to mine: have a special object (let's
call it installer) that will install software prior to proxy unmarshalling.
1. For some reason unclear to me you want to constrain the way how this
"installer object" is passed only via the route of ServiceRegistrar (as
attributes)
But why not:
public interface RemoteEventProducer {
void registerListener(SmartProxyInstaller installer, byte[]
remoteEventListenerBytes);
}
I cannot see a difference at all and this is why I say that mixing
ServiceRegistrar and ServiceDiscoveryManager
into it is really mixing levels of abstraction (and does not add
anything to the solution).
2. If you allow to pass "installer" to unmarshall proxies - then the
next question is - why do you require doing it explicitly in the
application code???
THIS is really mixing levels of abstraction. A programmer expects a
natural programming model (and will not buy into Jini if it does not
offer such):
registerListener(RemoteEventListener listener);
without any additional complexities that should be solved by the
infrastructure code.
3. But the above is easy to solve:
class UnmarshallingInvocationHandler implements InvocationHandler {
private Object unmarshalledProxy;
private SmartProxyInstaller installer;
writeObject(...) { write out installer and proxy bytes }
readObject(...) {read installer then bytes and then unmarshall the proxy}
}
4. But then the next question is why not put it in the object stream
implementation itself???
Thanks,
Michal
Peter wrote:
Also see the OSGi Enterprise specification, v6, Chapter 136, page 691,
there's some discussion about the NP-complete nature of dependency
resolution there as well.
https://www.osgi.org/developer/downloads/release-6/release-6-download/
On 13/02/2017 5:19 PM, Peter wrote:
OSGi Dependency resolution is.
http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
Which means if we want to support an OSGi environment properly, we
may need some time to resolve the dependencies for a smart proxy,
before deserializing the proxy, rather than downloading the proxy
dynamically during unmarshalling, it's better to delay unmarshalling
until the dependencies are resolved, so the client isn't impacted by
delays.
Cheers,
Peter.
On 13/02/2017 4:50 PM, Michał Kłeczek wrote:
Sorry, NP Completness of what?
I have been the first to mention NP hardness of constraint
satisfaction problem
but I am not sure if this is what you are asking about.
Thanks,
Michal
Patricia Shanahan wrote:
Are you literally claiming NP Completeness, or just using that as
an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Also see the OSGi Enterprise specification, v6, Chapter 136, page 691, there's some discussion about the NP-complete nature of dependency resolution there as well. https://www.osgi.org/developer/downloads/release-6/release-6-download/ On 13/02/2017 5:19 PM, Peter wrote: OSGi Dependency resolution is. http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html Which means if we want to support an OSGi environment properly, we may need some time to resolve the dependencies for a smart proxy, before deserializing the proxy, rather than downloading the proxy dynamically during unmarshalling, it's better to delay unmarshalling until the dependencies are resolved, so the client isn't impacted by delays. Cheers, Peter. On 13/02/2017 4:50 PM, Michał Kłeczek wrote: Sorry, NP Completness of what? I have been the first to mention NP hardness of constraint satisfaction problem but I am not sure if this is what you are asking about. Thanks, Michal Patricia Shanahan wrote: Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
OSGi Dependency resolution is. http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html Which means if we want to support an OSGi environment properly, we may need some time to resolve the dependencies for a smart proxy, before deserializing the proxy, rather than downloading the proxy dynamically during unmarshalling, it's better to delay unmarshalling until the dependencies are resolved, so the client isn't impacted by delays. Cheers, Peter. On 13/02/2017 4:50 PM, Michał Kłeczek wrote: Sorry, NP Completness of what? I have been the first to mention NP hardness of constraint satisfaction problem but I am not sure if this is what you are asking about. Thanks, Michal Patricia Shanahan wrote: Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Sorry, NP Completness of what? I have been the first to mention NP hardness of constraint satisfaction problem but I am not sure if this is what you are asking about. Thanks, Michal Patricia Shanahan wrote: Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult?
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Are you literally claiming NP Completeness, or just using that as an analogy for really, really difficult? On 2/11/2017 1:23 PM, Michał Kłeczek wrote: I am sorry but I think that to solve various issues we need to make sure fundamentals are right: 1. There is NO such a thing as "reflective non-smart" proxy - EVERY proxy is "smart" (even if it is "reflective") - there is an InvocationHandler down there, isn't there? 2. Solving this on service discovery level is trying to do it on the WRONG level of abstraction. Services DEPEND on class loading - not the other way around. 3. What you propose is a partial "solution". Not being able to register "smart" event listeners means no custom endpoints for example (UDPEndpoint anyone?) 4. Trying to squeeze partial solutions into the framework is IMHO a BIG no no. This is simply creating more code, more maintenance burden and more headache for users trying to workaround "edge, unsupported cases". Please - lets try to come up with the RIGHT solution that is going to REALLY fix class loading issues. Thanks, Michal Peter wrote: In a word, ServiceDiscoveryManager ServiceDiscoveryManager is the solution. ServiceDiscoveryManager performs discovery and looks up services from registrars based on filters. ServiceDiscoveryManager then performs local filtering. This allows time for proxy bundles to be installed, resolve, started and confirmed type compatible, prior to them being made available (via OSGi service registry if you so desire) for client use. The new interfaces that are part of JGDMS that I'd like to see their way into River, found here: https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java ServiceCodebaseAccessor is also used as part of secure discovery, but the codebase string and certs are transferred as primitives over the network. In this case codebase annotations don't need to be included in the stream, the JERI endpoints don't need them at all. How so? We can use ServiceItemFilter and ProxyPreparer to install, resolve and start out proxy codebase, before downloading the proxy. The interfaces listed above allow an array bootstrap proxy's (java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar. Firstly the bootstrap proxy's JERI endpoint will be loaded in the ServiceDiscoveryManager's ClassLoader, so after we've retrieved the codebase annotation and signers, created a bundle for the proxy, resolved it's dependencies (via OSGi resolution and repository services), we need to remarshall the bootstrap proxy into a MarshalledInstance, then unmarshall it using the ClassLoader of the recently started proxy bundle. Then when we cast the bootstrap proxy to ServiceProxyAccessor and retrieve the smart proxy, it will be loaded into the same ClassLoader that the bootstrap proxy uses, our newly provisioned and loaded bundle (the correct ClassLoader) without need to serialize any annotations, then the smart proxy can have constraints applied etc and be registered as an OSGi service with the OSGi registrar, where client code can interact with the remote proxy. Now if public Serializable classes that are imported by the proxy's bundle (service api) or private classes in the proxy's bundle can be deserialized and the JERI endpoint has a reference to the ClassLoader of the proxy. This should be good enough so we don't require the "bundle stack" proposed earlier, which also saves the need to explain it and simplifies the solution (the intent of the bundle stack was to allow deserialization of private classes within other bundles whose packages have been imported by the proxy). The client won't be able to pass a smart proxy to the service (like a Listener), but it can still pass a non smart proxy and it will still function. So clients can still export their own remote service (albiet without a codebase, excluding smart proxy's), but it'll be good enough for a listener etc. Cheers, Peter. On 8/02/2017 1:09 PM, Niclas Hedhman wrote: Maybe there are some misunderstanding somewhere... see below; On Wed, Feb 8, 2017 at 3:35 AM, Peter<j...@zeus.net.au> wrote: I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Ok, makes it a lot easier for me to follow. Not expecting the cl
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Interestingly ServiceCodebaseAccessor.getClassAnnotation() could be modified to accept a string parameter. This would allow different smart proxy codebases to be provisioned for different environments. By provisioning and deserializing into the ClassLoader for the provisioned codebase, a service can support multiple client environments. Although this doesn't provide the full gammut of seamlessly passing smart proxys and private serializable objects that extend the service api, as remote method arguments, which is an admirable and ideal goal, my pragmatic side tells me it's an acceptable compromise to constrain functionality within the known limitations of the jvm and ClassLoaders. Jim Waldo lamented (sorry don't have a )reference the change in Java 1.2 that tied class identity to ClassLoaders, see following paper: http://www.bracha.org/classloaders.ps The proxy ClassLoader of the local environment controlls class visibility, annotations are not required in the serialized object stream, if it is deserialised into the proxy ClassLoader. As Nic and Mic have mentioned OSGi's NP Complete resolution means that codebase provisioning is something that needs to performed by a background thread, it's not something that's practical to perform during unmarshalling, it needs to occur prior to deserialization. Gregg has demonstrated that delayed unmarshalling has significant performance benefits in desktop applications. Delayed unmarshalling also has security benefits, allowing authentication to be performed prior to codebase downloads and deserialization. Terms: Dynamic proxy - proxy that's generated dynamically, it doesn’t use or require a codebase. Smart proxy - proxy that performs local processing / computation, it requires a codebase. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 12/02/2017 08:42:23 am To: dev@river.apache.org Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy Thanks Michal, See inline below. On 12/02/2017 7:23 AM, Michał Kłeczek wrote: > I am sorry but I think that to solve various issues we need to make > sure fundamentals are right: > > 1. There is NO such a thing as "reflective non-smart" proxy - EVERY > proxy is "smart" (even if it is "reflective") - there is an > InvocationHandler down there, isn't there? Easy fixed, we change the terminology to a proxy without a codebase download, "dynamic proxy" was a term that Sun used for it previously. Sun used the terminology "Smart Proxy" to describe a proxy that had a codebase. Dynamic proxy's still have an InvocationHandler, it's part of one of the existing jeri implementations and already installed at both ends. > 2. Solving this on service discovery level is trying to do it on the > WRONG level of abstraction. Services DEPEND on class loading - not the > other way around. Can you explain a little more please? > 3. What you propose is a partial "solution". Not being able to > register "smart" event listeners means no custom endpoints for example > (UDPEndpoint anyone?) Yes, you can still have custom endpoints, but the endpoints need to be installed at both ends, prior. Initially, smart proxy's cannot be used for remote objects, only exported services. > 4. Trying to squeeze partial solutions into the framework is IMHO a > BIG no no. > This is simply creating more code, more maintenance burden and more > headache for users trying to workaround "edge, unsupported cases". Not really, it provides working solutions for the majority of cases, with minimal changes, low hanging fruit so to speak and allows adoption, the other cases can be worked on later, when there are more users and more developers to assist. The proxy implementation (provider) is abstracted behind the service api completely, typical of OSGi practise, ServiceDiscoveryManager allows time for service proxy's to be wired up, prior to deserialization using delayed unmarshalling. The consumer must use the service api to interact with the service and must honour the service api documentation, which may state which classes may be overridden, that providers must interact using certain classes in the local jvm and not transfer them remotely, a good way to do that is ensure they're not serializable. A listener is intended to be implemented, however the consumer would only be able to export a remote object that doesn't have a codebase and still be able expect the provider to deserialize it. Unless of course we create a new service that allows a client to request a bundle be installed, resolved and started in a remote end, allowing consumers to request remote provisioning. Still I'd like to allow the so
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Thanks Michal, See inline below. On 12/02/2017 7:23 AM, Michał Kłeczek wrote: I am sorry but I think that to solve various issues we need to make sure fundamentals are right: 1. There is NO such a thing as "reflective non-smart" proxy - EVERY proxy is "smart" (even if it is "reflective") - there is an InvocationHandler down there, isn't there? Easy fixed, we change the terminology to a proxy without a codebase download, "dynamic proxy" was a term that Sun used for it previously. Sun used the terminology "Smart Proxy" to describe a proxy that had a codebase. Dynamic proxy's still have an InvocationHandler, it's part of one of the existing jeri implementations and already installed at both ends. 2. Solving this on service discovery level is trying to do it on the WRONG level of abstraction. Services DEPEND on class loading - not the other way around. Can you explain a little more please? 3. What you propose is a partial "solution". Not being able to register "smart" event listeners means no custom endpoints for example (UDPEndpoint anyone?) Yes, you can still have custom endpoints, but the endpoints need to be installed at both ends, prior. Initially, smart proxy's cannot be used for remote objects, only exported services. 4. Trying to squeeze partial solutions into the framework is IMHO a BIG no no. This is simply creating more code, more maintenance burden and more headache for users trying to workaround "edge, unsupported cases". Not really, it provides working solutions for the majority of cases, with minimal changes, low hanging fruit so to speak and allows adoption, the other cases can be worked on later, when there are more users and more developers to assist. The proxy implementation (provider) is abstracted behind the service api completely, typical of OSGi practise, ServiceDiscoveryManager allows time for service proxy's to be wired up, prior to deserialization using delayed unmarshalling. The consumer must use the service api to interact with the service and must honour the service api documentation, which may state which classes may be overridden, that providers must interact using certain classes in the local jvm and not transfer them remotely, a good way to do that is ensure they're not serializable. A listener is intended to be implemented, however the consumer would only be able to export a remote object that doesn't have a codebase and still be able expect the provider to deserialize it. Unless of course we create a new service that allows a client to request a bundle be installed, resolved and started in a remote end, allowing consumers to request remote provisioning. Still I'd like to allow the solution to grow, we shouldn't expect to be able to do it all at once. We can still agree that at some point addressing the remaining functionality would be nice to have. Please - lets try to come up with the RIGHT solution that is going to REALLY fix class loading issues. Can you explain a little more please? I'm only trying to provide support for OSGi users at this stage, so might not be considering the problem you want solved. At least for the OSGi user, there shouldn't be any class loader issues, each endpoint has been designated a ClassLoader and it only needs the visibility it's been given in order to provide the service, this also has security benefits as an attacker is no longer given access to every class in the jvm. Regards, Peter. Thanks, Michal Peter wrote: In a word, ServiceDiscoveryManager ServiceDiscoveryManager is the solution. ServiceDiscoveryManager performs discovery and looks up services from registrars based on filters. ServiceDiscoveryManager then performs local filtering. This allows time for proxy bundles to be installed, resolve, started and confirmed type compatible, prior to them being made available (via OSGi service registry if you so desire) for client use. The new interfaces that are part of JGDMS that I'd like to see their way into River, found here: https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java ServiceCodebaseAccessor is also used as part of secure discovery, but the codebase string and certs are transferred as primitives
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
I am sorry but I think that to solve various issues we need to make sure fundamentals are right: 1. There is NO such a thing as "reflective non-smart" proxy - EVERY proxy is "smart" (even if it is "reflective") - there is an InvocationHandler down there, isn't there? 2. Solving this on service discovery level is trying to do it on the WRONG level of abstraction. Services DEPEND on class loading - not the other way around. 3. What you propose is a partial "solution". Not being able to register "smart" event listeners means no custom endpoints for example (UDPEndpoint anyone?) 4. Trying to squeeze partial solutions into the framework is IMHO a BIG no no. This is simply creating more code, more maintenance burden and more headache for users trying to workaround "edge, unsupported cases". Please - lets try to come up with the RIGHT solution that is going to REALLY fix class loading issues. Thanks, Michal Peter wrote: In a word, ServiceDiscoveryManager ServiceDiscoveryManager is the solution. ServiceDiscoveryManager performs discovery and looks up services from registrars based on filters. ServiceDiscoveryManager then performs local filtering. This allows time for proxy bundles to be installed, resolve, started and confirmed type compatible, prior to them being made available (via OSGi service registry if you so desire) for client use. The new interfaces that are part of JGDMS that I'd like to see their way into River, found here: https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java ServiceCodebaseAccessor is also used as part of secure discovery, but the codebase string and certs are transferred as primitives over the network. In this case codebase annotations don't need to be included in the stream, the JERI endpoints don't need them at all. How so? We can use ServiceItemFilter and ProxyPreparer to install, resolve and start out proxy codebase, before downloading the proxy. The interfaces listed above allow an array bootstrap proxy's (java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar. Firstly the bootstrap proxy's JERI endpoint will be loaded in the ServiceDiscoveryManager's ClassLoader, so after we've retrieved the codebase annotation and signers, created a bundle for the proxy, resolved it's dependencies (via OSGi resolution and repository services), we need to remarshall the bootstrap proxy into a MarshalledInstance, then unmarshall it using the ClassLoader of the recently started proxy bundle. Then when we cast the bootstrap proxy to ServiceProxyAccessor and retrieve the smart proxy, it will be loaded into the same ClassLoader that the bootstrap proxy uses, our newly provisioned and loaded bundle (the correct ClassLoader) without need to serialize any annotations, then the smart proxy can have constraints applied etc and be registered as an OSGi service with the OSGi registrar, where client code can interact with the remote proxy. Now if public Serializable classes that are imported by the proxy's bundle (service api) or private classes in the proxy's bundle can be deserialized and the JERI endpoint has a reference to the ClassLoader of the proxy. This should be good enough so we don't require the "bundle stack" proposed earlier, which also saves the need to explain it and simplifies the solution (the intent of the bundle stack was to allow deserialization of private classes within other bundles whose packages have been imported by the proxy). The client won't be able to pass a smart proxy to the service (like a Listener), but it can still pass a non smart proxy and it will still function. So clients can still export their own remote service (albiet without a codebase, excluding smart proxy's), but it'll be good enough for a listener etc. Cheers, Peter. On 8/02/2017 1:09 PM, Niclas Hedhman wrote: Maybe there are some misunderstanding somewhere... see below; On Wed, Feb 8, 2017 at 3:35 AM, Peter<j...@zeus.net.au> wrote: I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Ok, makes it a lot easier for me to follow. Not expecting the client calling bundle to resolve everything, hence the stack, so we have the full visibility of
Re: OSGi - deserialization remote invocation strategy
Thanks Nic, It looks like OSGi is generating a lot of interest. I'm spending a lot of time respoding to discussions, which is important, but I'd also like to get the the existing test suite working with the new JGDMS OGSi Bundles, so will have to excuse myself as I'm still in the learning understanding and experimenting stages, as you're aware the way things might appear before experimenting and learing can look a lot different afterwards, thanks for your suggestions and reccommedations, will follow up later. Cheers, Peter. On 8/02/2017 1:09 PM, Niclas Hedhman wrote: Maybe there are some misunderstanding somewhere... see below; On Wed, Feb 8, 2017 at 3:35 AM, Peter<j...@zeus.net.au> wrote: I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Ok, makes it a lot easier for me to follow. Not expecting the client calling bundle to resolve everything, hence the stack, so we have the full visibility of the bundle of the class that was last resolved, so we can resolve its fields from it's bundle. Eg it might import packages the client does not. What is "client calling bundle"? I assume that it is the bundle that contains the service lookup... First of all, is the OSGi integration intending to be at "OSGi service level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini service registrations and register those in the local OSGi registry and all clients would end up being rather ignorant about the remote lookup. It is also possible to "be involved" in the service lookup (and registration) through the Service Hook API (I might got the name of it wrong), so that only when services are requested or service listeners are registered, does the Jini support kick in. And this can therefore work both ways... So, that said; I agree that the "client calling bundle" does not resolve anything. And I would go further and say; IF the client calling bundle looks up Jini services directly, all bets are off. This introduces some workable constraints of what could happen. The "exact version" thing (only applies to the proxy bundle as we expect the framework to load its deps) can be relaxed to compatible versions to increase class sharing if you think it helps. The proxy bundle doesn't export anything at the client, only the server, it just seems to make sense to keep the latest proxy communicating in case that last bug fix release addresses a security issue. All proxy classes are implementation only classes. Yes, correct. The 'temporary' proxy bundle should not export any packages. Because the proxy bundle manifest declares version import ranges, I'm expecting the framework to favour already loaded bundles to satisfy package import deps. U... The OSGi framework doesn't "auto load" bundles. It is an explicit step. There are many "bundle loaders" around, such as Apache Karaf "Features" and "deploy/" directory. Most frameworks can also be instructed to load bundles at boot. So there is no "favor already loaded bundles". If there is no bundle satisfying the Import-Package, the bundle being resolved will not go to RESOLVED state. If there are many bundles (quite typical) satisfying an Import-Package, with all its additional contraints (versions, attributes, uses, ...), then you enter the NP-complete problem that Michal mentioned, finding a combination of wiring that satisfy as many bundles as possible. This is a problem mainly due to "uses" (since we seldom use attributes), where graphs of types must end up matching in the class space. See (especially) section 3.7 in OSGi 6.0 Core Specification. If anyone can get their head around those details in the first pass, it is you Peter. Not easy reading... If the client is matching service api with the correct import package version ranges (requirements defined by entry's), the proxy bundle should find the service api and other imported packages are already loaded. Eg the client may use the requirements to use the resource service or whatever the new bundle repository standard service is called now to preload the requirements. The client may also perform upgrades before downloading a service. I think this is a misunderstanding as well. By doing what I wrote in the beginning (listen on Jini Service Registrations and register "something" (Remote proxy or a local proxy) in the OSGi registry) then the client bundle doesn't need to know anything. Also, the Jini support bundle gets plenty of information both from the OSGi Registry as well as the Jini Registry. So, when something disappears from Reggie, remove it from the local Registry and vice versa. Now, the deserialization of the Reggie proxy should detect version changes and update a cache. And I think that Paremus idea of "Bundle Garbage Collection" is sound, but something for later discussion.
Re: OSGi - deserialization remote invocation strategy
> On Feb 7, 2017, at 8:56 AM, Michał Kłeczek (XPro Sp. z o. o.) > <michal.klec...@xpro.biz> wrote: > > Comments inline > > Niclas Hedhman wrote: >> 4. For Server(osgi)+Client(osgi), number of options goes up. In this space, >> Paremus has a lot of experience, and perhaps willing to share a bit, >> without compromising the secret sauce? Either way, Michal's talk about >> "wiring" becomes important and that wiring should possibly be >> re-established on the client side. The insistence on "must be exactly the >> same version" is to me a reflection of "we haven't cared about version >> management before", and I think it may not be in the best interest to load >> many nearly identical bundles just because they are a little off, say stuff >> like guava, commons-xyz, slf4j and many more common dependencies. > This problem is generally unsolvable because there are contradicting > requirements here: > 1. The need to transfer object graphs of (unknown) classes > 2. The need to optimize the number of class versions (and the number of > ClassLoaders) in the JVM > > It might be tempting to do the resolution on the client but it is (AFAIR) > NP-hard > - the object graph is a set of constraints on possible module (bundle) > versions. Plus there is a whole > set of constraints originating from the modules installed in the container > prior to graph deserialization. > > So the only good solution for a library is to provide a client with an > interface to implement: > Module resolve(Module candidate) (or Module[] resolve(Module[] candidates)) > and let it decide what to do. > >> >> Peter wrote; >>> This is why the bundle must be given first >>> attempt to resolve an objects class and rely on the bundle dependency >> resolution process. >>> OSGi must be allowed to wire up dependencies, we must avoid attempting to >> make decisions about >>> compatibility and use the current bundle wires instead (our stack). >> >> Well, not totally sure about that. The 'root object classloader' doesn't >> have visibility to serialized objects, and will fail if left to do it all >> by itself. And as soon as you delegate to another BundleClassLoader, you >> have made the resolution decision, not the framework. Michal's proposal to >> transfer the BundleWiring (available in runtime) from the server to the >> client, makes it somewhat possible to do the delegation. And to make >> matters worse, it is quite common that packages are exported from more than >> one bundle, so the question is what is included in the bundleWiring coming >> across the wire. > The whole issue with proposals based on the stream itself is the fact that to > resolve properly > one have to walk the whole graph first to gather all modules and their > dependencies. > > It is much better to simply provide the module graph (wiring) first (at the > beginning of the stream) > and only after resolution of all the modules - deserialize the objects. The missing notion of versioning in classloader was meant to be solved by URLs in the annotations. That would provide explicit versioning control and through the use of the TCCL, code and objects could be isolated from other versions. However, that’s not perfect, and so the preferred class loading mechanism is also a path that allows a “platform” or “hosting” environment to declare the classes that it will use to interact with a client/proxy. Practically, it’s exactly the NP-hard problem you say though. There are “pretty good” solutions, but realistically there is not a perfect solution until there is an exact dependency graph which allows perfect specification of dependencies. Gregg > > Thanks, > Michal
Re: OSGi - deserialization remote invocation strategy
Maybe there are some misunderstanding somewhere... see below; On Wed, Feb 8, 2017 at 3:35 AM, Peter <j...@zeus.net.au> wrote: > I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Ok, makes it a lot easier for me to follow. > Not expecting the client calling bundle to resolve everything, hence the > stack, so we have the full visibility of the bundle of the class that was > last resolved, so we can resolve its fields from it's bundle. Eg it > might import packages the client does not. What is "client calling bundle"? I assume that it is the bundle that contains the service lookup... First of all, is the OSGi integration intending to be at "OSGi service level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini service registrations and register those in the local OSGi registry and all clients would end up being rather ignorant about the remote lookup. It is also possible to "be involved" in the service lookup (and registration) through the Service Hook API (I might got the name of it wrong), so that only when services are requested or service listeners are registered, does the Jini support kick in. And this can therefore work both ways... So, that said; I agree that the "client calling bundle" does not resolve anything. And I would go further and say; IF the client calling bundle looks up Jini services directly, all bets are off. This introduces some workable constraints of what could happen. > The "exact version" thing (only applies to the proxy bundle as we > expect the framework to load its deps) can be relaxed to compatible > versions to increase class sharing if you think it helps. The proxy bundle > doesn't export anything at the client, only the server, it just seems to > make sense to keep the latest proxy communicating in case that last > bug fix release addresses a security issue. All proxy classes are > implementation only classes. Yes, correct. The 'temporary' proxy bundle should not export any packages. > Because the proxy bundle manifest declares version import ranges, > I'm expecting the framework to favour already loaded bundles to > satisfy package import deps. U... The OSGi framework doesn't "auto load" bundles. It is an explicit step. There are many "bundle loaders" around, such as Apache Karaf "Features" and "deploy/" directory. Most frameworks can also be instructed to load bundles at boot. So there is no "favor already loaded bundles". If there is no bundle satisfying the Import-Package, the bundle being resolved will not go to RESOLVED state. If there are many bundles (quite typical) satisfying an Import-Package, with all its additional contraints (versions, attributes, uses, ...), then you enter the NP-complete problem that Michal mentioned, finding a combination of wiring that satisfy as many bundles as possible. This is a problem mainly due to "uses" (since we seldom use attributes), where graphs of types must end up matching in the class space. See (especially) section 3.7 in OSGi 6.0 Core Specification. If anyone can get their head around those details in the first pass, it is you Peter. Not easy reading... > If the client is matching service api with the correct import package > version ranges (requirements defined by entry's), the proxy bundle > should find the service api and other imported packages are already > loaded. Eg the client may use the requirements to use the resource > service or whatever the new bundle repository standard service is > called now to preload the requirements. The client may also perform > upgrades before downloading a service. I think this is a misunderstanding as well. By doing what I wrote in the beginning (listen on Jini Service Registrations and register "something" (Remote proxy or a local proxy) in the OSGi registry) then the client bundle doesn't need to know anything. Also, the Jini support bundle gets plenty of information both from the OSGi Registry as well as the Jini Registry. So, when something disappears from Reggie, remove it from the local Registry and vice versa. Now, the deserialization of the Reggie proxy should detect version changes and update a cache. And I think that Paremus idea of "Bundle Garbage Collection" is sound, but something for later discussion. Point being; No need to figure out what can and can not be unloaded. ALSO, since OSGi mandates intermittent service availability, most OSGi applications are reasonably capable to handle that the Jini service will "disappear" and is required to release any held references to the object(s) so that regular GC can toss out the classes and classloader when bundle is unloaded. > In the majority of cases I don't think there's going to be much state > in the smart proxy that ca
Re: OSGi - deserialization remote invocation strategy
Correct in the second instance. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 08/02/2017 09:24:21 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy So in this case there is no authentication prior to code download? Or the client has to be authenticated before registering a listener. Am I correct? On Tue, 7 Feb 2017 at 22:32, Peter <j...@zeus.net.au> wrote: > Create three bundles, api (if any looks like your using jini platform > class as api), service and proxy. No split packages. > > Export it using a Jeri exporter. > > It's passed as a listener parameter to a service B ( which is also > exported using jeri and lets assume sent to the service end) the remote end > deserializes the smart proxy over it's remote connection, the jeri > InvocationDespatcher selects the service B implementation as it's > classloader or BundleReference, but your smart proxy cclasses are not > present, so the annotation will be used to load the bundle, which will > resolve to the imported jini api packages from River bundles in the server > back end. > > In other words, the same mechanism is used, a MarshalInputStream > RMiClassLoaderSpi, InvocationDespatcher instead of an invocation handler > (although your smart proxy will contain its own reflective proxy with its > own jeri endpoint and invocation handler and marshal streams) . > > Cheers, > > Peter. > Sent from my Samsung device. > > Include original message > Original message > From: Michał Kłeczek <mic...@kleczek.org> > Sent: 08/02/2017 06:51:55 am > To: dev@river.apache.org > Subject: Re: OSGi - deserialization remote invocation strategy > > I think you have misunderstood my question. > I want my implementation of RemoteEventListener to be a smart proxy. > How do u support that? > > On Tue, 7 Feb 2017 at 20:26, Peter <j...@zeusnet.au> wrote: > > > No, it's just an example, no such restriction. But it is restricted to > > JERI. > > > > RemoteEvents have been extended also in JGDMS to use MarshalledInstance > > and JErI instead of MarshalledObjects and JRMP. All methods that rely on > > MarshalledObject have been deprecated and replacements provided with the > > exception of Activation constructors. > > > > MarshalledInstance has also been made extensible to allow other > > serialization formats to be supported by the api. > > > > But we're diverging... > > > > Cheers, > > > > Peter. > > > > Sent from my Samsung device. > > > > Include original message > > Original message > > From: Michał Kłeczek <mic...@kleczek.org> > > Sent: 08/02/2017 06:07:19 am > > To: dev@river.apache.org > > Subject: Re: OSGi - deserialization remote invocation strategy > > > > Hmm.. I see. > > > > So your solution explicitly only handles cases of looking up a service > > in the ServiceRegistrar. > > > > How about smart RemoteEventListeners? They are not published in any > > lookup service. > > > > Thanks, > > Michal > > > > Peter wrote: > > > > > > In the JGDMS fork of River, in addition to Reggie's proxy returing >unmarshalled service proxy's from the ServiceRegistrar lookup method, >SafeServiceRegistrar provides a lookup method that returns instances of >java.lang.reflect.Proxy that implement interfaces to retrieve the service, >it's attributes and it's codebase uri and signer certs. > > > > > > > > > The client can authenticate, filter and grant permissions (for >deserialization and codebase download) before retrieving the service proxy >using a method call. > > > > > > > > > In this case the service proxy is obtained by from the reflection proxy >instead of MarshalledIstance. > > > > > > Cheers, > > > > > > Peter. > > > > > > Sent from my Samsung device. > > > > > >Include original message > > > Original message > > > From: "Michał Kłeczek (XPro Sp z o. o.)"<michal.klec...@xpro.biz> > > > Sent: 08/02/2017 05:51:07 am > > > To: dev@river.apache.org > > > Subject: Re: OSGi - deserialization remote invocation strategy > > > > > > > > > So I must have misunderstood the part about smart proxies being obtained >via "reflection proxies" or MarshalledInstances. > > > > > > Wha
Re: OSGi - deserialization remote invocation strategy
So in this case there is no authentication prior to code download? Or the client has to be authenticated before registering a listener. Am I correct? On Tue, 7 Feb 2017 at 22:32, Peter <j...@zeus.net.au> wrote: > Create three bundles, api (if any looks like your using jini platform > class as api), service and proxy. No split packages. > > Export it using a Jeri exporter. > > It's passed as a listener parameter to a service B ( which is also > exported using jeri and lets assume sent to the service end) the remote end > deserializes the smart proxy over it's remote connection, the jeri > InvocationDespatcher selects the service B implementation as it's > classloader or BundleReference, but your smart proxy cclasses are not > present, so the annotation will be used to load the bundle, which will > resolve to the imported jini api packages from River bundles in the server > back end. > > In other words, the same mechanism is used, a MarshalInputStream > RMiClassLoaderSpi, InvocationDespatcher instead of an invocation handler > (although your smart proxy will contain its own reflective proxy with its > own jeri endpoint and invocation handler and marshal streams) . > > Cheers, > > Peter. > Sent from my Samsung device. > > Include original message > Original message > From: Michał Kłeczek <mic...@kleczek.org> > Sent: 08/02/2017 06:51:55 am > To: dev@river.apache.org > Subject: Re: OSGi - deserialization remote invocation strategy > > I think you have misunderstood my question. > I want my implementation of RemoteEventListener to be a smart proxy. > How do u support that? > > On Tue, 7 Feb 2017 at 20:26, Peter <j...@zeus.net.au> wrote: > > > No, it's just an example, no such restriction. But it is restricted to > > JERI. > > > > RemoteEvents have been extended also in JGDMS to use MarshalledInstance > > and JErI instead of MarshalledObjects and JRMP. All methods that rely on > > MarshalledObject have been deprecated and replacements provided with the > > exception of Activation constructors. > > > > MarshalledInstance has also been made extensible to allow other > > serialization formats to be supported by the api. > > > > But we're diverging... > > > > Cheers, > > > > Peter. > > > > Sent from my Samsung device. > > > > Include original message > > Original message > > From: Michał Kłeczek <mic...@kleczek.org> > > Sent: 08/02/2017 06:07:19 am > > To: dev@river.apache.org > > Subject: Re: OSGi - deserialization remote invocation strategy > > > > Hmm.. I see. > > > > So your solution explicitly only handles cases of looking up a service > > in the ServiceRegistrar. > > > > How about smart RemoteEventListeners? They are not published in any > > lookup service. > > > > Thanks, > > Michal > > > > Peter wrote: > > > > > > In the JGDMS fork of River, in addition to Reggie's proxy returing > > > unmarshalled service proxy's from the ServiceRegistrar lookup method, > > > SafeServiceRegistrar provides a lookup method that returns instances of > > > java.lang.reflect.Proxy that implement interfaces to retrieve the > > > service, it's attributes and it's codebase uri and signer certs. > > > > > > > > > The client can authenticate, filter and grant permissions (for > > > deserialization and codebase download) before retrieving the service > > > proxy using a method call. > > > > > > > > > In this case the service proxy is obtained by from the reflection proxy > > > instead of MarshalledIstance. > > > > > > Cheers, > > > > > > Peter. > > > > > > Sent from my Samsung device. > > > > > >Include original message > > > Original message > > > From: "Michał Kłeczek (XPro Sp z o. o.)"<michal.klec...@xpro.biz> > > > Sent: 08/02/2017 05:51:07 am > > > To: dev@river.apache.org > > > Subject: Re: OSGi - deserialization remote invocation strategy > > > > > > > > > So I must have misunderstood the part about smart proxies being obtained > > > via "reflection proxies" or MarshalledInstances. > > > > > > What are these "reflection proxies"? > > > > > > Thanks, > > > Michal > > > > > > Peter wrote: > > > No, no bootstrap objects. > > > > > > Cheers, > > > > > > Peter. > > > > > > &
Re: OSGi - deserialization remote invocation strategy
No, it's just an example, no such restriction. But it is restricted to JERI. RemoteEvents have been extended also in JGDMS to use MarshalledInstance and JErI instead of MarshalledObjects and JRMP. All methods that rely on MarshalledObject have been deprecated and replacements provided with the exception of Activation constructors. MarshalledInstance has also been made extensible to allow other serialization formats to be supported by the api. But we're diverging... Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Michał Kłeczek <mic...@kleczek.org> Sent: 08/02/2017 06:07:19 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy Hmm.. I see. So your solution explicitly only handles cases of looking up a service in the ServiceRegistrar. How about smart RemoteEventListeners? They are not published in any lookup service. Thanks, Michal Peter wrote: > In the JGDMS fork of River, in addition to Reggie's proxy returing >unmarshalled service proxy's from the ServiceRegistrar lookup method, >SafeServiceRegistrar provides a lookup method that returns instances of >java.lang.reflect.Proxy that implement interfaces to retrieve the service, >it's attributes and it's codebase uri and signer certs. > > The client can authenticate, filter and grant permissions (for >deserialization and codebase download) before retrieving the service proxy >using a method call. > > In this case the service proxy is obtained by from the reflection proxy >instead of MarshalledIstance. > > Cheers, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: "Michał Kłeczek (XPro Sp z o. o.)"<michal.klec...@xpro.biz> > Sent: 08/02/2017 05:51:07 am > To: dev@river.apache.org > Subject: Re: OSGi - deserialization remote invocation strategy > > So I must have misunderstood the part about smart proxies being obtained via >"reflection proxies" or MarshalledInstances. > > What are these "reflection proxies"? > > Thanks, > Michal > > Peter wrote: > No, no bootstrap objects. > > Cheers, > > Peter. > > > > Sent from my Samsung device. > >Include original message > Original message > From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> > Sent: 08/02/2017 12:28:50 am > To: dev@river.apache.org > Subject: Re: OSGi - deserialization remote invocation strategy > > Are you proposing to provide a bootstrap object that will download some > meta information prior to class resolution? > > How does it differ from simply changing annotations to be those > "bootstrap objects" instead of Strings? > > Thanks, > Michal > > Peter wrote: > Proposed JERI OSGi class loading strategy during deserialization. > > Record caller context - this is the default bundle at the beginning of > the stack. It is obtained by the InvocationHandler on the > client side. The InvocationDispatcher on the server side has the > calling context of the Remote > implementation. The reflection dynamic proxy must be installed in the > client's class loader, so the > InvocationHandler knows exactly what it is, it will be passed to the > MarshalInputStream. Any > interfaces not found in the client's bundle can be safely shed. For a > smart proxy the reflection proxy will > be installed in the smart proxy loader. The smart proxy is obtained > either via a reflection proxy or a MarshalledInstance. > MarshalledInstance also passes in the callers loader to the > MarshalInputStream. > > The smart proxy classloader is not a child loader of the clients > loader, instead it's a bundle that imports > service api packages, with a version range that overlaps those already > imported by the client. > > Both Invocationhandler and InvocationDispatcher utilise > MarshalInputStream and MarshalOutputStream, for marshalling parameters > and return values. > > The codebase annotation bundle's manifest contains a list of package > imports. > > Do we need to make a list of package imports for every new bundle that > we load? > Do we need to record the wiring and packages and their imports from > the remote end? > > I don't think so, the bundles themselves contain this information, I > think we just need to keep the view of available classes relevant to > the current object being deserialized. > > Codebase Annotations are exact versions! They need to be to allo
Re: OSGi - deserialization remote invocation strategy
Clarification inline. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 08/02/2017 05:35:57 am To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi - deserialization remote invocation strategy Hi Nic, I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Not expecting the client calling bundle to resolve everything, hence the stack, so we have the full visibility of the bundle of the class that was last resolved, so we can resolve its fields from it's bundle Eg it might import packages the client does not. The "exact version" thing (only applies to the proxy bundle as we expect the framework to load its deps) can be relaxed to compatible versions to increase class sharing if you think it helps. The proxy bundle doesn't export anything at the client, only the server, it just seems to make sense to keep the latest This same proxy codebase version as the server is using, assuming it's using the latest. The proxy bundle codebase might not have been published to a repository, it may be an httpmd url for example. proxy communicating in case that last bug fix release addresses a security issue. All proxy classes are implementation only classes. Because the proxy bundle manifest declares version import ranges, I'm expecting the framework to favour already loaded bundles to satisfy package import deps. If the client is matching service api with the correct import package version ranges (requirements defined by entry's), the proxy bundle should find the service api and other imported packages are already loaded. Eg the client may use the requirements to use the resource service or whatever the new bundle repository standard service is called now to preload the requirements. The client may also perform upgrades before downloading a service. In the majority of cases I don't think there's going to be much state in the smart proxy that can't be loaded via the smart proxy bundle and it's package imports, except for the odd handback, which the client bundle should have the opportunity to resolve before resorting to using an annotation. I'm not quite ready to agree it's too complex and it's unsolveable, I think we should at least explore it and understand it before we junk the idea of supporting OSGi. Rather than utilise the Java2 class loading I was planning to cast ClassLoaders to BundleReference where appropriate and utilise the Bundle. I did notice you're interpretation of what I've written is different than mine, so I think I need to put some effort into communicating more effectively. I think you're interpretation of codebase annotation "version is fixed" ignores that the annotation is only consulted after determining that the current class is not available in our Bundles currently participating in deserialization. It doesn't apply to resolved imported packages as annotations aren't used for them at all For example, the first class we attempt to resolve during unmarshalling belongs to a smart proxy, the client Bundle can't find the class. Ask the framework to load the proxy bundle from the codebase annotation, it does so and resolves all necessary package imports declared in its manifest. We now continue deserializing the smart proxy class fields with the visibility of the smart proxy's bundle. The smart proxy may contain fields referencing objects resolved from its imports, we ensure those classes deserialize their fields with the visibility of their own bundles. Every time we can't resolve a class we first check if it's a handback or parameter from a preceeding object in the graph, thus we walk our graphs bundle stack. If we still haven't resolved a class only then do we load a bundle from it's codebase annotation url and check it can be cast to the field before asdigning it. If it can't be cast to that field, we throw an exception. In the case of a non smart proxy, there is no codebase, deserialization will be loaded by and rely completely on the visibility of the client bundle. I think OSGi will be a lot less dependant on annotations than say a std env. Still I guess wiring may be an an option, so as Michael suggests, annotate objects with their wiring graphs. What would we be considering if we hadn't been pre exposed to codebase annotations? Standard deserialization uses one classpath, each bundle has its own unique classpath. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Niclas Hedhman <nic...@hedhman.org> Sent: 08/02/2017 12:32:35 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy TL;DR 1. It sounds awfully complex, because my gut says that it is not a solvable problem, especially since I don't see 4 distinct cases; Server(osgi)+Client(osgi), Server(o
Re: OSGi - deserialization remote invocation strategy
Hmm... I see. So your solution explicitly only handles cases of looking up a service in the ServiceRegistrar. How about smart RemoteEventListeners? They are not published in any lookup service. Thanks, Michal Peter wrote: In the JGDMS fork of River, in addition to Reggie's proxy returing unmarshalled service proxy's from the ServiceRegistrar lookup method, SafeServiceRegistrar provides a lookup method that returns instances of java.lang.reflect.Proxy that implement interfaces to retrieve the service, it's attributes and it's codebase uri and signer certs. The client can authenticate, filter and grant permissions (for deserialization and codebase download) before retrieving the service proxy using a method call. In this case the service proxy is obtained by from the reflection proxy instead of MarshalledIstance. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 08/02/2017 05:51:07 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy So I must have misunderstood the part about smart proxies being obtained via "reflection proxies" or MarshalledInstances. What are these "reflection proxies"? Thanks, Michal Peter wrote: No, no bootstrap objects. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 08/02/2017 12:28:50 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy Are you proposing to provide a bootstrap object that will download some meta information prior to class resolution? How does it differ from simply changing annotations to be those "bootstrap objects" instead of Strings? Thanks, Michal Peter wrote: Proposed JERI OSGi class loading strategy during deserialization. Record caller context - this is the default bundle at the beginning of the stack. It is obtained by the InvocationHandler on the client side. The InvocationDispatcher on the server side has the calling context of the Remote implementation. The reflection dynamic proxy must be installed in the client's class loader, so the InvocationHandler knows exactly what it is, it will be passed to the MarshalInputStream. Any interfaces not found in the client's bundle can be safely shed. For a smart proxy the reflection proxy will be installed in the smart proxy loader. The smart proxy is obtained either via a reflection proxy or a MarshalledInstance. MarshalledInstance also passes in the callers loader to the MarshalInputStream. The smart proxy classloader is not a child loader of the clients loader, instead it's a bundle that imports service api packages, with a version range that overlaps those already imported by the client. Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream and MarshalOutputStream, for marshalling parameters and return values. The codebase annotation bundle's manifest contains a list of package imports. Do we need to make a list of package imports for every new bundle that we load? Do we need to record the wiring and packages and their imports from the remote end? I don't think so, the bundles themselves contain this information, I think we just need to keep the view of available classes relevant to the current object being deserialized. Codebase Annotations are exact versions! They need to be to allow the service to ensure the correct proxy codebase is used. Other proxy codebases will be installed in the client, possibly different versions, but these won't be visible through the resolved dependencies, because the proxy codebases only import packages at the client and OSGi restricts visibility to the current bundle's own classes and any imported packages. Instead of appending dependencies to the codebase annotation they'll need be defined in the proxy's bundle manifest. Of course if an identical version of a proxy codebase bundle is already installed at the client, this will be used again. Because a bundle generally imports packages (importing entire bundles is discouraged in OSGi), there may be classes that aren't visible from those bundles, such as transient imports, but also including private packages that aren't exported, private implementations need to be deserialized, but is it possible to do so safely, without causing package conflicts? Private implementation classes can be used as fields within an exported public object, but cannot and should not escape their private scope, doing so risks them being resolved to a bundle with the version of the remote end, instead of the locally resolved / wired package, causing
Re: OSGi - deserialization remote invocation strategy
In the JGDMS fork of River, in addition to Reggie's proxy returing unmarshalled service proxy's from the ServiceRegistrar lookup method, SafeServiceRegistrar provides a lookup method that returns instances of java.lang.reflect.Proxy that implement interfaces to retrieve the service, it's attributes and it's codebase uri and signer certs. The client can authenticate, filter and grant permissions (for deserialization and codebase download) before retrieving the service proxy using a method call. In this case the service proxy is obtained by from the reflection proxy instead of MarshalledIstance. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 08/02/2017 05:51:07 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy So I must have misunderstood the part about smart proxies being obtained via "reflection proxies" or MarshalledInstances. What are these "reflection proxies"? Thanks, Michal Peter wrote: No, no bootstrap objects. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 08/02/2017 12:28:50 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy Are you proposing to provide a bootstrap object that will download some meta information prior to class resolution? How does it differ from simply changing annotations to be those "bootstrap objects" instead of Strings? Thanks, Michal Peter wrote: Proposed JERI OSGi class loading strategy during deserialization. Record caller context - this is the default bundle at the beginning of the stack. It is obtained by the InvocationHandler on the client side. The InvocationDispatcher on the server side has the calling context of the Remote implementation. The reflection dynamic proxy must be installed in the client's class loader, so the InvocationHandler knows exactly what it is, it will be passed to the MarshalInputStream. Any interfaces not found in the client's bundle can be safely shed. For a smart proxy the reflection proxy will be installed in the smart proxy loader. The smart proxy is obtained either via a reflection proxy or a MarshalledInstance. MarshalledInstance also passes in the callers loader to the MarshalInputStream. The smart proxy classloader is not a child loader of the clients loader, instead it's a bundle that imports service api packages, with a version range that overlaps those already imported by the client. Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream and MarshalOutputStream, for marshalling parameters and return values. The codebase annotation bundle's manifest contains a list of package imports. Do we need to make a list of package imports for every new bundle that we load? Do we need to record the wiring and packages and their imports from the remote end? I don't think so, the bundles themselves contain this information, I think we just need to keep the view of available classes relevant to the current object being deserialized. Codebase Annotations are exact versions! They need to be to allow the service to ensure the correct proxy codebase is used. Other proxy codebases will be installed in the client, possibly different versions, but these won't be visible through the resolved dependencies, because the proxy codebases only import packages at the client and OSGi restricts visibility to the current bundle's own classes and any imported packages. Instead of appending dependencies to the codebase annotation they'll need be defined in the proxy's bundle manifest. Of course if an identical version of a proxy codebase bundle is already installed at the client, this will be used again. Because a bundle generally imports packages (importing entire bundles is discouraged in OSGi), there may be classes that aren't visible from those bundles, such as transient imports, but also including private packages that aren't exported, private implementations need to be deserialized, but is it possible to do so safely, without causing package conflicts? Private implementation classes can be used as fields within an exported public object, but cannot and should not escape their private scope, doing so risks them being resolved to a bundle with the version of the remote end, instead of the locally resolved / wired package, causing ClassClassExceptions. Initial (naive) first pass strategy of class resolution (for each branch in the serialized object graph)?: 1.Try current bundle on the stack (which will be the callers bundle if we haven't loaded any
Re: OSGi - deserialization remote invocation strategy
So I must have misunderstood the part about smart proxies being obtained via "reflection proxies" or MarshalledInstances. What are these "reflection proxies"? Thanks, Michal Peter wrote: No, no bootstrap objects. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 08/02/2017 12:28:50 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy Are you proposing to provide a bootstrap object that will download some meta information prior to class resolution? How does it differ from simply changing annotations to be those "bootstrap objects" instead of Strings? Thanks, Michal Peter wrote: Proposed JERI OSGi class loading strategy during deserialization. Record caller context - this is the default bundle at the beginning of the stack. It is obtained by the InvocationHandler on the client side. The InvocationDispatcher on the server side has the calling context of the Remote implementation. The reflection dynamic proxy must be installed in the client's class loader, so the InvocationHandler knows exactly what it is, it will be passed to the MarshalInputStream. Any interfaces not found in the client's bundle can be safely shed. For a smart proxy the reflection proxy will be installed in the smart proxy loader. The smart proxy is obtained either via a reflection proxy or a MarshalledInstance. MarshalledInstance also passes in the callers loader to the MarshalInputStream. The smart proxy classloader is not a child loader of the clients loader, instead it's a bundle that imports service api packages, with a version range that overlaps those already imported by the client. Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream and MarshalOutputStream, for marshalling parameters and return values. The codebase annotation bundle's manifest contains a list of package imports. Do we need to make a list of package imports for every new bundle that we load? Do we need to record the wiring and packages and their imports from the remote end? I don't think so, the bundles themselves contain this information, I think we just need to keep the view of available classes relevant to the current object being deserialized. Codebase Annotations are exact versions! They need to be to allow the service to ensure the correct proxy codebase is used. Other proxy codebases will be installed in the client, possibly different versions, but these won't be visible through the resolved dependencies, because the proxy codebases only import packages at the client and OSGi restricts visibility to the current bundle's own classes and any imported packages. Instead of appending dependencies to the codebase annotation they'll need be defined in the proxy's bundle manifest. Of course if an identical version of a proxy codebase bundle is already installed at the client, this will be used again. Because a bundle generally imports packages (importing entire bundles is discouraged in OSGi), there may be classes that aren't visible from those bundles, such as transient imports, but also including private packages that aren't exported, private implementations need to be deserialized, but is it possible to do so safely, without causing package conflicts? Private implementation classes can be used as fields within an exported public object, but cannot and should not escape their private scope, doing so risks them being resolved to a bundle with the version of the remote end, instead of the locally resolved / wired package, causing ClassClassExceptions. Initial (naive) first pass strategy of class resolution (for each branch in the serialized object graph)?: 1.Try current bundle on the stack (which will be the callers bundle if we haven't loaded any new bundles yet). 2.Then use the package name of a class to determine if the package is loaded by any of the bundles referenced by the callers bundle imports (to handle any private implementation packages that aren't in the current imports). Is this a good idea? Or should we go straight to step 3 and let the framework resolve common classes, what if we use a different version to the client's imported bundle? Should we first compare our bundle annotation to the currently imported bundles and select one of those if it's a compatible version? Yes, this could be an application bundle, otherwise goto 3. 3.Load bundle from annotation (if already loaded, it will be an exact version match). Place the new bundle on top of the bundle stack, remove this bundle from the stack once all fields of this object have been deserialized, returning to t
Re: OSGi - deserialization remote invocation strategy
Hi Nic, I'm currently only considering OSGi server -> OSGi client. Mick's investigating all four options. Not expecting the client calling bundle to resolve everything, hence the stack, so we have the full visibility of the bundle of the class that was last resolved, so we can resolve its fields from it's bundle. Eg it might import packages the client does not. The "exact version" thing (only applies to the proxy bundle as we expect the framework to load its deps) can be relaxed to compatible versions to increase class sharing if you think it helps. The proxy bundle doesn't export anything at the client, only the server, it just seems to make sense to keep the latest proxy communicating in case that last bug fix release addresses a security issue. All proxy classes are implementation only classes. Because the proxy bundle manifest declares version import ranges, I'm expecting the framework to favour already loaded bundles to satisfy package import deps. If the client is matching service api with the correct import package version ranges (requirements defined by entry's), the proxy bundle should find the service api and other imported packages are already loaded. Eg the client may use the requirements to use the resource service or whatever the new bundle repository standard service is called now to preload the requirements. The client may also perform upgrades before downloading a service. In the majority of cases I don't think there's going to be much state in the smart proxy that can't be loaded via the smart proxy bundle and it's package imports, except for the odd handback, which the client bundle should have the opportunity to resolve before resorting to using an annotation. I'm not quite ready to agree it's too complex and it's unsolveable, I think we should at least explore it and understand it before we junk the idea of supporting OSGi. Rather than utilise the Java2 class loading I was planning to cast ClassLoaders to BundleReference where appropriate and utilise the Bundle. I did notice you're interpretation of what I've written is different than mine, so I think I need to put some effort into communicating more effectively. I think you're interpretation of codebase annotation "version is fixed" ignores that the annotation is only consulted after determining that the current class is not available in our Bundles currently participating in deserialization. It doesn't apply to resolved imported packages as annotations aren't used for them at all. For example, the first class we attempt to resolve during unmarshalling belongs to a smart proxy, the client Bundle can't find the class. Ask the framework to load the proxy bundle from the codebase annotation, it does so and resolves all necessary package imports declared in its manifest. We now continue deserializing the smart proxy class fields with the visibility of the smart proxy's bundle. The smart proxy may contain fields referencing objects resolved from its imports, we ensure those classes deserialize their fields with the visibility of their own bundles. Every time we can't resolve a class we first check if it's a handback or parameter from a preceeding object in the graph, thus we walk our graphs bundle stack. If we still haven't resolved a class only then do we load a bundle from it's codebase annotation url and check it can be cast to the field before asdigning it. If it can't be cast to that field, we throw an exception. In the case of a non smart proxy, there is no codebase, deserialization will be loaded by and rely completely on the visibility of the client bundle. I think OSGi will be a lot less dependant on annotations than say a std env. Still I guess wiring may be an an option, so as Michael suggests, annotate objects with their wiring graphs. What would we be considering if we hadn't been pre exposed to codebase annotations? Standard deserialization uses one classpath, each bundle has its own unique classpath. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Niclas Hedhman <nic...@hedhman.org> Sent: 08/02/2017 12:32:35 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy TL;DR 1. It sounds awfully complex, because my gut says that it is not a solvable problem, especially since I don't see 4 distinct cases; Server(osgi)+Client(osgi), Server(osgi)+Client(plain), Server(plain)+Client(osgi) and Server(plain)+Client(plain), where the last one is what we currently have. I am not sure how many of those you are discussing. 2. For Server(osgi)+Client(plain), I don't think there is a generic solution, due to "uses" in OSGi terms. The same object graph can contain multiple versions of the same classes in non-hierarchical order, which to me seems to be incompatible with the Java2 classloading mechanism. Replicating
Re: OSGi - deserialization remote invocation strategy
No, no bootstrap objects. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 08/02/2017 12:28:50 am To: dev@river.apache.org Subject: Re: OSGi - deserialization remote invocation strategy Are you proposing to provide a bootstrap object that will download some meta information prior to class resolution? How does it differ from simply changing annotations to be those "bootstrap objects" instead of Strings? Thanks, Michal Peter wrote: > Proposed JERI OSGi class loading strategy during deserialization. > > Record caller context - this is the default bundle at the beginning of > the stack. It is obtained by the InvocationHandler on the > client side. The InvocationDispatcher on the server side has the > calling context of the Remote > implementation. The reflection dynamic proxy must be installed in the > client's class loader, so the > InvocationHandler knows exactly what it is, it will be passed to the > MarshalInputStream. Any > interfaces not found in the client's bundle can be safely shed. For a > smart proxy the reflection proxy will > be installed in the smart proxy loader. The smart proxy is obtained > either via a reflection proxy or a MarshalledInstance. > MarshalledInstance also passes in the callers loader to the > MarshalInputStream. > > The smart proxy classloader is not a child loader of the clients > loader, instead it's a bundle that imports > service api packages, with a version range that overlaps those already > imported by the client. > > Both Invocationhandler and InvocationDispatcher utilise > MarshalInputStream and MarshalOutputStream, for marshalling parameters > and return values. > > The codebase annotation bundle's manifest contains a list of package > imports. > > Do we need to make a list of package imports for every new bundle that > we load? > Do we need to record the wiring and packages and their imports from > the remote end? > > I don't think so, the bundles themselves contain this information, I > think we just need to keep the view of available classes relevant to > the current object being deserialized. > > Codebase Annotations are exact versions! They need to be to allow the > service to ensure the correct proxy codebase is used. Other proxy > codebases will be installed in the client, possibly different > versions, but these won't be visible through the resolved > dependencies, because the proxy codebases only import packages at the > client and OSGi restricts visibility to the current bundle's own > classes and any imported packages. > Instead of appending dependencies to the codebase annotation they'll > need be defined in the proxy's bundle manifest. Of course if an > identical version of a proxy codebase bundle is already installed at > the client, this will be used again. > > Because a bundle generally imports packages (importing entire bundles > is discouraged in OSGi), there may be classes > that aren't visible from those bundles, such as transient imports, but > also including private packages that aren't exported, private > implementations need to be deserialized, but is it possible to do so > safely, without causing package > conflicts? Private implementation classes can be used as fields > within an exported public object, but cannot and should not > escape their private scope, doing so risks them being resolved to a > bundle with the version of the remote end, instead of the locally > resolved / wired package, causing ClassClassExceptions. > > Initial (naive) first pass strategy of class resolution (for each > branch in the serialized object graph)?: > 1.Try current bundle on the stack (which will be the callers > bundle if we haven't loaded any new bundles yet). > 2.Then use the package name of a class to determine if the package > is loaded by any of the bundles > referenced by the callers bundle imports (to handle any private > implementation packages > that aren't in the current imports). Is this a good idea? Or should > we go straight to step 3 > and let the framework resolve common classes, what if we use a > different version to the > client's imported bundle? Should we first compare our bundle > annotation to the currently > imported bundles and select one of those if it's a compatible > version? Yes, this could be an > application bundle, otherwise goto 3. > 3.Load bundle from annotation (if already loaded, it will be an > exact version match). Place the > new bundle on top of the bundle s
Re: OSGi - deserialization remote invocation strategy
Comments inline Niclas Hedhman wrote: 4. For Server(osgi)+Client(osgi), number of options goes up. In this space, Paremus has a lot of experience, and perhaps willing to share a bit, without compromising the secret sauce? Either way, Michal's talk about "wiring" becomes important and that wiring should possibly be re-established on the client side. The insistence on "must be exactly the same version" is to me a reflection of "we haven't cared about version management before", and I think it may not be in the best interest to load many nearly identical bundles just because they are a little off, say stuff like guava, commons-xyz, slf4j and many more common dependencies. This problem is generally unsolvable because there are contradicting requirements here: 1. The need to transfer object graphs of (unknown) classes 2. The need to optimize the number of class versions (and the number of ClassLoaders) in the JVM It might be tempting to do the resolution on the client but it is (AFAIR) NP-hard - the object graph is a set of constraints on possible module (bundle) versions. Plus there is a whole set of constraints originating from the modules installed in the container prior to graph deserialization. So the only good solution for a library is to provide a client with an interface to implement: Module resolve(Module candidate) (or Module[] resolve(Module[] candidates)) and let it decide what to do. Peter wrote; This is why the bundle must be given first attempt to resolve an objects class and rely on the bundle dependency resolution process. OSGi must be allowed to wire up dependencies, we must avoid attempting to make decisions about compatibility and use the current bundle wires instead (our stack). Well, not totally sure about that. The 'root object classloader' doesn't have visibility to serialized objects, and will fail if left to do it all by itself. And as soon as you delegate to another BundleClassLoader, you have made the resolution decision, not the framework. Michal's proposal to transfer the BundleWiring (available in runtime) from the server to the client, makes it somewhat possible to do the delegation. And to make matters worse, it is quite common that packages are exported from more than one bundle, so the question is what is included in the bundleWiring coming across the wire. The whole issue with proposals based on the stream itself is the fact that to resolve properly one have to walk the whole graph first to gather all modules and their dependencies. It is much better to simply provide the module graph (wiring) first (at the beginning of the stream) and only after resolution of all the modules - deserialize the objects. Thanks, Michal
Re: OSGi - deserialization remote invocation strategy
Are you proposing to provide a bootstrap object that will download some meta information prior to class resolution? How does it differ from simply changing annotations to be those "bootstrap objects" instead of Strings? Thanks, Michal Peter wrote: Proposed JERI OSGi class loading strategy during deserialization. Record caller context - this is the default bundle at the beginning of the stack. It is obtained by the InvocationHandler on the client side. The InvocationDispatcher on the server side has the calling context of the Remote implementation. The reflection dynamic proxy must be installed in the client's class loader, so the InvocationHandler knows exactly what it is, it will be passed to the MarshalInputStream. Any interfaces not found in the client's bundle can be safely shed. For a smart proxy the reflection proxy will be installed in the smart proxy loader. The smart proxy is obtained either via a reflection proxy or a MarshalledInstance. MarshalledInstance also passes in the callers loader to the MarshalInputStream. The smart proxy classloader is not a child loader of the clients loader, instead it's a bundle that imports service api packages, with a version range that overlaps those already imported by the client. Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream and MarshalOutputStream, for marshalling parameters and return values. The codebase annotation bundle's manifest contains a list of package imports. Do we need to make a list of package imports for every new bundle that we load? Do we need to record the wiring and packages and their imports from the remote end? I don't think so, the bundles themselves contain this information, I think we just need to keep the view of available classes relevant to the current object being deserialized. Codebase Annotations are exact versions! They need to be to allow the service to ensure the correct proxy codebase is used. Other proxy codebases will be installed in the client, possibly different versions, but these won't be visible through the resolved dependencies, because the proxy codebases only import packages at the client and OSGi restricts visibility to the current bundle's own classes and any imported packages. Instead of appending dependencies to the codebase annotation they'll need be defined in the proxy's bundle manifest. Of course if an identical version of a proxy codebase bundle is already installed at the client, this will be used again. Because a bundle generally imports packages (importing entire bundles is discouraged in OSGi), there may be classes that aren't visible from those bundles, such as transient imports, but also including private packages that aren't exported, private implementations need to be deserialized, but is it possible to do so safely, without causing package conflicts? Private implementation classes can be used as fields within an exported public object, but cannot and should not escape their private scope, doing so risks them being resolved to a bundle with the version of the remote end, instead of the locally resolved / wired package, causing ClassClassExceptions. Initial (naive) first pass strategy of class resolution (for each branch in the serialized object graph)?: 1.Try current bundle on the stack (which will be the callers bundle if we haven't loaded any new bundles yet). 2.Then use the package name of a class to determine if the package is loaded by any of the bundles referenced by the callers bundle imports (to handle any private implementation packages that aren't in the current imports). Is this a good idea? Or should we go straight to step 3 and let the framework resolve common classes, what if we use a different version to the client's imported bundle? Should we first compare our bundle annotation to the currently imported bundles and select one of those if it's a compatible version? Yes, this could be an application bundle, otherwise goto 3. 3.Load bundle from annotation (if already loaded, it will be an exact version match). Place the new bundle on top of the bundle stack, remove this bundle from the stack once all fields of this object have been deserialized, returning to the previous bundle context. We are relying on the current bundle to wire itself up to the same package versions of the clients bundle imports, for shared classes. Classes that use different bundles will not be visible to the client, but will need to be visible to the current object's bundle. 4.Place a bundle reference on the stack when a new object is deserialized from the stream and remove it once all fields have been deserialized. (we might need to remember stack depth). 5.Don't place non bundle references on the stack. For example system class loader or any other class loader, we want resolution to occur via the OSGi resolution process. What about a simpler strategy (again nai
Re: OSGi - deserialization remote invocation strategy
TL;DR 1. It sounds awfully complex, because my gut says that it is not a solvable problem, especially since I don't see 4 distinct cases; Server(osgi)+Client(osgi), Server(osgi)+Client(plain), Server(plain)+Client(osgi) and Server(plain)+Client(plain), where the last one is what we currently have. I am not sure how many of those you are discussing. 2. For Server(osgi)+Client(plain), I don't think there is a generic solution, due to "uses" in OSGi terms. The same object graph can contain multiple versions of the same classes in non-hierarchical order, which to me seems to be incompatible with the Java2 classloading mechanism. Replicating this, would effectively need (as Michal suggested) the OSGi framework to be booted up on the client (not impossible per se). 3. For Server(plain)+Client(osgi), I think tthat the 'easy' solution is collapse all dependencies into a single bundle and load that bundle on the OSGi framework. The exception being the API classes, i.e. those that must have been present on the client to be able to use the service. Exactly how to figure that out could be complicated and I have no good answers. 4. For Server(osgi)+Client(osgi), number of options goes up. In this space, Paremus has a lot of experience, and perhaps willing to share a bit, without compromising the secret sauce? Either way, Michal's talk about "wiring" becomes important and that wiring should possibly be re-established on the client side. The insistence on "must be exactly the same version" is to me a reflection of "we haven't cared about version management before", and I think it may not be in the best interest to load many nearly identical bundles just because they are a little off, say stuff like guava, commons-xyz, slf4j and many more common dependencies. Peter wrote; > This is why the bundle must be given first > attempt to resolve an objects class and rely on the bundle dependency resolution process. > OSGi must be allowed to wire up dependencies, we must avoid attempting to make decisions about > compatibility and use the current bundle wires instead (our stack). Well, not totally sure about that. The 'root object classloader' doesn't have visibility to serialized objects, and will fail if left to do it all by itself. And as soon as you delegate to another BundleClassLoader, you have made the resolution decision, not the framework. Michal's proposal to transfer the BundleWiring (available in runtime) from the server to the client, makes it somewhat possible to do the delegation. And to make matters worse, it is quite common that packages are exported from more than one bundle, so the question is what is included in the bundleWiring coming across the wire. HTH On Tue, Feb 7, 2017 at 8:14 PM, Peter <j...@zeus.net.au> wrote: > Proposed JERI OSGi class loading strategy during deserialization. > > Record caller context - this is the default bundle at the beginning of the > stack. It is obtained by the InvocationHandler on the > client side. The InvocationDispatcher on the server side has the calling > context of the Remote > implementation. The reflection dynamic proxy must be installed in the > client's class loader, so the > InvocationHandler knows exactly what it is, it will be passed to the > MarshalInputStream. Any > interfaces not found in the client's bundle can be safely shed. For a > smart proxy the reflection proxy will > be installed in the smart proxy loader. The smart proxy is obtained > either via a reflection proxy or a MarshalledInstance. > MarshalledInstance also passes in the callers loader to the > MarshalInputStream. > > The smart proxy classloader is not a child loader of the clients loader, > instead it's a bundle that imports > service api packages, with a version range that overlaps those already > imported by the client. > > Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream > and MarshalOutputStream, for marshalling parameters and return values. > > The codebase annotation bundle's manifest contains a list of package > imports. > > Do we need to make a list of package imports for every new bundle that we > load? > Do we need to record the wiring and packages and their imports from the > remote end? > > I don't think so, the bundles themselves contain this information, I think > we just need to keep the view of available classes relevant to the current > object being deserialized. > > Codebase Annotations are exact versions! They need to be to allow the > service to ensure the correct proxy codebase is used. Other proxy > codebases will be installed in the client, possibly different versions, but > these won't be visible through the resolved dependencies, because the proxy > codebases only import packages at the client and OSGi restricts visibility > to the
OSGi - deserialization remote invocation strategy
Proposed JERI OSGi class loading strategy during deserialization. Record caller context - this is the default bundle at the beginning of the stack. It is obtained by the InvocationHandler on the client side. The InvocationDispatcher on the server side has the calling context of the Remote implementation. The reflection dynamic proxy must be installed in the client's class loader, so the InvocationHandler knows exactly what it is, it will be passed to the MarshalInputStream. Any interfaces not found in the client's bundle can be safely shed. For a smart proxy the reflection proxy will be installed in the smart proxy loader. The smart proxy is obtained either via a reflection proxy or a MarshalledInstance. MarshalledInstance also passes in the callers loader to the MarshalInputStream. The smart proxy classloader is not a child loader of the clients loader, instead it's a bundle that imports service api packages, with a version range that overlaps those already imported by the client. Both Invocationhandler and InvocationDispatcher utilise MarshalInputStream and MarshalOutputStream, for marshalling parameters and return values. The codebase annotation bundle's manifest contains a list of package imports. Do we need to make a list of package imports for every new bundle that we load? Do we need to record the wiring and packages and their imports from the remote end? I don't think so, the bundles themselves contain this information, I think we just need to keep the view of available classes relevant to the current object being deserialized. Codebase Annotations are exact versions! They need to be to allow the service to ensure the correct proxy codebase is used. Other proxy codebases will be installed in the client, possibly different versions, but these won't be visible through the resolved dependencies, because the proxy codebases only import packages at the client and OSGi restricts visibility to the current bundle's own classes and any imported packages. Instead of appending dependencies to the codebase annotation they'll need be defined in the proxy's bundle manifest. Of course if an identical version of a proxy codebase bundle is already installed at the client, this will be used again. Because a bundle generally imports packages (importing entire bundles is discouraged in OSGi), there may be classes that aren't visible from those bundles, such as transient imports, but also including private packages that aren't exported, private implementations need to be deserialized, but is it possible to do so safely, without causing package conflicts? Private implementation classes can be used as fields within an exported public object, but cannot and should not escape their private scope, doing so risks them being resolved to a bundle with the version of the remote end, instead of the locally resolved / wired package, causing ClassClassExceptions. Initial (naive) first pass strategy of class resolution (for each branch in the serialized object graph)?: 1.Try current bundle on the stack (which will be the callers bundle if we haven't loaded any new bundles yet). 2.Then use the package name of a class to determine if the package is loaded by any of the bundles referenced by the callers bundle imports (to handle any private implementation packages that aren't in the current imports). Is this a good idea? Or should we go straight to step 3 and let the framework resolve common classes, what if we use a different version to the client's imported bundle? Should we first compare our bundle annotation to the currently imported bundles and select one of those if it's a compatible version? Yes, this could be an application bundle, otherwise goto 3. 3.Load bundle from annotation (if already loaded, it will be an exact version match). Place the new bundle on top of the bundle stack, remove this bundle from the stack once all fields of this object have been deserialized, returning to the previous bundle context. We are relying on the current bundle to wire itself up to the same package versions of the clients bundle imports, for shared classes. Classes that use different bundles will not be visible to the client, but will need to be visible to the current object's bundle. 4.Place a bundle reference on the stack when a new object is deserialized from the stream and remove it once all fields have been deserialized. (we might need to remember stack depth). 5.Don't place non bundle references on the stack. For example system class loader or any other class loader, we want resolution to occur via the OSGi resolution process. What about a simpler strategy (again naive), where we don't attempt to resolve private implementation classes? 1.The calling class' bundle, is given priority. 2.Load bundle from annotation (exact version), when not found in calling class. 3.No stack, what if an application bundle from server
Re: OSGi
I still feel that RMIClassLoaderSPI can provide this mechanism. There is an execution context required, but predominately, the URL string can still reflect the source of the code you want to use. Gregg > On Feb 5, 2017, at 11:34 PM, Michał Kłeczek (XPro Sp. z o. o.) > <michal.klec...@xpro.biz> wrote: > > Once you realize you need some codebase metadata different than mere list of > URLs > the next conclusion is that annotations should be something different than... > a String :) > > The next thing to ask is: "what about mixed OSGI and non-OSGI environments" > Then you start to realize you need to abstract over the class loading > environment itself. > > Then you start to realize that to support all the scenarios you need to > provide a class loading environment that is "pluggable" > - ie allows using it with other class loading environments and allow the user > to decide which classes should be loaded > by which environment. > > This is what I am working on right now :) > > Thanks, > Michal > > Peter wrote: >> My phone sent the previous email before I completed editing. >> >> ...If api classes are already loaded locally by client code, then a smart >> proxy codebase bundle will resolve imports to those packages (if they're >> within the imported version range), when the proxy bundle is downloaded, >> resolved and loaded. >> >> The strategy should be, deserialize using the callers context until a class >> is not found, then switch to the object containing the current field being >> deserialized (which may be a package private implementation class in the >> service api bundle) and if that fails use the codebase annotation (the smart >> proxy). This is similar in some ways to never preferred, where locally >> visible classes will be selected first. >> >> The strategy is to let OSGi do all the dependency wiring from bundle >> manifests. Classes not visible will be visible from a common package import >> class, except for poorly designed services, which is outside of scope. >> >> Only match api version compatible services. >> >> No allowances made for split packages or other complexities. >> >> If deserialization doesn't succeed, look up another service. >> >> Cheers, >> >> Peter. >> >> Sent from my Samsung device. >> Include original message >> Original message >> From: Peter<j...@zeus.net.au> >> Sent: 06/02/2017 02:59:09 pm >> To: dev@river.apache.org<dev@river.apache.org> >> Subject: Re: OSGi >> >> Thanks Nic, >> >> If annot >> >> You've identified the reason we need an OSGi specific RMIClassLoaderSpi >> implementation; so we can capture and provide Bundle specific annotation >> information. >> >> Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, >> the context ClassLoader is used by PreferredClassProvider when the >> ClassLoader argument is null. >> >> Standard Java serialization's OIS walks the call stack and selects the first >> non system classloader (it's looking for the application class loader), it >> deserializes into the application ClassLoader's context. This doesn't work >> in OSGi because the application classes are loaded by a multitude of >> ClassLoaders. >> >> It also looks like we'll need an OSGi specific InvocationLayerFactory to >> capture ClassLoader information to pass to our MarshalInputStream then to >> our RMIClassLoaderSpi during deserialization at both endpoints. >> >> We also need to know the bundle (ClassLoader) of the class that calls a >> java.lang.reflect.Proxy on the client side, this is actually quite easy to >> find, walk the stack, find the Proxy class and obtain the BundleReference / >> ClassLoader of the caller. >> >> Currently the java.lang.reflectProxy dynamically generated subclass instance >> proxy's ClassLoader is used, this is acceptable when the proxy bytecode is >> loaded by the the Client's ClassLoader or smart proxy ClassLoader in the >> case where a smart proxy is utilised >> >> >> >> If the caller changes, so does the calling context. >> >> >> Each bundle provides access to all classes within that bundle, including any >> public classes from imported packages. >> >> >> >> >> >> Sent from my Samsung device. >> Include original message >> Original message >> From: Niclas Hedhman<nic...@hedhman.org> >> Sent:
Re: OSGi
The upside is that it simplifies the overall architecture. For example it makes the whole part of River related to proxy trust verification obsolete. All these ProxyTrustIterators executed in an untrusted security context, Verifier implementations loaded using a proper ClassLoader etc. - this is not needed anymore. Thanks, Michal Michał Kłeczek (XPro Sp. z o. o.) wrote: Well - times changed since original Jini has been developed. There is a whole lot of amazing libraries out there - so the undertaking is much easier than doing it without them. I am specifically talking about Google Guava, JBoss Modules and RxJava. As River is concerned - once you get past the assumption that codebase annotations are Strings - it has all the necessary extension points available. I've already started writing the test suite for the thing and hope to present it soon. Thanks, Michal Peter wrote: For the sake of simplicity it's probably best if OSGi and non interact only using reflection proxy's and have their own djinn groups so code downloading is unnecessary between them. At least that's how I'd consider introducing it into an existing djinn. A jvm that doesn't have version management of some sort may have a lot of difficulty interacting with services from a framework that can use incompatible library versions (and that includes service api) side by side. My concern is interacting with non versioned env's will probably cause the developer to have to continue dealing with the problems the modular framework they selected intended solving Maven and OSGi can probably interact using mvn: codebase annotations, provided all modules have bundle manifests. I still support what your doing and find it interesting and don't wish to discourage you, I think you're likely to admit it will be a difficult undertaking, but that's probably an attraction right? Maybe River could provide some interfaces for extensibility where you could plug in? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 06/02/2017 03:34:54 pm To:dev@river.apache.org Subject: Re: OSGi Once you realize you need some codebase metadata different than mere list of URLs the next conclusion is that annotations should be something different than... a String :) The next thing to ask is: "what about mixed OSGI and non-OSGI environments" Then you start to realize you need to abstract over the class loading environment itself. Then you start to realize that to support all the scenarios you need to provide a class loading environment that is "pluggable" - ie allows using it with other class loading environments and allow the user to decide which classes should be loaded by which environment. This is what I am working on right now :) Thanks, Michal Peter wrote: My phone sent the previous email before I completed editing. ...If api classes are already loaded locally by client code, then a smart proxy codebase bundle will resolve imports to those packages (if they're within the imported version range), when the proxy bundle is downloaded, resolved and loaded. The strategy should be, deserialize using the callers context until a class is not found, then switch to the object containing the current field being deserialized (which may be a package private implementation class in the service api bundle) and if that fails use the codebase annotation (the smart proxy). This is similar in some ways to never preferred, where locally visible classes will be selected first. The strategy is to let OSGi do all the dependency wiring from bundle manifests. Classes not visible will be visible from a common package import class, except for poorly designed services, which is outside of scope. Only match api version compatible services. No allowances made for split packages or other complexities. If deserialization doesn't succeed, look up another service. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter<j...@zeus.net.au> Sent: 06/02/2017 02:59:09 pm To:dev@river.apache.org<dev@river.apache.org> Subject: Re: OSGi Thanks Nic, If annot You've identified the reason we need an OSGi specific RMIClassLoaderSpi implementation; so we can capture and provide Bundle specific annotation information. Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the context ClassLoader is used by PreferredClassProvider when the ClassLoader argument is null. Standard Java serialization's OIS walks the call stack and selects the first non system classloader (it's looking for the application class loader), it deserializes into the application ClassLoader's context. This doesn't work in OSGi because the application classes are loaded
Re: OSGi
Well - times changed since original Jini has been developed. There is a whole lot of amazing libraries out there - so the undertaking is much easier than doing it without them. I am specifically talking about Google Guava, JBoss Modules and RxJava. As River is concerned - once you get past the assumption that codebase annotations are Strings - it has all the necessary extension points available. I've already started writing the test suite for the thing and hope to present it soon. Thanks, Michal Peter wrote: For the sake of simplicity it's probably best if OSGi and non interact only using reflection proxy's and have their own djinn groups so code downloading is unnecessary between them. At least that's how I'd consider introducing it into an existing djinn. A jvm that doesn't have version management of some sort may have a lot of difficulty interacting with services from a framework that can use incompatible library versions (and that includes service api) side by side. My concern is interacting with non versioned env's will probably cause the developer to have to continue dealing with the problems the modular framework they selected intended solving Maven and OSGi can probably interact using mvn: codebase annotations, provided all modules have bundle manifests. I still support what your doing and find it interesting and don't wish to discourage you, I think you're likely to admit it will be a difficult undertaking, but that's probably an attraction right? Maybe River could provide some interfaces for extensibility where you could plug in? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz> Sent: 06/02/2017 03:34:54 pm To: dev@river.apache.org Subject: Re: OSGi Once you realize you need some codebase metadata different than mere list of URLs the next conclusion is that annotations should be something different than... a String :) The next thing to ask is: "what about mixed OSGI and non-OSGI environments" Then you start to realize you need to abstract over the class loading environment itself. Then you start to realize that to support all the scenarios you need to provide a class loading environment that is "pluggable" - ie allows using it with other class loading environments and allow the user to decide which classes should be loaded by which environment. This is what I am working on right now :) Thanks, Michal Peter wrote: My phone sent the previous email before I completed editing. ...If api classes are already loaded locally by client code, then a smart proxy codebase bundle will resolve imports to those packages (if they're within the imported version range), when the proxy bundle is downloaded, resolved and loaded. The strategy should be, deserialize using the callers context until a class is not found, then switch to the object containing the current field being deserialized (which may be a package private implementation class in the service api bundle) and if that fails use the codebase annotation (the smart proxy). This is similar in some ways to never preferred, where locally visible classes will be selected first. The strategy is to let OSGi do all the dependency wiring from bundle manifests. Classes not visible will be visible from a common package import class, except for poorly designed services, which is outside of scope. Only match api version compatible services. No allowances made for split packages or other complexities. If deserialization doesn't succeed, look up another service. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter<j...@zeus.net.au> Sent: 06/02/2017 02:59:09 pm To: dev@river.apache.org<dev@river.apache.org> Subject: Re: OSGi Thanks Nic, If annot You've identified the reason we need an OSGi specific RMIClassLoaderSpi implementation; so we can capture and provide Bundle specific annotation information. Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the context ClassLoader is used by PreferredClassProvider when the ClassLoader argument is null. Standard Java serialization's OIS walks the call stack and selects the first non system classloader (it's looking for the application class loader), it deserializes into the application ClassLoader's context. This doesn't work in OSGi because the application classes are loaded by a multitude of ClassLoaders. It also looks like we'll need an OSGi specific InvocationLayerFactory to capture ClassLoader information to pass to our MarshalInputStream then to our RMIClassLoaderSpi during deserialization at both endpoints. We also need to know the bundle (ClassLoader) of the class that calls a java.lang.reflect.Proxy on the client side, this is ac
Re: OSGi
For the sake of simplicity it's probably best if OSGi and non interact only using reflection proxy's and have their own djinn groups so code downloading is unnecessary between them. At least that's how I'd consider introducing it into an existing djinn. A jvm that doesn't have version management of some sort may have a lot of difficulty interacting with services from a framework that can use incompatible library versions (and that includes service api) side by side. My concern is interacting with non versioned env's will probably cause the developer to have to continue dealing with the problems the modular framework they selected intended solving Maven and OSGi can probably interact using mvn: codebase annotations, provided all modules have bundle manifests. I still support what your doing and find it interesting and don't wish to discourage you, I think you're likely to admit it will be a difficult undertaking, but that's probably an attraction right? Maybe River could provide some interfaces for extensibility where you could plug in? Regards, Peter. Sent from my Samsung device. Include original message Original message From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz> Sent: 06/02/2017 03:34:54 pm To: dev@river.apache.org Subject: Re: OSGi Once you realize you need some codebase metadata different than mere list of URLs the next conclusion is that annotations should be something different than... a String :) The next thing to ask is: "what about mixed OSGI and non-OSGI environments" Then you start to realize you need to abstract over the class loading environment itself. Then you start to realize that to support all the scenarios you need to provide a class loading environment that is "pluggable" - ie allows using it with other class loading environments and allow the user to decide which classes should be loaded by which environment. This is what I am working on right now :) Thanks, Michal Peter wrote: > My phone sent the previous email before I completed editing. > > ...If api classes are already loaded locally by client code, then a smart >proxy codebase bundle will resolve imports to those packages (if they're >within the imported version range), when the proxy bundle is downloaded, >resolved and loaded. > > The strategy should be, deserialize using the callers context until a class >is not found, then switch to the object containing the current field being >deserialized (which may be a package private implementation class in the >service api bundle) and if that fails use the codebase annotation (the smart >proxy). This is similar in some ways to never preferred, where locally >visible classes will be selected first. > > The strategy is to let OSGi do all the dependency wiring from bundle >manifests. Classes not visible will be visible from a common package import >class, except for poorly designed services, which is outside of scope. > > Only match api version compatible services. > > No allowances made for split packages or other complexities. > > If deserialization doesn't succeed, look up another service. > > Cheers, > > Peter. > > Sent from my Samsung device. > >Include original message > Original message > From: Peter<j...@zeus.net.au> > Sent: 06/02/2017 02:59:09 pm > To: dev@river.apache.org<dev@river.apache.org> > Subject: Re: OSGi > > > Thanks Nic, > > If annot > > You've identified the reason we need an OSGi specific RMIClassLoaderSpi >implementation; so we can capture and provide Bundle specific annotation >information. > > Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, >the context ClassLoader is used by PreferredClassProvider when the ClassLoader >argument is null. > > Standard Java serialization's OIS walks the call stack and selects the first >non system classloader (it's looking for the application class loader), it >deserializes into the application ClassLoader's context. This doesn't work >in OSGi because the application classes are loaded by a multitude of >ClassLoaders. > > It also looks like we'll need an OSGi specific InvocationLayerFactory to >capture ClassLoader information to pass to our MarshalInputStream then to our >RMIClassLoaderSpi during deserialization at both endpoints. > > We also need to know the bundle (ClassLoader) of the class that calls a >java.lang.reflect.Proxy on the client side, this is actually quite easy to >find, walk the stack, find the Proxy class and obtain the BundleReference / >ClassLoader of the caller. > > Currently the java.lang.reflectProxy dynamically generated subclass instance >proxy's ClassLoader is u
Re: OSGi
Once you realize you need some codebase metadata different than mere list of URLs the next conclusion is that annotations should be something different than... a String :) The next thing to ask is: "what about mixed OSGI and non-OSGI environments" Then you start to realize you need to abstract over the class loading environment itself. Then you start to realize that to support all the scenarios you need to provide a class loading environment that is "pluggable" - ie allows using it with other class loading environments and allow the user to decide which classes should be loaded by which environment. This is what I am working on right now :) Thanks, Michal Peter wrote: My phone sent the previous email before I completed editing. ...If api classes are already loaded locally by client code, then a smart proxy codebase bundle will resolve imports to those packages (if they're within the imported version range), when the proxy bundle is downloaded, resolved and loaded. The strategy should be, deserialize using the callers context until a class is not found, then switch to the object containing the current field being deserialized (which may be a package private implementation class in the service api bundle) and if that fails use the codebase annotation (the smart proxy). This is similar in some ways to never preferred, where locally visible classes will be selected first. The strategy is to let OSGi do all the dependency wiring from bundle manifests. Classes not visible will be visible from a common package import class, except for poorly designed services, which is outside of scope. Only match api version compatible services. No allowances made for split packages or other complexities. If deserialization doesn't succeed, look up another service. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter<j...@zeus.net.au> Sent: 06/02/2017 02:59:09 pm To: dev@river.apache.org<dev@river.apache.org> Subject: Re: OSGi Thanks Nic, If annot You've identified the reason we need an OSGi specific RMIClassLoaderSpi implementation; so we can capture and provide Bundle specific annotation information. Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the context ClassLoader is used by PreferredClassProvider when the ClassLoader argument is null. Standard Java serialization's OIS walks the call stack and selects the first non system classloader (it's looking for the application class loader), it deserializes into the application ClassLoader's context. This doesn't work in OSGi because the application classes are loaded by a multitude of ClassLoaders. It also looks like we'll need an OSGi specific InvocationLayerFactory to capture ClassLoader information to pass to our MarshalInputStream then to our RMIClassLoaderSpi during deserialization at both endpoints. We also need to know the bundle (ClassLoader) of the class that calls a java.lang.reflect.Proxy on the client side, this is actually quite easy to find, walk the stack, find the Proxy class and obtain the BundleReference / ClassLoader of the caller. Currently the java.lang.reflectProxy dynamically generated subclass instance proxy's ClassLoader is used, this is acceptable when the proxy bytecode is loaded by the the Client's ClassLoader or smart proxy ClassLoader in the case where a smart proxy is utilised If the caller changes, so does the calling context. Each bundle provides access to all classes within that bundle, including any public classes from imported packages. Sent from my Samsung device. Include original message Original message From: Niclas Hedhman<nic...@hedhman.org> Sent: 04/02/2017 12:43:28 pm To: dev@river.apache.org Subject: Re: OSGi Further, I think the only "sane" approach in a OSGi environment is to create a new bundle for the Remote environment, all codebases not part of the API goes into that bundle and that the API is required to be present in the OSGi environment a priori. I.e. treat the Remote objects in OSGi as it is treated in plain Java; one classloader, one chunk, sort out its own serialization woes. Likewise for the server; treat it as ordinary RMI, without any mumbo-jambo OSGi stuff to be figured out at a non-OSGi-running JVM. An important difference is that in OSGi, the BundleClassLoader is not (required to be) a URLClassLoader, so the Java serialization's auto annotation of globally reachable URLs won't work, and one need to rely on java.rmi.server.codebase property, but a bundle could watch for loaded bundles and build that up for URLs that can be resolved globally. Cheers -- Niclas Hedhman, Software Developer http://polygeneapache.org<http://zest.apache.org> - New Energy for Java
Re: OSGi
My phone sent the previous email before I completed editing. ...If api classes are already loaded locally by client code, then a smart proxy codebase bundle will resolve imports to those packages (if they're within the imported version range), when the proxy bundle is downloaded, resolved and loaded. The strategy should be, deserialize using the callers context until a class is not found, then switch to the object containing the current field being deserialized (which may be a package private implementation class in the service api bundle) and if that fails use the codebase annotation (the smart proxy). This is similar in some ways to never preferred, where locally visible classes will be selected first. The strategy is to let OSGi do all the dependency wiring from bundle manifests. Classes not visible will be visible from a common package import class, except for poorly designed services, which is outside of scope. Only match api version compatible services. No allowances made for split packages or other complexities. If deserialization doesn't succeed, look up another service. Cheers, Peter. Sent from my Samsung device. Include original message Original message From: Peter <j...@zeus.net.au> Sent: 06/02/2017 02:59:09 pm To: dev@river.apache.org <dev@river.apache.org> Subject: Re: OSGi Thanks Nic, If annot You've identified the reason we need an OSGi specific RMIClassLoaderSpi implementation; so we can capture and provide Bundle specific annotation information. Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the context ClassLoader is used by PreferredClassProvider when the ClassLoader argument is null. Standard Java serialization's OIS walks the call stack and selects the first non system classloader (it's looking for the application class loader), it deserializes into the application ClassLoader's context. This doesn't work in OSGi because the application classes are loaded by a multitude of ClassLoaders. It also looks like we'll need an OSGi specific InvocationLayerFactory to capture ClassLoader information to pass to our MarshalInputStream then to our RMIClassLoaderSpi during deserialization at both endpoints. We also need to know the bundle (ClassLoader) of the class that calls a java.lang.reflect.Proxy on the client side, this is actually quite easy to find, walk the stack, find the Proxy class and obtain the BundleReference / ClassLoader of the caller. Currently the java.lang.reflectProxy dynamically generated subclass instance proxy's ClassLoader is used, this is acceptable when the proxy bytecode is loaded by the the Client's ClassLoader or smart proxy ClassLoader in the case where a smart proxy is utilised If the caller changes, so does the calling context. Each bundle provides access to all classes within that bundle, including any public classes from imported packages. Sent from my Samsung device. Include original message Original message From: Niclas Hedhman <nic...@hedhman.org> Sent: 04/02/2017 12:43:28 pm To: dev@river.apache.org Subject: Re: OSGi Further, I think the only "sane" approach in a OSGi environment is to create a new bundle for the Remote environment, all codebases not part of the API goes into that bundle and that the API is required to be present in the OSGi environment a priori. I.e. treat the Remote objects in OSGi as it is treated in plain Java; one classloader, one chunk, sort out its own serialization woes. Likewise for the server; treat it as ordinary RMI, without any mumbo-jambo OSGi stuff to be figured out at a non-OSGi-running JVM. An important difference is that in OSGi, the BundleClassLoader is not (required to be) a URLClassLoader, so the Java serialization's auto annotation of globally reachable URLs won't work, and one need to rely on java.rmi.server.codebase property, but a bundle could watch for loaded bundles and build that up for URLs that can be resolved globally. Cheers -- Niclas Hedhman, Software Developer http://polygeneapache.org <http://zest.apache.org> - New Energy for Java
Re: OSGi
Thanks Nic, If annot You've identified the reason we need an OSGi specific RMIClassLoaderSpi implementation; so we can capture and provide Bundle specific annotation information. Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the context ClassLoader is used by PreferredClassProvider when the ClassLoader argument is null. Standard Java serialization's OIS walks the call stack and selects the first non system classloader (it's looking for the application class loader), it deserializes into the application ClassLoader's context. This doesn't work in OSGi because the application classes are loaded by a multitude of ClassLoaders. It also looks like we'll need an OSGi specific InvocationLayerFactory to capture ClassLoader information to pass to our MarshalInputStream then to our RMIClassLoaderSpi during deserialization at both endpoints. We also need to know the bundle (ClassLoader) of the class that calls a java.lang.reflect.Proxy on the client side, this is actually quite easy to find, walk the stack, find the Proxy class and obtain the BundleReference / ClassLoader of the caller. Currently the java.lang.reflectProxy dynamically generated subclass instance proxy's ClassLoader is used, this is acceptable when the proxy bytecode is loaded by the the Client's ClassLoader or smart proxy ClassLoader in the case where a smart proxy is utilised. If the caller changes, so does the calling context. Each bundle provides access to all classes within that bundle, including any public classes from imported packages. Sent from my Samsung device. Include original message Original message From: Niclas Hedhman <nic...@hedhman.org> Sent: 04/02/2017 12:43:28 pm To: dev@river.apache.org Subject: Re: OSGi Further, I think the only "sane" approach in a OSGi environment is to create a new bundle for the Remote environment, all codebases not part of the API goes into that bundle and that the API is required to be present in the OSGi environment a priori. I.e. treat the Remote objects in OSGi as it is treated in plain Java; one classloader, one chunk, sort out its own serialization woes. Likewise for the server; treat it as ordinary RMI, without any mumbo-jambo OSGi stuff to be figured out at a non-OSGi-running JVM. An important difference is that in OSGi, the BundleClassLoader is not (required to be) a URLClassLoader, so the Java serialization's auto annotation of globally reachable URLs won't work, and one need to rely on java.rmi.server.codebase property, but a bundle could watch for loaded bundles and build that up for URLs that can be resolved globally. Cheers -- Niclas Hedhman, Software Developer http://polygeneapache.org <http://zest.apache.org> - New Energy for Java
