Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

Hi Peter,

> On 16 Feb 2022, at 10:01, Peter Firmstone  wrote:
> 
> Inline below.
> 
> On 16/02/2022 5:24 pm, Michał Kłeczek wrote:
>> 
>>> On 16 Feb 2022, at 04:25, Peter Firmstone  
>>> wrote:
>>> 
>>> From the CodebaseAccessor service.
>>> 
>>> The CodebaseAccessor proxy (local code) is passed as a parameter along with 
>>> a MarshalledInstance of the proxy, by ProxySerializer to ProxyCodebaseSpi.
>>> 
>>> https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/jgdms-platform/src/main/java/net/jini/export/CodebaseAccessor.java
>>>  
>>> <https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/jgdms-platform/src/main/java/net/jini/export/CodebaseAccessor.java>
>> Ok, so you have introduced a level of indirection to retrieve String 
>> codebase annotations. Why not go one step further and instead of:
>> 
>> interface CodebaseAccessor {
>>   String getClassAnnotation() throws IOException;
>> }
>> 
>> have something along the lines of (conceptually):
>> 
>> interface Codebase extends RemoteMethodControl {
>>   ClassLoader getClassLoader() throws IOException;
>> }
> 
> 
> I personally wouldn't take this step, because CodebaseAccessor is a Remote 
> interface and ClassLoader is a class of high privilege, so it presents a 
> security risk. 

Not really as implementation is constrained by the same security rules as any 
other code: you can constrain it via policy that only grants create ClassLoader 
permission to specific implementations.

>> See above:
>> you can move this code from the client to the service and let it provide the 
>> implementation of class resolution algorithm.
> 
> 
> I would advise against that, the remote service JVM knows little about the 
> client JVM, both are likely to have completely different ClassLoader 
> hierarchies.

To be able to communicate they have to understand each other class loading 
mechanism anyway.
In particular the class annotation String syntax and semantics has to be the 
same for both.

[...]
> 
>>> 
>>> If using OSGi, OSGi will resolve the required dependencies and download 
>>> them if not already present on the client, OSGi will give preference if a 
>>> compatible version of the bundle dependencies already loaded at the client. 
>>>   If using preferred classes the preferred class list will determine the 
>>> order of preference, whether classes are loaded from the proxy codebase or 
>>> the client ClassLoader (the parent loader of the proxy ClassLoader) first.
>> I also tried this route and it is a dead end because
>> 
>> * it is not possible to statically (ie. as part of the software package like 
>> OSGi manifest) provide dependency resolution constraints to be able to 
>> exchange arbitrarily constructed object graphs *
> 
> 
> This is a limitation and compromise I have accepted, JGDMS doesn't attempt to 
> load arbitrarily constructed object graphs, instead it ensures that both 
> endpoints of a Service have the same class resolution view, the same proxy 
> bundle version is used at the Server and client.

This breaks once there are multiple parties (services) involved because it 
means _all_ of them have to have their software versions synchronised in 
advance - which in turn makes the excercise moot: if you have to do that there 
is no need for mobile code anymore.
What is really IMHO needed is a practical way of independent evolution of 
system components.

> At the server, the proxy bundle is depended upon by the service 
> implementation, but  nothing at the client depends upon the proxy bundle 
> loaded there, instead the proxy bundle depends on the api loaded by the 
> client.  That is why JGDMS discourages marshaling of client subclasses that 
> override service method parameter classes, because they cannot be exported as 
> remote objects and are subject to codebase annotation loss and class 
> resolution problems. 

There is no subclassing in my example of RemoteEventSpacePublisher.

> Sometimes, less is more, I've chosen this compromise in this instance to 
> avoid complexity.  I saw little to be gained from the added complexity, it 
> can be worked around by improving service api design and practices.

It cannot. How would you improve RemoteEventListener or JavaSpace API to avoid 
these issues?

>   Instead the client can export method parameter interface types as remote 
> objects, with independent ClassLoader visibility, that will resolve to common 
> super interface types.
> 
> If there is a versioning problem at the client, where it uses an incompatible 
> API with the client, ServiceDiscoveryManager will recognise the service is 
> the 

Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

> On 16 Feb 2022, at 04:25, Peter Firmstone  wrote:
> 
> From the CodebaseAccessor service.
> 
> The CodebaseAccessor proxy (local code) is passed as a parameter along with a 
> MarshalledInstance of the proxy, by ProxySerializer to ProxyCodebaseSpi.
> 
> https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/jgdms-platform/src/main/java/net/jini/export/CodebaseAccessor.java
>  
> 

Ok, so you have introduced a level of indirection to retrieve String codebase 
annotations. Why not go one step further and instead of:

interface CodebaseAccessor {
  String getClassAnnotation() throws IOException;
}

have something along the lines of (conceptually):

interface Codebase extends RemoteMethodControl {
  ClassLoader getClassLoader() throws IOException;
}

The reasoning is:
In principle class annotation is a program in (unspecified) language that is 
executed by the client to create a ClassLoader instance.
You have to make sure all participants share this language and can execute it 
in the same way.
If the type of class annotation is String - it only means this is some kind of 
obscure interpreted scripting language.

The insight here is that we _already have_ the language that we _know_ all 
participants can execute: Java bytecode.
There is no reason not to use it.

>>> 
>> Ok, so the client has to know in advance how to load the service code?
> 
> Yes, the client knows how to load the service code dynamically just prior to 
> proxy unmarshalling, a default ProxyCodebaseSpi is provided for that purpose 
> and it is used by default.

See above:
you can move this code from the client to the service and let it provide the 
implementation of class resolution algorithm.

> 
> If you are using Maven, eg Rio Resolver, or OSGi, then a ProxyCodebaseSpi 
> implementation specific to one of these should be used.  If you have a mixed 
> environment, then you can use the codebase string to determine which to use.

See above: you can also abstract it away behind an interface.

> 
>>> 
 Does it require to have it installed in advance? If so - how?
>>> 
>>> Only JGDMS platform and JERI.
>>> 
>>> 
>> How are service proxy classes loaded then?
> 
> ProxyCodebaseSpi::resolve does this by provisioning a ClassLoader then 
> unmarshalling the proxy into it.
> 
> OSGi:
> 
> https://github.com/pfirmstone/JGDMS/blob/a774a9141e6571f1d7f9771f74b714850d447d3e/JGDMS/jgdms-osgi-proxy-bundle-provider/src/main/java/org/apache/river/osgi/ProxyBundleProvider.java#L131
>  
> 

[...]

> 
>> I am asking about something different - the smart proxy class depends on 
>> _two_ interfaces:
>> 
>> RemoteEventListener <—— proxy class ——> JavaSpace
>> 
>> RemoteEventListener is its service interface.
>> But it is not known in advance what interfaces the client already has:
>> 
>> 1) Just RemoteEventListener
>> 2) Both RemoteEventListener and JavaSpace
>> 3) None
>> 
>> How is class resolution implemented in JGDMS so that it works properly in 
>> _all_ of the above cases?
> 
> This is a responsibility of the underlying platform used for class resolution 
> or modularity.
> 
> If using OSGi, OSGi will resolve the required dependencies and download them 
> if not already present on the client, OSGi will give preference if a 
> compatible version of the bundle dependencies already loaded at the client.   
> If using preferred classes the preferred class list will determine the order 
> of preference, whether classes are loaded from the proxy codebase or the 
> client ClassLoader (the parent loader of the proxy ClassLoader) first.

I also tried this route and it is a dead end because

* it is not possible to statically (ie. as part of the software package like 
OSGi manifest) provide dependency resolution constraints to be able to exchange 
arbitrarily constructed object graphs *

Take for example the RemoteEventSpacePublisher class from my previous email:

interface RemoteEventListener {
}

package net.jini.space;
interface JavaSpace {
}

Bundle manifest:
Import-Package: net.jini.space;version=“[1.0,1.7)”

//implements JavaSpace up to version 1.7
class JavaSpaceImplProxy implements JavaSpace, Serializable {
}

Bundle manifest:
Import-Package: net.jini.space;version=“[1.5,2.0)”

//Requires JavaSpace of at least version 1.5
class RemoteEventSpacePublisher implements RemoteEventListener, Serializable {
  private JavaSpace space;
}


During deserialisation of RemoteEventSpacePublisher the resolution of a bundle 
containing JavaSpace interface has to result in a _single_ bundle that provides 
net.jini.space package obeying _two_ constraints:
Import-Package: net.jini.space;version=“[1.5,2.0)” - from 
RemoteEventSpacePublisher
Import-Package: 

Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

Follow up:

> On 15 Feb 2022, at 13:05, Peter Firmstone  wrote:
> 
> 
>> 
>> - How do you provide the above mentioned JavaSpaceEventPublisher
>> - How would you provide a java.sql.DataSource as a service?
> 
> 
> If you don't have an ObjectEndpoint, then there is no one to authenticate, 
> you only have bytes to de-serialize, so how do you establish trust, ie who 
> did the bytes come from?  However it is possible to have a service that has 
> an ObjectEndpoint and only uses it for authentication of the proxy serializer 
> and codebase provisioning after which the deserialized bytes become objects 
> and don't make remote method calls.  I think that would be an acceptable 
> alternative; someone needs to vouch for the serialized bytes.

You still need to load code to be able to deserialise.
How do you do that in your solution?

> 
> 
>> 
>>> Now the proxy serializer is itself a service (bootstrap proxy), that is 
>>> authenticated when using secure endpoints.  You could quite easily add an 
>>> interface to the proxy serializer to return your object annotation.
>>> 
>>> Note that I use a string, because I also use it in secure multicast 
>>> discovery protocols (typically IPv6), which don't include objects, for 
>>> authentication and provisioning a ClassLoader for a lookup service proxy 
>>> prior to any Object de-serialization.
>>> 
>>> https://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xhtml
>>>  
>>> 
>>> 
>>> Summing up to simplify JGDMS and solve some very difficult issues, it had 
>>> to give up:
>>> 
>>> 1. Support for circular references in serialized object graphs, was
>>>dropped.
>>> 
>> My solution supports any object graph.
> 
> 
> What capability do you need circular references for? 

This is besides the point.
The point is that code loading should be orthogonal to serialisation format an 
whether it supports circular references.

[...]
>> 
>> It would be great to be able to merge the ideas and work on common solution.
>> The question is whether it should be Apache River project…
> 
> 
> I think the PMC has already decided River's fate, and I tend to agree with 
> their decision, the problem is that historically, it hasn't been possible to 
> innovate inside the Apache River project, innovation has been forced to 
> happen elsewhere and it wasn't just what I was doing, there was an attempt to 
> do some container work in River but that also got shut down.  People had 
> trouble in the past agreeing on Rivers direction and there are no currently 
> active developers.   It is still possible to get a group of people together 
> to create an Apache project, but I don't think the code needs it.   github 
> and other sites like it are better for loose collaboration, where developers 
> can feed off each others ideas and innovations and the best solutions survive.

I am with you here.

Michal

Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

> On 15 Feb 2022, at 13:05, Peter Firmstone  wrote:
> 
> 
>> How the client knows the code needed to deserialise?
> The service provides this information, typically in a services configuration,
> 
How is this configuration provided to the client and when?
> by default this is a space separated list of URI, similar to a codebase 
> annotation, but it doesn't have to be.  JERI manages the deserialization of 
> code through a default ProxyCodebaseSpi implementation,
> 
How does the default ProxyCodebaseSpi implementation know where to download 
code from if there are no annotations?
> the client applies constraints, to ensure that input validation is used as 
> well as any other constraints, such as principals, or encryption strength.  
> ProxyCodebaseSpi can be customized by the client, so the client may implement 
> ProxyCodebaseSpi if it wants to do something different, eg use OSGi or Maven 
> to manage dependency resolution.
> 
Ok, so the client has to know in advance how to load the service code?
> 
>> Does it require to have it installed in advance? If so - how?
> Only JGDMS platform and JERI.
> 
> 
> 
How are service proxy classes loaded then?

>> How is the following scenario handled:
>> 
>> class JavaSpaceEventPublisher implements RemoteEventListener, Serializable {
>>   private final JavaSpace space;
>> 
>> //… publish event in JavaSpace implementation
>> }
>> 
>> The smart proxy class has dependencies on RemoteEventListener and on 
>> JavaSpace. How do you properly resolve classes in this case?
> Typically the client has the ServiceAPI it needs already installed locally, 
> however this may not always be the case, depending on how you want to resolve 
> the proxy classes and how much you want to share with the client, you can 
> include additional jar files in the annotation, and use preferred.list or you 
> can use Maven or OSGi to resolve dependencies and provision the ClassLoader 
> used for proxy deserialization. 
> 
I am asking about something different - the smart proxy class depends on _two_ 
interfaces:

RemoteEventListener <—— proxy class ——> JavaSpace

RemoteEventListener is its service interface.
But it is not known in advance what interfaces the client already has:

1) Just RemoteEventListener
2) Both RemoteEventListener and JavaSpace
3) None

How is class resolution implemented in JGDMS so that it works properly in _all_ 
of the above cases?
> …[snip]
> 
> If you want to use non final classes for your service method arguments and 
> allow clients to override these classes, then you will need to enable 
> codebase annotations in AtomicILFactory in your configuration.   The caveat 
> is there is no guarantee, the service will be able to resolve these classes 
> at the server endpoint, or that codebase annotation loss won't occur, it will 
> try using existing mechanisms, such as RMIClassLoaderSPI, which is probably 
> fine for seasoned Jini vets, but not so user friendly for the newbie, who now 
> has to debug ClassNotFoundExceptions.
> 
> 


That’s why I got rid of RMIClassLoaderSPI completely - this is simply a wrong 
API and has to be thrown away.


Michal

Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

Hi Peter,
> JGDMS uses a new implementation of a subset of the Java Serialization’s 
> stream format, with input validation and defenses against malicious data (all 
> connections are first authenticated when using secure endpoints).   Codebase 
> annotations are no longer appended in serialization streams, this feature is 
> deprecated but it can still be enabled. 
> 
How the client knows the code needed to deserialise?
Does it require to have it installed in advance? If so - how?
How is the following scenario handled:

class JavaSpaceEventPublisher implements RemoteEventListener, Serializable {
  private final JavaSpace space;

//… publish event in JavaSpace implementation
}

The smart proxy class has dependencies on RemoteEventListener and on JavaSpace. 
How do you properly resolve classes in this case?

> This paper documents the problems with this approach: 
> https://dl.acm.org/doi/pdf/10./1698139 
> 
> JGDMS provisions a ClassLoader at each Endpoint, the ClassLoader is solely 
> responsible for class resolution, once it has been assigned to the relevant 
> ObjectEndpoint.  A provider mechanism allows customization.
> 
> JGDMS doesn't suffer from codebase annotation loss, nor class resolution 
> issues.   But it did have to give up some functionality; it cannot resolve 
> classes that do not belong to a service proxy or its service api and are not 
> resolvable from the Endpoint ClassLoader, if they are not present on the 
> remote machine.   The solution is to always use a service, for parameters 
> passed to a service, if they are not part of the service api, eg the client 
> overrides the type of parameter arguments for a service.  This means that if 
> the parameter is not an interface, you cannot create a service that 
> implements it and pass it as an argument.  That’s why its still possible, but 
> not recommended to use codebase annotations appended to the serialization 
> stream.  The solution is to create service api that uses only interfaces for 
> parameter arguments.   For example a remote events and listeners use this 
> pattern.  To prevent unexpected breakages, either use interfaces, or final 
> classes, or both, for service api remote method parameters.  Then you won’t 
> get into the situation where you need codebase annotations appended in the 
> stream.
> 

I am not sure I follow but...

What I am trying to achieve is exactly the opposite - place as little 
constraints as possible on service implementors and make the whole thing 
“magically work” :)

> For example if a service proxy is serialized within a serialization stream, 
> it will be replaced by a proxy serializer and it will be assigned its own 
> independent stream, with ClassLoader, independent of the stream in which it 
> was serialized.  This is based on the ObjectEndpoint identity, so it will 
> always resolve to the same ClassLoader.  Note that ProxyCodebaseSpi can be a 
> provider or OSGi service.
> 
Does it mean you cannot provide services that don’t have any ObjectEndpoint 
(ie. local only)?
This would be IMO unacceptable constraint. For example:

- How do you provide the above mentioned JavaSpaceEventPublisher
- How would you provide a java.sql.DataSource as a service?

> Now the proxy serializer is itself a service (bootstrap proxy), that is 
> authenticated when using secure endpoints.  You could quite easily add an 
> interface to the proxy serializer to return your object annotation.
> 
> Note that I use a string, because I also use it in secure multicast discovery 
> protocols (typically IPv6), which don't include objects, for authentication 
> and provisioning a ClassLoader for a lookup service proxy prior to any Object 
> de-serialization.
> 
> https://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xhtml
>  
> 
> Summing up to simplify JGDMS and solve some very difficult issues, it had to 
> give up:
> 
> Support for circular references in serialized object graphs, was dropped.
My solution supports any object graph.
> 
> Extensible classes in service api method parameters are not advised.
Yes - this one is tricky although my solution supports that as well (there are 
edge cases though).
> 
> ProxyTrust - deprecated and replaced with secure authentication and httpmd 
> (SHA-256) or signer certificates using ProxySerializer.
Deprecated in my solution as well: code is validated _before_ execution.
> Untrusted machines are not allowed in a djinn, some level of trust is 
> required, with authentication and authorisation constraints.
Not necessary in my solution.



In general I think the differences are caused by the different perspective:

I see software distribution (ie. mobile code) as orthogonal to networking.
You see River primarily as networking solution with mobile code dependent on it.

It would be great to be able to merge the ideas and 

Re: [DISCUSS] Moving Apache River to the Attic

[Michał Kłeczek]

Hi All,

Based on the excelent work of (mainly) Peter Firmstone (et al) I am personally 
working on “River 4.0” right now and was planning to share my work with the 
community soon(tm).

1. Implementation of my old idea of making codebase annotations objects 
implementing a specific interface (instead of Strings containing space 
delimited list of URLs). One might think of it as “smart codebase annotations” 
being installers of software necessary to deserialise an object.

2. Refactoring of River codebase to make it modular (in terms of JPMS modules).

3. Some cleanup to modernise the codebase (ie. use generics instead of raw 
types, enums, records where possible/useful)

What it gives us is:
a) uniform representation of code and data: there is no need to have separate 
specifications for code metadata languages (manifests etc.). Examples:
- module dependency graph is represented as serialised Java objects
- Expressions specifying module compatibility (such as semantic version ranges) 
are provided as objects implementing a specific interface
- in particular it is possible not to rely on version numbers but have “animal 
sniffers” that decide if local code is compatible with downloaded classes
- installation packages can be provided as serialised objects (either with code 
serialised in-band or out-of-band depending on the needs)

In other words: “everything is an object”.

b) security
To instantiate any class and deserialise an object - installer first makes sure 
code is verified before execution (the verification algorithm is pluggable as 
the installer can be any object implementing an interface).
Installers are treated the same way - the code of the installer has to be 
verified first.
The installer itself grants permissions to the code it installs and is 
constrained by any GrantPermissions it has.

So no more unverified code execution - contrary to old Jini way of 
instantiating an unknown class to ask for a verifier.

c) great flexibility of the way how code is downloaded and installed:
- in particular it is possible to express a module dependency graph that can be 
instantiated in the client VM
- since codebase annotations are objects that are also annotated - it is now 
possible to provide new protocols to download and verify software.

d) since we can now express non-hierarchical module dependencies it is possible 
to implement scenarios not possible (or really cumbersome) to implement 
previously:
- “adapter/wrapper services” - ie. smart proxies that wrap other services 
proxies. For example RemoteEventListener that pushes events to a JavaSpace.



I am only doing it for fun and sentiment to Jini idea of mobile code.
From reading the mailing list and taking part in some discussions here I can 
see there is almost no interest in moving River forward.
And even if there is - there is no clear idea on what direction it should be.

My own opinion on the matter is that the single DEFINING idea of Jini/River is 
mobile code. Without mobile code there is no reason for Jini to exist:
- it is 2022 - there are so many excellent networking libraries on the market 
that River does not bring anything to the table
- web protocols and new developments such as QUIC make River as a networking 
solution (even if excellent) less and less relevant.
- River codebase is outdated and does not play well with the broad Java 
ecosystem (DI frameworks, pluggable logging, JFR etc.)
- looks like “static linking” is the current buzz in the industry (either as 
executables or container images).

On the other hand - having worked quite extensively with 
k8s/Helm/containers/Docker/microservices and what not - I still think there is 
still a (big) case for Jini style mobile code. But only if it is re-designed to 
be secure, robust and flexible.
There is also Project Loom - it will make River scalable and - hopefully - 
relevant again.


Michal


> On 8 Feb 2022, at 01:41, Roy T. Fielding  wrote:
> 
> Hello everyone,
> 
> It's that time of year when I try to figure out what I am doing and
> what I am not, and try to cut back on the stuff that seems unlikely
> to succeed. I suspect the same is true of others.
> 
> I had hoped that more new people at River would result in more activity,
> but that hasn't occurred over the past 9 months and doesn't seem likely
> in the future. Aside from ASF echoes and Infra-driven website replacement,
> there has been nothing to report about River the entire time that I have
> been the chair pro tem, and there hasn't been any chatter by users either.
> 
> Please feel free to let me know if I am missing something.
> 
> If not, I'd like us to accept the reality of this situation and move the
> River project to the Attic. The code will still be available there, and
> folks are welcome to copy it under the license, move it to Github, or
> otherwise seek to re-mold it into a collaborative project wherever is
> most convenient for them.
> 
> Cheers,
> 
> Roy
> 

Re: JEP 411 - end of Apache River?

[Michał Kłeczek]

Yeah, this is not the right place for this rant.
But was curious what the River community thinks about it.

Michał 

> On 30 Apr 2021, at 17:59, Jeremy R. Easton-Marks  
> wrote:
> 
> Hi Michal,
> I think you should definitely share your opinions about this (
> https://bugs.openjdk.java.net/browse/JDK-8264713). I think they may not
> know about Apache River and they probably should address the issues around
> dynamically loaded code. I would cite other Java examples other than just
> River. Android is able to operate without the Java Security Manager and
> maybe the JDK should look into adopting the Android way.
> 
> 
> 
>> On Fri, Apr 30, 2021 at 4:38 AM Michał Kłeczek  wrote:
>> 
>> 
>> 
>>> On 30 Apr 2021, at 04:42, Jeremy R. Easton-Marks <
>> j.r.eastonma...@gmail.com> wrote:
>>> 
>>> I'm not sure how much JEP 411 will impact River. I agree with the overall
>>> direction that it proposes in removing the security manager.
>> 
>> IMHO JEP 411 gives unconvincing reasons to ditch SecurityManager:
>> 
>> - brittle permission model
>> This one is really well summarized in the same paper: The Security Manager
>> does not allow negative permissions that could express "grant permission
>> for all operations except reading files”.
>> 
>> Why not fix that??? Especially that it has been done before:
>> https://github.com/olukas/pro-grade
>> 
>> - poor performance
>> Why not fix that??? Especially that it has been done before by… yes -
>> Apache River (thanks Peter!).
>> 
>> - difficult programming model
>> This one is difficult to sensibly assess without comparing alternatives.
>> Simply speaking: security is difficult - there is no free lunch.
>> 
>> Example from the document (ie. client code needs to be granted permissions
>> required by library code if doPrivileged is not used) has it completely
>> backwards as that’s the whole point of stack walking based security: avoid
>> confused deputee issue.
>> 
>> It also looks like there is another huge misconception in the industry:
>> "running untrusted code on the server is a big no-no and once you only run
>> trusted code SecurityManager is not needed anymore”.
>> The point really is: there is no such a thing as “trusted code” - that’s
>> illusion. See for example: https://news.ycombinator.com/item?id=26087064
>> 
>> Similar misconception: one should avoid dynamically loaded code (or even
>> from the JEP 411 itself: nobody is interested in dynamically loaded code).
>> Well… Docker images _are_ dynamically loaded code.
>> And k8s or Helm yaml descriptors defining what this dynamically loaded
>> code can access are by no means simpler than security policy files. Quite
>> the opposite.
>> 
>> 
>>> However, I am
>>> worried that removing security features is going to cause some other
>>> problems in the Java ecosystem. Beyond the potential impact of River.
>>> 
>>> I do think what affect it will have on Apache River should be explored.
>>> 
>>> I am of the opinion that Apache River should look beyond just being a
>> Java
>>> only project and that we may need to rethink the way that we approach
>>> building distributed systems.
>> 
>> Killing mobile code is killing Apache River as IMHO that’s the defining
>> feature of it.
>> 
>> The question is whether killing SecurityManager is killing mobile code.
>> 
>> Regards,
>> Michal
>> 
>> 
> 
> -- 
> Jeremy R. Easton-Marks
> 
> "être fort pour être utile"

Re: JEP 411 - end of Apache River?

[Michał Kłeczek]

> On 30 Apr 2021, at 04:42, Jeremy R. Easton-Marks  
> wrote:
> 
> I'm not sure how much JEP 411 will impact River. I agree with the overall
> direction that it proposes in removing the security manager.

IMHO JEP 411 gives unconvincing reasons to ditch SecurityManager:

- brittle permission model
This one is really well summarized in the same paper: The Security Manager does 
not allow negative permissions that could express "grant permission for all 
operations except reading files”.

Why not fix that??? Especially that it has been done before:
https://github.com/olukas/pro-grade

- poor performance
Why not fix that??? Especially that it has been done before by… yes - Apache 
River (thanks Peter!).

- difficult programming model
This one is difficult to sensibly assess without comparing alternatives.
Simply speaking: security is difficult - there is no free lunch. 

Example from the document (ie. client code needs to be granted permissions 
required by library code if doPrivileged is not used) has it completely 
backwards as that’s the whole point of stack walking based security: avoid 
confused deputee issue.

It also looks like there is another huge misconception in the industry:
"running untrusted code on the server is a big no-no and once you only run 
trusted code SecurityManager is not needed anymore”.
The point really is: there is no such a thing as “trusted code” - that’s 
illusion. See for example: https://news.ycombinator.com/item?id=26087064

Similar misconception: one should avoid dynamically loaded code (or even from 
the JEP 411 itself: nobody is interested in dynamically loaded code).
Well… Docker images _are_ dynamically loaded code.
And k8s or Helm yaml descriptors defining what this dynamically loaded code can 
access are by no means simpler than security policy files. Quite the opposite.


> However, I am
> worried that removing security features is going to cause some other
> problems in the Java ecosystem. Beyond the potential impact of River.
> 
> I do think what affect it will have on Apache River should be explored.
> 
> I am of the opinion that Apache River should look beyond just being a Java
> only project and that we may need to rethink the way that we approach
> building distributed systems.

Killing mobile code is killing Apache River as IMHO that’s the defining feature 
of it.

The question is whether killing SecurityManager is killing mobile code.

Regards,
Michal

JEP 411 - end of Apache River?

[Michał Kłeczek]

Hi All,

I’ve just learned JEP 411 (https://openjdk.java.net/jeps/411 
) moved to Candidate status.

Does this JEP mean end of mobile code and hence Apache River?

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

There is a huge misunderstanding here.

I am pretty confident I understand and know how River works (at least up 
to version 3.0 exclusive)
And my questions are not about how things are implemented but what 
changes Peter is proposing.
Especially in the context of all the discussions that are held here 
around security, OSGI, serialization etc.


The questions are very detailed and technical - because there is a lot 
of things that are not clear

in the proposed designs.

For example - saying that "deserializing an object is secure because
it is a dynamic proxy" is simply misleading. That is why I ask more 
questions.


You might call it "demanding your time" but it shouldn't stop anyone 
from asking questions.


Cheers,
Michal

Gregg Wonderly wrote:

The important detail is to understand that there is nearly a decade of 
development and design with experiences driving most of that around what exists 
today.  Peter, nor really anyone any longer, can not “answer” all the questions 
you might have in a short amount of time.  I sense that you have a view of how 
things should work, and believe that because that is not the case, that we need 
to “fix it.”   I am not suggesting, nor do I sense Peter is, that there is 
nothing to fix or improve with River.  However, it’s important to understand 
how River was designed to work, and that will require you to study, from 
several angles, the details.  Yes, the code is hard to read.  It doesn’t just 
calculate numbers, or arrange data in collections.  Instead, it is interacting 
with the various details of security, class loading and JVM reflection to 
provide a flexible mechanism for RPC.  I know, that you know, that there are a 
lot of technologies that exist today, which did not exist at the time that 
River was created as Jini.  Instead, people without knowledge of many things 
that already existed to solve their problems went off to make software that 
works and looks the way that they thought it should for RPC or messaging at 
some RPC like level, and now we have a diverse set of technologies which all, 
in the end, allow network based communications to happen.

River’s way of doing it, is but one.  It’s not perfect and it needs work.  
Please understand that Peter and others have ideas for things/changes which 
will improve the user experience of River.  What we are trying to do, is to 
understand your perspective better.  The questions and comments/answers here 
are not going to be very good if you are just demanding our time, and not 
spending your time to learn the details what Peter is pointing out about how 
River works.

Gregg



On Feb 15, 2017, at 1:00 AM, Michał Kłeczek<mic...@kleczek.org>  wrote:

They are valid questions and you haven't answered any of them.
I've described _your_ way of thinking (which I do not agree with).

Apache River has many problems both technical and organizational.
But I find the idea interesting and was expecting openness
for contributions and open discussion.

This is an open source project and there are no obligations to take part in the 
discussion nor answer any questions.
But I find your patronizing statement disincentive to contribute to this 
project - especially that you are one of its main contributors.

Regards,
Michal

Peter wrote:

Finding the answer to this question should assist you to discover answers to 
many of the other questions you've had.

While I've done my best to answer as many of your questions as I can, time is 
limited and I haven't had time to answer all of them or rebutt or confirm all 
arguments /  assumptions.  Sometimes the right questions are more important 
than answers.

Regards,

Peter.

Sent from my Samsung device.
 Include original message
 Original message 
From: Peter<j...@zeus.net.au>
Sent: 15/02/2017 12:58:55 pm
To: dev@river.apache.org<dev@river.apache.org>
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

The PreferredClassLoader will attempt to download the jar file in order to get 
the preferred list.

DownloadPermission should be called DefineClassPermission, I don't think it 
will prevent download of the jar per say.

Why must the bootstrap proxy be loaded by the codebase ClassLoader?

Regards,

Peter.

Sent from my Samsung device.
 Include original message
---- Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 15/02/2017 06:20:37 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

So I've given it some thought and the only explanation I can come up  with is:
1. To create an instance of the bootstrap proxy you need the codebase  
annotation. 2. Codebase annotation is needed because you want the bootstrap 
proxy's  class to be defined in the proper codebase ClassLoader 3. Since you do 
not want to allow any code downloads before placing  constraints on the 
bootstrap proxy - it has to be a dynami

Re: deserialization remote invocation strategy

[Michał Kłeczek]

Reviewing just the source code without any high level overview and 
explanation how and why it is implemented in a particular way

is difficult (if possible at all).

That is why it would be really helpful if the questions asked were answered.

Not only researchers are interested - also potential users and contributors.

Thanks,
Michal

Peter wrote:

I can't make any guarantee that it is secure, but the more people review it, 
the more likelihood bugs and flaws will be identified.

I'm especially interested in security researchers checking it out if they're 
interested.

Cheers,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 15/02/2017 08:04:39 pm
To: dev@river.apache.org
Subject: Re: deserialization remote invocation strategy



  The code actually does what I've described above, but don't take my word for 
it, check it for youself. :)

  If you disagree, don't use it.
It works the other way around - before I decide to use it - I have to 
understand how it works.

Even more so if we are talking about security.

That is why I consider scrutiny and questioning a Good Thing in this 
context.


"Check for yourself" is not an encouraging advice in the area of security :)

Cheers,
Michal

Re: deserialization remote invocation strategy

[Michał Kłeczek]

The code actually does what I've described above, but don't take my word for 
it, check it for youself. :)

If you disagree, don't use it.
It works the other way around - before I decide to use it - I have to 
understand how it works.

Even more so if we are talking about security.

That is why I consider scrutiny and questioning a Good Thing in this 
context.


"Check for yourself" is not an encouraging advice in the area of security :)

Cheers,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

They are valid questions and you haven't answered any of them.
I've described _your_ way of thinking (which I do not agree with).

Apache River has many problems both technical and organizational.
But I find the idea interesting and was expecting openness
for contributions and open discussion.

This is an open source project and there are no obligations to take part 
in the discussion nor answer any questions.
But I find your patronizing statement disincentive to contribute to this 
project - especially that you are one of its main contributors.


Regards,
Michal

Peter wrote:

Finding the answer to this question should assist you to discover answers to 
many of the other questions you've had.

While I've done my best to answer as many of your questions as I can, time is 
limited and I haven't had time to answer all of them or rebutt or confirm all 
arguments /  assumptions.  Sometimes the right questions are more important 
than answers.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Peter<j...@zeus.net.au>
Sent: 15/02/2017 12:58:55 pm
To: dev@river.apache.org<dev@river.apache.org>
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

The PreferredClassLoader will attempt to download the jar file in order to get 
the preferred list.

DownloadPermission should be called DefineClassPermission, I don't think it 
will prevent download of the jar per say.

Why must the bootstrap proxy be loaded by the codebase ClassLoader?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 15/02/2017 06:20:37 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

So I've given it some thought and the only explanation I can come up  
with is: 

1. To create an instance of the bootstrap proxy you need the codebase  
annotation. 
2. Codebase annotation is needed because you want the bootstrap proxy's  
class to be 
defined in the proper codebase ClassLoader 
3. Since you do not want to allow any code downloads before placing  
constraints on the 
bootstrap proxy - it has to be a dynamic proxy. That way its class can  
be defined by the codebase loader 
and yet no code is downloaded 

So the overall sequence is as follows: 
1. Get the codebase annotation and create the codebase loader 
2. Create an instance of a dynamic proxy of a class defined by the  
codebase loader 
3. IMPORTANT - before creating the proxy instance DO NOT grant any  
download permissions 
- that way we are sure the proxy does not triggers any code download and  
execution due 
to it implementing some foreign interfaces 
4. Once the proxy is instantiated - grant its ClassLoader download  
permissions 
5. Place the constraints on the proxy 
6. Invoke a remote method on the proxy 

I understand the whole thing is to make sure the bootstrap proxy 
is defined by the codebase ClassLoader - and the ClassLoader is needed  
to be able to 
dynamically grant download permissions. 

What I DO NOT understand is - why the download permissions are needed at  
all? 
Since the bootstrap proxy's code MUST be local code - why not simply  
have its class 
defined by the context ClassLoader? 
Since downloading code is done only after authentication anyway - I  
don't see the reason to 
use DownloadPermissions at all. 

The only thing that comes to mind is that it is to make sure the service 
is not able to download code from a codebase different than its own. 

Which is OK but redundant. The reasoning is: 
Since the code of the service proxy is already trusted (we have  
established trust before downloading it) - 
we can simply place the constraints on the service proxy that instructs  
it to only download 
code meeting certain criteria. 

Am I correct in my thinking? 

Thanks, 
Michal 


Michał Kłeczek wrote:
  Let me dig some deeper. Comments inline. 
  
  Peter wrote:
  Yes the dynamic proxy's are 100% local code.  Remember dynamic  
  proxy's don't have codebase s. :)
  Of course they do - look at PreferredClassProvider - the dynamic proxy  
  class is defined by the codebase loader! 
  
  Being a dynamic proxy does not mean there is no codebase.
  
  AtomicMarshalInputStream performs a special input validation on  
  java.lang.reflect.Proxy thus ensuring the InvocationHandler is also  
  trusted.  If the InvocationHandler doesn't pass the test the proxy's  
  never created.
  Why does it only verify dynamic proxies? Doesn't it verify normal  
  objects?
  
  The dynamic proxy's put you in direct contact with the service  
  provider using only local code with input validation constrained over  
  secure connections (as configured with constraints in force). 
  
  I think I've given you enough info now to investigate further.
  Ok - so your "token" is the same thing as my Smart

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

So I've given it some thought and the only explanation I can come up 
with is:


1. To create an instance of the bootstrap proxy you need the codebase 
annotation.
2. Codebase annotation is needed because you want the bootstrap proxy's 
class to be

defined in the proper codebase ClassLoader
3. Since you do not want to allow any code downloads before placing 
constraints on the
bootstrap proxy - it has to be a dynamic proxy. That way its class can 
be defined by the codebase loader

and yet no code is downloaded

So the overall sequence is as follows:
1. Get the codebase annotation and create the codebase loader
2. Create an instance of a dynamic proxy of a class defined by the 
codebase loader
3. IMPORTANT - before creating the proxy instance DO NOT grant any 
download permissions
- that way we are sure the proxy does not triggers any code download and 
execution due

to it implementing some foreign interfaces
4. Once the proxy is instantiated - grant its ClassLoader download 
permissions

5. Place the constraints on the proxy
6. Invoke a remote method on the proxy

I understand the whole thing is to make sure the bootstrap proxy
is defined by the codebase ClassLoader - and the ClassLoader is needed 
to be able to

dynamically grant download permissions.

What I DO NOT understand is - why the download permissions are needed at 
all?
Since the bootstrap proxy's code MUST be local code - why not simply 
have its class

defined by the context ClassLoader?
Since downloading code is done only after authentication anyway - I 
don't see the reason to

use DownloadPermissions at all.

The only thing that comes to mind is that it is to make sure the service
is not able to download code from a codebase different than its own.

Which is OK but redundant. The reasoning is:
Since the code of the service proxy is already trusted (we have 
established trust before downloading it) -
we can simply place the constraints on the service proxy that instructs 
it to only download

code meeting certain criteria.

Am I correct in my thinking?

Thanks,
Michal

Michał Kłeczek wrote:

Let me dig some deeper. Comments inline.

Peter wrote:
Yes the dynamic proxy's are 100% local code.  Remember dynamic 
proxy's don't have codebase s. :)
Of course they do - look at PreferredClassProvider - the dynamic proxy 
class is defined by the codebase loader!


Being a dynamic proxy does not mean there is no codebase.


AtomicMarshalInputStream performs a special input validation on 
java.lang.reflect.Proxy thus ensuring the InvocationHandler is also 
trusted.  If the InvocationHandler doesn't pass the test the proxy's 
never created.
Why does it only verify dynamic proxies? Doesn't it verify normal 
objects?


The dynamic proxy's put you in direct contact with the service 
provider using only local code with input validation constrained over 
secure connections (as configured with constraints in force).


I think I've given you enough info now to investigate further.
Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's 
call it a "bootstrap proxy", ok?


1. What interface this bootstrap proxy implements?
2. Why do you think it has to be a dynamic proxy (ie. an instance of a 
subclass of java.lang.Proxy)?
3. What and when are DownloadPermissions required? How do they add to 
the overall security?
I understand the security of service proxies is enforced by the 
constraints placed on the bootstrap proxy.

So where is the place for DownloadPermissions?

4. Finally - how is the lookup service proxy verified? Does it also 
provide the bootstrap proxy?

If so - what special role does it play in the architecture?
The bootstrap proxy does not have to be provided by the lookup 
service, does it?


If it is verified differently - why and how?

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Let me dig some deeper. Comments inline.

Peter wrote:

Yes the dynamic proxy's are 100% local code.  Remember dynamic proxy's don't 
have codebase s. :)
Of course they do - look at PreferredClassProvider - the dynamic proxy 
class is defined by the codebase loader!


Being a dynamic proxy does not mean there is no codebase.


AtomicMarshalInputStream performs a special input validation on 
java.lang.reflect.Proxy thus ensuring the InvocationHandler is also trusted.  
If the InvocationHandler doesn't pass the test the proxy's never created.

Why does it only verify dynamic proxies? Doesn't it verify normal objects?


The dynamic proxy's put you in direct contact with the service provider using 
only local code with input validation constrained over secure connections (as 
configured with constraints in force).

I think I've given you enough info now to investigate further.
Ok - so your "token" is the same thing as my SmartProxyWrapper. Let's 
call it a "bootstrap proxy", ok?


1. What interface this bootstrap proxy implements?
2. Why do you think it has to be a dynamic proxy (ie. an instance of a 
subclass of java.lang.Proxy)?
3. What and when are DownloadPermissions required? How do they add to 
the overall security?
I understand the security of service proxies is enforced by the 
constraints placed on the bootstrap proxy.

So where is the place for DownloadPermissions?

4. Finally - how is the lookup service proxy verified? Does it also 
provide the bootstrap proxy?

If so - what special role does it play in the architecture?
The bootstrap proxy does not have to be provided by the lookup service, 
does it?


If it is verified differently - why and how?

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

It seems like I am missing the high level view on the architecture of 
your software.

Would it be possible for you to write one?

Peter wrote:

There's a new constraint AtomicInputValidation that jeri uses to prevent 
standard serialization being used.

Reggie has been granted DownloadPermission and DeserializationPermission but 
only for its own codebase.

I do not understand this.
How does the client know it is "Reggie" it is talking to?
How Reggie can grant itself permissions in the client environment???

Could you be more precise?

Reggie has also been granted necessary permissions to communicate with its 
service.  No perms have been granted allowing Reggie to do anything else

Reggie gives the client a dynamic proxy token (via SafeServiceRegistrar's 
lookup method).
What is this "token"? Is it a Java object implementing a well known 
interface? What interface is it?


If your class is a client class you can give it the power to open network 
connections and do as you wish, but it's not a good idea to grant that power to 
Reggie.  But it's also worth remembering it's risky to deserialize into 
privileged context and you're going to have to do that in this case.
I do not understand this. I the SmartProxyWrapper there are no 
permissions granted to any downloaded code.
The code can be downloaded and classes defined only if the client 
imposed constraints are met.


SafeServiceRegistrar will still allow you to use your wrapper class, it just 
adds an additional step of allowing you to authenticate that service before you 
deserialize your service wrapper, making it a little safer.

How???
By using the "token" you talk about earlier? But you have to verify the 
token to be secure, don't you?


So you delegate "token" authentication to Reggie, but that requires 
verification of Reggie proxy.

But how do you verify Reggie proxy?

Does Reggie also provide the "token"? If so - how is it authenticated? 
If not - how do you authenticate reggie?
You have to break the chain somewhere - and the shorter the trust chain 
the more secure system is.




You can grant permissions to services only to the codebase certificate signers 
if you want.  Using the url in the permission grant just restricts it more, but 
it's not mandatory.

Fair enough.

The downside is that the permission check requires downloading the jar 
file and reading it to verify the code signers.


In my proposal the check is done even before actual code downloading.

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Comments inline.

Peter wrote:

Some of your assumptions around forbidding code downloading are incorrect, 
which means the other assumptions that rely upon it are also wrong.

Which assumptions?
I know about DownloadPermissions. And while using RequireDlPermProvider 
closes the security vulnerability,

the problem with DownloadPermissions is that they are not flexible enough.
They preclude scenarios like downloading code from BitTorrent or any 
other "locationless" source or

any source for which the client does not have a URL handler installed.

Scalability has to be understood in a broader sense. Not only in terms 
of runtime performance

but also in terms of flexibility and openness of the system for extension.

Contrary to RequireDlPermProvider the use of SmartProxyWrapper is both 
secure and flexible.




In existing River Jini applications, code download can be limited by using 
RequireDlPermProvider, it's not possible for a Serializable class to do so via 
deserialization api.  We must dynamically grant specific download permission 
after authentication, but prior to loading the remote code, this limits scope.  
If we using AtomicSerial we also need to dynamically grant 
DeserializationPermission.


I have an impression that you are only moving the problem around.

Is use of AtomicSerial mandatory? If not then it does not increase 
security in any way.


And secondly:
Based on what information are DownloadPermissions and 
DeserializationPermissions granted?

Only URL of the jar file? See above for why it does not scale.
Any other information would require changing the format of codebase 
annotations (like somehow encoding electronic signature in it).




Downloading smart proxy's directly from each service, instead of Reggie 
significantly reduces the network burden on the lookup service.
This is exactly what my SmartProxyWrapper is doing. The real proxy can 
even be downloaded from a third party service (neither the lookup 
service nor the service itself).
And the bootstrap proxy can be downloaded from whatever place the client 
wants to treat as the lookup service - it might be as well serialized 
base64 encoded TXT record in DNS.


The beauty of it is that it is orthogonal to the lookup service 
interface - the lookup service now is just another service on the 
network - not being special in any way.


Thanks,
Michal

Deserialization vulnerabilities

[Michał Kłeczek]

Right - presented code is just a demonstration of the idea and has to be 
written with all the precautions you mention.


Nevertheless:
Do you have any information about how all these deserialization
vulnerabilities are affected by simply running
the application with SecurityManager?

I know of DoS vulnerability that must be prevented additionally. But other?

Most materials on the web don't even mention SecurityManager
and if they do it is only about using SecurityManager only around 
deserialization code (which is not secure due to finalizers).


I have an impression that the worst mistake a vendor can do is
to allow turning off security at the very beginning
(to "make life of programmers easier"). After that the product
is blamed for being "insecure" while it is simply a matter of turning 
security on.


It happened to MS with Windows (contrary to Unix they made it easy to 
run everything with escalated privileges).

It happened to Java (by making SecurityManager turned off the default).

Thanks,
Michal

Peter wrote:
  
Input validation only works if failure is atomic.  If an instance of SmartProxyWrapper is created, when a CCE is thrown whilst trying to read in ServiceProxyDownloader because the attacker has chosen to deserialize a gadget chain, the attacker potentially wins.


Of course if you implement AtomicExternal and throw a CCE before the object 
superclass method is called, you can prevent your object from being created at 
all.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:55:56 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

It is as simple as:

interface ServiceProxyDownloader extends Remote, RemoteMethodControl {
Object downloadServiceProxy() throws RemoteException;
}

//this class is local to the client - to make it more secure it
//performs full input validation in readExternal
class SmartProxyWrapper implements Externalizable {

private ServiceProxyDownloader proxyDownloader;

//the constraints are NOT read from the stream
//but configured in the client environment
private MethodConstraints downloadConstraints;

public void readExternal(ObjectInput input) {

  //this is safe since no code downloading
  //is done by the ObjectInputStream

  ServiceProxyDownloader noConstraintsProxyDownloader = 
input.readObject();


  proxyDownloader = 
noConstraintsProxyDownloader.setConstraints(downloadConstraints);


}

Object readResolve() throws ObjectStreamException {
  try {
return proxyDownloader.downloadServiceProxy();
  }
  catch (Exception e) {
//handle aproprietly
  }
}

}

Thanks,
Michal

Peter wrote:

  It can impliment AtomicSerial and perform input validation.  How do you get 
from discovery to service without SafeServiceRegistrar lookup?

  How does your wrapper authenticate the service before codebase and smart 
proxy download?

  Sent from my Samsung device.

 Include original message

   Original message ----
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 01:21:46 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

  So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then 
  it is fine?

  Cool - lets do that.

  Thanks,
  Michal

  Peter wrote:

Any Serializable class that implements AtomicSerial can perform input 
validation.

Using one of the secure discovery providers with authentication and input 
validation.  Download and deserialization permissions are granted dynamically 
just after authentication, but before download.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:13:04 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

So your SaveUnmarshallingProxy can do input validation fist as well, 
can't it?


BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a 
secure way?


Thanks,
Michal

Peter wrote:

  I did notice that.

  Are you comnnected to a network and performing deserialization without 
input validation?   Does the secure endpoint allow anon clients?  That is even 
if you are using client certificates does the endpoint allow anon?  Does your 
endpoint allow insecure cyphers?

  Have a look at the changes in JGDMS.

  SafeServiceRegistrar authenticates and performs input validation first.

  Regards,

  Peter.

  Sent from my Samsung device.

 Include original message

  ---- Original message 
  From: Michał Kłeczek<mic...@klecze

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Wouldn't it be easier if you simply forbid code downloading during 
unmarshalling as in SmartProxyWrapper I've shown in another message?
Then u use the unmarshalled bootstrap object to securely download the 
real proxy using existing Jeri implementation.

Then you do not need any advanced discovery packet inspections at all.

What's more - you can apply the same technique to every service proxy - 
so your security does not depend on how secure the lookup service is.


Relying on the lookup service for security does not scale.
If you dream of internet scale Jini it has to work in a similar way to 
current web.
I can contact my bank securely even though an attacker might have taken 
control

over my home router (so both routing and naming service is not secure).
That's because it is built on end-to-end principle: it has to be secure 
even if the

intermediate services (whatever they are) are insecure.

The lookup service is an analogue to DNS. It shouldn't be necessary to 
make it secure

to make the whole system secure.

Thanks,
Michal

Peter wrote:

The certs aren't encoded in the codebase annotation, but sent in packets as 
strings and bytes that are used to reconstruct the certificates during 
discovery.

The certs are also included in the jar file. If Download permission hasn't been 
granted, the classes can't be defined.  DownloadPermission is incorrectly 
named, it should be called DefineClassPermission.

Regards,

Peter.

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

It is as simple as:

interface ServiceProxyDownloader extends Remote, RemoteMethodControl {
  Object downloadServiceProxy() throws RemoteException;
}

//this class is local to the client - to make it more secure it
//performs full input validation in readExternal
class SmartProxyWrapper implements Externalizable {

  private ServiceProxyDownloader proxyDownloader;

  //the constraints are NOT read from the stream
  //but configured in the client environment
  private MethodConstraints downloadConstraints;

  public void readExternal(ObjectInput input) {

//this is safe since no code downloading
//is done by the ObjectInputStream

ServiceProxyDownloader noConstraintsProxyDownloader = 
input.readObject();


proxyDownloader = 
noConstraintsProxyDownloader.setConstraints(downloadConstraints);


  }

  Object readResolve() throws ObjectStreamException {
try {
  return proxyDownloader.downloadServiceProxy();
}
catch (Exception e) {
  //handle aproprietly
}
  }

}

Thanks,
Michal

Peter wrote:

It can impliment AtomicSerial and perform input validation.  How do you get 
from discovery to service without SafeServiceRegistrar lookup?

How does your wrapper authenticate the service before codebase and smart proxy 
download?

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:21:46 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then 
it is fine?

Cool - lets do that.

Thanks,
Michal

Peter wrote:

  Any Serializable class that implements AtomicSerial can perform input 
validation.

  Using one of the secure discovery providers with authentication and input 
validation.  Download and deserialization permissions are granted dynamically 
just after authentication, but before download.

  Regards,

  Peter.

  Sent from my Samsung device.

 Include original message

   Original message 
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 01:13:04 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

  So your SaveUnmarshallingProxy can do input validation fist as well, 
  can't it?


  BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a 
  secure way?


  Thanks,
  Michal

  Peter wrote:

I did notice that.

Are you comnnected to a network and performing deserialization without 
input validation?   Does the secure endpoint allow anon clients?  That is even 
if you are using client certificates does the endpoint allow anon?  Does your 
endpoint allow insecure cyphers?

Have a look at the changes in JGDMS.

SafeServiceRegistrar authenticates and performs input validation first.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message ----
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:42:43 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

I fail to understand how you are more vulnerable because of trusted 
local class that securely downloads code on behalf of a service.


And how in terms of security it is different from your 
SecureServiceRegistrar.


Thanks,
Michal

Peter wrote:
  Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc. 


  JGDMS is as secure as possible with current cyphers, no anon certs, no 
known insecure cyphers (tlsv1.2), input validation during deserialization, 
delayed unmarshalling with authentication.

  I don't see why a compelling reason to give that up for a local class 
with a readResolve method?

  Sorry.

  Regards,

  Peter.
  Sent from my Samsung device.

 Include original message

   Original message 
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 12:14:41 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote 
invocation strategy


  Peter wrote:

In jgdms I've enabled support for https unicast lookup in LookupLocator 
this establishes a connection to a Registrar only, not any service.  This 
functionality doesn't exist in River.

How do you propose establishing a connection using one of these 
endpoints?

  I am not sure I understand the question.
  In exactly the same way how today the connection is established by for 
  example a ProxyTrust instance


  Thanks,
  Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Peter wrote:

The codebase is signed and download permission is granted only to the signed 
codebase.
What is "signed codebase"? How do you encode the signature in the 
codebase annotation?


Codebase of what service?
All of them?

Thanks,
Michal




Sent from my Samsung device.
  
   Include original message

 Original message ----
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:27:09 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

See below.

Peter wrote:

  Using one of the secure discovery providers with authentication and input 
validation.  Download and deserialization permissions are granted dynamically 
just after authentication, but before download.
But now you just moved trust decisions to SafeServiceRegistrar 
implementation.

It is even worse than with "CodeDownloadingSmartProxyWrapper" because
SafeServiceRegistrar implementation classes are dynamically downloaded
while the CodeDownloadingSmartProxyWrapper class is local.

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Both ways have their pros and cons.

But the decision whether there should be one proxy instance or more is 
independent

of the decision on how to resolve the proxy class.
The latter being much more difficult to design properly. So far we have:
1. Use existing PreferredClassLoader infrastructure
This has the issue that you are not reusing any OSGi container class 
loading infrastructure
2. Use OSGi container to install proxy's class bundle - which has other 
problems discusses so far


IMHO there is a third way - but I am still not finished with it :)

Thanks,
Michal

Peter wrote:

My take is a little different, instead services are discovered and registered, 
then the client can utilise them.

The approach you're now suggesting is exactly what Bharath has implented, this 
is a great first effort and I believe it can be improved.  It's only worked 
since River 3.0, I think we can do much more to support OSGi.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<michal@kleczekorg>
Sent: 14/02/2017 01:05:54 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

There are actually two things that we are discussing at the same time:

1. The need to have an "installer" object and how it should be provided 
to the client

2. The algorithm of class loader resolution in OSGi

These two things are orthogonal to each other and your question is about 
point 2.


My take on this is that in OSGi you have to delay the unmarshalling 
decision until a client bundle

asks for a service.
Then you simply create a proxy ClassLoader with the delegation parent 
set to the requesting bundle ClassLoader.
It means that many instances of the service proxy might be present in 
the client environment -
but that is fine since it is no different than deserializing the service 
proxy in two JVMs.


Thanks,
Michal

Peter wrote:

  service providers for each api version can still be loaded.

  But I don't see a way to force all clients to use a single service api 
version without a compatibility layer.  Why not just reload the clients so they 
can use the latest compatible version of a service?

  How does your proposed solution solve this problem?

  Regards,

  Peter.

  Sent from my Samsung device

 Include original message

   Original message ----
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 12:39:40 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

  But what if the client has MULTIPLE clients - each with its own exact 
  API version?


  OSGi handles this case just fine with service trackers.

  Do you want to give up on this?

  Thanks,
  Michal

  Peter wrote:

You can however for each service api package version, it's all in the smart 
proxy bundle manifest.

You are bound by the dependency resolution process, the client can only 
choose from compatible versions.  The service has the power to constrain its 
proxy bundle manifest if it wishes.

Regards,

Peter.

Sent from my Samsung device
  
   Include original message

 Original message 
    From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:24:58 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy


Peter wrote:

  There a multiple remote services, if one client cant obtain a service 
because there is also a later version installed then you need a service that 
doesn't import the later version.  You can still supply another service to 
cater.
This does not scale because you would have to have one service per each 
service interface version any client might require.


No... You have to be able to make this class resolution decision on the 
client.
And if the client VM allows to have many class loading context at the 
same time (as is the case with OSGI) then

the infrastructure has to take care of this resolution.

But you cannot force the service provider to provide separate instance 
for each case.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

See below.

Peter wrote:

Using one of the secure discovery providers with authentication and input 
validation.  Download and deserialization permissions are granted dynamically 
just after authentication, but before download.
But now you just moved trust decisions to SafeServiceRegistrar 
implementation.

It is even worse than with "CodeDownloadingSmartProxyWrapper" because
SafeServiceRegistrar implementation classes are dynamically downloaded
while the CodeDownloadingSmartProxyWrapper class is local.

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

So if the CodeDownloadingSmartProxyWrapper implements AtomicSerial then 
it is fine?

Cool - lets do that.

Thanks,
Michal

Peter wrote:

Any Serializable class that implements AtomicSerial can perform input 
validation.

Using one of the secure discovery providers with authentication and input 
validation.  Download and deserialization permissions are granted dynamically 
just after authentication, but before download.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 01:13:04 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

So your SaveUnmarshallingProxy can do input validation fist as well, 
can't it?


BTW - how does the client unmarshalls SafeServiceRegistrar proxy in a 
secure way?


Thanks,
Michal

Peter wrote:

  I did notice that.

  Are you comnnected to a network and performing deserialization without input 
validation?   Does the secure endpoint allow anon clients?  That is even if you 
are using client certificates does the endpoint allow anon?  Does your endpoint 
allow insecure cyphers?

  Have a look at the changes in JGDMS.

  SafeServiceRegistrar authenticates and performs input validation first.

  Regards,

  Peter.

  Sent from my Samsung device.

 Include original message

   Original message 
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 12:42:43 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

  I fail to understand how you are more vulnerable because of trusted 
  local class that securely downloads code on behalf of a service.


  And how in terms of security it is different from your 
  SecureServiceRegistrar.


  Thanks,
  Michal

  Peter wrote:
Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc. 


JGDMS is as secure as possible with current cyphers, no anon certs, no 
known insecure cyphers (tlsv1.2), input validation during deserialization, 
delayed unmarshalling with authentication.

I don't see why a compelling reason to give that up for a local class with 
a readResolve method?

Sorry.

Regards,

Peter.
Sent from my Samsung device.
  
   Include original message

 Original message ----
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:14:41 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy


Peter wrote:

  In jgdms I've enabled support for https unicast lookup in LookupLocator 
this establishes a connection to a Registrar only, not any service.  This 
functionality doesn't exist in River.

  How do you propose establishing a connection using one of these endpoints?

I am not sure I understand the question.
In exactly the same way how today the connection is established by for 
example a ProxyTrust instance


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

There are actually two things that we are discussing at the same time:

1. The need to have an "installer" object and how it should be provided 
to the client

2. The algorithm of class loader resolution in OSGi

These two things are orthogonal to each other and your question is about 
point 2.


My take on this is that in OSGi you have to delay the unmarshalling 
decision until a client bundle

asks for a service.
Then you simply create a proxy ClassLoader with the delegation parent 
set to the requesting bundle ClassLoader.
It means that many instances of the service proxy might be present in 
the client environment -
but that is fine since it is no different than deserializing the service 
proxy in two JVMs.


Thanks,
Michal

Peter wrote:

service providers for each api version can still be loaded.

But I don't see a way to force all clients to use a single service api version 
without a compatibility layer.  Why not just reload the clients so they can use 
the latest compatible version of a service?

How does your proposed solution solve this problem?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message ----
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:39:40 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

But what if the client has MULTIPLE clients - each with its own exact 
API version?


OSGi handles this case just fine with service trackers.

Do you want to give up on this?

Thanks,
Michal

Peter wrote:

  You can however for each service api package version, it's all in the smart 
proxy bundle manifest.

  You are bound by the dependency resolution process, the client can only 
choose from compatible versions.  The service has the power to constrain its 
proxy bundle manifest if it wishes.

  Regards,

  Peter.

  Sent from my Samsung device

 Include original message

   Original message ----
  From: Michał Kłeczek<mic...@kleczek.org>
  Sent: 14/02/2017 12:24:58 am
  To: dev@river.apache.org
  Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy


  Peter wrote:

There a multiple remote services, if one client cant obtain a service 
because there is also a later version installed then you need a service that 
doesn't import the later version.  You can still supply another service to 
cater.
  This does not scale because you would have to have one service per each 
  service interface version any client might require.


  No... You have to be able to make this class resolution decision on the 
  client.
  And if the client VM allows to have many class loading context at the 
  same time (as is the case with OSGI) then

  the infrastructure has to take care of this resolution.

  But you cannot force the service provider to provide separate instance 
  for each case.


  Thanks,
  Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

I fail to understand how you are more vulnerable because of trusted 
local class that securely downloads code on behalf of a service.


And how in terms of security it is different from your 
SecureServiceRegistrar.


Thanks,
Michal

Peter wrote:
Then you are vulnerable to deserialization gadget attacks, insecure cyphers anon certs etc. 


JGDMS is as secure as possible with current cyphers, no anon certs, no known 
insecure cyphers (tlsv1.2), input validation during deserialization, delayed 
unmarshalling with authentication.

I don't see why a compelling reason to give that up for a local class with a 
readResolve method?

Sorry.

Regards,

Peter.
Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:14:41 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy


Peter wrote:

  In jgdms I've enabled support for https unicast lookup in LookupLocator this 
establishes a connection to a Registrar only, not any service.  This 
functionality doesn't exist in River.

  How do you propose establishing a connection using one of these endpoints?

I am not sure I understand the question.
In exactly the same way how today the connection is established by for 
example a ProxyTrust instance.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

But what if the client has MULTIPLE clients - each with its own exact 
API version?


OSGi handles this case just fine with service trackers.

Do you want to give up on this?

Thanks,
Michal

Peter wrote:

You can however for each service api package version, it's all in the smart 
proxy bundle manifest.

You are bound by the dependency resolution process, the client can only choose 
from compatible versions.  The service has the power to constrain its proxy 
bundle manifest if it wishes.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 14/02/2017 12:24:58 am
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy


Peter wrote:

  There a multiple remote services, if one client cant obtain a service because 
there is also a later version installed then you need a service that doesn't 
import the later version.  You can still supply another service to cater.
This does not scale because you would have to have one service per each 
service interface version any client might require.


No... You have to be able to make this class resolution decision on the 
client.
And if the client VM allows to have many class loading context at the 
same time (as is the case with OSGI) then

the infrastructure has to take care of this resolution.

But you cannot force the service provider to provide separate instance 
for each case.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Peter wrote:

There a multiple remote services, if one client cant obtain a service because 
there is also a later version installed then you need a service that doesn't 
import the later version.  You can still supply another service to cater.
This does not scale because you would have to have one service per each 
service interface version any client might require.


No... You have to be able to make this class resolution decision on the 
client.
And if the client VM allows to have many class loading context at the 
same time (as is the case with OSGI) then

the infrastructure has to take care of this resolution.

But you cannot force the service provider to provide separate instance 
for each case.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Peter wrote:

In jgdms I've enabled support for https unicast lookup in LookupLocator this 
establishes a connection to a Registrar only, not any service.  This 
functionality doesn't exist in River.

How do you propose establishing a connection using one of these endpoints?

I am not sure I understand the question.
In exactly the same way how today the connection is established by for 
example a ProxyTrust instance.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

Nope - not at all. I am only trying to convince you that there is no 
reason to involve ServiceRegistrar or SDM for code downloading.


HOW the class resolution is done - is another story.
I actually tend to think in a similar way to what Niclas said:
Do not use OSGi to load proxy class - create a separate ClassLoader - 
but make sure it delegates to the client bundle ClassLoader for

non-preferred classes.

Thanks,
Michal

Peter wrote:

I changed it to highlite Nic's point that it's not feasible to resolve and 
provision osgi bundle transitive dependencies during deserialization because 
the time taken to do that can be excessive due to NP Complete nature of 
resolution.

It is incompatible with stream codebase annotations.

I think Mic's currently arguing for a solution that relies on resolution and 
provisioning to occur during deserialization and I'm arguing against it.

I'm arguing for a background task that preceeds deserialization of the proxy.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Patricia Shanahan<p...@acm.org>
Sent: 13/02/2017 11:27:27 pm
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

Sorry, I'm trying to find out the meaning of the current subject line. 
I'm not sure when it changed to "OSGi MP Complete".


On 2/12/2017 10:50 PM, Michał Kłeczek wrote:

  Sorry, NP Completness of what?
  I have been the first to mention NP hardness of constraint satisfaction
  problem
  but I am not sure if this is what you are asking about.

  Thanks,
  Michal

  Patricia Shanahan wrote:

  Are you literally claiming NP Completeness, or just using that as an
  analogy for really, really difficult?

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

KerberosEnpoint?
HttpsEndpoint?

Thanks,
Michal

Peter wrote:

How do you establish the secure jeri connection?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 13/02/2017 11:34:45 pm
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

1. The connection can be done using normal (secure) Jeri.
We do not have to verify the installer object since its classes were loaded 
locally and (by definition) are trusted.

2 The attacker cannot instantiate any non-local class. That is the whole point.
Since the "installer" classes must be local, then we can trust the installer to
honor any invocation constraints we place on it. So any code downloads
are secure - in the sense that the client can require 
authentication/integrity/confidentiality etc.

Note that (if necessary) we can apply the same logic recursively - we can provide an 
"installer of an installer"
and still be sure any code download is going to honor the security constraints 
we require.

Thanks,
Michal

Peter wrote:
So this object that you have with a locally installed class is tasked with 
authenticating the remote service, provisioning and resolving a bundle, 
deserializing the smart proxy and registering it with the OSGi service 
registrar in a readResolve or readObject method?

How do you propose the connection from the client to the service established in 
order to enable this to occur?

How do you prevent an attacker from choosing a different class to deserialize?

Regards,

Peter.

Sent from my Samsung device
  
   Include original message

---- Original message 
From: Michał Kłeczek<michal@kleczekorg>
Sent: 13/02/2017 10:07:28 pm
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

Comments inline.

Peter wrote:
  Mic,

  I'm attempting to get my head around your proposal:

  In the case of JERI, the InvocationHandler is part of the smart 
  proxy's serialized state.  A number of smart proxy classes will need 
  to be unmarshalled before the UnmarshallingInvocationHandler is 
  deserialized.


  The smart proxy contains a reference to a dynamic proxy (which sun 
  called the bootstrap proxy) and the dynamic proxy contains a reference 
  to your UnmarshallingInvocationHandler.This means the smart proxy 
  must be unmarshalled first.


  How do you get access to UnmarshallingInvocationHandler without 
  unmarshalling the smart proxy first?


No no - I am saying about wrapping the smart proxy inside another 
object. It can be either a dynamic proxy, or simply an object that 
implements "readResolve" returning the unmarshalled smart proxy.


  More comments inline below.

  On 13/02/2017 6:11 PM, Michał Kłeczek wrote:
  We are talking about the same thing.

  We are turning circles, Peter - all of this has been already discussed.

  1. Yes - you need to resolve bundles in advance (in OSGi it is not 
  possible to do otherwise anyway)

  Agree.
  2. You cannot decide upon the bundle chosen by the container to load 
  the proxy class (the container does the resolution)
  Disagree, nothing in the client depends on the proxy bundle, there's 
  no reason to provision a different version.
  3. The runtime graph of object places additional constraints on the 
  bundle resolution process (to what is specified in bundles' manifests).
  Since you do not have any way to pass these additional constraints to 
  the container - the case is lost.
  Disagree.  The proxy bundle contains a manifest with requirements.  
  The stream has no knowledge of versioning, nor does it need to, there 
  are no additional constraints.  If the service proxy dependencies 
  cannot be resolved, or it doesn't unmarshall, then it will not be 
  registered with the OSGi registry in the client, client code will not 
  discover it and the client will have no knowledge of it's existance 
  except for some logging.


This is totally backwards.
That way no client is able to find any service because there is a 
chicken and egg problem - we do not know the proxy interfaces until the 
proxy's bundle is resolved.


Understand that when you place a bundle identifier in the stream - it is 
equivalent to specifying a Require-Bundle constraint - nothing more 
nothing less.


  Additionally - to explain what I've said before about wrong level of 
  abstraction:


  Your general idea is very similar to mine: have a special object 
  (let's call it installer) that will install software prior to proxy 
  unmarshalling.


  1. For some reason unclear to me you want to constrain the way how 
  this "installer object" is passed only via the route of 
  ServiceRegistrar (as attributes)
  Disagree, I'm not proposing the service have any control over 
  installation at the client, other than

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

Comments inline.

Peter wrote:
N.B Can't see any chicken egg problem.  


If service doesn't resolve to same service api as client, then it isn't 
compatible.  The client isn't interested in incompatible services, only those 
that are compatible  This is just an artifact of the dependency resolution 
process.

But when do you perform resolution?

Lets say you have two client bundles:
Client1 is resolved to depend on package a.b.c ver 1.1.3 from bundle 
a.b.c 1.1.3
Client2 is resolved to depend on package a.b.c ver 1.2.0 from bundle 
a.b.c 1.2.0


Your service supports both clients. How do you resolve it so that it can 
be linked with both clients?


The answer is - you can not. You must create two different instances of 
the service proxy each one

instantiated from a class loaded in a different context.
But you cannot decide upon this context in advance!!! You have to know 
which client you are serving.


No bundle indentifiers are necesary in the stream, the smart proxy ClassLoader 
decides visibility and delegation, not RMIClassLoader.

But what ClassLoader is used to load the smart proxy???


You're attempting to do too many things with one class / object, there's a risk 
that this very powerful class could be leveraged by an attacker, best to break 
up the functionality.

Why do you think that it is handled by one class/object???


Also ServiceDiscoveryManager handles a lot of scenarios that occur with remote 
services, your class wont, there are thousands of lines of code.

Have a look at SDM in JGDMS.

Sorry for sarcasm, but... This is one class/object doing so many things...

Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

1. The connection can be done using normal (secure) Jeri.
We do not have to verify the installer object since its classes were 
loaded locally and (by definition) are trusted.


2. The attacker cannot instantiate any non-local class. That is the 
whole point.
Since the "installer" classes must be local, then we can trust the 
installer to

honor any invocation constraints we place on it. So any code downloads
are secure - in the sense that the client can require 
authentication/integrity/confidentiality etc.


Note that (if necessary) we can apply the same logic recursively - we 
can provide an "installer of an installer"
and still be sure any code download is going to honor the security 
constraints we require.


Thanks,
Michal

Peter wrote:

So this object that you have with a locally installed class is tasked with 
authenticating the remote service, provisioning and resolving a bundle, 
deserializing the smart proxy and registering it with the OSGi service 
registrar in a readResolve or readObject method?

How do you propose the connection from the client to the service established in 
order to enable this to occur?

How do you prevent an attacker from choosing a different class to deserialize?

Regards,

Peter.

Sent from my Samsung device
  
   Include original message

 Original message 
From: Michał Kłeczek<mic...@kleczek.org>
Sent: 13/02/2017 10:07:28 pm
To: dev@river.apache.org
Subject: Re: OSGi NP Complete Was: OSGi - deserialization remote invocation 
strategy

Comments inline.

Peter wrote:

  Mic,

  I'm attempting to get my head around your proposal:

  In the case of JERI, the InvocationHandler is part of the smart 
  proxy's serialized state.  A number of smart proxy classes will need 
  to be unmarshalled before the UnmarshallingInvocationHandler is 
  deserialized.


  The smart proxy contains a reference to a dynamic proxy (which sun 
  called the bootstrap proxy) and the dynamic proxy contains a reference 
  to your UnmarshallingInvocationHandler.This means the smart proxy 
  must be unmarshalled first.


  How do you get access to UnmarshallingInvocationHandler without 
  unmarshalling the smart proxy first?


No no - I am saying about wrapping the smart proxy inside another 
object. It can be either a dynamic proxy, or simply an object that 
implements "readResolve" returning the unmarshalled smart proxy.



  More comments inline below.

  On 13/02/2017 6:11 PM, Michał Kłeczek wrote:

  We are talking about the same thing.

  We are turning circles, Peter - all of this has been already discussed.

  1. Yes - you need to resolve bundles in advance (in OSGi it is not 
  possible to do otherwise anyway)

  Agree.
  2. You cannot decide upon the bundle chosen by the container to load 
  the proxy class (the container does the resolution)
  Disagree, nothing in the client depends on the proxy bundle, there's 
  no reason to provision a different version.
  3. The runtime graph of object places additional constraints on the 
  bundle resolution process (to what is specified in bundles' manifests).
  Since you do not have any way to pass these additional constraints to 
  the container - the case is lost.
  Disagree.  The proxy bundle contains a manifest with requirements.  
  The stream has no knowledge of versioning, nor does it need to, there 
  are no additional constraints.  If the service proxy dependencies 
  cannot be resolved, or it doesn't unmarshall, then it will not be 
  registered with the OSGi registry in the client, client code will not 
  discover it and the client will have no knowledge of it's existance 
  except for some logging.



This is totally backwards.
That way no client is able to find any service because there is a 
chicken and egg problem - we do not know the proxy interfaces until the 
proxy's bundle is resolved.


Understand that when you place a bundle identifier in the stream - it is 
equivalent to specifying a Require-Bundle constraint - nothing more 
nothing less.


  Additionally - to explain what I've said before about wrong level of 
  abstraction:


  Your general idea is very similar to mine: have a special object 
  (let's call it installer) that will install software prior to proxy 
  unmarshalling.


  1. For some reason unclear to me you want to constrain the way how 
  this "installer object" is passed only via the route of 
  ServiceRegistrar (as attributes)
  Disagree, I'm not proposing the service have any control over 
  installation at the client, other than the manifest in the proxy 
  bundle, nor am I proposing using service attributes, or the use of any 
  existing ServiceRegistar methods (see SafeServiceRegistrar link posted 
  earlier).
If you think about it from the higher architectural view - there is no 
difference. It does not really matter what steps are made - important 
thing is that:
a) you have a special object used to download code - this object is 
supposed t

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Comments inline.

Peter wrote:

Mic,

I'm attempting to get my head around your proposal:

In the case of JERI, the InvocationHandler is part of the smart 
proxy's serialized state.  A number of smart proxy classes will need 
to be unmarshalled before the UnmarshallingInvocationHandler is 
deserialized.


The smart proxy contains a reference to a dynamic proxy (which sun 
called the bootstrap proxy) and the dynamic proxy contains a reference 
to your UnmarshallingInvocationHandler.This means the smart proxy 
must be unmarshalled first.


How do you get access to UnmarshallingInvocationHandler without 
unmarshalling the smart proxy first?


No no - I am saying about wrapping the smart proxy inside another 
object. It can be either a dynamic proxy, or simply an object that 
implements "readResolve" returning the unmarshalled smart proxy.




More comments inline below.

On 13/02/2017 6:11 PM, Michał Kłeczek wrote:

We are talking about the same thing.

We are turning circles, Peter - all of this has been already discussed.

1. Yes - you need to resolve bundles in advance (in OSGi it is not 
possible to do otherwise anyway)

Agree.
2. You cannot decide upon the bundle chosen by the container to load 
the proxy class (the container does the resolution)
Disagree, nothing in the client depends on the proxy bundle, there's 
no reason to provision a different version.
3. The runtime graph of object places additional constraints on the 
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to 
the container - the case is lost.
Disagree.  The proxy bundle contains a manifest with requirements.  
The stream has no knowledge of versioning, nor does it need to, there 
are no additional constraints.  If the service proxy dependencies 
cannot be resolved, or it doesn't unmarshall, then it will not be 
registered with the OSGi registry in the client, client code will not 
discover it and the client will have no knowledge of it's existance 
except for some logging.



This is totally backwards.
That way no client is able to find any service because there is a 
chicken and egg problem - we do not know the proxy interfaces until the 
proxy's bundle is resolved.


Understand that when you place a bundle identifier in the stream - it is 
equivalent to specifying a Require-Bundle constraint - nothing more 
nothing less.




Additionally - to explain what I've said before about wrong level of 
abstraction:


Your general idea is very similar to mine: have a special object 
(let's call it installer) that will install software prior to proxy 
unmarshalling.


1. For some reason unclear to me you want to constrain the way how 
this "installer object" is passed only via the route of 
ServiceRegistrar (as attributes)


Disagree, I'm not proposing the service have any control over 
installation at the client, other than the manifest in the proxy 
bundle, nor am I proposing using service attributes, or the use of any 
existing ServiceRegistar methods (see SafeServiceRegistrar link posted 
earlier).
If you think about it from the higher architectural view - there is no 
difference. It does not really matter what steps are made - important 
thing is that:
a) you have a special object used to download code - this object is 
supposed to be of a class installed locally in advance
b) the above object is used to create a ClassLoader that you will use it 
load the actual deserialized object's class


It does not matter where the first object is taken from - be it 
"SafeServiceRegistrar", the stream itself, a JavaSpace or the Moon.


Thanks,
Michal

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

We are talking about the same thing.

We are turning circles, Peter - all of this has been already discussed.

1. Yes - you need to resolve bundles in advance (in OSGi it is not 
possible to do otherwise anyway)
2. You cannot decide upon the bundle chosen by the container to load the 
proxy class (the container does the resolution)
3. The runtime graph of object places additional constraints on the 
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to 
the container - the case is lost.


Additionally - to explain what I've said before about wrong level of 
abstraction:


Your general idea is very similar to mine: have a special object (let's 
call it installer) that will install software prior to proxy unmarshalling.


1. For some reason unclear to me you want to constrain the way how this 
"installer object" is passed only via the route of ServiceRegistrar (as 
attributes)

But why not:

public interface RemoteEventProducer {
  void registerListener(SmartProxyInstaller installer, byte[] 
remoteEventListenerBytes);

}

I cannot see a difference at all and this is why I say that mixing 
ServiceRegistrar and ServiceDiscoveryManager
into it is really mixing levels of abstraction (and does not add 
anything to the solution).


2. If you allow to pass "installer" to unmarshall proxies - then the 
next question is - why do you require doing it explicitly in the 
application code???
THIS is really mixing levels of abstraction. A programmer expects a 
natural programming model (and will not buy into Jini if it does not 
offer such):


registerListener(RemoteEventListener listener);

without any additional complexities that should be solved by the 
infrastructure code.


3. But the above is easy to solve:

class UnmarshallingInvocationHandler implements InvocationHandler {

  private Object unmarshalledProxy;
  private SmartProxyInstaller installer;

  writeObject(...) { write out installer and proxy bytes }
  readObject(...) {read installer then bytes and then unmarshall the proxy}

}

4. But then the next question is why not put it in the object stream 
implementation itself???


Thanks,
Michal


Peter wrote:
Also see the OSGi Enterprise specification, v6, Chapter 136, page 691, 
there's some discussion about the NP-complete nature of dependency 
resolution there as well.


https://www.osgi.org/developer/downloads/release-6/release-6-download/

On 13/02/2017 5:19 PM, Peter wrote:

OSGi Dependency resolution is.

http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html 



Which means if we want to support an OSGi environment properly, we 
may need some time to resolve the dependencies for a smart proxy, 
before deserializing the proxy, rather than downloading the proxy 
dynamically during unmarshalling, it's better to delay unmarshalling 
until the dependencies are resolved, so the client isn't impacted by 
delays.


Cheers,

Peter.

On 13/02/2017 4:50 PM, Michał Kłeczek wrote:

Sorry, NP Completness of what?
I have been the first to mention NP hardness of constraint 
satisfaction problem

but I am not sure if this is what you are asking about.

Thanks,
Michal

Patricia Shanahan wrote:
Are you literally claiming NP Completeness, or just using that as 
an analogy for really, really difficult?

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Sorry, NP Completness of what?
I have been the first to mention NP hardness of constraint satisfaction 
problem

but I am not sure if this is what you are asking about.

Thanks,
Michal

Patricia Shanahan wrote:
Are you literally claiming NP Completeness, or just using that as an 
analogy for really, really difficult?

Re: OSGi NP Complete Was: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

I am sorry but I think that to solve various issues we need to make sure 
fundamentals are right:


1. There is NO such a thing as "reflective non-smart" proxy - EVERY 
proxy is "smart" (even if it is "reflective") - there is an 
InvocationHandler down there, isn't there?
2. Solving this on service discovery level is trying to do it on the 
WRONG level of abstraction. Services DEPEND on class loading - not the 
other way around.
3. What you propose is a partial "solution". Not being able to register 
"smart" event listeners means no custom endpoints for example 
(UDPEndpoint anyone?)
4. Trying to squeeze partial solutions into the framework is IMHO a BIG 
no no.
This is simply creating more code, more maintenance burden and more 
headache for users trying to workaround "edge, unsupported cases".


Please - lets try to come up with the RIGHT solution that is going to 
REALLY fix class loading issues.


Thanks,
Michal

Peter wrote:

In a word, ServiceDiscoveryManager

ServiceDiscoveryManager is the solution.

ServiceDiscoveryManager performs discovery and looks up services from 
registrars based on filters.  ServiceDiscoveryManager then performs 
local filtering.  This allows time for proxy bundles to be installed, 
resolve, started and confirmed type compatible, prior to them being 
made available (via OSGi service registry if you so desire) for client 
use.


The new interfaces that are part of JGDMS that I'd like to see their 
way into River, found here:


https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java 



https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java 

https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java 

https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java 

https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java 



ServiceCodebaseAccessor is also used as part of secure discovery, but 
the codebase string and certs are transferred as primitives over the 
network.


In this case codebase annotations don't need to be included in the 
stream, the JERI endpoints don't need them at all.


How so?

We can use ServiceItemFilter and ProxyPreparer to install, resolve and 
start out proxy codebase, before downloading the proxy.  The 
interfaces listed above allow an array bootstrap proxy's 
(java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar.


Firstly the bootstrap proxy's JERI endpoint will be loaded in the 
ServiceDiscoveryManager's ClassLoader, so after we've retrieved the 
codebase annotation and signers, created a bundle for the proxy, 
resolved it's dependencies (via OSGi resolution and repository 
services), we need to remarshall the bootstrap proxy into a 
MarshalledInstance, then unmarshall it using the ClassLoader of the 
recently started proxy bundle.  Then when we cast the bootstrap proxy 
to ServiceProxyAccessor and retrieve the smart proxy, it will be 
loaded into the same ClassLoader that the bootstrap proxy uses, our 
newly provisioned and loaded bundle (the correct ClassLoader) without 
need to serialize any annotations, then the smart proxy can have 
constraints applied etc and be registered as an OSGi service with the 
OSGi registrar, where client code can interact with the remote proxy.


Now if public Serializable classes that are imported by the proxy's 
bundle (service api) or private classes in the proxy's bundle can be 
deserialized and the JERI endpoint has a reference to the ClassLoader 
of the proxy.


This should be good enough so we don't require the "bundle stack" 
proposed earlier, which also saves the need to explain it and 
simplifies the solution (the intent of the bundle stack was to allow 
deserialization of private classes within other bundles whose packages 
have been imported by the proxy).


The client won't be able to pass a smart proxy to the service (like a 
Listener), but it can still pass a non smart proxy and it will still 
function.  So clients can still export their own remote service 
(albiet without a codebase, excluding smart proxy's), but it'll be 
good enough for a listener etc.


Cheers,

Peter.


On 8/02/2017 1:09 PM, Niclas Hedhman wrote:

Maybe there are some misunderstanding somewhere... see below;

On Wed, Feb 8, 2017 at 3:35 AM, Peter  wrote:

I'm currently only considering OSGi server ->  OSGi client.  Mick's

investigating all four options.

Ok, makes it a lot easier for me to follow.

Not expecting the client calling bundle to resolve everything, hence 
the
stack, so we have the full visibility of the bundle of the class 
that was

last resolved, so we can resolve its fields 

Re: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

So in this case there is no authentication prior to code download?
Or the client has to be authenticated before registering a listener. Am I
correct?

On Tue, 7 Feb 2017 at 22:32, Peter <j...@zeus.net.au> wrote:

> Create three bundles, api (if any looks like your using jini platform
> class as api), service and proxy.  No split packages.
>
> Export it using a Jeri exporter.
>
> It's passed as a listener parameter to a service B ( which is also
> exported using jeri and lets assume sent to the service end) the remote end
> deserializes the smart proxy over it's remote connection, the jeri
> InvocationDespatcher selects the service B implementation as it's
> classloader or BundleReference, but your smart proxy cclasses are not
> present, so the annotation will be used to load the bundle, which will
> resolve to the imported jini api packages from River bundles in the server
> back end.
>
> In other words, the same mechanism is used, a MarshalInputStream
> RMiClassLoaderSpi, InvocationDespatcher instead of an invocation handler
> (although your smart proxy will contain its own reflective proxy with its
> own jeri endpoint and invocation handler and marshal streams) .
>
> Cheers,
>
> Peter.
> Sent from my Samsung device.
>
>   Include original message
>  Original message 
> From: Michał Kłeczek <mic...@kleczek.org>
> Sent: 08/02/2017 06:51:55 am
> To: dev@river.apache.org
> Subject: Re: OSGi - deserialization remote invocation strategy
>
> I think you have misunderstood my question.
> I want my implementation of RemoteEventListener to be a smart proxy.
> How do u support that?
>
> On Tue, 7 Feb 2017 at 20:26, Peter <j...@zeus.net.au> wrote:
>
> > No, it's just an example, no such restriction.  But it is restricted to
> > JERI.
> >
> > RemoteEvents have been extended also in JGDMS to use MarshalledInstance
> > and JErI instead of MarshalledObjects and JRMP.  All methods that rely on
> > MarshalledObject have been deprecated and replacements provided with the
> > exception of Activation constructors.
> >
> > MarshalledInstance has also been made extensible to allow other
> > serialization formats to be supported by the api.
> >
> > But we're diverging...
> >
> > Cheers,
> >
> > Peter.
> >
> > Sent from my Samsung device.
> >
> >   Include original message
> >  Original message 
> > From: Michał Kłeczek <mic...@kleczek.org>
> > Sent: 08/02/2017 06:07:19 am
> > To: dev@river.apache.org
> > Subject: Re: OSGi - deserialization remote invocation strategy
> >
> > Hmm.. I see.
> >
> > So your solution explicitly only handles cases of looking up a service
> > in the ServiceRegistrar.
> >
> > How about smart RemoteEventListeners? They are not published in any
> > lookup service.
> >
> > Thanks,
> > Michal
> >
> > Peter wrote:
> >
>
> > > In the JGDMS fork of River, in addition to Reggie's proxy returing 
> > > unmarshalled service proxy's from the ServiceRegistrar lookup method, 
> > > SafeServiceRegistrar provides a lookup method that returns instances of 
> > > java.lang.reflect.Proxy that implement interfaces to retrieve the 
> > > service, it's attributes and it's codebase uri and signer certs.
> > >
> >
>
> > > The client can authenticate, filter and grant permissions (for 
> > > deserialization and codebase download) before retrieving the service 
> > > proxy using a method call.
> > >
> >
>
> > > In this case the service proxy is obtained by from the reflection proxy 
> > > instead of MarshalledIstance.
> > >
> > > Cheers,
> > >
> > > Peter.
> > >
> > > Sent from my Samsung device.
> > >
> > >Include original message
> > >  Original message 
> > > From: "Michał Kłeczek (XPro Sp z o. o.)"<michal.klec...@xpro.biz>
> > > Sent: 08/02/2017 05:51:07 am
> > > To: dev@river.apache.org
> > > Subject: Re: OSGi - deserialization remote invocation strategy
> > >
> >
>
> > > So I must have misunderstood the part about smart proxies being obtained 
> > > via "reflection proxies" or MarshalledInstances.
> > >
> > > What are these "reflection proxies"?
> > >
> > > Thanks,
> > > Michal
> > >
> > > Peter wrote:
> > > No, no bootstrap objects.
> > >
> > > Cheers,
> > >
> > > Peter.
> > >
> > >
&

Re: OSGi - deserialization remote invocation strategy

[Michał Kłeczek]

Hmm... I see.

So your solution explicitly only handles cases of looking up a service 
in the ServiceRegistrar.


How about smart RemoteEventListeners? They are not published in any 
lookup service.


Thanks,
Michal

Peter wrote:

In the JGDMS fork of River, in addition to Reggie's proxy returing unmarshalled 
service proxy's from the ServiceRegistrar lookup method, SafeServiceRegistrar 
provides a lookup method that returns instances of java.lang.reflect.Proxy that 
implement interfaces to retrieve the service, it's attributes and it's codebase 
uri and signer certs.

The client can authenticate, filter and grant permissions (for deserialization 
and codebase download) before retrieving the service proxy using a method call.

In this case the service proxy is obtained by from the reflection proxy instead 
of MarshalledIstance.

Cheers,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 08/02/2017 05:51:07 am
To: dev@river.apache.org
Subject: Re: OSGi - deserialization remote invocation strategy

So I must have misunderstood the part about smart proxies being obtained via 
"reflection proxies" or MarshalledInstances.

What are these "reflection proxies"?

Thanks,
Michal

Peter wrote:
No, no bootstrap objects.

Cheers,

Peter.



Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 08/02/2017 12:28:50 am
To: dev@river.apache.org
Subject: Re: OSGi - deserialization remote invocation strategy

Are you proposing to provide a bootstrap object that will download some 
meta information prior to class resolution?


How does it differ from simply changing annotations to be those 
"bootstrap objects" instead of Strings?


Thanks,
Michal

Peter wrote:
  Proposed JERI OSGi class loading strategy during deserialization.

  Record caller context - this is the default bundle at the beginning of 
  the stack.  It is obtained by the InvocationHandler on the
  client side.  The InvocationDispatcher on the server side has the 
  calling context of the Remote
  implementation.  The reflection dynamic proxy must be installed in the 
  client's class loader, so the
  InvocationHandler knows exactly what it is, it will be passed to the 
  MarshalInputStream.  Any
  interfaces not found in the client's bundle can be safely shed.  For a 
  smart proxy the reflection proxy will
  be installed in the smart proxy loader.  The smart proxy is obtained 
  either via a reflection proxy or a MarshalledInstance.
  MarshalledInstance also passes in the callers loader to the 
  MarshalInputStream.


  The smart proxy classloader is not a child loader of the clients 
  loader, instead it's a bundle that imports
  service api packages, with a version range that overlaps those already 
  imported by the client.


  Both Invocationhandler and InvocationDispatcher utilise 
  MarshalInputStream and MarshalOutputStream, for marshalling parameters 
  and return values.


  The codebase annotation bundle's manifest contains a list of package 
  imports.


  Do we need to make a list of package imports for every new bundle that 
  we load?
  Do we need to record the wiring and packages and their imports from 
  the remote end?


  I don't think so, the bundles themselves contain this information, I 
  think we just need to keep the view of available classes relevant to 
  the current object being deserialized.


  Codebase Annotations are exact versions!  They need to be to allow the 
  service to ensure the correct proxy codebase is used.  Other proxy 
  codebases will be installed in the client, possibly different 
  versions, but these won't be visible through the resolved 
  dependencies, because the proxy codebases only import packages at the 
  client and OSGi restricts visibility to the current bundle's own 
  classes and any imported packages.
  Instead of appending dependencies to the codebase annotation they'll 
  need be defined in the proxy's bundle manifest.  Of course if an 
  identical version of a proxy codebase bundle is already installed at 
  the client, this will be used again.


  Because a bundle generally imports packages (importing entire bundles 
  is discouraged in OSGi), there may be classes
  that aren't visible from those bundles, such as transient imports, but 
  also including private packages that aren't exported, private
  implementations need to be deserialized, but is it possible to do so 
  safely, without causing package
  conflicts?   Private implementation classes can be used as fields 
  within an exported public object, but cannot and should not
  escape their private scope, doing so risks them being resolved to a 
  bundle with the version of the remote end, instead of the locally 
  resolved / wired package, causing

Re: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

So I must have misunderstood the part about smart proxies being obtained 
via "reflection proxies" or MarshalledInstances.


What are these "reflection proxies"?

Thanks,
Michal

Peter wrote:

No, no bootstrap objects.

Cheers,

Peter.



Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 08/02/2017 12:28:50 am
To: dev@river.apache.org
Subject: Re: OSGi - deserialization remote invocation strategy

Are you proposing to provide a bootstrap object that will download some 
meta information prior to class resolution?


How does it differ from simply changing annotations to be those 
"bootstrap objects" instead of Strings?


Thanks,
Michal

Peter wrote:

  Proposed JERI OSGi class loading strategy during deserialization.

  Record caller context - this is the default bundle at the beginning of 
  the stack.  It is obtained by the InvocationHandler on the
  client side.  The InvocationDispatcher on the server side has the 
  calling context of the Remote
  implementation.  The reflection dynamic proxy must be installed in the 
  client's class loader, so the
  InvocationHandler knows exactly what it is, it will be passed to the 
  MarshalInputStream.  Any
  interfaces not found in the client's bundle can be safely shed.  For a 
  smart proxy the reflection proxy will
  be installed in the smart proxy loader.  The smart proxy is obtained 
  either via a reflection proxy or a MarshalledInstance.
  MarshalledInstance also passes in the callers loader to the 
  MarshalInputStream.


  The smart proxy classloader is not a child loader of the clients 
  loader, instead it's a bundle that imports
  service api packages, with a version range that overlaps those already 
  imported by the client.


  Both Invocationhandler and InvocationDispatcher utilise 
  MarshalInputStream and MarshalOutputStream, for marshalling parameters 
  and return values.


  The codebase annotation bundle's manifest contains a list of package 
  imports.


  Do we need to make a list of package imports for every new bundle that 
  we load?
  Do we need to record the wiring and packages and their imports from 
  the remote end?


  I don't think so, the bundles themselves contain this information, I 
  think we just need to keep the view of available classes relevant to 
  the current object being deserialized.


  Codebase Annotations are exact versions!  They need to be to allow the 
  service to ensure the correct proxy codebase is used.  Other proxy 
  codebases will be installed in the client, possibly different 
  versions, but these won't be visible through the resolved 
  dependencies, because the proxy codebases only import packages at the 
  client and OSGi restricts visibility to the current bundle's own 
  classes and any imported packages.
  Instead of appending dependencies to the codebase annotation they'll 
  need be defined in the proxy's bundle manifest.  Of course if an 
  identical version of a proxy codebase bundle is already installed at 
  the client, this will be used again.


  Because a bundle generally imports packages (importing entire bundles 
  is discouraged in OSGi), there may be classes
  that aren't visible from those bundles, such as transient imports, but 
  also including private packages that aren't exported, private
  implementations need to be deserialized, but is it possible to do so 
  safely, without causing package
  conflicts?   Private implementation classes can be used as fields 
  within an exported public object, but cannot and should not
  escape their private scope, doing so risks them being resolved to a 
  bundle with the version of the remote end, instead of the locally 
  resolved / wired package, causing ClassClassExceptions.


  Initial (naive) first pass strategy of class resolution (for each 
  branch in the serialized object graph)?:
  1.Try current bundle on the stack (which will be the callers 
  bundle if we haven't loaded any new bundles yet).
  2.Then use the package name of a class to determine if the package 
  is loaded by any of the bundles
  referenced by the callers bundle imports (to handle any private 
  implementation packages
  that aren't in the current imports).  Is this a good idea? Or should 
  we go straight to step 3
  and let the framework resolve common classes, what if we use a 
  different version to the
  client's imported bundle?  Should we first compare our bundle 
  annotation to the currently
  imported bundles and select one of those if it's a compatible 
  version?  Yes, this could be an

  application bundle, otherwise goto 3.
  3.Load bundle from annotation (if already loaded, it will be an 
  exact version match).  Place the
  new bundle on top of the bundle stack, remove this bundle from the 
  stack once all fields of
  this object have been deserialized, returning to t

Re: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

Comments inline

Niclas Hedhman wrote:

4. For Server(osgi)+Client(osgi), number of options goes up. In this space,
Paremus has a lot of experience, and perhaps willing to share a bit,
without compromising the secret sauce? Either way, Michal's talk about
"wiring" becomes important and that wiring should possibly be
re-established on the client side. The insistence on "must be exactly the
same version" is to me a reflection of "we haven't cared about version
management before", and I think it may not be in the best interest to load
many nearly identical bundles just because they are a little off, say stuff
like guava, commons-xyz, slf4j and many more common dependencies.
This problem is generally unsolvable because there are contradicting 
requirements here:

1. The need to transfer object graphs of (unknown) classes
2. The need to optimize the number of class versions (and the number of 
ClassLoaders) in the JVM


It might be tempting to do the resolution on the client but it is 
(AFAIR) NP-hard
- the object graph is a set of constraints on possible module (bundle) 
versions. Plus there is a whole
set of constraints originating from the modules installed in the 
container prior to graph deserialization.


So the only good solution for a library is to provide a client with an 
interface to implement:

Module resolve(Module candidate) (or Module[] resolve(Module[] candidates))
and let it decide what to do.



Peter wrote;

This is why the bundle must be given first
attempt to resolve an objects class and rely on the bundle dependency

resolution process.

OSGi must be allowed to wire up dependencies, we must avoid attempting to

make decisions about

compatibility and use the current bundle wires instead (our stack).


Well, not totally sure about that. The 'root object classloader' doesn't
have visibility to serialized objects, and will fail if left to do it all
by itself. And as soon as you delegate to another BundleClassLoader, you
have made the resolution decision, not the framework. Michal's proposal to
transfer the BundleWiring (available in runtime) from the server to the
client, makes it somewhat possible to do the delegation. And to make
matters worse, it is quite common that packages are exported from more than
one bundle, so the question is what is included in the bundleWiring coming
across the wire.
The whole issue with proposals based on the stream itself is the fact 
that to resolve properly
one have to walk the whole graph first to gather all modules and their 
dependencies.


It is much better to simply provide the module graph (wiring) first (at 
the beginning of the stream)

and only after resolution of all the modules - deserialize the objects.

Thanks,
Michal

Re: Changing TCCL during deserialization

[Michał Kłeczek (XPro Sp. z o. o.)]

This is fine for me. I am asking not about one interaction where 
multiple instances of MarshalledInputStreams are used (each with its own 
TCCL)
I am asking about the situation described in another email - that during 
a deserialization using a single instance of the stream the TCCL is changed.


Thanks,
Michal

Gregg Wonderly wrote:

I am not sure about “locked”.  In my example about ServiceUI, imagine that 
there is a common behavior that you ServiceUI hosting environment provides to 
all ServiceUI Components.  It can be that there is a button press or something 
else where an AWTEvent thread is going to take action.  It’s that specific 
thread whose TCCL must be changed, each time, to the codebase of the service 
you are interacting with.  If it calls out the service proxy and that is a 
smart proxy, imagine that the smart proxy might use a different service each 
time, and thats where the TCCL must be set appropriately so that any newly 
created classes are parented by the correct environment in your ServiceUI 
hosting platform.

Gregg

Re: OSGi - deserialization remote invocation strategy

[Michał Kłeczek (XPro Sp. z o. o.)]

Are you proposing to provide a bootstrap object that will download some 
meta information prior to class resolution?


How does it differ from simply changing annotations to be those 
"bootstrap objects" instead of Strings?


Thanks,
Michal

Peter wrote:

Proposed JERI OSGi class loading strategy during deserialization.

Record caller context - this is the default bundle at the beginning of 
the stack.  It is obtained by the InvocationHandler on the
client side.  The InvocationDispatcher on the server side has the 
calling context of the Remote
implementation.  The reflection dynamic proxy must be installed in the 
client's class loader, so the
InvocationHandler knows exactly what it is, it will be passed to the 
MarshalInputStream.  Any
interfaces not found in the client's bundle can be safely shed.  For a 
smart proxy the reflection proxy will
be installed in the smart proxy loader.  The smart proxy is obtained 
either via a reflection proxy or a MarshalledInstance.
MarshalledInstance also passes in the callers loader to the 
MarshalInputStream.


The smart proxy classloader is not a child loader of the clients 
loader, instead it's a bundle that imports
service api packages, with a version range that overlaps those already 
imported by the client.


Both Invocationhandler and InvocationDispatcher utilise 
MarshalInputStream and MarshalOutputStream, for marshalling parameters 
and return values.


The codebase annotation bundle's manifest contains a list of package 
imports.


Do we need to make a list of package imports for every new bundle that 
we load?
Do we need to record the wiring and packages and their imports from 
the remote end?


I don't think so, the bundles themselves contain this information, I 
think we just need to keep the view of available classes relevant to 
the current object being deserialized.


Codebase Annotations are exact versions!  They need to be to allow the 
service to ensure the correct proxy codebase is used.  Other proxy 
codebases will be installed in the client, possibly different 
versions, but these won't be visible through the resolved 
dependencies, because the proxy codebases only import packages at the 
client and OSGi restricts visibility to the current bundle's own 
classes and any imported packages.
Instead of appending dependencies to the codebase annotation they'll 
need be defined in the proxy's bundle manifest.  Of course if an 
identical version of a proxy codebase bundle is already installed at 
the client, this will be used again.


Because a bundle generally imports packages (importing entire bundles 
is discouraged in OSGi), there may be classes
that aren't visible from those bundles, such as transient imports, but 
also including private packages that aren't exported, private
implementations need to be deserialized, but is it possible to do so 
safely, without causing package
conflicts?   Private implementation classes can be used as fields 
within an exported public object, but cannot and should not
escape their private scope, doing so risks them being resolved to a 
bundle with the version of the remote end, instead of the locally 
resolved / wired package, causing ClassClassExceptions.


Initial (naive) first pass strategy of class resolution (for each 
branch in the serialized object graph)?:
1.Try current bundle on the stack (which will be the callers 
bundle if we haven't loaded any new bundles yet).
2.Then use the package name of a class to determine if the package 
is loaded by any of the bundles
referenced by the callers bundle imports (to handle any private 
implementation packages
that aren't in the current imports).  Is this a good idea? Or should 
we go straight to step 3
and let the framework resolve common classes, what if we use a 
different version to the
client's imported bundle?  Should we first compare our bundle 
annotation to the currently
imported bundles and select one of those if it's a compatible 
version?  Yes, this could be an

application bundle, otherwise goto 3.
3.Load bundle from annotation (if already loaded, it will be an 
exact version match).  Place the
new bundle on top of the bundle stack, remove this bundle from the 
stack once all fields of
this object have been deserialized, returning to the previous bundle 
context.  We are relying
on the current bundle to wire itself up to the same package versions 
of the clients bundle
imports, for shared classes.  Classes that use different bundles will 
not be visible to the client,

but will need to be visible to the current object's bundle.
4.Place a bundle reference on the stack when a new object is 
deserialized from the stream and
remove it once all fields have been deserialized. (we might need to 
remember stack depth).
5.Don't place non bundle references on the stack.  For example 
system class loader or any
other class loader, we want resolution to occur via the OSGi 
resolution process.


What about a simpler strategy (again naive), where 

Re: Changing TCCL during deserialization

[Michał Kłeczek (XPro Sp. z o. o.)]

I am not sure how OSGI relates to this question. But I can imagine the 
situation like this:


class MySmartAssWrappingObject implements Serializable {

  Object myMember;
...

private void readObject(ObjectInputStream ois) {
  
Thread.currentThread().setContextClassLoader(getClass().getClassLoader());

  myMember = ois.readObject();
}

}

That would allow you to do something similar to what you wanted to do 
with class resolution by remembering the stack of class loaders.


So my question is:
is it something that people do?

Thanks,
Michal

Peter wrote:
  
In PreferredClassProvider, no the callers ClassLoader (context) is the parent ClassLoader of the codebase loader.


It depends on the ClassLoader hierarchy and chosen strategy used to resolve 
annotations.

But the index key for PreferrefClassProvider is  URI[] and parent loader 
(callers loader).

This strategy allows codebases to be duplicated for different calling context.

OSGi however, only loads one Bundle per URL, but as Bharath has demonstrated, 
the codebase loader doesn't have to be a BundleReference.

There are some caveats if the proxy codebase loader isn't a BundleReference, 
one is your dependencies aren't version managed for you, and you can only see 
public classes imported by the parent BundleReference.

The strategy of switching context wouldn't work with PreferredClassProvider.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 07/02/2017 07:20:59 am
To: dev@river.apache.org
Subject: Re: Changing TCCL during deserialization

This still does not answer my question - maybe I am not clear enough.
Do you have a need to set a TCCL DURING a remote call that is in progress?
Ie. you execute a remote call and DURING deserialization of the return value 
you change the TCCL (so one class is resolved using one context loader and 
another using a different one when reading THE SAME stream)

Thanks,
Michal

Gregg Wonderly wrote:
Anytime that a thread might end up being the one to download code, you need 
that threads CCL to be set.   The AWTEvent thread(s) in particular are a 
sticking point.  I have a class which I use for managing threading in 
AWT/Swing.  It’s called ComponentUpdateThread.  It works as follows.

new ComponentUpdateThread<List>( itemList, actionButton1, actionButton2, 
checkbox1 ) {
public void setup() {
// In event thread
setBusyCursorOn( itemList );
}
public Listconstruct() {
try {
return service.getListOfItems( filterParm1 );
} catch( Exception ex ) {
reportException(ex);
}
return null;
}
public void finished() {
List  let;
if( (lst = get()) != null ) {
itemList.getModel().setContents( lst );
}
}
}.start();

This class will make the passed components disabled to keep them from being 
clicked on again, setup for processing use a non AWTEvent thread for getting 
data with other components of the UI still working, and finally mark the 
disabled components back to enabled, and load the list with the returned items, 
if there where any returned.

There is the opportunity for 3 or more threads to be involved here.  First, 
there is the calling thread.  It doesn’t do anything but start the work.  Next, 
there is an AWTEvent thread which will invoke setup().  Next there is a worker 
thread which will invoke construct().   Finally, there is (possible another) 
AWTEventThread which will invoke finished().

In total there could be up to four different threads involved, all of which 
must have TCCL set to the correct class loader.  My convention in the 
implementation, is that that will be this.getClass()getClassLoader().

This is all managed inside of the implementation of ComponentUpdateThread so 
that I don’t have to worry about it, any more.  But it’s important to 
understand that if you don’t do that, then the classes that the calling thread 
can resolve, and Item in this specific case in particular, and you would thus 
potentially see Item come from another class loader than you intended (the 
services class loader with “null” as the parent), and this will result in 
either a CNFE or CCE.

Gregg

On Feb 6, 2017, at 11:28 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

What I was specifically asking for is whether this is needed during 
deserialization or after deserialization.

In other words - if I can lock the TCCL to an instance of MarshalInputStream 
existing for the duration of a single remote call.

Thanks,
Michal

Gregg Wonderly wrote:
The predominant place where it is needed is when you download a serviceUI 
component from a proxy service which just advertises some kind of “browsing”

Re: Changing TCCL during deserialization

[Michał Kłeczek (XPro Sp. z o. o.)]

This still does not answer my question - maybe I am not clear enough.
Do you have a need to set a TCCL DURING a remote call that is in progress?
Ie. you execute a remote call and DURING deserialization of the return 
value you change the TCCL (so one class is resolved using one context 
loader and another using a different one when reading THE SAME stream)


Thanks,
Michal

Gregg Wonderly wrote:

Anytime that a thread might end up being the one to download code, you need 
that threads CCL to be set.   The AWTEvent thread(s) in particular are a 
sticking point.  I have a class which I use for managing threading in 
AWT/Swing.  It’s called ComponentUpdateThread.  It works as follows.

new ComponentUpdateThread<List>( itemList, actionButton1, actionButton2, 
checkbox1 ) {
public void setup() {
// In event thread
setBusyCursorOn( itemList );
}
public Listconstruct() {
try {
return service.getListOfItems( filterParm1 );
} catch( Exception ex ) {
reportException(ex);
}
return null;
}
public void finished() {
List  let;
if( (lst = get()) != null ) {
itemList.getModel().setContents( lst );
}
}
}.start();

This class will make the passed components disabled to keep them from being 
clicked on again, setup for processing use a non AWTEvent thread for getting 
data with other components of the UI still working, and finally mark the 
disabled components back to enabled, and load the list with the returned items, 
if there where any returned.

There is the opportunity for 3 or more threads to be involved here.  First, 
there is the calling thread.  It doesn’t do anything but start the work.  Next, 
there is an AWTEvent thread which will invoke setup().  Next there is a worker 
thread which will invoke construct().   Finally, there is (possible another) 
AWTEventThread which will invoke finished().

In total there could be up to four different threads involved, all of which 
must have TCCL set to the correct class loader.  My convention in the 
implementation, is that that will be this.getClass().getClassLoader().

This is all managed inside of the implementation of ComponentUpdateThread so 
that I don’t have to worry about it, any more.  But it’s important to 
understand that if you don’t do that, then the classes that the calling thread 
can resolve, and Item in this specific case in particular, and you would thus 
potentially see Item come from another class loader than you intended (the 
services class loader with “null” as the parent), and this will result in 
either a CNFE or CCE.

Gregg


On Feb 6, 2017, at 11:28 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

What I was specifically asking for is whether this is needed during 
deserialization or after deserialization.

In other words - if I can lock the TCCL to an instance of MarshalInputStream 
existing for the duration of a single remote call.

Thanks,
Michal

Gregg Wonderly wrote:

The predominant place where it is needed is when you download a serviceUI 
component from a proxy service which just advertises some kind of “browsing” 
interface to find specific services and interact with them, and that serviceUI 
is embedded in another application with it’s own codebase

appl->serviceUI-for-browsing->Service-to-use->That-Services-ServiceUI

In this case, TCCL must be set to the serviceui classes classloader so that the 
“serviceui-for-browsing” will have a proper parent class pointer.

Anytime that downloaded code might download more code, it should always set 
TCCL to its own class loader so that the classes it downloads reflect against 
the existing class definitions.

Gregg


On Feb 6, 2017, at 12:03 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  <mailto:michal.klec...@xpro.biz>  wrote:

Hi,

During my work on object based annotations I realized it would be more efficient not to 
look for TCCL upon every call to "load class" (when default loader does not 
match the annotation).
It might be more effective to look it up upon stream creation and using it 
subsequently for class loader selection.

But this might change semantics of deserialization a little bit - it would not 
be possible to change the context loader during deserialization.
My question is - are there any scenarios that require that?
I cannot think of any but...

Thanks,
Michal

Re: Changing TCCL during deserialization

[Michał Kłeczek (XPro Sp. z o. o.)]

What I was specifically asking for is whether this is needed during 
deserialization or after deserialization.


In other words - if I can lock the TCCL to an instance of 
MarshalInputStream existing for the duration of a single remote call.


Thanks,
Michal

Gregg Wonderly wrote:

The predominant place where it is needed is when you download a serviceUI 
component from a proxy service which just advertises some kind of “browsing” 
interface to find specific services and interact with them, and that serviceUI 
is embedded in another application with it’s own codebase

appl->serviceUI-for-browsing->Service-to-use->That-Services-ServiceUI

In this case, TCCL must be set to the serviceui classes classloader so that the 
“serviceui-for-browsing” will have a proper parent class pointer.

Anytime that downloaded code might download more code, it should always set 
TCCL to its own class loader so that the classes it downloads reflect against 
the existing class definitions.

Gregg


On Feb 6, 2017, at 12:03 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

Hi,

During my work on object based annotations I realized it would be more efficient not to 
look for TCCL upon every call to "load class" (when default loader does not 
match the annotation).
It might be more effective to look it up upon stream creation and using it 
subsequently for class loader selection.

But this might change semantics of deserialization a little bit - it would not 
be possible to change the context loader during deserialization.
My question is - are there any scenarios that require that?
I cannot think of any but...

Thanks,
Michal

Re: AbstractILFactory bug?

[Michał Kłeczek (XPro Sp. z o. o.)]

I'm talking about this:
Util.checkPackageAccess(interfaces[i].getClass()); //NOTE the getClass() 
here!!!


It should be:
Util.checkPackageAccess(interfaces[i]);

Michal

Michał Kłeczek (XPro Sp. z o. o.) wrote:

I understand the check is needed.

It is that we are not checking the right package but "java.lang"

Thanks,
Michal

Peter wrote:
Ok, worked out why, java.lang.reflect.Proxy's newProxyInstance 
permission check  is caller sensitive.  In this case 
AbstractILFactory is the caller, so not checking it would allow an 
attacker to bypass the check using AbstractILFactory.

Cheers,

Peter.

Sent from my Samsung device.
 Include original message
 Original message ----
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michalklec...@xpro.biz>
Sent: 06/02/2017 05:06:32 pm
To: dev@river.apache.org
Subject: AbstractILFactory bug?

I have just found this piece of code in AbstractILFactory:

Class[] interfaces = getProxyInterfaces(impl);
...
for (int i = 0; i<  interfaces.length; i++) {
  Util.checkPackageAccess(interfaces[i].getClass());
}

So we check "java.lang" package access.

A bug?

Thanks,
Michal

Re: AbstractILFactory bug?

[Michał Kłeczek (XPro Sp. z o. o.)]

I understand the check is needed.

It is that we are not checking the right package but "java.lang"

Thanks,
Michal

Peter wrote:
Ok, worked out why, java.lang.reflect.Proxy's newProxyInstance permission check  is caller sensitive.  In this case AbstractILFactory is the caller, so not checking it would allow an attacker to bypass the check using AbstractILFactory. 


Cheers,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message ----
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michalklec...@xpro.biz>
Sent: 06/02/2017 05:06:32 pm
To: dev@river.apache.org
Subject: AbstractILFactory bug?

I have just found this piece of code in AbstractILFactory:

Class[] interfaces = getProxyInterfaces(impl);
...
for (int i = 0; i<  interfaces.length; i++) {
  Util.checkPackageAccess(interfaces[i].getClass());
}

So we check "java.lang" package access.

A bug?

Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

The upside is that it simplifies the overall architecture.
For example it makes the whole part of River related to proxy trust 
verification obsolete.
All these ProxyTrustIterators executed in an untrusted security context, 
Verifier implementations loaded using a proper ClassLoader etc. - this 
is not needed anymore.


Thanks,
Michal

Michał Kłeczek (XPro Sp. z o. o.) wrote:

Well - times changed since original Jini has been developed.
There is a whole lot of amazing libraries out there - so the 
undertaking is much easier than doing it without them.

I am specifically talking about Google Guava, JBoss Modules and RxJava.

As River is concerned - once you get past the assumption that codebase 
annotations are Strings - it has all the necessary extension points 
available.


I've already started writing the test suite for the thing and hope to 
present it soon.


Thanks,
Michal

Peter wrote:
For the sake of simplicity it's probably best if OSGi and non interact only using reflection proxy's and have their own djinn groups so code downloading is unnecessary between them. 


At least that's how I'd consider introducing it into an existing djinn.

A jvm that doesn't have version management of some sort may have a lot of 
difficulty interacting with services from a framework that can use incompatible 
library versions (and that includes service api) side by side.

My concern is interacting with non versioned env's will probably cause the 
developer to have to continue dealing with the problems the modular framework 
they selected intended solving

Maven and OSGi can probably interact using mvn: codebase annotations, provided 
all modules have bundle manifests.

I still support what your doing and find it interesting and don't wish to 
discourage you, I think you're likely to admit it will be a difficult 
undertaking, but that's probably an attraction right?  Maybe River could 
provide some interfaces for extensibility where you could plug in?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 06/02/2017 03:34:54 pm
To:dev@river.apache.org
Subject: Re: OSGi

Once you realize you need some codebase metadata different than mere 
list of URLs
the next conclusion is that annotations should be something different 
than... a String :)


The next thing to ask is: "what about mixed OSGI and non-OSGI environments"
Then you start to realize you need to abstract over the class loading 
environment itself.


Then you start to realize that to support all the scenarios you need to 
provide a class loading environment that is "pluggable"
- ie allows using it with other class loading environments and allow the 
user to decide which classes should be loaded

by which environment.

This is what I am working on right now :)

Thanks,
Michal

Peter wrote:

  My phone sent the previous email before I completed editing.

  ...If api classes are already loaded locally by client code, then a smart 
proxy codebase bundle will resolve imports to those packages (if they're within 
the imported version range), when the proxy bundle is downloaded, resolved and 
loaded.

  The strategy should be, deserialize using the callers context until a class 
is not found, then switch to the object containing the current field being 
deserialized (which may be a package private implementation class in the 
service api bundle) and if that fails use the codebase annotation (the smart 
proxy).  This is similar in some ways to never preferred, where locally visible 
classes will be selected first.

  The strategy is to let OSGi do all the dependency wiring from bundle 
manifests.  Classes not visible will be visible from a common package import 
class, except for poorly designed services, which is outside of scope.

  Only match api version compatible services.

  No allowances made for split packages or other complexities.

  If deserialization doesn't succeed, look up another service.

  Cheers,

  Peter.

  Sent from my Samsung device.

 Include original message

   Original message 
  From: Peter<j...@zeus.net.au>
  Sent: 06/02/2017 02:59:09 pm
  To:dev@river.apache.org<dev@river.apache.org>
  Subject: Re: OSGi


  Thanks Nic,


  If annot

  You've identified the reason we need an OSGi specific RMIClassLoaderSpi 
implementation; so we can capture and provide Bundle specific annotation 
information.

  Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, 
the context ClassLoader is used by PreferredClassProvider when the ClassLoader 
argument is null.

  Standard Java serialization's OIS walks the call stack and selects the first 
non system classloader (it's looking for the application class loader), it 
deserializes into the application ClassLoader's context.  This doesn't  work in 
OSGi because the application classes are loaded 

AbstractILFactory bug?

[Michał Kłeczek (XPro Sp. z o. o.)]

I have just found this piece of code in AbstractILFactory:

Class[] interfaces = getProxyInterfaces(impl);
...
for (int i = 0; i < interfaces.length; i++) {
Util.checkPackageAccess(interfaces[i].getClass());
}

So we check "java.lang" package access.

A bug?

Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Well - times changed since original Jini has been developed.
There is a whole lot of amazing libraries out there - so the undertaking 
is much easier than doing it without them.

I am specifically talking about Google Guava, JBoss Modules and RxJava.

As River is concerned - once you get past the assumption that codebase 
annotations are Strings - it has all the necessary extension points 
available.


I've already started writing the test suite for the thing and hope to 
present it soon.


Thanks,
Michal

Peter wrote:
For the sake of simplicity it's probably best if OSGi and non interact only using reflection proxy's and have their own djinn groups so code downloading is unnecessary between them. 


At least that's how I'd consider introducing it into an existing djinn.

A jvm that doesn't have version management of some sort may have a lot of 
difficulty interacting with services from a framework that can use incompatible 
library versions (and that includes service api) side by side.

My concern is interacting with non versioned env's will probably cause the 
developer to have to continue dealing with the problems the modular framework 
they selected intended solving

Maven and OSGi can probably interact using mvn: codebase annotations, provided 
all modules have bundle manifests.

I still support what your doing and find it interesting and don't wish to 
discourage you, I think you're likely to admit it will be a difficult 
undertaking, but that's probably an attraction right?  Maybe River could 
provide some interfaces for extensibility where you could plug in?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 06/02/2017 03:34:54 pm
To: dev@river.apache.org
Subject: Re: OSGi

Once you realize you need some codebase metadata different than mere 
list of URLs
the next conclusion is that annotations should be something different 
than... a String :)


The next thing to ask is: "what about mixed OSGI and non-OSGI environments"
Then you start to realize you need to abstract over the class loading 
environment itself.


Then you start to realize that to support all the scenarios you need to 
provide a class loading environment that is "pluggable"
- ie allows using it with other class loading environments and allow the 
user to decide which classes should be loaded

by which environment.

This is what I am working on right now :)

Thanks,
Michal

Peter wrote:

  My phone sent the previous email before I completed editing.

  ...If api classes are already loaded locally by client code, then a smart 
proxy codebase bundle will resolve imports to those packages (if they're within 
the imported version range), when the proxy bundle is downloaded, resolved and 
loaded.

  The strategy should be, deserialize using the callers context until a class 
is not found, then switch to the object containing the current field being 
deserialized (which may be a package private implementation class in the 
service api bundle) and if that fails use the codebase annotation (the smart 
proxy).  This is similar in some ways to never preferred, where locally visible 
classes will be selected first.

  The strategy is to let OSGi do all the dependency wiring from bundle 
manifests.  Classes not visible will be visible from a common package import 
class, except for poorly designed services, which is outside of scope.

  Only match api version compatible services.

  No allowances made for split packages or other complexities.

  If deserialization doesn't succeed, look up another service.

  Cheers,

  Peter.

  Sent from my Samsung device.

 Include original message

   Original message 
  From: Peter<j...@zeus.net.au>
  Sent: 06/02/2017 02:59:09 pm
  To: dev@river.apache.org<dev@river.apache.org>
  Subject: Re: OSGi


  Thanks Nic,


  If annot

  You've identified the reason we need an OSGi specific RMIClassLoaderSpi 
implementation; so we can capture and provide Bundle specific annotation 
information.

  Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, 
the context ClassLoader is used by PreferredClassProvider when the ClassLoader 
argument is null.

  Standard Java serialization's OIS walks the call stack and selects the first 
non system classloader (it's looking for the application class loader), it 
deserializes into the application ClassLoader's context.  This doesn't  work in 
OSGi because the application classes are loaded by a multitude of ClassLoaders.

  It also looks like we'll need an OSGi specific InvocationLayerFactory to 
capture ClassLoader information to pass to our MarshalInputStream then to our 
RMIClassLoaderSpi during deserialization at both endpoints.

  We also need to know the bundle (ClassLoader) of the class that calls a 
java.lang.reflect.Proxy on the client side, this is ac

Changing TCCL during deserialization

[Michał Kłeczek (XPro Sp. z o. o.)]

Hi,

During my work on object based annotations I realized it would be more 
efficient not to look for TCCL upon every call to "load class" (when 
default loader does not match the annotation).
It might be more effective to look it up upon stream creation and using 
it subsequently for class loader selection.


But this might change semantics of deserialization a little bit - it 
would not be possible to change the context loader during deserialization.

My question is - are there any scenarios that require that?
I cannot think of any but...

Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Once you realize you need some codebase metadata different than mere 
list of URLs
the next conclusion is that annotations should be something different 
than... a String :)


The next thing to ask is: "what about mixed OSGI and non-OSGI environments"
Then you start to realize you need to abstract over the class loading 
environment itself.


Then you start to realize that to support all the scenarios you need to 
provide a class loading environment that is "pluggable"
- ie allows using it with other class loading environments and allow the 
user to decide which classes should be loaded

by which environment.

This is what I am working on right now :)

Thanks,
Michal

Peter wrote:

My phone sent the previous email before I completed editing.

...If api classes are already loaded locally by client code, then a smart proxy 
codebase bundle will resolve imports to those packages (if they're within the 
imported version range), when the proxy bundle is downloaded, resolved and 
loaded.

The strategy should be, deserialize using the callers context until a class is 
not found, then switch to the object containing the current field being 
deserialized (which may be a package private implementation class in the 
service api bundle) and if that fails use the codebase annotation (the smart 
proxy).  This is similar in some ways to never preferred, where locally visible 
classes will be selected first.

The strategy is to let OSGi do all the dependency wiring from bundle manifests. 
 Classes not visible will be visible from a common package import class, except 
for poorly designed services, which is outside of scope.

Only match api version compatible services.

No allowances made for split packages or other complexities.

If deserialization doesn't succeed, look up another service.

Cheers,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: Peter
Sent: 06/02/2017 02:59:09 pm
To: dev@river.apache.org
Subject: Re: OSGi

  
Thanks Nic,


If annot

You've identified the reason we need an OSGi specific RMIClassLoaderSpi 
implementation; so we can capture and provide Bundle specific annotation 
information.

Rmiclassloaderspi's loadClass method expects a ClassLoader to be passed in, the 
context ClassLoader is used by PreferredClassProvider when the ClassLoader 
argument is null.

Standard Java serialization's OIS walks the call stack and selects the first 
non system classloader (it's looking for the application class loader), it 
deserializes into the application ClassLoader's context.  This doesn't  work in 
OSGi because the application classes are loaded by a multitude of ClassLoaders.

It also looks like we'll need an OSGi specific InvocationLayerFactory to 
capture ClassLoader information to pass to our MarshalInputStream then to our 
RMIClassLoaderSpi during deserialization at both endpoints.

We also need to know the bundle (ClassLoader) of the class that calls a 
java.lang.reflect.Proxy on the client side, this is actually quite easy to 
find, walk the stack, find the Proxy class and obtain the BundleReference / 
ClassLoader of the caller.

Currently the java.lang.reflectProxy dynamically generated subclass instance 
proxy's ClassLoader is used, this is acceptable when the proxy bytecode is 
loaded by the the Client's ClassLoader or smart proxy ClassLoader in the case 
where a smart proxy is utilised



If the caller changes, so does the calling context.


Each bundle provides access to all classes within that bundle, including any 
public classes from imported packages.





Sent from my Samsung device.
  
   Include original message

 Original message 
From: Niclas Hedhman
Sent: 04/02/2017 12:43:28 pm
To: dev@river.apache.org
Subject: Re: OSGi



Further, I think the only "sane" approach in a OSGi environment is to 
create a new bundle for the Remote environment, all codebases not part of 
the API goes into that bundle and that the API is required to be present in 
the OSGi environment a priori. I.e. treat the Remote objects in OSGi as it 
is treated in plain Java; one classloader, one chunk, sort out its own 
serialization woes. Likewise for the server; treat it as ordinary RMI, 
without any mumbo-jambo OSGi stuff to be figured out at a non-OSGi-running 
JVM. An important difference is that in OSGi, the BundleClassLoader is not 
(required to be) a URLClassLoader, so the Java serialization's auto 
annotation of globally reachable URLs won't work, and one need to rely on 
java.rmi.server.codebase property, but a bundle could watch for loaded 
bundles and build that up for URLs that can be resolved globally. 



Cheers 
--  
Niclas Hedhman, Software Developer 
http://polygeneapache.org  - New Energy for Java

Re: Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

It is performant - no doubt about it.
But it is not scalable because your scalability is limited not by 
network speed but the maximum number of threads.


Thanks,
Michal

Peter wrote:

The other catch is that shared mutable state also needs to be synchronized.

Still River 3.0 should be running at close to raw socket speed, it has the 
worlds most scalable security manager and fastest URLClassLoader.  The 
multiplexer in jeri allows 127 different remote objects to share the same 
connection between two endpoints.

Cheers,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 05/02/2017 04:04:03 am
To: dev@river.apache.org
Subject: Re: Serialization issues

You do not have to do any IO in readObject/writeObject.

The fact that you have readObject/writeObject methods means that you are forced 
to do blocking IO.
It is simple:

readObject(...) {
   ois.defaultReadObject();
   //the above line MUST be blocking because
   verifyMyState();
   //this line expects the data to be read
}

Siilarly:

writeObject(ObjectOutputStream oos) {
   oos.writeInt(whateverField);
   //buffers full? You need to block, sorry
   oos.writeObject(...)
}

Thanks,
Michal

Niclas Hedhman wrote:
I am asking what Network I/O you are doing in the readObject/writeObject
methods. Because to me I can't figure out any use-case where that is a
problem...

On Sun, Feb 5, 2017 at 1:14 AM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:

Don't know about other serialization uses but my issue with it is that it
precludes using it in non-blocking IO.
Sorry if I haven't been clear enough.


Thanks,
Michal

Niclas Hedhman wrote:

And what I/O (network I/O I presume) are you doing during the serialization
(without RMI)?

On Sun, Feb 5, 2017 at 12:48 AM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  wrote:


It is not possible to do non-blocking as in "non blocking IO" - meaning -
threads do not block on IO operations.
Just google "C10K problem"

Thanks,
Michal

Niclas Hedhman wrote:

I don't follow. What does readObject/writeObject got to do with blocking or
not? You could spin up executors to do the work in parallel if you so wish.
And why is "something else" less blocking? And what are you doing that is
"blocking" since the "work" is (or should be) CPU only, there is limited
(still) opportunity to do that non-blocking (whatever that would mean in
CPU-only circumstance). Feel free to elaborate... I am curious.



On Sat, Feb 4, 2017 at 8:38 PM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  <michal.klec...@xpro.biz>
  wrote:


Unfortunately due to "writeObject" and "readObject" methods that have to
be handled (to comply with the spec) - it is not possible to
serialize/deserialize in a non-blocking fashion.
So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.

I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

1. Yes, you can buffer the whole object graph as long as it is small enough.
2. In the end threads are just abstraction on top of inherently serial 
machine that produces asynchronous events (CPU with interrupts)

providing a nice programming in languages that do not support monads.
It might be worth investigating http://www.paralleluniverse.co/quasar/ 
for River.


OTOH I have also read papers about handling C10M problem. These guys are 
serious :) .
The general conclusion is that any "general" abstraction (such as 
threads) breaks. Any context switches are no-no.
So you implement the full event (interrupt) driven network stack in user 
space - the architecture somewhat similar to an exokernel.

See https://www.freebsd.org/cgi/man.cgi?query=netmap=4

But we are diverging...
Cheers,
Michal

Niclas Hedhman wrote:

Ok, but assuming that you are not talking about GB-sized object graphs, it
is more an issue with RMI than Serialization, because you can create
non-blocking I/O "on top", just like Jetty has non-blocking I/O "on top" of
the equally blocking Servlet API. Right? I assume that there is a similar
thing in Tomcat, because AFAIK Google AppEngine runs on Tomcat
It is not required (even it is) that the ObjectOutputStream is directly
connected to the underlying OS file descriptor. I am pretty sure that it
would be a mistake trying to re-design all software that writes to a stream
to have a non-blocking design.

Additionally, while looking into this, I came across
https://www.usenix.org/legacy/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/index.html,
which might be to your interest. Not totally relevant, but still an
interesting read.

Cheers

On Sun, Feb 5, 2017 at 2:04 AM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


You do not have to do any IO in readObject/writeObject.

The fact that you have readObject/writeObject methods means that you are
forced to do blocking IO.
It is simple:

readObject(...) {
   ois.defaultReadObject();
   //the above line MUST be blocking because
   verifyMyState();
   //this line expects the data to be read
}

Siilarly:

writeObject(ObjectOutputStream oos) {
   oos.writeInt(whateverField);
   //buffers full? You need to block, sorry
   oos.writeObject(...)

}

Thanks,
Michal

Niclas Hedhman wrote:

I am asking what Network I/O you are doing in the readObject/writeObject
methods. Because to me I can't figure out any use-case where that is a
problem...

On Sun, Feb 5, 2017 at 1:14 AM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  wrote:


Don't know about other serialization uses but my issue with it is that it
precludes using it in non-blocking IO.
Sorry if I haven't been clear enough.


Thanks,
Michal

Niclas Hedhman wrote:

And what I/O (network I/O I presume) are you doing during the serialization
(without RMI)?

On Sun, Feb 5, 2017 at 12:48 AM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  <michal.klec...@xpro.biz>
  wrote:


It is not possible to do non-blocking as in "non blocking IO" - meaning -
threads do not block on IO operations.
Just google "C10K problem"

Thanks,
Michal

Niclas Hedhman wrote:

I don't follow. What does readObject/writeObject got to do with blocking or
not? You could spin up executors to do the work in parallel if you so wish.
And why is "something else" less blocking? And what are you doing that is
"blocking" since the "work" is (or should be) CPU only, there is limited
(still) opportunity to do that non-blocking (whatever that would mean in
CPU-only circumstance). Feel free to elaborate... I am curious.



On Sat, Feb 4, 2017 at 8:38 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>  
<michal.klec...@xpro.biz>  <michal.klec...@xpro.biz>  <michal.klec...@xpro.biz>
  wrote:


Unfortunately due to "writeObject" and "readObject" methods that have to
be handled (to comply with the spec) - it is not possible to
serialize/deserialize in a non-blocking fashion.
So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.

I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

You do not have to do any IO in readObject/writeObject.

The fact that you have readObject/writeObject methods means that you are 
forced to do blocking IO.

It is simple:

readObject(...) {
  ois.defaultReadObject();
  //the above line MUST be blocking because
  verifyMyState();
  //this line expects the data to be read
}

Siilarly:

writeObject(ObjectOutputStream oos) {
  oos.writeInt(whateverField);
  //buffers full? You need to block, sorry
  oos.writeObject(...)
}

Thanks,
Michal

Niclas Hedhman wrote:

I am asking what Network I/O you are doing in the readObject/writeObject
methods. Because to me I can't figure out any use-case where that is a
problem...

On Sun, Feb 5, 2017 at 1:14 AM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


Don't know about other serialization uses but my issue with it is that it
precludes using it in non-blocking IO.
Sorry if I haven't been clear enough.


Thanks,
Michal

Niclas Hedhman wrote:

And what I/O (network I/O I presume) are you doing during the serialization
(without RMI)?

On Sun, Feb 5, 2017 at 12:48 AM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  wrote:


It is not possible to do non-blocking as in "non blocking IO" - meaning -
threads do not block on IO operations.
Just google "C10K problem"

Thanks,
Michal

Niclas Hedhman wrote:

I don't follow. What does readObject/writeObject got to do with blocking or
not? You could spin up executors to do the work in parallel if you so wish.
And why is "something else" less blocking? And what are you doing that is
"blocking" since the "work" is (or should be) CPU only, there is limited
(still) opportunity to do that non-blocking (whatever that would mean in
CPU-only circumstance). Feel free to elaborate... I am curious.



On Sat, Feb 4, 2017 at 8:38 PM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  <michal.klec...@xpro.biz>
  wrote:


Unfortunately due to "writeObject" and "readObject" methods that have to
be handled (to comply with the spec) - it is not possible to
serialize/deserialize in a non-blocking fashion.
So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.

I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: Serialization Formats, Previously: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I cannot disagree with rants about software industry state.

Let's get back to technical solutions to non-technical problems. I am 
interested in providing tools - whether will be used... is a different 
story.


That said...
IMHO Jini - in all its greatness - DID NOT solve the problem of Java 
code mobility in any way.
As has been discussed on this list several time the way how it "solved" 
it is:
- inherently insecure (because object validation is done _after_ code 
execution)
- is not capable of transferring complicated object graphs - hence it 
cannot be used in many different interesting scenarios.


Partial solutions are worse than lack of solutions - they confuse users 
(in our case programmers) and in the end people loose interest.


I am not a big fan of Java containers - be it JEE or any other (OSGI 
included)
The industry seems to understand they are a dead end - escpecially in 
the age of Docker etc - and is moving away from them (not that in a very 
meaningful direction :) ).


I have worked with OSGI for several years and it was a difficult 
relationship :)
Today I prefer simpler solutions: "java -jar 
my-very-complicated-and-important-service.jar" is the way to go.


Thanks,
Michal


Niclas Hedhman wrote:

(I think wrong thread, so to please Peter, I copied it all into here)

Correct, it is not different. But you are missing the point; CONSTRAINTS.
And without constraints, developers are capable of doing these good deeds
(such as your example) and many very poor ones. The knife cuts your meat or
butcher your neighbor... It is all about the constraints, something that
few developers are willing to admit that makes our work better.

As for the "leasable and you have..."; The problem is that you are probably
wrong on that front too, like the OSGi community have learned the hard way.
There are too many ways software entangle classloading. All kinds of shit
"registers itself" in the bowels of the runtime, such as with the
DriverManager, Loggers, URLHandlers or PermGenSpace (might be gone in Java
8). Then add 100s of common libraries that also does a poor job in
releasing "permanent" resources/instances/classes... The stain sticks, but
the smell is weak, so often we can't tell other than memory leak during
class updates.
And why do we have all this mess? IMHO; Lack of constraints, lack of
lifecycle management in "everything Java" (and most languages) and lack of
discipline (something Gregg has, and I plus 5 million other Java devs don't
have). OSGi is not as successful as it "should" (SpringSource gave up)
because it makes the "stain" stink really badly. OSGi introduces
constraints and fails spectacular when we try to break or circumvent them.

River as it currently stands has "solved" Java code mobility, Java leases,
dynamic service registry with query capabilities and much more. But as with
a lot of good technology, the masses don't buy it. The ignorant masses are
now in Peter Deutsch's Fallacies of Distributed Computing territory,
thinking that microservices on JAX-RS is going to save the day (it isn't, I
am rescuing a project out of it now).
Distributed OSGi tried to solve this problem, and AFAICT has serious
problems to work reliably in production environments. What do I learn? This
is hard, but every 5 years we double in numbers, so half the developer
population is inexperienced and repeat the same mistakes again and again.

Sorry for highlighting problems, mostly psychological/philosophical rather
than technological. I don't have the answers, other than; Without
Constraints Technology Fails. And the better the constraints are defined,
the better likelihood that it can succeed.




On Sat, Feb 4, 2017 at 8:59 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


Comments below.

Niclas Hedhman wrote:

see below

On Sat, Feb 4, 2017 at 6:21 PM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  wrote:

Once you transfer the code with your data - the issue of code version

synchronization disappears, doesn't it?

It also makes the wire data format irrelevant. At least for "short lived

serialized states".

Only works if you have no exchange with the environment it is executing.
And this is where "sandboxing" concern kicks in. What is the sandbox? In a
web browser they try to define it to DOM + handful of other well-defined
objects. In case of Java Serialization, it is all classes reachable from
the loading classloader. And I think Gregg is trying to argue that if one
is very prudent, one need to manage this well.


But how is "exchange with the environment it is executing"
actually different when installing code on demand from installing it in
advance???

The whole point IMHO is to shift thinking from "moving data" to "exchange
configured software" -
think Java specific Docker on ster

Re: Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

Don't know about other serialization uses but my issue with it is that 
it precludes using it in non-blocking IO.

Sorry if I haven't been clear enough.

Thanks,
Michal

Niclas Hedhman wrote:

And what I/O (network I/O I presume) are you doing during the serialization
(without RMI)?

On Sun, Feb 5, 2017 at 12:48 AM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


It is not possible to do non-blocking as in "non blocking IO" - meaning -
threads do not block on IO operations.
Just google "C10K problem"

Thanks,
Michal

Niclas Hedhman wrote:

I don't follow. What does readObject/writeObject got to do with blocking or
not? You could spin up executors to do the work in parallel if you so wish.
And why is "something else" less blocking? And what are you doing that is
"blocking" since the "work" is (or should be) CPU only, there is limited
(still) opportunity to do that non-blocking (whatever that would mean in
CPU-only circumstance). Feel free to elaborate... I am curious.



On Sat, Feb 4, 2017 at 8:38 PM, "Michał Kłeczek (XPro Sp. z o. 
o.)"<michal.klec...@xpro.biz>  wrote:


Unfortunately due to "writeObject" and "readObject" methods that have to
be handled (to comply with the spec) - it is not possible to
serialize/deserialize in a non-blocking fashion.
So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter<j...@zeus.net.au>  <j...@zeus.net.au>  
<j...@zeus.net.au>  <j...@zeus.net.au>  wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.

I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

For those not following my emails on this list :) :

My "codebase annotations" are actually objects of (sub)classes of:

abstract class CodeBase implements Serializable {
...
  abstract Class loadClass(String name, ClassLoader defaultLoader) 
throws IOException, ClassNotFoundException;

...
}

The interface is actually different for several reasons but the idea is 
the same.


So my AnnotatedInputStream is something like:

class AnnotatedInputStream extends ObjectInputStream {

  protected Class resolveClass(...) {
((CodeBase)readObject()).loadClass(...);
  }

}

Simply speaking I allow the _service_ to provide an object that can 
download the code.


Peter proposed to provide serialized CodeBase instances as Base64 
encoded strings (or something similar) - to maintain the assumption that 
codebase annotation is String.
But I do not see it as important at this moment - if needed - might be 
implemented.


Thanks,
Michal

Gregg Wonderly wrote:

Okay, then I think you should investigate my replacement of the 
RMIClassLoaderSPI implementation with a pluggable mechanism.

public interface CodebaseClassAccess {
public Class loadClass( String codebase,
  String name ) throws IOException, 
ClassNotFoundException;
public Class loadClass(String codebase,
  String name,
  ClassLoader defaultLoader) throws 
IOException,ClassNotFoundException;
 public Class loadProxyClass(String codebase,
   String[] interfaceNames,
   ClassLoader defaultLoader ) throws 
IOException,ClassNotFoundException;
public String getClassAnnotation( Class cls );
public ClassLoader getClassLoader(String codebase) throws IOException;
public ClassLoader createClassLoader( URL[] urls,
ClassLoader parent,
boolean requireDlPerm,
AccessControlContext ctx );
/**
 * This should return the class loader that represents the system
 * environment.  This might often be the same as {@link 
#getSystemContextClassLoader()}
 * but may not be in certain circumstances where container mechanisms 
isolate certain
 * parts of the classpath between various contexts.
 * @return
 */
 public ClassLoader getParentContextClassLoader();
/**
 * This should return the class loader that represents the local system
 * environment that is associated with never-preferred classes
 * @return
 */
 public ClassLoader getSystemContextClassLoader( ClassLoader defaultLoader 
);
}

I have forgotten what Peter renamed it to.  But this base interface is what all 
of the Jini codebase uses to load classes.  The annotation is in the “codebase” 
parameter.  From this you can explore how the annotation can move from being a 
URL, which you could recognize and still use, but substitute your own indicator 
for another platform such as a maven or OSGi targeted codebase.

Thus, you can still use the annotation, but use it to specify the type of 
stream instead of what to download via HTTP.

Gregg



On Feb 4, 2017, at 2:02 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

My annotated streams replace codebase resolution with object based one (ie - 
not using RMIClassLoader).

Michal

Gregg Wonderly wrote:

Why specific things do you want your AnnotatedStream to provide?

Gregg

Re: Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

It is not possible to do non-blocking as in "non blocking IO" - meaning 
- threads do not block on IO operations.

Just google "C10K problem"

Thanks,
Michal

Niclas Hedhman wrote:

I don't follow. What does readObject/writeObject got to do with blocking or
not? You could spin up executors to do the work in parallel if you so wish.
And why is "something else" less blocking? And what are you doing that is
"blocking" since the "work" is (or should be) CPU only, there is limited
(still) opportunity to do that non-blocking (whatever that would mean in
CPU-only circumstance). Feel free to elaborate... I am curious.



On Sat, Feb 4, 2017 at 8:38 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


Unfortunately due to "writeObject" and "readObject" methods that have to
be handled (to comply with the spec) - it is not possible to
serialize/deserialize in a non-blocking fashion.
So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter<j...@zeus.net.au>  <j...@zeus.net.au>  
wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.

I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Comments below.

Niclas Hedhman wrote:

see below

On Sat, Feb 4, 2017 at 6:21 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:

Once you transfer the code with your data - the issue of code version

synchronization disappears, doesn't it?

It also makes the wire data format irrelevant. At least for "short lived

serialized states".

Only works if you have no exchange with the environment it is executing.
And this is where "sandboxing" concern kicks in. What is the sandbox? In a
web browser they try to define it to DOM + handful of other well-defined
objects. In case of Java Serialization, it is all classes reachable from
the loading classloader. And I think Gregg is trying to argue that if one
is very prudent, one need to manage this well.


But how is "exchange with the environment it is executing"
actually different when installing code on demand from installing it in 
advance???


The whole point IMHO is to shift thinking from "moving data" to 
"exchange configured software" -

think Java specific Docker on steroids.

Transferable objects allow you for example to do things like
downloading your JDBC driver automagically without the fuss of 
installing it and managing upgrades.

Just publish a DataSource object in your ServiceRegistrar and you are done.
Make it leasable and you have automatic upgrades and/or reconfiguration.

Thanks,
Michal

Serialization issues

[Michał Kłeczek (XPro Sp. z o. o.)]

Unfortunately due to "writeObject" and "readObject" methods that have to 
be handled (to comply with the spec) - it is not possible to 
serialize/deserialize in a non-blocking fashion.

So yes... - it is serialization per se.

Thanks,
Michal

Niclas Hedhman wrote:

Oh, well that is not "Serialization" per se... No wonder i didn't get it.

On Sat, Feb 4, 2017 at 7:20 PM, Peter  wrote:


On 4/02/2017 9:09 PM, Niclas Hedhman wrote:


but rather with the APIs - it is inherently blocking by design.
I am not sure I understand what you mean by that.



He means the client thread that makes the remote call blocks waiting for
the remote end to process the request and respond.

Cheers,

Peter.

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Once you transfer the code with your data - the issue of code version 
synchronization disappears, doesn't it?
It also makes the wire data format irrelevant. At least for "short lived 
serialized states".


I fail to understand how JSON or XML changes anything here.

In the end all of the arguments against Java Object Serialization boil 
down to:
"It is easy to use but if not used carefully it will bite you - so it is 
too easy to use"


What I do not like about Java Object Serialization has nothing to do 
with the format of persistent data

but rather with the APIs - it is inherently blocking by design.

Thanks,
Michal

Niclas Hedhman wrote:

Gregg,
I know that you can manage to "evolve" the binary format if you are
incredibly careful and not make mistakes. BUT, that seems really hard,
since EVEN Sun/Oracle state that using Serilazation for "long live objects"
are highly discouraged. THAT is a sign that it is not nearly as easy as you
make it sound to be, and it is definitely different from XML/JSON as once
the working codebase is lost (i.e. either literally lost (yes, I have been
involved trying to restore that), or modified so much that compatibility
broke, which happens when serialization is not the primary focus of a
project) then you are pretty much screwed forever, unlike XML/JSON.

Now, you may say, that is for "long lived serialized states" but we are
dealing with "short lived" ones. However, in today's architectures and
platforms, almost no organization manages to keep all parts of a system
synchronized when it comes to versioning. Different parts of a system is
upgraded at different rates. And this is essentially the same as "long
lived objects" ---  "uh this was serialized using LibA 1.1, LibB 2.3 and
JRE 1.4, and we are now at LibA 4.6, LibB 3.1 and Java 8", do you see the
similarity? If not, then I will not be able to convince you. If you do,
then ask "why did Sun/Oracle state that long-lived objects with Java
Serialization was a bad idea?", or were they also clueless on how to do it
right, which seems to be your actual argument.

And I think (purely speculative) that many people saw exactly this problem
quite early on, whereas myself I was at the time mostly in relatively small
confined and controlled environments, where up-to-date was managed. And
took me much longer to realize the downsides that are inherent.

Cheers
Niclas

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

My annotated streams replace codebase resolution with object based one 
(ie - not using RMIClassLoader).


Michal

Gregg Wonderly wrote:

Why specific things do you want your AnnotatedStream to provide?

Gregg

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I know that.
And while it is better than Java RMI for several reasons (extensibility 
being one of them) - it is still not perfect:


1) It is inherently blocking
2) Does not support data streaming (in general you need a separate comm 
channel for this)
3) invocation layer depends on particular object serialization 
implementation - Marshall input/output streams (this is my favorite - to 
plug in my new AnnotatedStream implementations I must basically rewrite 
the invocation layer)


Thanks,
Michal


Peter wrote:

FYI.  JERI != Java RMI.

There's no reason these layers couldn't be provided as OSGi services 
and selected from the service registry either.


Cheers,

Peter.


   Protocol Stack

The Jini ERI architecture has a protocol stack with three layers as 
shown in the following table, with interfaces representing the 
abstractions of each layer on the client side and the server side as 
shown:


Layer Client-side abstractions Server-side abstractions
Invocation layer |InvocationHandler| 
 
|InvocationDispatcher|

Object identification layer |ObjectEndpoint| |RequestDispatcher|
Transport layer |Endpoint|, |OutboundRequestIterator|, 
|OutboundRequest| |ServerCapabilities|, |ServerEndpoint|, 
|InboundRequest|


The client-side and server-side implementations of each layer are 
chosen for a particular remote object as part of exporting the remote 
object. The design is intended to allow plugging in different 
implementations of one layer without affecting the implementations of 
the other layers.


The client side abstractions correspond to the structure of the 
client-side proxy for a remote object exported with Jini ERI, with the 
invocation layer implementation containing the object identification 
layer implementation and that, in turn, containing the transport layer 
implementation.


Which invocation constraints are supported for remote invocations to a 
particular remote object exported with Jini ERI is partially dependent 
on the particular implementations of these layers used for the remote 
object (most especially the transport layer implementation).




On 4/02/2017 3:51 PM, Peter wrote:

Thanks Nic,

JERI shouldn't be considered as being limited to or dependant on Java 
Serialization, it's only a transport layer, anything that can write 
to an OutputStream and read from an InputStream will do.


The JSON document could be compressed and sent as bytes, or UTF 
strings sent as bytes.


See the interfaces InboundRequest and OutboundRequest.

Cheers,

Peter.

On 4/02/2017 3:35 PM, Niclas Hedhman wrote:
FYI in case you didn't know; Jackson ObjectMapper takes a POJO 
structure
and creates a (for instance) JSON document, or the other way around. 
It is

not meant for "any object to binary and back".
My point was, Java Serialization (and by extension JERI) has a scope 
that
is possibly wrongly defined in the first place. More constraints 
back then

might have been a good thing...



On Sat, Feb 4, 2017 at 12:36 PM, Peter  wrote:


On 4/02/2017 12:43 PM, Niclas Hedhman wrote:


On Fri, Feb 3, 2017 at 12:23 PM, Peter   wrote:

No serialization or Remote method invocation framework currently 
supports
OSGi very well, one that works well and can provide security 
might gain a

lot of new interest from that user base.

What do you mean by this? Jackson's ObjectMapper doesn't have 
problems on

OSGi. You are formulating the problem wrongly, and if formulated
correctly,
perhaps one realizes why Java Serialization fell out of fashion 
rather
quickly 10-12 years ago, when people realized that code mobility 
(as done

in Java serialization/RMI) caused a lot of problems.


Hmm, I didn't know that, sounds like an option for JERI.


IMHO, RMI/Serialization's design is flawed. Mixing too many 
concerns in the

same abstraction; sandboxing w/ integration , code mobility, class
resolution, versioning and deserialization, with very little hooks to
cusomize any or all of these aspects. And these aspects should not 
have

been wrapped into one monolith.

Further, I think the only "sane" approach in a OSGi environment is to
create a new bundle for the Remote environment, all codebases not 
part of
the API goes into that bundle and that the API is required to be 
present

in
the OSGi environment a priori. I.e. treat the Remote objects in 
OSGi as it
is treated in plain Java; one classloader, one chunk, sort out its 
own
serialization woes. Likewise for the server; treat it as ordinary 
RMI,
without any mumbo-jambo OSGi stuff to be figured out at a 
non-OSGi-running
JVM. An important difference is that in OSGi, the 
BundleClassLoader is not

(required to be) a URLClassLoader, so the Java serialization's auto
annotation of globally reachable URLs won't work, and one need to 
rely on
java.rmi.server.codebase property, but a bundle could watch for 
loaded

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Are you opposing the whole idea of sending data and code (or 
instructions how to download it) bundled together? (the spec)

Or just the way how it is done in Java today. (the impl)

If it is the first - we are in an absolute disagreement.
If the second - I agree wholeheartedly.

Thanks,
Michal

Niclas Hedhman wrote:

FYI in case you didn't know; Jackson ObjectMapper takes a POJO structure
and creates a (for instance) JSON document, or the other way around. It is
not meant for "any object to binary and back".
My point was, Java Serialization (and by extension JERI) has a scope that
is possibly wrongly defined in the first place. More constraints back then
might have been a good thing...

Re: object based annotations

[Michał Kłeczek (XPro Sp. z o. o.)]

Object based annotations allow creating ClassLoaders - it is not only a 
way to download code but to:

1. Make sure only trusted code is executed
2. Resolve any dependencies and create the whole ClassLoader structure 
when deserializing objects


So URL handler is not enough - alternative implementation of 
RMIClassProviderSpi is necessary.


But anyway - if there is a need to encode annotations as Strings - I do 
not see anything wrong with that except that I personally do not see it 
necessary.


Thanks,
Michal

Peter wrote:

Bitwise operations shouldn't have much performance impact.
  
You could also create a url scheme and handler.


When a ClassLoader retrieves a URL, it doesn't care how it's done, just that it 
retrieves the class code.

eg:

obj:[length][bytes]

Then install your URL handler in your client jvms.

The community agreed to break compatibility with the com.sun namespace change, 
so it can happen, although admittedly concensus is often difficult, if someone 
believes a breaking change is necessary, state the reasons, impacts and request 
a vote..

I've found in the majority of cases with some additional thought and work that 
the api is extensible, so it may not be necessary to cause breakage.

The performance improvent you'll get with your own URL scheme will vastly 
outstrip standard url schemes by avoiding dns calls, which will exceed the cost 
of encoding by orders of magnitude.

Jini/River 2.x.x has a significant number of unnecessary dns calls. 


But it sounds like you may have found an alternative option.

Regards,

Peter.


Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 02/02/2017 06:29:55 am
To: dev@river.apache.org
Subject: Re: object based annotations

I have actually given up on the idea of object annotations encoded as 
Strings (in whatever form).

Simply speaking it does not make any sense really:
- it would complicate the solution because of additional encoding and 
decoding logic

- it would influence performance because of additional encoding and decoding
- it would complicate maintaining codebase objects identity (since 
encoded annotation would be a separate stream)


- it would be incompatible with existing clients (if there are any :) ) 
anyway - all expect the annotation to be a space separated list of URLs


I have starting working on a compatibility layer that would allow 
existing clients to download the class loading infrastructure from a 
codebase URL magically (using existing RMI infrastructure).
TBH - I do not see any benefit in maintaining backwards compatibility - 
Jini/River is out of favor nowadays and existing software needs upgrade 
anyway because of security and concurrency fixes.


Thanks,
Michal

Peter Firmstone wrote:

  Mike, I recall the last time I looked at object based annotations, there was 
a backward compatibility issue because both ends of the Marshal streams expect 
string based annotations as does RMIClassLoader.

  However if you are still keen to investigate object based annotations there's 
no reason you couldn't treat a string like a char array = byte array (beware 
signed byte) and have a RMIClassLoaderSPI deserialize the objects after they 
were sent in string form?

  Regards,

  Peter.

  Sent from my Samsung device.

Re: object based annotations

[Michał Kłeczek (XPro Sp. z o. o.)]

I have actually given up on the idea of object annotations encoded as 
Strings (in whatever form).

Simply speaking it does not make any sense really:
- it would complicate the solution because of additional encoding and 
decoding logic

- it would influence performance because of additional encoding and decoding
- it would complicate maintaining codebase objects identity (since 
encoded annotation would be a separate stream)


- it would be incompatible with existing clients (if there are any :) ) 
anyway - all expect the annotation to be a space separated list of URLs


I have starting working on a compatibility layer that would allow 
existing clients to download the class loading infrastructure from a 
codebase URL magically (using existing RMI infrastructure).
TBH - I do not see any benefit in maintaining backwards compatibility - 
Jini/River is out of favor nowadays and existing software needs upgrade 
anyway because of security and concurrency fixes.


Thanks,
Michal

Peter Firmstone wrote:

Mike, I recall the last time I looked at object based annotations, there was a 
backward compatibility issue because both ends of the Marshal streams expect 
string based annotations as does RMIClassLoader.

However if you are still keen to investigate object based annotations there's 
no reason you couldn't treat a string like a char array = byte array (beware 
signed byte) and have a RMIClassLoaderSPI deserialize the objects after they 
were sent in string form?

Regards,

Peter.

Sent from my Samsung device.

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Rant aside...

This is what I am saying all along... Bundles are not good candidates 
for codebase annotations.
For exactly the reason you describe: bundles represent a template that 
may produce different wirings.


But to recreate an object graph you need the _wiring_ - not the template.

And this is also why any kind of statically (ie. at bundle build time) 
locking of the dependencies is not going to work either.
The ImplementationDetailServiceProxy runtime dependencies are not yet 
known when creating the bundle manifest
since its interface to be useful must be resolved in each client 
environment differently (or all parties in the distributed system
must share the same versions of the code - which makes the whole point 
of OSGI moot)


The only way I can see around this is not to treat jar files downloaded 
dynamically as bundles
but rather create fake bundles from them and generate artificial 
manifests based on information in the stream.

Kind of an ugly hack really...

The situation might change if OSGI defined a way to provide bundle 
manifest not as a statically prepared MANIFEST.MF
inside the jar file but as separate piece of information provided 
dynamically at runtime.


A little rant now...

This container centric view is IMHO quite restricting.
There is no reason why resolution of code dependencies (aka preparing 
the wiring)
must be done by the particular container where the client software is 
executed.
It might as well be done somewhere else (ie. in the service provider 
environment)

and sent to the client already prepared for execution.

OSGI way is not the only way.

I do not find any particular problem with downloading data together with 
the code that manipulates this data.

I am doing it very often when viewing tens (if not more) of web sites daily.
Haven't found any major issues with that - quite the opposite - it seems 
like this is a pretty damn good way of doing things.


End of rant...

Thanks,
Michal

Niclas Hedhman wrote:

As I think you know, the whole purpose of OSGi is to NOT tie the resolution
to Bundles, but to explicitly Exported (with versions) packages. If people
screw up and don't declare the Import/Export package meta data correctly,
then ordinary OSGi may fail just as spectacularly. The difference being
that Java serialization is slightly more sensitive to class changes than
the code itself, and that was an issue from the start. "Back then" it was
quickly realized that "long-term" storage of serialized state was a bad
idea, but "long-term" is relative to code releases and with highly
distributed systems, this is now a reality in networked applications as
well as storing serialized objects to disk. With that in mind, one should
seriously reconsider the status of Java Serialization in one's application,
realize that every serialized class, incl possible exceptions, is a
published API and needs to be treated as such, just as a REST API, SOAP
interface or HTTP itself. Sorry for the rant...

Back to OSGi; even if you really need to tie a class to a particular
bundle, you can do that with attributes and the 'mandatory' directive.

Cheers
Niclas

On Wed, Feb 1, 2017 at 3:29 AM, Michał Kłeczek<michal.klec...@xpro.biz>
wrote:


Unfortunately it is

Re: OSGi

[Michał Kłeczek]

gt; codebase.  This is why I mentioned Objectspace Voyage earlier.  I wanted
> to
> > use it as an example of a mechanism which always packages class
> definitions
> > into the byte stream that is used for sending objects between VMs.
> Voyager
> > would extract the class definitions from the jars, wrap them into the
> > stream, and the remote JVM would be able to then resolve the classes by
> > constructing instances of the class using the byte[] data for the class
> > definition.
> >
> > Ultimately, no matter what the source of the byte[] data for the class
> > definition is, it has to be present, at some point in all VMs using that
> > definition/version of the class.  That’s what I am trying to say.  The
> > issue is simply where would the class resolve from?  I think that class
> > definition conveyance, between JVMs is something that we have choices on.
> > But, practically, you can’t change “annotations” to make this work.  If
> the
> > middle man above is a “proxy” service which bridges two different
> networks,
> > neither JVM on each network would have routing to get to the one on the
> > other side of the proxy JVM.  This is why a mechanism like Objectspace
> > Voyager would be one way to send class definitions defined on one network
> > to another JVM on another network via this proxy service.
> >
> > Of course other mechanisms for class conveyance are possible and in fact
> > already exist.  Maven and even OSGi provide class, version oriented
> > conveyance from a distribution point, into a particular JVM instance.
> Once
> > the class definition exists inside of one of those JVMs then we have all
> > the other details about TCCL and creation of proper versions and
> resolution
> > from proper class loaders.
> >
> > I don’t think we have to dictate that a particular class conveyance
> > mechanism is the only one.  But, to solve the problem of how to allow
> > classes hop between multiple JVMs, we have to  specify how that might
> work
> > at the level that service instances are resolved and some kind of class
> > loading context is attached to that service.
> >
> > The reason I am talking specifically about directed graphs of class
> > loading is because I am first focused on the fact that there is a lot
> less
> > flexibility in trying to resolve through a large collection of specific
> > classes rather than an open set of classes resolved through a directed
> > graph of the code execution path which exposes the places and moments of
> > object use in a much more controlled and natural way to me.
> >
> > Gregg
> >
> > > On Jan 30, 2017, at 9:14 AM, Michał Kłeczek (XPro Sp. z o. o.) <
> > michal.klec...@xpro.biz> wrote:
> > >
> > > It looks to me like we are talking past each other.
> > >
> > > Thread local resolution context is needed - we both agree on this.
> > > What we do not agree on is that the context should be a single
> > ClassLoader. It has to be a set of ClassLoaders to support situations
> when
> > dependencies are not hierarchical.
> > >
> > > The use case is simple - I want to implement "decorator" services that
> > provide smart proxies wrapping (smart) proxies of other services.
> > > I also want to have Exporters provided as dynamic services which would
> > allow my services to adapt to changing network environment.
> > >
> > > And I would like to stress - I am actually quite negative about OSGI
> > being the right environment for this.
> > >
> > > Thanks,
> > > Michal
> > >
> > > Gregg Wonderly wrote:
> > >> Maybe you can help me out here by explaining how it is that execution
> > context and class visibility are both handled by OSGi bundles.  For
> > example, one of my client applications is a desktop environment.  It does
> > service look up for all services registrations providing a “serviceUI”.
> It
> > then integrates all of those services into a desktop view where the UIs
> are
> > running at the same time with each one imbedded in a JDesktopPane or a
> > JTabbedPane or a JFrame or JDialog.  There are callbacks from parts of
> that
> > environment into my application which in turn is interacting with the
> > ServiceUI component.  You have AWT event threads which are calling out,
> > into the ServiceUIs and lots of other threads of execution which all,
> > ultimately, must have different class loading environments so that the
> > ServiceUI components can know where to load code from.
> > >>
> > >&

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I meant "of course it is NOT too intelligent". Freudian mistake :D

Michał Kłeczek (XPro Sp. z o. o.) wrote:

Of course it is too intelligent.

What I am saying is that it is at service provider's discretion to 
decide how to load its own proxy classes.
If a service decides that the full container is necessary and the 
client does not have it - well...


On the other hand. How do you manage upgrades of the OSGI container 
due to - let's say - a security issue in current implementation?


Thanks,
Michal

Niclas Hedhman wrote:

It doesn't sound very intelligent to download an OSGi Container to a
client. It surely is something wrong with that... Proxy should depend on
the deployed services, locally some more... What am I missing, other than
you are trying to convey an absurdity?

On Tue, Jan 31, 2017 at 4:10 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


My point throughout the whole thread is that to support these scenarios:

1. Manipulating class streams (like in Voyager) is not necessary (quite
franky - I think it is a bad idea actually since it assumes a single
namespace for classes what precludes class evolution)
2. Dictating a particular "class conveyance mechanism" is not necessary
either

What I am proposing is:
1. Abstract over a "class conveyance mechanism" (by making codebases
serializable objects which classes implement a specific contract)
2. Change ClassProvider API to support the above (accept abstract
codebases instead of only Strings)
3. (Optionally) - provide a default class conveyance mechanism that:
a) allows resolving classes in non-hierarchical way (similar to
ClassWorlds or JBossModules or... OSGI)
b) supports coexisting of other "class conveyance mechanisms" in the same
JVM

Point 1 and 3b) will make the whole solution really dynamic allowing a
"class conveyance mechanism" to be dynamically downloaded by the client.
So - how do you make sure a service deployed in OSGi container may send
its proxy to a non-OSGI client? Yes! You let the client download the OSGI
container dynamically!

What's more - once you abstract over how the classes are downloaded - it
is possible to support downloading code through relays etc.

Thanks,
Michal

Gregg Wonderly wrote:


The annotation for the exported services/classes is what is at issue
here.  Here’s the perspectives I’m trying to make sure everyone sees.

1) Somehow, exported classes from one JVM need to be resolved in another
JVM (at a minimum).  The source of those classes today, is the codebase
specified by the service.  A directed graph of JVMs exchanging classes
demands that all service like JVMs provide a codebase for client like JVMs
to be able to resolve the classes for objects traveling to the client form
the service.  This is nothing we all don’t already know I believe.

2) If there is a 3rd party user of a class from one JVM which is handed
objects resolved by a middle man JVM (as Michal is mentioning here), there
is now a generally required class which all 3 JVMs need to be able to
resolve.  As we know, Jini’s current implementation and basic design is
that a services codebase has to provide a way for clients to resolve the
classes it exports in its service implementation.  In the case Michal is
mentioning, the demand would be for the middle man service to have the
classes that it wants the 3rd service to resolve, in some part of its
codebase.  This is why I mentioned Objectspace Voyage earlier.  I wanted to
use it as an example of a mechanism which always packages class definitions
into the byte stream that is used for sending objects between VMs.  Voyager
would extract the class definitions from the jars, wrap them into the
stream, and the remote JVM would be able to then resolve the classes by
constructing instances of the class using the byte[] data for the class
definition.

Ultimately, no matter what the source of the byte[] data for the class
definition is, it has to be present, at some point in all VMs using that
definition/version of the class.  That’s what I am trying to say.  The
issue is simply where would the class resolve from?  I think that class
definition conveyance, between JVMs is something that we have choices on.
But, practically, you can’t change “annotations” to make this work.  If the
middle man above is a “proxy” service which bridges two different networks,
neither JVM on each network would have routing to get to the one on the
other side of the proxy JVM.  This is why a mechanism like Objectspace
Voyager would be one way to send class definitions defined on one network
to another JVM on another network via this proxy service.

Of course other mechanisms for class conveyance are possible and in fact
already exist.  Maven and even OSGi provide class, version oriented
conveyance from a distribution point, into a particular JVM instance.  Once
the class definition exists inside of one of those JVMs then we hav

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Of course it is too intelligent.

What I am saying is that it is at service provider's discretion to 
decide how to load its own proxy classes.
If a service decides that the full container is necessary and the client 
does not have it - well...


On the other hand. How do you manage upgrades of the OSGI container due 
to - let's say - a security issue in current implementation?


Thanks,
Michal

Niclas Hedhman wrote:

It doesn't sound very intelligent to download an OSGi Container to a
client. It surely is something wrong with that... Proxy should depend on
the deployed services, locally some more... What am I missing, other than
you are trying to convey an absurdity?

On Tue, Jan 31, 2017 at 4:10 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


My point throughout the whole thread is that to support these scenarios:

1. Manipulating class streams (like in Voyager) is not necessary (quite
franky - I think it is a bad idea actually since it assumes a single
namespace for classes what precludes class evolution)
2. Dictating a particular "class conveyance mechanism" is not necessary
either

What I am proposing is:
1. Abstract over a "class conveyance mechanism" (by making codebases
serializable objects which classes implement a specific contract)
2. Change ClassProvider API to support the above (accept abstract
codebases instead of only Strings)
3. (Optionally) - provide a default class conveyance mechanism that:
a) allows resolving classes in non-hierarchical way (similar to
ClassWorlds or JBossModules or... OSGI)
b) supports coexisting of other "class conveyance mechanisms" in the same
JVM

Point 1 and 3b) will make the whole solution really dynamic allowing a
"class conveyance mechanism" to be dynamically downloaded by the client.
So - how do you make sure a service deployed in OSGi container may send
its proxy to a non-OSGI client? Yes! You let the client download the OSGI
container dynamically!

What's more - once you abstract over how the classes are downloaded - it
is possible to support downloading code through relays etc.

Thanks,
Michal

Gregg Wonderly wrote:


The annotation for the exported services/classes is what is at issue
here.  Here’s the perspectives I’m trying to make sure everyone sees.

1) Somehow, exported classes from one JVM need to be resolved in another
JVM (at a minimum).  The source of those classes today, is the codebase
specified by the service.  A directed graph of JVMs exchanging classes
demands that all service like JVMs provide a codebase for client like JVMs
to be able to resolve the classes for objects traveling to the client form
the service.  This is nothing we all don’t already know I believe.

2) If there is a 3rd party user of a class from one JVM which is handed
objects resolved by a middle man JVM (as Michal is mentioning here), there
is now a generally required class which all 3 JVMs need to be able to
resolve.  As we know, Jini’s current implementation and basic design is
that a services codebase has to provide a way for clients to resolve the
classes it exports in its service implementation.  In the case Michal is
mentioning, the demand would be for the middle man service to have the
classes that it wants the 3rd service to resolve, in some part of its
codebase.  This is why I mentioned Objectspace Voyage earlier.  I wanted to
use it as an example of a mechanism which always packages class definitions
into the byte stream that is used for sending objects between VMs.  Voyager
would extract the class definitions from the jars, wrap them into the
stream, and the remote JVM would be able to then resolve the classes by
constructing instances of the class using the byte[] data for the class
definition.

Ultimately, no matter what the source of the byte[] data for the class
definition is, it has to be present, at some point in all VMs using that
definition/version of the class.  That’s what I am trying to say.  The
issue is simply where would the class resolve from?  I think that class
definition conveyance, between JVMs is something that we have choices on.
But, practically, you can’t change “annotations” to make this work.  If the
middle man above is a “proxy” service which bridges two different networks,
neither JVM on each network would have routing to get to the one on the
other side of the proxy JVM.  This is why a mechanism like Objectspace
Voyager would be one way to send class definitions defined on one network
to another JVM on another network via this proxy service.

Of course other mechanisms for class conveyance are possible and in fact
already exist.  Maven and even OSGi provide class, version oriented
conveyance from a distribution point, into a particular JVM instance.  Once
the class definition exists inside of one of those JVMs then we have all
the other details about TCCL and creation of proper versions and resolution
from proper class loaders.

I don’t think we have

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

My point throughout the whole thread is that to support these scenarios:

1. Manipulating class streams (like in Voyager) is not necessary (quite 
franky - I think it is a bad idea actually since it assumes a single 
namespace for classes what precludes class evolution)
2. Dictating a particular "class conveyance mechanism" is not necessary 
either


What I am proposing is:
1. Abstract over a "class conveyance mechanism" (by making codebases 
serializable objects which classes implement a specific contract)
2. Change ClassProvider API to support the above (accept abstract 
codebases instead of only Strings)

3. (Optionally) - provide a default class conveyance mechanism that:
a) allows resolving classes in non-hierarchical way (similar to 
ClassWorlds or JBossModules or... OSGI)
b) supports coexisting of other "class conveyance mechanisms" in the 
same JVM


Point 1 and 3b) will make the whole solution really dynamic allowing a 
"class conveyance mechanism" to be dynamically downloaded by the client.
So - how do you make sure a service deployed in OSGi container may send 
its proxy to a non-OSGI client? Yes! You let the client download the 
OSGI container dynamically!


What's more - once you abstract over how the classes are downloaded - it 
is possible to support downloading code through relays etc.


Thanks,
Michal

Gregg Wonderly wrote:

The annotation for the exported services/classes is what is at issue here.  
Here’s the perspectives I’m trying to make sure everyone sees.

1) Somehow, exported classes from one JVM need to be resolved in another JVM 
(at a minimum).  The source of those classes today, is the codebase specified 
by the service.  A directed graph of JVMs exchanging classes demands that all 
service like JVMs provide a codebase for client like JVMs to be able to resolve 
the classes for objects traveling to the client form the service.  This is 
nothing we all don’t already know I believe.

2) If there is a 3rd party user of a class from one JVM which is handed objects 
resolved by a middle man JVM (as Michal is mentioning here), there is now a 
generally required class which all 3 JVMs need to be able to resolve.  As we 
know, Jini’s current implementation and basic design is that a services 
codebase has to provide a way for clients to resolve the classes it exports in 
its service implementation.  In the case Michal is mentioning, the demand would 
be for the middle man service to have the classes that it wants the 3rd service 
to resolve, in some part of its codebase.  This is why I mentioned Objectspace 
Voyage earlier.  I wanted to use it as an example of a mechanism which always 
packages class definitions into the byte stream that is used for sending 
objects between VMs.  Voyager would extract the class definitions from the 
jars, wrap them into the stream, and the remote JVM would be able to then 
resolve the classes by constructing instances of the class using the byte[] 
data for the class definition.

Ultimately, no matter what the source of the byte[] data for the class 
definition is, it has to be present, at some point in all VMs using that 
definition/version of the class.  That’s what I am trying to say.  The issue is 
simply where would the class resolve from?  I think that class definition 
conveyance, between JVMs is something that we have choices on.  But, 
practically, you can’t change “annotations” to make this work.  If the middle 
man above is a “proxy” service which bridges two different networks, neither 
JVM on each network would have routing to get to the one on the other side of 
the proxy JVM.  This is why a mechanism like Objectspace Voyager would be one 
way to send class definitions defined on one network to another JVM on another 
network via this proxy service.

Of course other mechanisms for class conveyance are possible and in fact 
already exist.  Maven and even OSGi provide class, version oriented conveyance 
from a distribution point, into a particular JVM instance.  Once the class 
definition exists inside of one of those JVMs then we have all the other 
details about TCCL and creation of proper versions and resolution from proper 
class loaders.

I don’t think we have to dictate that a particular class conveyance mechanism 
is the only one.  But, to solve the problem of how to allow classes hop between 
multiple JVMs, we have to  specify how that might work at the level that 
service instances are resolved and some kind of class loading context is 
attached to that service.

The reason I am talking specifically about directed graphs of class loading is 
because I am first focused on the fact that there is a lot less flexibility in 
trying to resolve through a large collection of specific classes rather than an 
open set of classes resolved through a directed graph of the code execution 
path which exposes the places and moments of object use in a much more 
controlled and natural way to me.

Gregg

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Let me once again provide a simple example:

interface ForClient {
}

interface ImplementationDetail {
}

class ServiceProxy implements ForClient {
  private ImplementationDetail implementationDetail;
}

class ServiceBackend { //not implementing any remote interface for 
simplicity


  public void start() {
ServiceRegistrar reggie = ...;
ImplementationDetail implDetail = lookupImplIn(reggie);
ServiceProxy myProxy = new ServiceProxy(implDetail);
publish(myProxy, reggie);
  }

}

We have two codebases:
1) ImplementationDetailServiceCodebase
2) ServiceProxyCodebase

Now the question:
How to make it possible to deserialize ServiceProxy in client 
environment assuming:
1) ServiceBackend dynamically downloads classes of ImplementationDetail 
proxy

2) Client dynamically downloads classes of ServiceProxy
3) Client is NOT aware ImplementationDetail interface (well... since it 
is an implementation detail)


Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

It looks to me like we are talking past each other.

Thread local resolution context is needed - we both agree on this.
What we do not agree on is that the context should be a single 
ClassLoader. It has to be a set of ClassLoaders to support situations 
when dependencies are not hierarchical.


The use case is simple - I want to implement "decorator" services that 
provide smart proxies wrapping (smart) proxies of other services.
I also want to have Exporters provided as dynamic services which would 
allow my services to adapt to changing network environment.


And I would like to stress - I am actually quite negative about OSGI 
being the right environment for this.


Thanks,
Michal

Gregg Wonderly wrote:

Maybe you can help me out here by explaining how it is that execution context 
and class visibility are both handled by OSGi bundles.  For example, one of my 
client applications is a desktop environment.  It does service look up for all 
services registrations providing a “serviceUI”.  It then integrates all of 
those services into a desktop view where the UIs are running at the same time 
with each one imbedded in a JDesktopPane or a JTabbedPane or a JFrame or 
JDialog.  There are callbacks from parts of that environment into my 
application which in turn is interacting with the ServiceUI component.  You 
have AWT event threads which are calling out, into the ServiceUIs and lots of 
other threads of execution which all, ultimately, must have different class 
loading environments so that the ServiceUI components can know where to load 
code from.

It’s exactly TCCL that allows them to know that based on all the other class 
loading standards.  The ClassLoader is exactly the thing that all of them have 
in common if you include OSGi bundles as well.  The important detail, is that 
if the TCCL is not used as new ClassLoaders are created, then there is no 
context for those new ClassLoaders to reference, universally.

The important details are:

1) The desktop application has to be able to prefer certain Entry 
classes which define details that are presented to the user.
2) When the user double clicks on a services icon, or right clicks and 
selects “Open in new Frame”, an async worker thread needs a TCCL pointing at 
the correct parent class loader for the service’s URLClassLoader to reference 
so that the preferred classes work.
3) Anytime that the AWT Event thread might be active inside of the 
services UI implementation, it also needs to indicate the correct parent class 
loader if that UI component causes other class loading to occur.
4) I am speaking specifically in the context of deferred class loading 
which is controlled outside of the service discovery moment.



On Jan 30, 2017, at 4:04 AM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

What I think Jini designers did not realize is that class loading can be 
treated exactly as any other capability provided by a (possibly remote) service.
Once you realize that - it is possible to provide a kind of a "universal container 
infrastructure" where different class loading implementations may co-exist in a 
single JVM.


That’s precisely what ClassLoader is for.  TCCL is precisely to allow “some 
class” to know what context to associate newly loaded classes with, so that in 
such an environment, any code can load classes on behalf of some other 
code/context.  It doesn’t matter if it is TCCL or some other class management 
scheme such as OSGi bundles.  We are talking about the same detail, just 
implemented in a different way.


What's more - these class loading implementations may be dynamic themselves - 
ie. it is a service that provides the client with a way to load its own (proxy) 
classes.

In other words: "there not enough Jini in Jini itself”.


I am not sure I understand where the short coming is at then.  Maybe you can 
illustrate with an example where TCCL fails to allow some piece of code to load 
classes on behalf of another piece of code?

In my desktop application environment, there is a abstract class which is used 
by each serviceUI to allow the desktop to know if it provides the ability to 
open into one of the above mentioned JComponent subclasses.  That class is 
preferred and provided and resolved using the codebase of the desktop client.  
That class loading environment is then the place where the service is finally 
resolved and classes created so that the proxy can be handed to the serviceUI 
component which ultimately only partially resolves from the services codebase.

It’s this class compatibility which needs to be lightweight.


We have _all_ the required pieces in place:
- dynamic code loading and execution (ClassLoaders),
- security model and implementation that allows restricting rights of the 
downloaded code,
- and a serialization/deserialization which allows sending arbitrary data (and 
yes - code too) over the wire.

It is just t

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

What I think Jini designers did not realize is that class loading can be 
treated exactly as any other capability provided by a (possibly remote) 
service.
Once you realize that - it is possible to provide a kind of a "universal 
container infrastructure" where different class loading implementations 
may co-exist in a single JVM.
What's more - these class loading implementations may be dynamic 
themselves - ie. it is a service that provides the client with a way to 
load its own (proxy) classes.


In other words: "there not enough Jini in Jini itself".

We have _all_ the required pieces in place:
- dynamic code loading and execution (ClassLoaders),
- security model and implementation that allows restricting rights of 
the downloaded code,
- and a serialization/deserialization which allows sending arbitrary 
data (and yes - code too) over the wire.


It is just the matter of glueing the pieces together.

Thanks,
Michal


Gregg Wonderly wrote:


I am not an OSGi user.  I am not trying to be an OSGi opponent.  What I am 
trying to say is that I consider all the commentary in those articles about 
TCCL not working to be just inexperience and argument to try and justify a 
different position or interpretation of what the real problem is.

The real problem is that there is not one “module” concept in Java (another one 
is almost here in JDK 9/Jigsaw).  No one is working together on this, and OSGi 
is solving problems in a small part of the world of software.   It works well 
for embedded, static systems.  I think OSGi misses the mark on dynamic systems 
because of the piecemeal loading and resolving of classes.  I am not sure that 
OSGi developers really understand everything that Jini can do because of the 
choices made (and not made) in the design.  The people who put Jini together 
had a great deal of years of experience piecing together systems which needed 
to work well with a faster degree of variability and adaptation to the 
environment then what most people seem to experience in their classes and work 
environments which are locked down by extremely controlled distribution 
strategies which end up slowing development in an attempt to control everything 
that doesn’t actually cause quality to suffer.

Gregg

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I absolutely agree with the requirements you state.

The problem with Jini (and hence River) usage of TCCL is that it assumes 
a parent-child relationship between class loaders - which in turn causes 
the issues with transferring object graphs I've described earlier.


What I understood when working on this is also that OSGI is not the 
right choice either :)
What I also understood is that even having a "module" concept in Java is 
not enough :)


Any solution needs to make a distinction between:
- a static notion of a module (in OSGI it is a bundle, in current Jini - 
a module represented by a codebase string)
- a dynamic notion of a module (in OSGI its a BundleWiring, in current 
Jini it would be the _hierarchy_ of class loaders) - it is a runtime 
state representing a module with its resolved dependencies
And the most important conclusion is that it is the latter that must be 
sent between parties in a distributed system to make it all work.


To support class (API) evolution the solution must also allow to provide 
"open ends" in the module dependency graph. The difference from 
PreferredClassProvider is that these "open ends" must allow selecting 
one of many existing class loaders instead of only one particular 
ClassLoader that a client has set as TCCL.
TCCL is still needed to support many separate services in a single JVM. 
It is used to select a proper subset of class loaders as the set of 
candidates to choose from when resolving code bases. It allows to make 
sure that a single "static module" may produce many instances of 
"dynamic module" resolved differently in the same JVM.


I am working on this and hope to be able to provide an initial 
implementation soon.
The solution I am working on assumes code bases are represented as 
serializable objects, so any example I am giving is based on that.


The basic idea might be presented as follows (of course details related 
to concurrency, weak references and proper method visibility to make the 
thing secure are left out):


class ClassResolver {
  static Map globalResolverMap;
  static ClassResolver getContextClassResolver() {...} //impl returns a 
resolver based on the TCCL


  final Map codeBaseMap;
  final Map existingLoadersMap;
  final Set apiImplementations;

  ClassLoader getClassLoader(CodeBase cb) {
ClassLoader loader = lookup(cb);
if (loader == null) {
  loader = resolve(cb).createLoader(this);
  //update the caches etc
}
return loader;
  }

  CodeBase resolve(CodeBase cb) {
if (cb instanceof ApiCodeBase) {
  return resolveApi((ApiCodeBase) cb);
}
return cb;
  }

  CodeBase resolveApi(ApiCodeBase apiCb) {
//java 8 style
//example is simplified since we want to select a "best match" in 
reality

existingLoadersMap.keySet().stream().filter(apiCb::matchesCodeBase).first().orElse(apiCb);

  }
}

class CodeBase {
  protected abstract ClassLoader createLoader(ClassResolver resolver); 
//creates a ClassLoader using provided resolver to resolve any dependencies

}

class ApiCodeBase {
  Predicate matcher;
  CodeBase defaultImplementation;

  ClassLoader createLoader(ClassResolver resolver) {
return defaultImplementation.createLoader(resolver);
  }

  boolean matchesCodeBase(CodeBase cb) {
return matcher.test(cb);
  }

}

So now when a service provider creates initializes its runtime 
environment it creates a set of instances of CodeBase subclasses
connected to each other in a way specific to a particular class loading 
implementation.
Some of which might be ApiCodeBase instances that will use their default 
implementations to create a class loader in the service provider 
environment (since they are resolved in a "clean" ClassResolver on startup).


Any client that will deserialize an object graph provided by the service 
will either:
1. Not have matching CodeBase to select from when resolving ApiCodeBase 
(the situation similar to service provider startup)

2. Have a matching CodeBase to select from:
a) it is an ApiCodeBase - will possibly be again resolved when 
transferring further

b) it is not an ApiCodeBase - will "lock" an "open end"

So any party is free to mark modules as "private" or "public".

The tricky thing is still handling code base bounce backs.
If not careful when specifying API a service might encounter a "lost 
codebase" problem which might be mitigated by having API modules only 
consist of interfaces.


Thanks,
Michal

Gregg Wonderly wrote:

But codebase identity is not a single thing.  If you are going to allow a 
client to interact with multiple services and use those services, together, to 
create a composite service or just be a client application, you need all of the 
classes to interact.  One of the benefits of dynamic class loading and the 
selling point of how Jini was first presented (and I still consider this to the 
a big deal), is the notion that you can introduce a new version 

Re: OSGi

[Michał Kłeczek]

IMHO it is possible to have both.

The prerequisite is to encode a snapshot of module wiring in the
serialization stream.

The service provider has to explicitly mark certain dependencies as "API
dependencies" which are going to be resolved from modules (bundles)
installed in the client environment.
Dependencies not marked as "API" would be resolved using strict module
identity.

Once resolved in the client the dependency is "locked" i.e. no longer
marked as "API" when serialising (as long as they are not dependencies of a
client's API)

The only problem then are dependencies that are both consumed and exposed
by a service proxy which might cause class cast exceptions.

Thanks,
Michał

On Sun, 29 Jan 2017 at 01:18, Peter <j...@zeus.net.au> wrote:

> The best way to handle that situation would be to throw a
> ClassCastException, but also gather as much information as possible re
> Bundles and Bundle versions to assist the developer to debug and
> unentangle their design.
>
> In this case BImpl was resolved first which has caused Child2Api version
> 1.0.0 to be installed.
>
> If the package that Child2Api resides is in the version range 1.0.0 <= X
> < 2.0.0, then the only reason that a higher bundle version may have been
> chosen is that bundle 1.0.1 etc was already installed at the client,
> prior to 1.0.0 being installed and C2 has selected it during resolution.
>
> This is also a good demonstration of why version ranges should be
> chosen, why would any sensible developer avoid an internal bug fix, or
> even a backward compatible api change in an api bundle, unless the
> bundle is broken?
>
> If we try to ensure replication of wiring from one JVM in another, then
> if we don't have full control over both JVM's then we may introduce
> incompatibility at the root object, thus causing a ClassCastException
> when the caller in the deserializing jvm tries to assign it to Root.
>
> We can't design a system for all scenarios, all systems have limitations
> and must make compromises.
>
> Cheers,
>
> Peter.
>
> On 28/01/2017 10:46 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:
> > Sorry - you haven't shown how to avoid situation when a single
> > instance of BImpl is assignable to Child1.myImpl but NOT assignable to
> > Child2.myImpl.
> >
> > This is simply NOT possible without having information about exact
> > bundle wiring on the other end.
> >
> > Thanks,
> > Michal
> >
> > Peter wrote:
> >> Hmm, bundle wiring, not sure, will think about it.
> >>
> >> Yep, guilty of lazyness.
> >>
> >> So continuing on, with the deserialization process...
> >> The stream reads in the stream class information, now Root's
> >> ClassLoader is on the stack,
> >> Child2 is loaded using Root's Bundle.
> >> Next, Child2's bundle C2's ClassLoader is pushed onto the stack.
> >> Child2's fields are now read from the stream, in this case a
> >> reference number is read in from the stream,
> >> this reference number is looked up from the stream reference table,
> >> and the same instance of BImpl contained
> >> in Class1's field is returned.
> >> Serialization doesn't send an object twice, unless it's written
> >> unshared.
> >> Now Class2's instance is created, it's ClassLoader popped off the
> >> stack, root's second field is written, it's ClassLoader popped off
> >> the stack and its instance returned.
> >>
> >> If Class2's field was another instance of BImpl, the class is stored
> >> in the ObjectStreamClass's cache and has been verified and loaded, so
> >> we don't need to resolve it again.  If BImpl was actually a different
> >> version and was loaded by Bundle BImpl2.0.0, then the annotation
> >> won't imply BImpl1.0.0,
> >> so it will load a new bundle.
> >>
> >> The Bundle wiring will depend on a combination of the local Bundle's
> >> or downloaded Bundle's manifest package import declarations.
> >>
> >> The wiring may be slightly different at each end.  A PackageAdmin
> >> service may decide to change the loaded bundles, we need to ensure
> >> that we allow them to communicate accross compatible versions.
> >>
> >> Hope this helps clarify it a little better.
> >>
> >> Cheers & thanks,
> >>
> >> Peter.
> >>
> >> On 28/01/2017 10:11 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:
> >>> Ahh... You've missed the important part :) - child2.
> >>> You cannot assume BImpl class loaded in context of C1 is assignable

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Sorry - you haven't shown how to avoid situation when a single instance 
of BImpl is assignable to Child1.myImpl but NOT assignable to Child2.myImpl.


This is simply NOT possible without having information about exact 
bundle wiring on the other end.


Thanks,
Michal

Peter wrote:

Hmm, bundle wiring, not sure, will think about it.

Yep, guilty of lazyness.

So continuing on, with the deserialization process...
The stream reads in the stream class information, now Root's 
ClassLoader is on the stack,

Child2 is loaded using Root's Bundle.
Next, Child2's bundle C2's ClassLoader is pushed onto the stack.
Child2's fields are now read from the stream, in this case a reference 
number is read in from the stream,
this reference number is looked up from the stream reference table, 
and the same instance of BImpl contained

in Class1's field is returned.
Serialization doesn't send an object twice, unless it's written unshared.
Now Class2's instance is created, it's ClassLoader popped off the 
stack, root's second field is written, it's ClassLoader popped off the 
stack and its instance returned.


If Class2's field was another instance of BImpl, the class is stored 
in the ObjectStreamClass's cache and has been verified and loaded, so 
we don't need to resolve it again.  If BImpl was actually a different 
version and was loaded by Bundle BImpl2.0.0, then the annotation won't 
imply BImpl1.0.0,

so it will load a new bundle.

The Bundle wiring will depend on a combination of the local Bundle's 
or downloaded Bundle's manifest package import declarations.


The wiring may be slightly different at each end.  A PackageAdmin 
service may decide to change the loaded bundles, we need to ensure 
that we allow them to communicate accross compatible versions.


Hope this helps clarify it a little better.

Cheers & thanks,

Peter.

On 28/01/2017 10:11 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:

Ahh... You've missed the important part :) - child2.
You cannot assume BImpl class loaded in context of C1 is assignable 
to child2 - it depends on what other unrelated bundles have been 
loaded earlier and how the container chooses to resolve them.


Imagine:

BImpl has:
Require-Bundle bundle.with.exact.child2Api.version 1.0.0

Child2 has:
Import-Package packageOf.child2Api range [1.0.0, 2.0.0)

It might be perfectly fine in one container where only 
bundle.with.exact.child2Api.version 1.0.0 was installed. But might be 
wrong in another where there are two "packageOf.child2Api" exporting 
bundles installed.


So:
1. The information about BImpl is no longer on the stack and not 
available when loading Child2
2. Even if it was available (because for example you modify your 
algorithm so that you remember all previously loaded bundles) - it 
means that when loading Child2 you no longer can depend on the 
container bundle resolution.


And 2) is the point I am trying to make from the beginning: when 
transferring object graphs you MUST serialize the BundleWiring 
information and recreate it when deserializing. You need the snapshot 
of runtime state - declarative dependency information is not enough.


Thanks,
Michal

Peter wrote:

Ah yes, that's easy, assuming all classes are Serializable :)

Firstly we register a BundleListener with our OSGi framework and 
keep track of installed Bundles.


So Root, is the object graph root, Classes Child1 and Child2 are 
visible to Root.

The first thing we do is push the ClassLoader of Root onto our stack.
When the stream reads in the fields of Root, the first field child1, 
it will read the stream class information, the class Child1 will 
resolve from Bundle BR, since it is visible from there.
Now we obtain Child1's ClassLoader (which provides a BundleReference 
to Bundle C1) and push it onto our stack.
The stream reads in the fields of Child1, the first field Child1Api, 
is an instance of BImpl, it first attempts to use Bundle C1, however 
assuming that BImpl isn't visible from Bundle C1, we use the 
codebase annotation we read in from the stream (read in for each 
class), to find any installed bundles implied by the annotation's URI.
Upon finding BImpl's bundle from the BundleContext, we first ask the 
bundle to load the class, then we check that the class is assignable 
to field myImpl in Child1, before Child1 is instantiated, using 
class.isAssignableFrom.
If this fails and there are other versions of BImpl installed we 
might iterate through them until we have a match.
If there are no BImpl bundles installed, then we use the annotation 
to load a new Bundle.

At this time BImpl's ClassLoader is pushed onto the stack.
However BImpl has no fields to deserialize, so a new instance of 
BImpl is created and before it is used to set the field myImpl in 
Child1, BImpl's ClassLoader is popped off the stack.
Now that all fields of Child1 have been created, Child1 can be 
created, and bundle C1's ClassLoader is popped off the stack.

The same process is repeated for

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Ahh... You've missed the important part :) - child2.
You cannot assume BImpl class loaded in context of C1 is assignable to 
child2 - it depends on what other unrelated bundles have been loaded 
earlier and how the container chooses to resolve them.


Imagine:

BImpl has:
Require-Bundle bundle.with.exact.child2Api.version 1.0.0

Child2 has:
Import-Package packageOf.child2Api range [1.0.0, 2.0.0)

It might be perfectly fine in one container where only 
bundle.with.exact.child2Api.version 1.0.0 was installed. But might be 
wrong in another where there are two "packageOf.child2Api" exporting 
bundles installed.


So:
1. The information about BImpl is no longer on the stack and not 
available when loading Child2
2. Even if it was available (because for example you modify your 
algorithm so that you remember all previously loaded bundles) - it means 
that when loading Child2 you no longer can depend on the container 
bundle resolution.


And 2) is the point I am trying to make from the beginning: when 
transferring object graphs you MUST serialize the BundleWiring 
information and recreate it when deserializing. You need the snapshot of 
runtime state - declarative dependency information is not enough.


Thanks,
Michal

Peter wrote:

Ah yes, that's easy, assuming all classes are Serializable :)

Firstly we register a BundleListener with our OSGi framework and keep 
track of installed Bundles.


So Root, is the object graph root, Classes Child1 and Child2 are 
visible to Root.

The first thing we do is push the ClassLoader of Root onto our stack.
When the stream reads in the fields of Root, the first field child1, 
it will read the stream class information, the class Child1 will 
resolve from Bundle BR, since it is visible from there.
Now we obtain Child1's ClassLoader (which provides a BundleReference 
to Bundle C1) and push it onto our stack.
The stream reads in the fields of Child1, the first field Child1Api, 
is an instance of BImpl, it first attempts to use Bundle C1, however 
assuming that BImpl isn't visible from Bundle C1, we use the codebase 
annotation we read in from the stream (read in for each class), to 
find any installed bundles implied by the annotation's URI.
Upon finding BImpl's bundle from the BundleContext, we first ask the 
bundle to load the class, then we check that the class is assignable 
to field myImpl in Child1, before Child1 is instantiated, using 
class.isAssignableFrom.
If this fails and there are other versions of BImpl installed we might 
iterate through them until we have a match.
If there are no BImpl bundles installed, then we use the annotation to 
load a new Bundle.

At this time BImpl's ClassLoader is pushed onto the stack.
However BImpl has no fields to deserialize, so a new instance of BImpl 
is created and before it is used to set the field myImpl in Child1, 
BImpl's ClassLoader is popped off the stack.
Now that all fields of Child1 have been created, Child1 can be 
created, and bundle C1's ClassLoader is popped off the stack.

The same process is repeated for Child2.

Cheers,

Peter.




On 28/01/2017 7:41 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:
I fail to see how it could possibly work. Could you walk step-by-step 
serialize/deserialize with the following object graph:


Bundle API:
interface Api {}

Bundle BR:
class Root {
  Child1 child1;
  Child2 child2;

  Api getApi() {
return isEven(getRandom()) ? child1.impl : child2.impl;
  }

}

Bundle C1
class Child1 {
  Child1Api myImpl;
}

Bundle C2
class Child2 {
  Child2Api myImpl;
}

Bundle Api1:
interface Child1Api extends Api {}

Bundle Api2:
interface Child2Api extends Api {}

Bundle BImpl:
class Impl implements Child1Api, Child2Api {}

Object graph:
impl = new BImpl()
root = new Root(new Child1(impl), new Child2(impl));

Serialize and deserialize root.

Thanks,
Michal

Peter wrote:

So here's how we can put it together:

Our OIS contains a stack we can use to track ClassLoader's at each 
branch in our serialized object graph.


  1. First object being read in by the OIS, check the cache see if the
 URI annotation is implied.
  2. If yes, use this Bundle to deserialize, place this Bundle's
 BundleReference [ClassLoader] on the top of the stack.
  3. If no, walk the stack, find the first BundleReference, dereference
 it's Bundle, obtain the BundleContext and request a new Bundle
 with the new URL.
  4. For each field in the current object:

   * Try to load it's class from the current Bundle .
   * If successful push Bundle's ClassLoader onto stack and return
 class.
   * Catch ClassNotFoundException, it's likely this is a dependency
 injected class the parent object's ClassLoader or Bundle
 doesn't have type visibility for.
   * Check the Bundle cache to see if the URI annotation is 
implied.

   * If yes, then iterate through each, load a class from this
 bundle, check that the returned class can be assigned to 

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I would say that using TCCL as is a poor man's approach to class 
resolution. Once you have codebase identity done right - it is not 
needed anymore.


Thanks,
Michal

Gregg Wonderly wrote:

The commentary in the first document indicates that there is no rhyme or reason 
to the use of the context class loader.  For me, the reason was very obvious.  
Anytime that you are going to create a new class loader, you should set the 
parent class loader to the context class loader so that the calling thread 
environments class loading context will allow for classes referenced by the new 
class loader to resolve to classes that thread’s execution already can resolve.

What other use of a context class loader would happen?

Gregg


On Jan 27, 2017, at 11:39 AM, Bharath Kumar  wrote:

Yes Peter. Usage of thread context class loader is discouraged in OSGi
environment.

http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html

Some of the problems are hard to solve in OSGi environment. For example,
creating dynamic java proxy from 2 or more interfaces that are located in
different bundles.

http://blog.osgi.org/2008/08/classy-solutions-to-tricky-proxies.html?m=1

This problem can be solved using composite class loader. But it is
difficult to write it correctly. Because OSGi environment is dynamic.

I believe that it is possible to provide enough abstraction in river code,
so that service developers don't even require to use context class loader
in their services.



Thanks&  Regards,
Bharath


On 27-Jan-2017 6:25 PM, "Peter"  wrote:


Thanks Gregg,

Thoughts inline below.

Cheers,

Peter.


On 27/01/2017 12:35 AM, Gregg Wonderly wrote:


Is there any thought here about how a single client might use both an
OSGi deployed service and a conventionally deployed service?


Not yet, I'm currently considering how to support OSGi by implementing an
RMIClassLoaderSPI, similar to how Dennis has for Maven in Rio.

I think once a good understanding of OSGi has developed, we can consider
how an implementation could support that, possibly by exploiting something
like Pax URL built into PreferredClassProvider.


  The ContextClassLoader is a good abstraction mechanism for finding “the”

approriate class loader.  It allows applications to deploy a composite
class loader in some form that would be able to resolve classes from many
sources and even provide things like preferred classes.


Yes, it works well for conventional frameworks and is utilised by
PreferredClassProvider, but it's use in OSGi is discouraged, I'd like to
consider how it's use can be avoided in an OSGi env.



In a Java desktop application, would a transition from a background
thread, interacting with a service to get an object from a service which is
not completely resolved to applicable loaders still resolve correctly in an
EventDispatch Thread?  That event dispatch thread can have the context
class loader set on it by the thread which got the object, to be the class
loader of the service object, to make sure that the resolution of classes
happens with the correct class loader such that there will not be a problem
with the service object having one set of definitions and another service
or the application classpath having a conflicting class definition by the
same name.

I’ve had to spend quite a bit of time to make sure that these scenarios
work correctly in my Swing applications.


Have you got more information?  I'm guessing this relates to delayed
unmarshalling into the EventDispatch thread.

It's early days yet, I'm still working it out what information is required
to resolve the correct ClassLoaders&  bundles, but this is an important
question, Bharath mentioned Entry's can be utilised for versioning and this
seems like a good idea.

What follows are thoughts and observations.

A bundle can be created from a URL, provided the codebase the URL refers
to has an OSGi bundle manifest, so this could allow any of the existing URL
formats to deliver a proxy codebase for an OSGi framework.  When OSGi loads
the bundle, the package dependencies will be wired up by the local env.  If
the URL doesn't reference a bundle, then we could use Bharath's approach
and subclass the client's ClassLoader, this does make all the clients
classes visible to the proxy however, but that would happen anyway in a
standard Jini / River environment.

OSGi gives preference to already loaded bundles when resolving package
dependencies, so the client should be careful not to unmarshall any proxy's
that might require a higher version than the bundle already installed when
the client bundle resolved its dependencies.

One of the proxy bundle dependencies will be the service api bundle.  The
proxy bundle can limit the service api package / packages to a specific
version or version range, which it could advertise in an Entry.  Boolean
logic comparison of Entry's would have to be performed locally, after
matching on the service type (this 

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

In general I think implementing class resolution logic in stream 
implementation is bad. It has to be decoupled.


Thanks,
Michal

Peter wrote:

So here's how we can put it together:

Our OIS contains a stack we can use to track ClassLoader's at each 
branch in our serialized object graph.


  1. First object being read in by the OIS, check the cache see if the
 URI annotation is implied.
  2. If yes, use this Bundle to deserialize, place this Bundle's
 BundleReference [ClassLoader] on the top of the stack.
  3. If no, walk the stack, find the first BundleReference, dereference
 it's Bundle, obtain the BundleContext and request a new Bundle
 with the new URL.
  4. For each field in the current object:

   * Try to load it's class from the current Bundle .
   * If successful push Bundle's ClassLoader onto stack and return
 class.
   * Catch ClassNotFoundException, it's likely this is a dependency
 injected class the parent object's ClassLoader or Bundle
 doesn't have type visibility for.
   * Check the Bundle cache to see if the URI annotation is implied.
   * If yes, then iterate through each, load a class from this
 bundle, check that the returned class can be assigned to the
 current field's type.  If not, discard and continue until a
 class is found that can be assigned to the current field.
   * If not, load the new bundle.
   * Push the current Bundle's ClassLoader on the top of the stack.

  5. Pop the last (Object in the graph) field's ClassLoader off the 
stack.


When the process completes we have a resolved object graph with 
correct class visiblity.


Regards,

Peter.

On 28/01/2017 1:49 PM, Peter wrote:
Having implemented an ObjectInputStream (OIS), I can share the 
following insights:


   * The object at the head of the graph is externally visible and
 encapsulates all other objects and primitives.
   * The objects and primitives at the tails of the graph are fields.
   * Circular references in object graphs are unsafe with untrusted
 input...
   * A serialized object graph without circular references is a tree.
   * Fields may have been dependency injected.
   * Each branch in the tree has an object at the head of the branch.
   * The OIS knows the types of fields in the local class as well as
 the types of fields in the deserialized class before it has been
 resolved.

The information required to deserialized an OSGi graph is in the OIS 
implementation.  The OIS implementation is not visible to the 
extending MarshalInputStream implementation.


MarshalInputStream contains a ClassLoader field, defaultLoader.

So the codebase annotation is critical for the identity of the first 
object,


Regards,

Peter.

On 28/01/2017 3:39 AM, Bharath Kumar wrote:

Yes Peter. Usage of thread context class loader is discouraged in OSGi
environment.

http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html 



Some of the problems are hard to solve in OSGi environment. For 
example,
creating dynamic java proxy from 2 or more interfaces that are 
located in

different bundles.

http://blog.osgi.org/2008/08/classy-solutions-to-tricky-proxies.html?m=1 



This problem can be solved using composite class loader. But it is
difficult to write it correctly. Because OSGi environment is dynamic.

I believe that it is possible to provide enough abstraction in river 
code,
so that service developers don't even require to use context class 
loader

in their services.



Thanks&  Regards,
Bharath


On 27-Jan-2017 6:25 PM, "Peter"  wrote:


Thanks Gregg,

Thoughts inline below.

Cheers,

Peter.


On 27/01/2017 12:35 AM, Gregg Wonderly wrote:


Is there any thought here about how a single client might use both an
OSGi deployed service and a conventionally deployed service?

Not yet, I'm currently considering how to support OSGi by 
implementing an

RMIClassLoaderSPI, similar to how Dennis has for Maven in Rio.

I think once a good understanding of OSGi has developed, we can 
consider
how an implementation could support that, possibly by exploiting 
something

like Pax URL built into PreferredClassProvider.


   The ContextClassLoader is a good abstraction mechanism for 
finding “the”
approriate class loader.  It allows applications to deploy a 
composite
class loader in some form that would be able to resolve classes 
from many

sources and even provide things like preferred classes.


Yes, it works well for conventional frameworks and is utilised by
PreferredClassProvider, but it's use in OSGi is discouraged, I'd 
like to

consider how it's use can be avoided in an OSGi env.



In a Java desktop application, would a transition from a background
thread, interacting with a service to get an object from a service 
which is
not completely resolved to applicable loaders still resolve 
correctly in an
EventDispatch Thread?  That event dispatch thread can have the 
context
class loader 

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I fail to see how it could possibly work. Could you walk step-by-step 
serialize/deserialize with the following object graph:


Bundle API:
interface Api {}

Bundle BR:
class Root {
  Child1 child1;
  Child2 child2;

  Api getApi() {
return isEven(getRandom()) ? child1.impl : child2.impl;
  }

}

Bundle C1
class Child1 {
  Child1Api myImpl;
}

Bundle C2
class Child2 {
  Child2Api myImpl;
}

Bundle Api1:
interface Child1Api extends Api {}

Bundle Api2:
interface Child2Api extends Api {}

Bundle BImpl:
class Impl implements Child1Api, Child2Api {}

Object graph:
impl = new BImpl()
root = new Root(new Child1(impl), new Child2(impl));

Serialize and deserialize root.

Thanks,
Michal

Peter wrote:

So here's how we can put it together:

Our OIS contains a stack we can use to track ClassLoader's at each 
branch in our serialized object graph.


  1. First object being read in by the OIS, check the cache see if the
 URI annotation is implied.
  2. If yes, use this Bundle to deserialize, place this Bundle's
 BundleReference [ClassLoader] on the top of the stack.
  3. If no, walk the stack, find the first BundleReference, dereference
 it's Bundle, obtain the BundleContext and request a new Bundle
 with the new URL.
  4. For each field in the current object:

   * Try to load it's class from the current Bundle .
   * If successful push Bundle's ClassLoader onto stack and return
 class.
   * Catch ClassNotFoundException, it's likely this is a dependency
 injected class the parent object's ClassLoader or Bundle
 doesn't have type visibility for.
   * Check the Bundle cache to see if the URI annotation is implied.
   * If yes, then iterate through each, load a class from this
 bundle, check that the returned class can be assigned to the
 current field's type.  If not, discard and continue until a
 class is found that can be assigned to the current field.
   * If not, load the new bundle.
   * Push the current Bundle's ClassLoader on the top of the stack.

  5. Pop the last (Object in the graph) field's ClassLoader off the 
stack.


When the process completes we have a resolved object graph with 
correct class visiblity.


Regards,

Peter.

On 28/01/2017 1:49 PM, Peter wrote:
Having implemented an ObjectInputStream (OIS), I can share the 
following insights:


   * The object at the head of the graph is externally visible and
 encapsulates all other objects and primitives.
   * The objects and primitives at the tails of the graph are fields.
   * Circular references in object graphs are unsafe with untrusted
 input...
   * A serialized object graph without circular references is a tree.
   * Fields may have been dependency injected.
   * Each branch in the tree has an object at the head of the branch.
   * The OIS knows the types of fields in the local class as well as
 the types of fields in the deserialized class before it has been
 resolved.

The information required to deserialized an OSGi graph is in the OIS 
implementation.  The OIS implementation is not visible to the 
extending MarshalInputStream implementation.


MarshalInputStream contains a ClassLoader field, defaultLoader.

So the codebase annotation is critical for the identity of the first 
object,


Regards,

Peter.

On 28/01/2017 3:39 AM, Bharath Kumar wrote:

Yes Peter. Usage of thread context class loader is discouraged in OSGi
environment.

http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html 



Some of the problems are hard to solve in OSGi environment. For 
example,
creating dynamic java proxy from 2 or more interfaces that are 
located in

different bundles.

http://blog.osgi.org/2008/08/classy-solutions-to-tricky-proxies.html?m=1 



This problem can be solved using composite class loader. But it is
difficult to write it correctly. Because OSGi environment is dynamic.

I believe that it is possible to provide enough abstraction in river 
code,
so that service developers don't even require to use context class 
loader

in their services.



Thanks&  Regards,
Bharath


On 27-Jan-2017 6:25 PM, "Peter"  wrote:


Thanks Gregg,

Thoughts inline below.

Cheers,

Peter.


On 27/01/2017 12:35 AM, Gregg Wonderly wrote:


Is there any thought here about how a single client might use both an
OSGi deployed service and a conventionally deployed service?

Not yet, I'm currently considering how to support OSGi by 
implementing an

RMIClassLoaderSPI, similar to how Dennis has for Maven in Rio.

I think once a good understanding of OSGi has developed, we can 
consider
how an implementation could support that, possibly by exploiting 
something

like Pax URL built into PreferredClassProvider.


   The ContextClassLoader is a good abstraction mechanism for 
finding “the”
approriate class loader.  It allows applications to deploy a 
composite
class loader in some form that would be able to resolve classes 
from many

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I also think about adding leasing to the scheme.
If CodeBaseModule can be leased (and the client is capable of handling 
declines of lease renewals) - it would be quite straightforward to 
implement auto-upgrade: the lease for a module "mymodule" ver 1.1 
expires and you have to ask the code server for a new CodeBaseModule - 
which in turn could return a newer patched version of it.


Cheers,
Michal

Michał Kłeczek (XPro Sp. z o. o.) wrote:
So for a client and a service to be able to communicate they must 
agree on a common set of interchangeable CodeRepositories that would 
allow them to have a common understanding of names.
In other words - to be able to work - any party first has to contact a 
CodeRepository that can authenticate itself as a particular principal. 
The issue is that to find the CodeRepository one needs to communicate 
with ServiceRegistrar. And to communicate with ServiceRegistrar you 
need a CodeRepository!!!. So there needs to be some bootstrapping in 
place:

- either ServiceRegistrar and CodeRepository constitute as single entity
- there is a bootstrap well known CodeRepository (Maven central?) - 
its implementation is based on a well known URL and its implementation 
code is shipped with the framework.


Thanks,
Michal

Michał Kłeczek (XPro Sp. z o. o.) wrote:
Honestly - since I am fixed ( :-) ) on having mobile code treated as 
any other object - I see it something like:


interface CodeBaseModule {
  ClassLoader createLoader() throws AnyImportantException;
}

interface CodeRepository {
  CodeBaseModule getCodeBaseModule(String name, Version version);
  boolean isSameNamespace(CodeRepository other);
}

class NamedCodeBase {
  String name; Version version;
  CodeRepository repository;
  boolean equals(Object other) { //check name, version and repo }
}

Now the question is about the implementation of "isSameNamespace". 
Since the protocol(s) to access code repository might differ (and 
there might be multiple available at the same time), location based 
equality won't work (although is the easiest to implement). My rough 
idea is for the CodeRepository to be able to authenticate as any of a 
set of Principals ( ie. satisfy the ServerMinPrincipal constraint ). 
Two CodeRepository instances are interchangeable if intersection of 
their principal sets is non-empty.


At first I thought about having a global naming scheme - then 
cryptographic hash would constitute the part of the name. But that 
would make names obscure and difficult to remember and write by hand.
So I came up with an idea to abstract it away - according to "all 
problems in CS can be solved by introducing another level of 
indirection" :)


Thanks,
Michal

Peter wrote:

codebase identity

So River codebase identity is currently any number of space delimited RFC 3986 
normalised URI strings.

httpmd uses a location filename and message digest.

But should location be part of identity?  How can you relocate a codebase once 
remote objects are deployed?

OSGi and Maven use a name and version to identify a codebase.  


Might we also need codebase signers (if any) to be part of identity?

If no, why not and if yes why?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 26/01/2017 08:30:58 am
To:d...@riverapache.org
Subject: Re: OSGi

I haven't been aware of ObjectSpace Voyager. I just briefly looked at it 
and it seems like it is based on Java 1.x (ancient beast) and - as I 
understand it - the issues you describe are mainly caused by having only 
a single class name space (single ClassLoader).


But sending IMHO class bytes in-band is not necessary (nor good).

What is needed is:
1. Encoding dependency information in codebases (either in-band or by 
providing a downloadable descriptor) so that it is possible to recreate 
proper ClassLoader structure (hierarchy or rather graph - see below) on 
the client.
2. Provide non-hierarchical class loading to support arbitrary object 
graph deserialization (otherwise there is a problem with "diamond 
shaped" object graphs).


A separate issue is with the definition of codebase identity. I guess 
originally Jini designers wanted to avoid this issue and left it 
undefined... but it is unavoidable :)


Thanks,
Michal

Gregg Wonderly wrote:

  That’s what I was suggesting.  The code works, but only if you put the 
required classes into codebases or class paths.  It’s not a problem with mobile 
code, it’s a problem with resolution of objects in mobile code references.  
That’s why I mentioned ObjectSpace Voyager.  It automatically sent/sends class 
definitions with object graphs to the remote VM.

  Gregg


  On Jan 23, 2017, at 3:03 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>   wrote:

  The problem is that we only support (smart) proxies that reference only 
objects of cl

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Unfortunatelly this is not that easy - the assumption that "the same 
bundle is going to be resolved in the same way in another OSGI 
container" is false.
Not only the containers must share common provisioning infrastructure 
(single view of the names) but - most importantly - the particular 
wiring of a bundle is TIME dependent. It can depend on what other 
bundles were loaded earlier and how the container resolved them. The 
container is free to choose any wiring that meets the requirements.


Thanks,
Michal

Peter wrote:
  
So lets say for argument sake, that we've got River "bundles" that are annotated with package imports (dependencies) and exports.


Using Bharath's proposed 3 bundle nomenclature for services...

Lets say that a third party services defines a service api in a bundle.  
Service api must only change in a backward compatible manner.

A client imports the service api packages.

A service proxy imports the service api packages.

The service api classes are already loaded in the client jvm because the client 
imported them.

The service proxy is deserialised in the client jvm.  Before the proxy can be 
deserialized the RMIClassLoader must first determine whether the proxy's bundle 
(exact version) exists, if not it needs to request OSGi to provision&  load 
that bundle.

When the proxy bundle is loaded, it imports the same service api packages 
visible to the client.

But how do we ensure we have a compatible service api in the client jvm?

Because the lookup service finds matching interfaces.  When those service api 
interfaces were marshalled as arguments to the lookup service by the client, 
they were matched on serial form, so the client will only ever receive 
compatible results.

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message ----
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 26/01/2017 08:30:58 am
To: dev@river.apache.org
Subject: Re: OSGi

I haven't been aware of ObjectSpace Voyager. I just briefly looked at it 
and it seems like it is based on Java 1.x (ancient beast) and - as I 
understand it - the issues you describe are mainly caused by having only 
a single class name space (single ClassLoader).


But sending IMHO class bytes in-band is not necessary (nor good).

What is needed is:
1. Encoding dependency information in codebases (either in-band or by 
providing a downloadable descriptor) so that it is possible to recreate 
proper ClassLoader structure (hierarchy or rather graph - see below) on 
the client.
2. Provide non-hierarchical class loading to support arbitrary object 
graph deserialization (otherwise there is a problem with "diamond 
shaped" object graphs).


A separate issue is with the definition of codebase identity. I guess 
originally Jini designers wanted to avoid this issue and left it 
undefined... but it is unavoidable :)


Thanks,
Michal

Gregg Wonderly wrote:

  That’s what I was suggesting.  The code works, but only if you put the 
required classes into codebases or class paths.  It’s not a problem with mobile 
code, it’s a problem with resolution of objects in mobile code references.  
That’s why I mentioned ObjectSpace Voyager.  It automatically sent/sends class 
definitions with object graphs to the remote VM.

  Gregg


  On Jan 23, 2017, at 3:03 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>   wrote:

  The problem is that we only support (smart) proxies that reference only 
objects of classes from their own code base.
  We do not support cases when a (smart) proxy wraps a (smart) proxy of another 
service (annotated with different codebase).

  This precludes several scenarios such as for example "dynamic exporters" - 
exporters that are actually smart proxies.

  Thanks,
  Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

So for a client and a service to be able to communicate they must agree 
on a common set of interchangeable CodeRepositories that would allow 
them to have a common understanding of names.
In other words - to be able to work - any party first has to contact a 
CodeRepository that can authenticate itself as a particular principal. 
The issue is that to find the CodeRepository one needs to communicate 
with ServiceRegistrar. And to communicate with ServiceRegistrar you need 
a CodeRepository!!!. So there needs to be some bootstrapping in place:

- either ServiceRegistrar and CodeRepository constitute as single entity
- there is a bootstrap well known CodeRepository (Maven central?) - its 
implementation is based on a well known URL and its implementation code 
is shipped with the framework.


Thanks,
Michal

Michał Kłeczek (XPro Sp. z o. o.) wrote:
Honestly - since I am fixed ( :-) ) on having mobile code treated as 
any other object - I see it something like:


interface CodeBaseModule {
  ClassLoader createLoader() throws AnyImportantException;
}

interface CodeRepository {
  CodeBaseModule getCodeBaseModule(String name, Version version);
  boolean isSameNamespace(CodeRepository other);
}

class NamedCodeBase {
  String name; Version version;
  CodeRepository repository;
  boolean equals(Object other) { //check name, version and repo }
}

Now the question is about the implementation of "isSameNamespace". 
Since the protocol(s) to access code repository might differ (and 
there might be multiple available at the same time), location based 
equality won't work (although is the easiest to implement). My rough 
idea is for the CodeRepository to be able to authenticate as any of a 
set of Principals ( ie. satisfy the ServerMinPrincipal constraint ). 
Two CodeRepository instances are interchangeable if intersection of 
their principal sets is non-empty.


At first I thought about having a global naming scheme - then 
cryptographic hash would constitute the part of the name. But that 
would make names obscure and difficult to remember and write by hand.
So I came up with an idea to abstract it away - according to "all 
problems in CS can be solved by introducing another level of 
indirection" :)


Thanks,
Michal

Peter wrote:

codebase identity

So River codebase identity is currently any number of space delimited RFC 3986 
normalised URI strings.

httpmd uses a location filename and message digest.

But should location be part of identity?  How can you relocate a codebase once 
remote objects are deployed?

OSGi and Maven use a name and version to identify a codebase.  


Might we also need codebase signers (if any) to be part of identity?

If no, why not and if yes why?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 26/01/2017 08:30:58 am
To:d...@riverapache.org
Subject: Re: OSGi

I haven't been aware of ObjectSpace Voyager. I just briefly looked at it 
and it seems like it is based on Java 1.x (ancient beast) and - as I 
understand it - the issues you describe are mainly caused by having only 
a single class name space (single ClassLoader).


But sending IMHO class bytes in-band is not necessary (nor good).

What is needed is:
1. Encoding dependency information in codebases (either in-band or by 
providing a downloadable descriptor) so that it is possible to recreate 
proper ClassLoader structure (hierarchy or rather graph - see below) on 
the client.
2. Provide non-hierarchical class loading to support arbitrary object 
graph deserialization (otherwise there is a problem with "diamond 
shaped" object graphs).


A separate issue is with the definition of codebase identity. I guess 
originally Jini designers wanted to avoid this issue and left it 
undefined... but it is unavoidable :)


Thanks,
Michal

Gregg Wonderly wrote:

  That’s what I was suggesting.  The code works, but only if you put the 
required classes into codebases or class paths.  It’s not a problem with mobile 
code, it’s a problem with resolution of objects in mobile code references.  
That’s why I mentioned ObjectSpace Voyager.  It automatically sent/sends class 
definitions with object graphs to the remote VM.

  Gregg


  On Jan 23, 2017, at 3:03 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>   wrote:

  The problem is that we only support (smart) proxies that reference only 
objects of classes from their own code base.
  We do not support cases when a (smart) proxy wraps a (smart) proxy of another 
service (annotated with different codebase).

  This precludes several scenarios such as for example "dynamic exporters" - 
exporters that are actually smart proxies.

  Thanks,
  Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Honestly - since I am fixed ( :-) ) on having mobile code treated as any 
other object - I see it something like:


interface CodeBaseModule {
  ClassLoader createLoader() throws AnyImportantException;
}

interface CodeRepository {
  CodeBaseModule getCodeBaseModule(String name, Version version);
  boolean isSameNamespace(CodeRepository other);
}

class NamedCodeBase {
  String name; Version version;
  CodeRepository repository;
  boolean equals(Object other) { //check name, version and repo }
}

Now the question is about the implementation of "isSameNamespace". Since 
the protocol(s) to access code repository might differ (and there might 
be multiple available at the same time), location based equality won't 
work (although is the easiest to implement). My rough idea is for the 
CodeRepository to be able to authenticate as any of a set of Principals 
( ie. satisfy the ServerMinPrincipal constraint ). Two CodeRepository 
instances are interchangeable if intersection of their principal sets is 
non-empty.


At first I thought about having a global naming scheme - then 
cryptographic hash would constitute the part of the name. But that would 
make names obscure and difficult to remember and write by hand.
So I came up with an idea to abstract it away - according to "all 
problems in CS can be solved by introducing another level of indirection" :)


Thanks,
Michal

Peter wrote:

codebase identity

So River codebase identity is currently any number of space delimited RFC 3986 
normalised URI strings.

httpmd uses a location filename and message digest.

But should location be part of identity?  How can you relocate a codebase once 
remote objects are deployed?

OSGi and Maven use a name and version to identify a codebase.  


Might we also need codebase signers (if any) to be part of identity?

If no, why not and if yes why?

Regards,

Peter.

Sent from my Samsung device.
  
   Include original message

 Original message 
From: "Michał Kłeczek (XPro Sp. z o. o.)"<michal.klec...@xpro.biz>
Sent: 26/01/2017 08:30:58 am
To: d...@riverapache.org
Subject: Re: OSGi

I haven't been aware of ObjectSpace Voyager. I just briefly looked at it 
and it seems like it is based on Java 1.x (ancient beast) and - as I 
understand it - the issues you describe are mainly caused by having only 
a single class name space (single ClassLoader).


But sending IMHO class bytes in-band is not necessary (nor good).

What is needed is:
1. Encoding dependency information in codebases (either in-band or by 
providing a downloadable descriptor) so that it is possible to recreate 
proper ClassLoader structure (hierarchy or rather graph - see below) on 
the client.
2. Provide non-hierarchical class loading to support arbitrary object 
graph deserialization (otherwise there is a problem with "diamond 
shaped" object graphs).


A separate issue is with the definition of codebase identity. I guess 
originally Jini designers wanted to avoid this issue and left it 
undefined... but it is unavoidable :)


Thanks,
Michal

Gregg Wonderly wrote:

  That’s what I was suggesting.  The code works, but only if you put the 
required classes into codebases or class paths.  It’s not a problem with mobile 
code, it’s a problem with resolution of objects in mobile code references.  
That’s why I mentioned ObjectSpace Voyager.  It automatically sent/sends class 
definitions with object graphs to the remote VM.

  Gregg


  On Jan 23, 2017, at 3:03 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>   wrote:

  The problem is that we only support (smart) proxies that reference only 
objects of classes from their own code base.
  We do not support cases when a (smart) proxy wraps a (smart) proxy of another 
service (annotated with different codebase).

  This precludes several scenarios such as for example "dynamic exporters" - 
exporters that are actually smart proxies.

  Thanks,
  Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

I haven't been aware of ObjectSpace Voyager. I just briefly looked at it 
and it seems like it is based on Java 1.x (ancient beast) and - as I 
understand it - the issues you describe are mainly caused by having only 
a single class name space (single ClassLoader).


But sending IMHO class bytes in-band is not necessary (nor good).

What is needed is:
1. Encoding dependency information in codebases (either in-band or by 
providing a downloadable descriptor) so that it is possible to recreate 
proper ClassLoader structure (hierarchy or rather graph - see below) on 
the client.
2. Provide non-hierarchical class loading to support arbitrary object 
graph deserialization (otherwise there is a problem with "diamond 
shaped" object graphs).


A separate issue is with the definition of codebase identity. I guess 
originally Jini designers wanted to avoid this issue and left it 
undefined... but it is unavoidable :)


Thanks,
Michal

Gregg Wonderly wrote:

That’s what I was suggesting.  The code works, but only if you put the required 
classes into codebases or class paths.  It’s not a problem with mobile code, 
it’s a problem with resolution of objects in mobile code references.  That’s 
why I mentioned ObjectSpace Voyager.  It automatically sent/sends class 
definitions with object graphs to the remote VM.

Gregg


On Jan 23, 2017, at 3:03 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:

The problem is that we only support (smart) proxies that reference only objects 
of classes from their own code base.
We do not support cases when a (smart) proxy wraps a (smart) proxy of another 
service (annotated with different codebase).

This precludes several scenarios such as for example "dynamic exporters" - 
exporters that are actually smart proxies.

Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

The problem is that we only support (smart) proxies that reference only 
objects of classes from their own code base.
We do not support cases when a (smart) proxy wraps a (smart) proxy of 
another service (annotated with different codebase).


This precludes several scenarios such as for example "dynamic exporters" 
- exporters that are actually smart proxies.


Thanks,
Michal

Gregg Wonderly wrote:

I guess I am not sure then what you are trying to show with your example.

Under what case would the SpacePublisher be sent to another VM, and how is that 
different from normal SmartProxy deserialization?

Gregg


On Jan 23, 2017, at 2:39 PM, Michał Kłeczek (XPro Sp. z o. 
o.)<michal.klec...@xpro.biz>  wrote:



Gregg Wonderly wrote:

michal.klec...@xpro.biz<mailto:michal.klec...@xpro.biz>  
<mailto:michal.klec...@xpro.biz>  <mailto:michal.klec...@xpro.biz>>  wrote:

The use case and the ultimate test to implement is simple - have a

listener that publishes remote events to a JavaSpace acquired dynamically
from a lookup service:

class SpacePublisher implements RemoteEventListener, Serializable {
   private final JavaSpace space;
   public void notify(RemoteEvent evt) {
 space.write(createEntry(evt), ...);
   }
}

It is NOT possible to do currently. It requires non-hierarchical class

loading. It is not easy to solve. It would open a whole lot of
possibilities.

I am probably too ignorant to see it; What exactly is "NOT possible" with
the above use-case snippet?

With currently implemented PreferredClassProvider it is not possible to 
deserialize such an object graph.

This can happen, but what’s necessary is that the codebase of the 
SpacePublisher needs to include all the possible RemoteEvent classes, or the 
javaspace’s classpath has to include them.

I am not sure I understand.
The problem does not have anything to do with RemoteEvent (sub)classes. The 
issue is that SpacePublisher cannot be deserialized at all ( except one case 
when JavaSpace interface is available from context class loader and it is not 
marked as preferred in SpacePublisher code base).

Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Gregg Wonderly wrote:

michal.klec...@xpro.biz > wrote:

The use case and the ultimate test to implement is simple - have a

listener that publishes remote events to a JavaSpace acquired dynamically
from a lookup service:

class SpacePublisher implements RemoteEventListener, Serializable {
   private final JavaSpace space;
   public void notify(RemoteEvent evt) {
 space.write(createEntry(evt), ...);
   }
}

It is NOT possible to do currently. It requires non-hierarchical class

loading. It is not easy to solve. It would open a whole lot of
possibilities.

I am probably too ignorant to see it; What exactly is "NOT possible" with
the above use-case snippet?

With currently implemented PreferredClassProvider it is not possible to 
deserialize such an object graph.


This can happen, but what’s necessary is that the codebase of the 
SpacePublisher needs to include all the possible RemoteEvent classes, or the 
javaspace’s classpath has to include them.

I am not sure I understand.
The problem does not have anything to do with RemoteEvent (sub)classes. 
The issue is that SpacePublisher cannot be deserialized at all ( except 
one case when JavaSpace interface is available from context class loader 
and it is not marked as preferred in SpacePublisher code base).


Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Hi,

comments below.

Niclas Hedhman wrote:

On Mon, Jan 23, 2017 at 1:48 AM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


I would say fully declarative approach in OSGI would be to only annotate

with a package version range (and let the OSGI container do the resolution
- it is designed to do it efficiently).

Of course then we have a problem with classes that are not from any

exported package but from internal bundle class path - then bundle id and
version might be necessary.

Then of course there is a question of classes from fragment bundles -

this is totally unsolvable since AFAIK there is no way to get the fragment
information based on the class loader of a class.

Not that I grasp why it is needed, but you can get the Bundle information
from the classloader, IF the class was loaded by OSGi. OSGi defines
BundleClassLoader, so you simply cast the classloader to that, and with
that you can get the Bundle and the BundleContext, which together should be
able to give you all the meta info that you could need.


It is needed to properly annotate a class during serialization.
AFAIK in OSGI you can not find out which fragment (if any) was used to 
load a class.

Based on the class loader you can get the bundle, but not the fragment.

The concept of "download the proxy, both code and state" was quite 
novel in 1999 when I first heard of Jini. But I think it should be 
recognized that it both introduces a lot of security problems as well 
as not being the only way this can be done. For instance, the proxy 
could be an HTML page, which downloads JavaScript which binds into a 
sandbox provided. I think that the "serialized object" should not be a 
requirement eventually, and with that removed, the OSGi environment 
can help quite considerably aiding the dynamic nature of River. 

My view is different:

1. Without this "serialized object" approach there is no such a thing as 
River since there is nothing interesting and novel left.
2. Indeed - the "serialized object" approach is not popular... But my 
take on this is that it is because the concept has not been implemented 
correctly - both in terms of security and in terms of class loading.


As Peter observed, Pax URL allows a whole bunch of URL schemes, which 
you could be annotated in the serialized objects, just like URL 
codebases are today. For instance, Maven "coordinates" could be in the 
annotations and OSGi loads bundles on-demand. Paremus also implemented 
a "bundle garbage collector" in some manner, which unloaded unused 
bundles eventually. Furthermore, there are defined hooks in OSGi for 
the resolution of service registration and service lookup, which I 
think River should exploit. There seems to be a huge complementary 
intersection right there. 

There are no hooks for _class_ resolution - that's what is needed.


The use case and the ultimate test to implement is simple - have a

listener that publishes remote events to a JavaSpace acquired dynamically
from a lookup service:

class SpacePublisher implements RemoteEventListener, Serializable {
   private final JavaSpace space;
   public void notify(RemoteEvent evt) {
 space.write(createEntry(evt), ...);
   }
}

It is NOT possible to do currently. It requires non-hierarchical class

loading. It is not easy to solve. It would open a whole lot of
possibilities.

I am probably too ignorant to see it; What exactly is "NOT possible" with
the above use-case snippet?
With currently implemented PreferredClassProvider it is not possible to 
deserialize such an object graph.


Thanks,
Michal

Re: OSGi

[Michał Kłeczek (XPro Sp. z o. o.)]

Hi,

Bharath Kumar wrote:


2. We can annotate the proxy object using osgi bundle symbolic name and 


version.
3. RMIClassLoader provider can check whether the proxy bundle is
installed or not, If it is not installed, it can install it from configured 

repo ( like OBR). We can even use pax-url which adds different URL handlers. 


4. Load class/proxy class from correct proxy bundle


If it only was that easy it would have been done a long time ago :)

The question "how to annotate a class" is difficult to answer.
I can see two approaches:

1. Declarative specification of _requirements_ to successfully 
deserialize an object (ie. a proxy). The client must have all the 
necessary mechanisms in place to resolve, download and install anything 
required to satisfy these requirements.


2. Fully specifying what code to download (and ideally - how). This is 
the approach of River at this moment. The client simply uses provided 
URLs to create an URLClassLoader (of course adding security and 
preferred resources but in general that's how it works)


Annotating with bundle id and version is actually half way - it neither 
is self contained and sufficient to download code and it does not allow 
the client to fully decide how the requirement should be satisfied.


And since the client bundle and the service bundle might be resolved 
differently by the container - it may lead to all sorts of pseudorandom 
class cast exceptions.


I would say fully declarative approach in OSGI would be to only annotate 
with a package version range (and let the OSGI container do the 
resolution - it is designed to do it efficiently).
Of course then we have a problem with classes that are not from any 
exported package but from internal bundle class path - then bundle id 
and version might be necessary.
Then of course there is a question of classes from fragment bundles - 
this is totally unsolvable since AFAIK there is no way to get the 
fragment information based on the class loader of a class.


And the main issue with this approach IMHO is that it requires a central 
authority that governs the naming and versioning of bundles.


Approach 2 in OSGI would require annotating a class not with a bundle 
but with a piece of information that would allow the client to download 
the _BundleWiring_ of the proxy - it unambiguously specifies the class 
loader graph that is required to be recreated by the client to 
deserialize an object..


When I tried to implement it - it appeared to me that the main issue is 
that it simply makes the whole OSGI with its declarative dependency 
resolution moot - we need to totally bypass it to transfer objects 
between containers anyway. So the only thing we actually would use from 
an OSGI container is its capability of non-hierarchical class loading.


One step towards solving the above issues is to implement the idea of 
code base annotations being objects implementing a well known interface 
- that would at least allow us to abstract away from the exact format of 
the annotation data. But I do not have a clear idea on how to solve 
other OSGI integration issues.


The use case and the ultimate test to implement is simple - have a 
listener that publishes remote events to a JavaSpace acquired 
dynamically from a lookup service:


class SpacePublisher implements RemoteEventListener, Serializable {
  private final JavaSpace space;
  public void notify(RemoteEvent evt) {
space.write(createEntry(evt), ...);
  }
}

It is NOT possible to do currently. It requires non-hierarchical class 
loading. It is not easy to solve. It would open a whole lot of 
possibilities.


My 2 cents :)
Michal

Re: site revamp

[Michał Kłeczek (XPro Sp. z o. o.)]

Hi,

Great job!

Could you link the mother of all Jini guides?
https://jan.newmarch.name/java/jini/tutorial/Jini.html

Thanks,
Michal


Geoffrey Arnold 
December 22, 2016 at 8:44 PM
Hey Zsolt, really fantastic job. Well done!






Zsolt Kúti 
December 22, 2016 at 6:24 PM
Hello,

The revamped site is now staged and can be reviewed here:
http://river.staging.apache.org/

Community decides when to publish it.

Cheers,
Zsolt

Re: Maven Build

[Michał Kłeczek (XPro Sp. z o. o.)]

Indeed - it is possible to specify your dependencies as tightly as you 
want in OSGi. The issue is that:

1) It defeats one of the main purposes of OSGi in the first place
2) Nobody does that which would mean the only objects you can send over 
the wire would be instances of classes from specially crafted bundles


I agree the same problems are present in standard Java class loading - 
this is one of the reasons why I say River class loading is broken and 
needs to be fixed :)


The discussion about JBoss Modules goals is not really important in this 
context. I am only using it as an implementation of non-hierarchical 
class loading. Module identification and dependency resolution needs to 
be River specific anyway. After doing some experiments and analysis I 
dumped OSGi (which was my first choice and I am really willing to 
discuss OSGi integration - would love to see it done properly). JBoss 
Modules is a "bare bones" library that provides the mechanism while not 
imposing any policy. OSGi container can be (and is) implemented on top 
of it. River can be implemented on top of it as well.


The requirements for me generally are:
- downloaded code must not be executed before it is verified
- it should be easy to provide "composite services" - for example 
RemoteEventListener implementation wrapping JavaSpace proxy and 
publishing events to the space. At this moment this is impossible and in 
general requires non-hierarchical class loading.
- programmers should not be required to provide any River specific 
information inside jar files or any River specific "descriptors". This 
is difficult so if any such descriptor is absolutely necessary - build 
tools should be provided that automate this process.

- no specific deployment format and/or structure should be imposed

Two first are must have and two later - nice to have (or be as close as 
possible).


Some design decisions (in no particular order):

- codebase is an implementation of an interface (instead of being a 
String). This will make the mechanism extensible and - hopefully - 
future proof.
- codebase implementation classes are going to be resolved recursively 
using exactly the same algorithms (eat your own dog food). Recursion 
depth limited by a (configurable?) constant.
- codebase identity based on object equality (codebase implementation 
required to implement equals and hashcode). The implementation should be 
(directly or indirectly) based on cryptographic hashes equality. It is 
important not to base codebase identity on names only.
- since codebases are objects - they can be verified before use so no 
downloaded code is executed before use. What's more - since classes of 
objects are known to be trusted already - objects may safely verify 
themselves! What it means in general is that all TrustVerifier concept 
and implementation is not needed anymore in River. Objects can simply 
use built in serialization/ObjectInputValidation mechanisms and/or 
implement a River specific interface for this.
- JBoss Modules as an implementation of non-hierarchical class loading - 
BUT it is only one of possible implementations. Implementation based on 
PreferredClassLoader is possible as well (and might be provided as 
legacy fallback - but not that important at this moment since it is 
broken and I see the whole effort as River 3.0 breaking backwards 
compatibility anyway).
- basically "CodeBase" implementation needs to provide only one method: 
"ClassLoader createClassLoader()" (some more might be required to handle 
related issues like granting permissions to codebases so that they can 
create class loaders)


Some outstanding design questions:
- how to handle "private" codebases: for example my RemoteEventListener 
wraps a JavaSpace proxy and we want to treat the JavaSpace proxy 
codebase as "local" to the wrapper - it should get a separate 
ClassLoader even if the actual code is the same and is downloaded from 
the same location. It basically means location or content itself is not 
enough to establish identity. We also need something more to distinguish 
them. Changing the identity of the codebase might cause "lost codebase" 
issues so it has to be done properly.
- How to implement permission grants properly? Granting to class loaders 
requires duplicating class loaders just to have separate permission sets 
per object. Maybe something else is required based on object (not class) 
identity. That would also allow solving the problem above.


Thanks,
Michal

Niclas Hedhman <mailto:nic...@hedhman.org>
November 16, 2016 at 11:53 PM
On Wed, Nov 16, 2016 at 8:43 PM, "Michał Kłeczek (XPro Sp. z o. o.)"<
michal.klec...@xpro.biz>  wrote:


3. My comment about OSGi being "non-deterministic" in resolving

dependencies means that the

same bundle installed in two different environments is going to be linked

with different dependent

bundles. I

Re: Maven Build

[Michał Kłeczek (XPro Sp. z o. o.)]

1. I am aware of both Remote Services and Remote Services Admin 
specifications. I would be happy to see how they deal with dynamic code 
downloading but unfortunately they simply don't:
"This specification does not mandate the technique used to proxy an 
Endpoint as a service in the OSGi

framework." - taken from "OSGi Compendium Release 6 Specification"
See point 3. below for some inherent issues with OSGi that make it 
difficult to implement marshalling/unmarshalling as required by River.


2. It has been a couple of years now since I last checked how Paremus 
implemented Jini components. At that time the whole dynamic code 
downloading has been based on PreferredClassProvider and 
"java.rmi.codebase" property which would suggest it did not really solve 
issues with code downloading in Jini at all. I would be happy to be 
proven wrong.


3. My comment about OSGi being "non-deterministic" in resolving 
dependencies means that the same bundle installed in two different 
environments is going to be linked with different dependent bundles. It 
basically means the same object graph valid in one VM instance cannot be 
deserialized in another VM instance unless:
a) all bundles have their dependencies specified in such a way that 
exact same dependency graph is going to be present everywhere (meaning 
no Package-Imports, no version ranges etc)
b) all JVMs have the same set of bundles installed (ie - there is no 
dynamic code downloading at all - the code is preloaded)
c) classes of serialized objects are not loaded by OSGi framework class 
loaders but by other non-related class loading framework (which is what 
original old OSGi-Jini spec did)
d) classes of all objects in the object graph are from the same bundle 
(special case of a) )


In general all efforts I am aware of regarding "remoting" in Java didn't 
really even try to solve issues of dynamically downloaded code. They 
were based on all parties having the same code installed and/or having a 
central authority providing a shared consistent view of all available 
software.
I am not saying it is wrong. I am only saying that IMHO it is not 
something River should do - it is simply a solved problem and there is 
not point of re-doing it (JERI is cool but it is simply 
yet-another-RPC-stack)


Regarding JBoss Modules:
I am not really advocating this particular library - it is just that it 
is the only (non-OSGi - see above :) ) implementation of 
non-hierarchical class loading that I am aware of which is of good 
quality and actively maintained. I wouldn't want to reimplement it by 
myself. Thought about ClassWorlds but it doesn't seem to be too active 
and had some performance problems in the past. JBoss Modules is Apache 
licensed so it can be used by River.


Thanks,
Michal


On Wednesday, 16 November 2016, Richard Nicholson 
<richard.nichol...@paremus.com <mailto:richard.nichol...@paremus.com>> 
wrote:


   Agree with Niclas. I don’t understand the resolution comment.

   Not only is OSGi <> Jini integration feasible but it is a historical
   fact. Paremus did this - and a pretty good job we did - in …
   something like 2006 :-/

   If the River community decided that there is interest in OSGi - then
   I’d suggest reading the Remote Service and Remote Service Admin
   specifications and thinking about how Jini concepts might enhance
   that world view. There are well over 10 million OSGi enabled IoT
   gateways out there!

   Sorry - I see no compelling technical or commercial arguments for
   the JBoss Module route.


> On 16 Nov 2016, at 08:11, Niclas Hedhman <nic...@hedhman.org
   <javascript:;>> wrote:
>
> I am curious, what do you mean by "non-deterministic dependency
   resolution"
> ?  You can make it as predictable as you wish with attributes and
> directives.
>
> Cheers
> Niclas
>
> On Wed, Nov 16, 2016 at 4:07 PM, Michał Kłeczek
   <michal.klec...@xpro.biz <javascript:;>>
> wrote:
>
>> While non-hierarchical class loading is crucial, OSGI with its
>> non-deterministic dependency resolution is very difficult ( if not
>> impossible ) to target.
>> I'm working on JBoss Module based class loading for River which
   I'm going
>> to propose as contribution soon.
>>
>> Thanks,
>> Michal
>>
>> On Wednesday, 16 November 2016, Dawid Loubser
   <da...@travellinck.com <javascript:;>>
>> wrote:
>>
>>> +1 for OSGi providing the best solution to the class resolution
   problem,
>>> though I think some work will have to be done around trust, as
   you say.
>>>
>>>
>>> On 16/11/2016 02:23, Peter wrote:
>>>>
>>>> The conventional alternatives wi

Re: Maven Build

[Michał Kłeczek]

While non-hierarchical class loading is crucial, OSGI with its
non-deterministic dependency resolution is very difficult ( if not
impossible ) to target.
I'm working on JBoss Module based class loading for River which I'm going
to propose as contribution soon.

Thanks,
Michal

On Wednesday, 16 November 2016, Dawid Loubser <da...@travellinck.com> wrote:

> +1 for OSGi providing the best solution to the class resolution problem,
> though I think some work will have to be done around trust, as you say.
>
>
> On 16/11/2016 02:23, Peter wrote:
> >
> > The conventional alternatives will remain;  the existing ClassLoader
> isolation and the complexities surrounding multiple copies of the same or
> different versions of the same classes interacting within the same jvm.
> Maven will present a new alternative of maximum sharing, where different
> service principals will share the same identity.
> >
> > Clearly, the simplest solution is to avoid code download and only use
> reflection proxy's
> >
> > An inter process call isn't remote, but there is a question of how a
> reflection proxy should behave when a subprocess is terminated.
> >
> > UndeclaredThrowableException seems appropriate.
> >
> > It would plug in via the existing ClassLoading RMIClassLoader provider
> mechanism, it would be a client concern, transparent to the service or
> server.
> >
> > The existing behaviour would remain default.
> >
> > So there can be multiple class resolution options:
> >
> > 1. Existing PrefferedClassProvider.
> > 2. Maven class resolution, where maximum class sharing exists.  This may
> be preferable in situations where there is one domain of trust, eg within
> one corporation or company.  Max performance.
> > 3. Process Isolation.  Interoperation between trusted entities, where
> code version incompatibilities may exist, because of separate development
> teams and administrators.  Each domain of trust has it's own process
> domain.  Max compatibility, but slower.
> > 4. OSGi.
> >
> > There may be occassions where simpler (because developers don't need to
> understand ClassLoaders), slow, compatible and reliable wins over fast and
> complex or broken.
> >
> > A subprocess may host numerous proxy's and codebases from one principal
> trust domain (even a later version of River could be provisioned using
> Maven).  A subprocess would exist for each trust domain. So if there are
> two companies, code from each remains isolated and communicates only using
> common api.  No unintended code versioning conflicts.
> >
> > This choice would not prevent or exclude other methods of communication,
> the service, even if isolated within it's own process will still
> communicate remotely over the network using JERI, JSON etc.  This is
> orthogonal to and independant of remote communication protocols.
> >
> > OSGi would of course be an alternative option, if one wished to execute
> incompatible versions of libraries etc within one process, but different
> trust domains will have a shared identity, again this may not matter
> depending on the use case.
> >
> > Cheers,
> >
> > Peter.
> >
> > ESent from my Samsung device.
> >
> >   Include original message
> >  Original message 
> > From: "Michał Kłeczek (XPro Sp. z o. o.)" <michal.klec...@xpro.biz
> <javascript:;>>
> > Sent: 15/11/2016 10:30:29 pm
> > To: dev@river.apache.org <javascript:;>
> > Subject: Re: Maven Build
> >
> > While I also thought about out-of-process based mechanism for execution
> of dynamically downloaded code, I came to the conclusion that in the
> context of River/Java in-process mechanism is something that MUST be done
> right. All other things can (and should) be built on that.
> >
> > I think that the proposal to implement "remote calls on smart proxy
> interfaces that aren't remote" is somewhat a misnomer. The call is either
> remote or local - you cannot have both at the same time. AFAIK Jini
> community always believed there is no possibility to have local/remote
> transparency. That is why there exists java.rmi.Remote marker interface in
> the first place.
> >
> > There is also the question about the level of isolation you want to
> achieve. Simple "out-of-process" is not enough, chroot is not enough,
> CGROUPS/containers/jails/zones might be not enough, virtual machines might
> be not enough :) - going the route you propose opens up the whole world of
> new questions to answer. At the same time you loose the most important
> advantages of in-process execution:
> > - simplicity of communica

Re: Maven Build

[Michał Kłeczek (XPro Sp. z o. o.)]

While I also thought about out-of-process based mechanism for execution 
of dynamically downloaded code, I came to the conclusion that in the 
context of River/Java in-process mechanism is something that MUST be 
done right. All other things can (and should) be built on that.


I think that the proposal to implement "remote calls on smart proxy 
interfaces that aren't remote" is somewhat a misnomer. The call is 
either remote or local - you cannot have both at the same time. AFAIK 
Jini community always believed there is no possibility to have 
local/remote transparency. That is why there exists java.rmi.Remote 
marker interface in the first place.


There is also the question about the level of isolation you want to 
achieve. Simple "out-of-process" is not enough, chroot is not enough, 
CGROUPS/containers/jails/zones might be not enough, virtual machines 
might be not enough :) - going the route you propose opens up the whole 
world of new questions to answer. At the same time you loose the most 
important advantages of in-process execution:
- simplicity of communication between components (basic function call, 
no need to do anything complicated to implement callbacks etc.)

- performance

In the end you either standardize on the well known set of communication 
protocols (such as JERI) OR you say "end of protocols" by allowing 
execution of dynamically downloaded code in-process.
If River is going to choose the first route - IMHO it is going to fail 
since it does not propose anything competitive comparing to current 
mainstream HTTP(S)/REST/JSON stack.


Thanks,
Michal

Peter 
November 15, 2016 at 8:28 AM
I've been thinking about process isolation (instead of using 
ClassLoader's for isolation).  Typically, smart proxy's are isolated 
in their own ClassLoader, with their own copies of classes, however 
with Maven, a lot more class sharing occurs.  Since River uses 
codebase annotations for identity, using maven codebase annotations 
will result in proxy's from different services sharing identity.


A better way to provide for different identities coexisting on the 
same node, would be to use subprocess jvm's for each Service's server 
principal identity, to keep classes from different services in 
different processes.


This way, each principal would have their own process & Maven 
namespace for their proxy's.


Presently JERI only exports interfaces in reflection proxy's that 
implement Remote, so I'd need an endpoint that can export all 
interfaces, accross a local interprocess connection to allow remote 
calls on smart proxy interfaces that aren't remote.


This also means that memory resource consumption of smart proxy's can 
be controlled by the client and a smart proxy's process killed without 
killing the client jvm.


Cheers,

Peter.



Dawid Loubser 
November 15, 2016 at 8:50 AM
As a very heavy Maven user, I wanted to say that this is great news.
This is encouraging indeed!

Dawid


Peter 
November 15, 2016 at 4:08 AM
Some other news that might encourage participation, I've been working 
on Dennis Reedy's script to modularise the codebase, I haven't run the 
test suites against it and it isn't generating stubs yet, and I'll 
need to modify the platform modules for the IoT effort after the 
conversion is complete.


Here's the output of the River maven build:

Reactor Summary:

River-Internet Project  SUCCESS [0.689s]
Module :: River Policy  SUCCESS [8.395s]
Module :: River Resources . SUCCESS [0.607s]
Module :: River Platform .. SUCCESS [23.521s]
Module :: River Service DL Library  SUCCESS [8.999s]
Module :: River Service Library ... SUCCESS [8.014s]
Module :: River Service Starter ... SUCCESS [3.930s]
Module :: River SharedGroup Destroy ... SUCCESS [3.018s]
Module :: Outrigger ... SUCCESS [0.056s]
Module :: Outrigger Service Download classes .. SUCCESS [2.416s]
Module :: Outrigger Service Implementation  SUCCESS [4.118s]
Module :: Outrigger Snaplogstore .. SUCCESS [3.273s]
Module :: Lookup Service .. SUCCESS [0.048s]
Module :: Reggie Service Download classes . SUCCESS [3.966s]
Module :: Reggie Service Implementation ... SUCCESS [3.621s]
Module :: Mahalo .. SUCCESS [0.436s]
Module :: Mahalo Service Download classes . SUCCESS [2.059s]
Module :: Mahalo Service Implementation ... SUCCESS [4.175s]
Module :: Mercury the Event Mailbox ... SUCCESS [0.497s]
Module :: Mercury Service Download classes  SUCCESS [3.622s]
Module :: Mercury Service Implementation .. SUCCESS [3.562s]
Module :: Norm  SUCCESS [0.013s]
Module :: Norm Service Download classes 

Re: another interesting link

[Michał Kłeczek]

 >> 2. proxy codebase jars contain a list of requested permissions to be
granted to the jar signer and url (client need not know in advance).

This one is tricky:
1) It is not always possible to specify fine grained permissions statically
(for example we want to restrict the connect permission to certain hosts
only and the hosts are only known at runtime)
2) It is really an all or nothing approach: since we grant any permission
that a (authenticated) service wants, we might equally grant it
"AllPermission"

What is needed is a way to dynamically decide what permissions are granted
to a downloaded "object".
The word "object" is again tricky. Today permissions are granted to
"codebases" - meaning we have no way of separating objects that share
codebases (which is a common thing when using Maven repositories as code
servers).

Proper separation of "trust domains" in a single virtual machine is
difficult (and the good question is really whether it is feasible at all to
do properly using current Java security model).

Of course - eliminating dynamic code downloading solves those issues - but
I would say without dynamic code downloading there is no point in using
River :)

IMHO any discussions and changes that do not solve the fundamental issues
faced when dynamically downloading code are really moot - they just mask
the underlying problems which are going to come out sooner or later.
And right now the architecture is fundamentally broken because of:
- hierarchical class loaders
- untrusted code execution (which Peter is working on)
- not flexible enough security model

>> 5. DownloadPermission automatically granted to authenticated registrars
(to signer and url, very specific) during multicast discovery.

IMHO security model should be orthogonal to any other functionality
(service discovery being a specific example) - so any security related code
should have absolutely no knowledge whatsoever about
- the interfaces implemented by a particular downloaded class
- any network protocols used to implement discovery
- the concept of a "discovery"

My two cents.

Thanks,
Michal

Peter 
July 26, 2016 at 2:58 AM
Note the comment about security on the blog?

Steps I've taken to simplify security (that could also be adopted by river):
1. Deprecate proxy trust, replace with authenticate service prior to
obtaining proxy.
2. proxy codebase jars contain a list of requested permissions to be
granted to the jar signer and url (client need not know in advance).
3. Policy file generation, least privilege principles (need to set up
command line based output for admin verification of each permission during
policy generation).
4 Input validation for serialization.
5. DownloadPermission automatically granted to authenticated registrars (to
signer and url, very specific) during multicast discovery.

Need to more work around simplification of certificate management.

Regards,

Peter.
Sent from my Samsung device.

  Include original message
 Original message 
From: Peter  
Sent: 26/07/2016 10:27:59 am
To: dev@river.apache.org  
Subject: another interesting link

https://blogs.oracle.com/hinkmond/entry/jini_iot_edition_connecting_the


Sent from my Samsung device.

Re: another interesting link

[Michał Kłeczek (XPro Sp. z o. o.)]

I am well aware of StartNow since that is the first Jini "support 
library" I have used. Indeed - it is really easy to use.
But it is only one side of the issue - the API and some support support 
code that is supposed to be linked statically with the service 
implementation.


What I am talking about is actually "externalizing" most aspects of a 
service implementation so that:
- you do not have to package any (for some meaning of "any" :) ) 
libraries statically (since all code can be downloaded dynamically)
- you do not have to provide any (for some meaning of "any" :) ) static 
configuration (ie. configuration files) - a service should simply use 
other services and "reconfigure" itself when those change
It would go towards some kind of an "agent architecture", with movable 
objects (ie "services") being "hosted" by well... other movable objects 
:).The idea is less appealing today when we have all the cloud 
infrastructure, virtualization, software defined networking etc. 
Nevertheless still interesting IMHO.


Thanks,
Michal

Gregg Wonderly 
July 26, 2016 at 1:28 PM
My StartNow project on Java.net aimed directly at this mode of 
operation a decade ago. I wanted conventions that provided use of 
configuration with defaults.


You just extend PersistantJiniService and call start(serviceName). 
Subclasses could override default implementation for how the 
conventions in the APIs created implementation objects through code or 
configuration.


The intent was to create THE API to provide the conventions of service 
creation.


We have a Window/JWindow class and don't have to do all the decorating 
ourselves.


Jini service construction should work the same way!

Gregg

Sent from my iPhone


Tom Hobbs 
July 26, 2016 at 11:50 AM
I would say the comment on that blog sums everything about Jini up.

It’s just too hard to set up and get working.

That’s why I think simplifying reggie is possibly a first step. Make a 
/small/ and simple reggie jar that just handled service registration 
and not proxy downloading etc. Make it really easy to register your 
services without needing class loaders etc, preferably via some 
convention rather than configuration. (This is what I’m trying to find 
the time to work on.)


I’d really like to be able to type;

$ java -jar reggie.jar

And have a reggie running with all the defaults ready to register my 
services with. Or perhaps, as an option;


$ java -jar reggie.jar —ipv6

Security, class loading, proxy downloading and all the rest of it 
could then be put back in by specifying more advanced configuration 
options.


My Scala service would be great if I could define it just as;

object MyCoolService extends LazyLogging with ReggieRegistration with 
ReggieLookup


Or in Java with default interface methods;

class MyCoolService implements ReggieRegistration, ReggieLookup

And that would be it, congratulations you’ve started a reggie and 
registered your service and have methods available to help you find 
other services.


This would satisfy use cases where the network was private and/or 
trusted. And security on top would, ideally, be up to configuration 
again or perhaps injecting some alternative implementation of some 
bean somewhere. But the core premise is, make it easy to startup, demo 
and see if it fits what you want it for.





Peter 
July 26, 2016 at 3:58 AM
Note the comment about security on the blog?

Steps I've taken to simplify security (that could also be adopted by 
river):
1. Deprecate proxy trust, replace with authenticate service prior to 
obtaining proxy.
2. proxy codebase jars contain a list of requested permissions to be 
granted to the jar signer and url (client need not know in advance).
3. Policy file generation, least privilege principles (need to set up 
command line based output for admin verification of each permission 
during policy generation).

4 Input validation for serialization.
5. DownloadPermission automatically granted to authenticated 
registrars (to signer and url, very specific) during multicast discovery.


Need to more work around simplification of certificate management.

Regards,

Peter.
Sent from my Samsung device.

  Include original message
 Original message 
From: Peter 
Sent: 26/07/2016 10:27:59 am
To: dev@river.apache.org 
Subject: another interesting link

https://blogs.oracle.com/hinkmond/entry/jini_iot_edition_connecting_the


Sent from my Samsung device.

Re: another interesting link

[Michał Kłeczek (XPro Sp. z o. o.)]

In my dreams I always thought of "self configuring" and "adapting" 
services. So instead of reading a "configuration" a service would simply 
search for other services and use them. Exporter service being an example.
Ideally - the only thing that should be configured would be the 
"identity" (ie. credentials) of a service principal(s).


That would be possible once dynamic code downloading is done right :)

And one more remark - dynamic proxy does not imply avoiding codebases 
and code downloading. You have to download service interface classes. 
You have to download invocation handler implementation class.


Thanks,
Michal

Peter 
July 26, 2016 at 12:43 PM

Perhaps a script that detects the environment, asks a few questions 
and creates the config files?  These can be edited for more complex 
configurations.


I've added a couple of default methods to ServiceRegistrar, the new 
lookup method doesn't return the service proxy's, so the registrar is 
basically used just for search, the client then contacts each service 
it's interested in.


Starting off with simple services that only use dynamic proxy's 
(java.lang.reflect.Proxy instances), avoids codebases.


Reggie still requires a codebase, however if we sign it, this 
addresses code trust.


Regards,

Peter.

Sent from my Samsung device

  Include original message
 Original message 
From: Tom Hobbs 
Sent: 26/07/2016 07:50:43 pm
To: dev@river.apache.org
Subject: Re: another interesting link

I would say the comment on that blog sums everything about Jini up.

It’s just too hard to set up and get working.

That’s why I think simplifying reggie is possibly a first step.  Make a /small/ and simple reggie jar that just handled service registration and not proxy downloading etc.  Make it really easy to register your services without needing class loaders etc, preferably via some convention rather than configuration.  (This is what I’m trying to find the time to work on.) 



I’d really like to be able to type;

$ java -jar reggie.jar

And have a reggie running with all the defaults ready to register my services with.  Or perhaps, as an option; 



$ java -jar reggie.jar —ipv6

Security, class loading, proxy downloading and all the rest of it could then be put back in by specifying more advanced configuration options. 



My Scala service would be great if I could define it just as;

object MyCoolService extends LazyLogging with ReggieRegistration with ReggieLookup 



Or in Java with default interface methods;

class MyCoolService implements ReggieRegistration, ReggieLookup

And that would be it, congratulations you’ve started a reggie and registered your service and have methods available to help you find other services. 



This would satisfy use cases where the network was private and/or trusted.  And security on top would, ideally, be up to configuration again or perhaps injecting some alternative implementation of some bean somewhere.  But the core premise is, make it easy to startup, demo and see if it fits what you want it for.