Dan Creswell wrote:
On 4 January 2011 12:04, Peter Firmstone <[email protected]> wrote:
Dan Creswell wrote:
So....
On 3 January 2011 22:31, Peter Firmstone <[email protected]> wrote:
Dan Creswell wrote:
Hi Peter,
On 3 January 2011 01:56, Peter Firmstone <[email protected]> wrote:
I'm currently experimenting with a modular build, I've laid out the
framework in skunk and I've got a local build where I'm trying to
define
what to include in the platform.
The most obvious is to create the platform module based on what's
included
in jsk-platform.jar
As has been pointed out there's also jsk-lib.jar and jsk-dl.jar
Services that utilise the jsk-lib also need to have jsk-dl in their
codebase for clients to download.
There are also a number of utility classes shared by service
implementations, included in their proxy's and it wouldn't make sense
to
have these duplicated in each service implementation for maintenance
reasons. This also presents an interesting situation, when these
classes
already exist in the client's classpath, the additional classes are not
loaded into the proxy classloader, since the parent classloader can
resolve
them, however that could also introduce versioning conflicts, if we
have
a
library that experiences version changes over time. There's nothing
currently that prevents a client from also utilising these library
classes.
I think the general solution for much of the versioning comes down to
"proper" use of PreferredClassLoader.
I think probably anything that is generally required/assumed by all of
service, proxy and client is platform. Utility classes for service
implementations you discuss above possibly aren't platform unless they
are
used by all services always.
This is why I think the client needs to be provided with a standard way
of
being run from a child classloader of the jini platform class loader,
in
this way, a service, proxy and client running within the same jvm, only
share the jini platform (& policy) classes, everything else becomes a
private implementation concern, including which version of a library to
use.
I imagine that the Service Starter framework would provide support for
most
of that already. Would still need some tweaks though....
Penny for your thoughts on tweaks?
Only a foggy thought about the current set of ServiceDescriptors and
whether
they are best for clients. I could imagine using (and have used) a
NonActivatableServiceDescriptor to fire up a client but the need for the
object constructor as it is, the spec'ing of codebase and so on feels like
overkill or at least there could be a simpler option for clients that
don't
have codebases etc.
Ok, that sounds like a good starting point, thanks.
From a versioning standpoint, we need a clean separation of name spaces
to
avoid runtime issues.
Modularity will reduce the burden of maintenance, but only if done
properly.
The most obvious places to break up the codebase are the points of
abstraction, where dynamic dependency injection is performed, these are
Jini
Services and Service Provider Interfaces (not to be confused with a
jini
service).
From observing recent improvements, the classes in com.sun.* change
more
often than those in net.jini.*, this was my reasoning for suggesting
including all net.jini.* in the platform, because I wanted to know your
thoughts. But doing so may drag more of the com.sun.* namespace into
platform, which is bad, because these are then visible in all
namespaces.
I've had thoughts of putting platform implementation classes into a
child
classloader, to make it invisible from client, proxy and service
namespaces,
but this also presents its challenges as it requires a superclass
present
in
the platform classloader. This is in some ways similar to the way OSGi
exports a package, while keeping implementation packages private.
Using
OSGi to control package visibility is one option, there's also netbeans
modularity or service providers. Of course mentioning these utilities
is
akin to provoking off topic arguments which shows how strongly people
feel
about River and Jini, but I'd first like to discuss the actual problem
and
listen to solutions.
Okay, so if we're going that way, we all should read Mike Warres' paper
way
back in the day that covers much of the classloader ball of mud,
preferred
class loading and outstanding issues:
http://labs.oracle.com/techrep/2006/smli_tr-2006-149.pdf
Damn good suggestion, I was actually going over it again yesterday.
Code base annotation loss is a big problem, when non preferred classes
are
resolved by the application class loader, that and code base location
changes over time and codebase configuration problems.
Preferred class loading for preferred classes doesn't follow the rule of
delegating class loading first to the parent class loader.
Preferred class loading is used when proxy classes also exist in the
parent
class loader.
The best option appears to be to limit the classes in the application
class
loader to jini platform classes, this minimizes the classes that need to
be
preferred.
One problem with using a child class loader for jini platform
implementation classes, is some of these classes may invariably be
serialized with api classes from the platform and need to be
deserialized,
if they're not visible to the application class loader... I've been
going
over these classes one by one, I'm not finding any serializable classes
yet... I'm starting to think a child class loader for the platform
implementation would be an unnecessary complication.
Codebase services was intended to be a solution to codebase migration and
configuration issues, the codebase service was found using discovery.
I've been having some thoughts about a custom URL handler, that utilises
DNS-SRV discovery to locate codebases. Each domain could provide a list
of
codebase servers, these could be redundant.
A custom URL format could have the following information:
* The domain in which to discover the codebase server.
* The jar archive name (or other file type name).
* A message digest of the file.
* A file size limit. (download is aborted after this limit is exceeded)
* A hashcode of the Server's Subject's certificate (the cert itself
could be too large)
The Certificate hashcode provides an identity for the URL (which provides
a
reasonable probability of being unique when combined with other
information), we might consider using a Subject to uniquely identify a
ClassLoader namespace, to avoid class sharing between proxies with
identical
code, but different server subjects. The Subject certificate hashcode
would
need to be checked as part of the verification process, albeit after
unmarshalling.
The problem here is we're bumping into the limitations of the java
platform, which the missing Isolates API was designed to address,
identical
classes could be used by separate identities in separate isolates.
If we didn't have a hashcode relating to the server subject, would it be
reasonable to expect all servers from the same domain to have the same
server Subject?
No real comment other than to say, feels like a narrowing assumption that
could provide a nasty surprise later so I would be inclined not to make
it.
My thoughts exactly, I wanted to see if someone else thought the same way.
This is precisely what would occur if two different proxy's with different
Subject's used the same code base, how could we tell their ClassLoaders
apart? This might occur if we used maven to provision the codebase or used
DNS-SRV discovered codebase servers.
Okay so I'm not sure maven and such are the trigger's for the above. It
would be some conscious decisions someone had made about packaging and
deployment, wouldn't it?
If two services and thus two proxies have the same codebase, someone is
choosing to either:
(1) Have multiple instances of the same service run off the same codebase -
not sure of the value except maybe for load-handling. Single codebase for
each service is potentially a single point of failure.
(2) Have multiple services of different types sharing a codebase - well this
could happen if:
(a) We choose to leave all classes sitting under a URL in unjar'd state (we
all know that means lots of roundtrips during class resolution etc and so
generally it's not done) or
(b) We're keeping multiple codebases inside of a single .jar (which
increases the size of the download required for class resolution etc and
makes version management a difficult affair as touching one service
Or (3) Use DNS-SRV to locate codebases in a domain, such that the URL is
the domain, not an individual IP address and DNS-SRV is used to locate
one of a number of redundant codebase servers that can provide the same
jar file. In this case, the location is the Domain, since multiple
services might use the same code to provide the same service to multiple
clients, it is possible for a client to consume more than one service
and it is possible that for some reason someone might want to have
multiple Subjects for their Jini Services from one domain, utilising
common code or libraries. DNS spoof attacks can be avoided using
message digests.
So do you think a Certificate would be better than a hashcode? If so
I'm happy to go with that.
If Services have the same Subject, then there is no danger of two
distinct service object's sharing the same classes and static variables,
since Permission is granted to Principals, that the Subject has. It is
only when run as a Subject after authentication that the proxy gain's
Permissions. It could be argued that the unmarshalling period of the
second proxy is a window where an unauthenticated proxy has a thread, to
transfer information back to it's host from sensitive static variables,
that have been populated by the first proxy. However the first proxy
and second proxy have to use the same code, the second proxy can't do
something unpredictable unless the code is not trusted, but in this case
it is, since the first proxy trusts the code and the client trust the
first proxy, so the only thing an attacking second proxy can do is,
deserialize a large amount of data, causing a stack overflow error.
The advantage of DNS-SRV discovery for codebase servers is that
codebases and services can be deployed separately and have codebase
server redundancy.
You're right, I'm trying to solve a problem that doesn't exist, at least
not yet...
For versioning, a proxy might include the version number in the name of
it's jar file as Dennis has previously suggested. We could provide a
refresh operation on the proxy that remarshalled it's current state and
updated its codebase annotation, then unmarshalled it in another
ClassLoader with an updated codebase, all while the client holds a
reference to the proxy (a reflective proxy), which internally is changed
to refer to the new proxy. The original proxy is then garbage collected.
The new codebase might include new functionality that a Service UI might
take advantage of.
Another possibility is having a reflective verifier proxy in the
Marshalled Instance, where the client platform is provided as a
parameter, and the remote end advises a codebase annotation that is more
suited to the local platform (might take advantage of some local
platform feature).
We might use a very simple client that looks for Services with Service
UI, where the Service UI, is the Service, in this way software could be
extremely dynamic, Registrar's in Domain's could be discovered using
DNS-SRV and Unicast Discovery V2, users could browse services from a
variety of clients with different UI requirements. Gregg's Jini
desktop, might become the cloud desktop.
Hence the subject change to stargazing.
Cheers,
Peter.
potentially impacts another).
Now in case of (1) I cannot see any reason why one would have different
subjects for each of those services. In essence they are the same service in
all about UUID so housing their proxies in the same classloader on a client
doesn't seem like a problem.
In case (2), I think the obvious way to bundle a service, one jar for -dl
and one jar for back-end is likely the prevailing one and these cases though
they could occur would not be recommended practice.
For maven, I can see how we could certainly build archetypes and such to
support the recommended bundling approach. For DNS SRV, there are means by
which codebase paths (.jars and such) can be put into the records such that
we could still construct a unique codebase to pull from.
In essence I think we might be solving a non-problem...
But is a Certificate hashcode enough?
My reasoning for using a Certificate hashcode, rather than a certificate:
1. Certificates are large, and would make Codebase URL debugging
output harder to understand.
Unless one builds tools to support this case. If it's a dominating case,
that's what we'd have to do in any case I reckon so I'm not fearful.
2. A Certificate cannot be checked until after proxy verification, it
would be unlikely that two different certificates would share the
same hashcode and codebase and be trusted by the client, but in
the case they do, a special ClassLoader might contain a reference
to the authenticated subject, enabling the client to check that a
proxy's objects unmarshalled into the classloader can be
authenticated as that subject.
Typically a Service must authenticate as a Principal, during proxy
verification, the Principal is known prior to authentication and is set as a
min server principal constraint, the Subject and it's certificate are not
known until after authentication and after class loading.
There is one more thing to consider. If a classloader namespace is already
in use for a proxy, and a new proxy is unmarshalled into this classloader,
that class loader has already been verified by another proxy, during the
period of unmarshalling, the classes in that class loader are trusted,
they're not local code, but they're trusted and permissions have been
granted to a Principal, assuming the code is trustworthy, it should be safe
to unmarshall unauthenticated proxy objects into this class loader, because
we trust the code and because if it isn't run as a subject during
unmarshalling, it doesn't gain any privileges. After unmarshalling if the
second proxy doesn't authenticate as the same Subject as the first, all
references to it should be set null and the garbage collector allowed to
collect it. Remember this is the case where proxy's share the same codebase
URL and Subject.
In pepe/skunk, I have a backward compatible experimental MarshalledInstance
that contains a verifier proxy, used to authenticate the original creator of
the MarshalledInstance, here the Subject could be supplied to something
similar to RMIClassLoader, Gregg has donated his CodebaseAccessClassloader,
this could have a method added to accept a Subject along with the
annotation, or alternately classloading could be performed using the
Subject. This authenticated verifier proxy then verifies the codebase
annotation using a message digest. It is expected that the Subject would be
the same as that which authenticates the proxy and it's every method call on
the proxy. Authentication prior to unmarshalling remote code is intended to
avoid unmarshalling attacks.
I plan to experiment, writing some more utility code, based on my proxy
isolate in skunk/pepe, that allows unmarshalling to be performed in an
isolated Thread, which catches a StackOverflowError, in the case of a remote
end deliberately sending a spuriously large object stream. Currently this
only handles the unmarshalling process (MarshalledInstance.get()), however
I'm looking at how to extend it to deserialization of MarshalledInstance
during discovery.
But what about Exported objects, not discovered through a lookup service,
but returned from another Service, if these are from a different remote host
with a different Subject?
All the above is a thought experiment, feel free to participate and correct
any errors or fallacies.
Cheers,
Peter.
Cheers,
Peter.
Then of course there's also the argument that we should do nothing, so
consider this a thought experiment to discover how it might be done, not
that it will or must be, that can be decided later.
What are your thoughts?
Cheers,
Peter.
Jeff Ramsdale wrote:
Chiming in here, perhaps off topic...
Sometime back (maybe within the past year?) there seemed to be
agreement on removing the ClassPath manifest attributes moving forward
in order to not make assumptions concerning relative jar locations
(e.g. classpath built from local Maven repo).
-jeff
On Sun, Jan 2, 2011 at 8:36 AM, Greg Trasuk <[email protected]>
wrote:
On Sun, 2011-01-02 at 11:15, Tom Hobbs wrote:
Am I right in thinking/remembering that, with the exception of the
*-dl.jar files, the only others that are needed are the jsk-*.jar
ones.
I'm pretty sure that many of the JARs contain the same class files,
I
think that there's definitely scope to reduce the number JAR files
that the build creates.
I think you might be mistaken about that. The *-dl.jar files often
contain duplications of classes in *.jar files, but that's reasonable
and expected. The few service implementation jar files that I've
looked
at contain ClassPath manifest attributes that reference jsk-lib etc.
The only real duplication I'm aware of is in the jini-core.jar,
jini-ext.jar and sun-utils.jar files, that duplicate the contents of
jsk-platform and jsk-lib. This is done for backwards compatability
(that's the way it was in Jini 1.0-days), and could probably be
deprecated at this point, after consulting with the user community.
Cheers,
Greg.
On Sun, Jan 2, 2011 at 8:51 AM, Peter Firmstone <[email protected]>
wrote:
I agree that dynamic proxy classes should remain dynamic downloads,
however
much of net.jini.* isn't in the jsk-platform.jar
Should we expand the platform to contain all net.jini.*?
Except for providers? (com.sun.jini.resource.Service, similar to
Java's
sun.misc.Service and java.util.ServiceLoader)
Perhaps we can include more in the platform and reduce the number
of
jar
archives we've got?
Any thoughts?
Cheers,
Peter.
[email protected] wrote:
Isn't that already jsk-platform.jar? I would object to anything
that
subverts the dynamic proxy loading concept that is central to
Jini.
It is imperative that people don't, for instance, get the
service-registrar proxy impls in their local class path. That
would
break
compatibility with future or alternate impls.
Cheers,
Greg
------Original Message------
From: Sim IJskes - QCG
To: [email protected]
ReplyTo: [email protected]
Subject: river.jar
Sent: Dec 31, 2010 10:07 AM
Hello,
anybody have an objection against a river.jar in the build that
contains
all river runtime classes?
Gr. Sim
