Interesting article

[Peter Firmstone]

https://dzone.com/articles/whats-new-in-owasp-apis-and-mitigation

Sent from my Samsung device.
 

Success stories

[Peter Firmstone]

We should add Blazegraph and Sorcer to our success stories and remove dead 
links, we can still mention older stories, but I think we need to focus on more 
recent stories.

Anyone know of other success stories not listed?

Does anyone work for a company that utilises River who can tell us their story?

Thanks & Regards,

Peter.

Sent from my Samsung device.
 

Testing Rio

[Peter Firmstone]

The only changes made are to POM file dependencies, no changes to Rio code.

A compatiblity layer has been used for the com.sun.jini namespace.

I'm not entirely sure whether the test failures are a result of a poor 
network connection (mobile network), perhaps those more familiar with 
Rio will know.


Cheers,

Peter.

[INFO] 


[INFO] Building Rio Distribution 5.2.3-SNAPSHOT
[INFO] 

[INFO] 


[INFO] Reactor Summary:
[INFO]
[INFO] Rio Project  SUCCESS [  
0.067 s]
[INFO] Module :: Rio Logging Support .. SUCCESS [  
0.653 s]
[INFO] Module :: Rio Platform . SUCCESS [  
3.734 s]
[INFO] Module :: Rio service starters . SUCCESS [  
0.097 s]
[INFO] Module :: Webster .. SUCCESS [  
0.097 s]
[INFO] Module :: Rio API .. SUCCESS [ 
22.014 s]
[INFO] Module :: Rio Proxy  SUCCESS [  
0.036 s]
[INFO] Module :: Rio Library .. SUCCESS [  
3.806 s]
[INFO] Module :: Aether Resolver .. SUCCESS [  
0.386 s]
[INFO] Module :: Project Module Resolver .. SUCCESS [  
0.583 s]
[INFO] Module :: Event Collector Service API .. SUCCESS [  
0.329 s]
[INFO] Module :: Event Collector Service Proxy  SUCCESS [  
0.034 s]
[INFO] Module :: Cybernode Service API  SUCCESS [  
0.039 s]
[INFO] Module :: Cybernode Service Proxy .. SUCCESS [  
0.056 s]
[INFO] Module :: Cybernode Service Implementation . SUCCESS [  
0.070 s]
[INFO] Module :: Monitor Service API .. SUCCESS [  
0.037 s]
[INFO] Module :: Rio Test Infrastructure .. SUCCESS [  
1.825 s]
[INFO] Module :: Event Collector Service Implementation ... SUCCESS [  
0.057 s]
[INFO] Module :: Watch User Interface . SUCCESS [  
0.041 s]
[INFO] Module :: Cybernode Service UI . SUCCESS [  
0.035 s]
[INFO] Module :: Gnostic .. SUCCESS [  
0.003 s]
[INFO] Module :: Gnostic API .. SUCCESS [  
0.285 s]
[INFO] Module :: Gnostic Service Implementation ... SUCCESS [ 
12.793 s]
[INFO] Module :: Monitor Service Proxy  SUCCESS [  
1.686 s]
[INFO] Module :: Monitor Service Implementation ... SUCCESS [  
4.657 s]
[INFO] Module :: Rio Command Line Interface ... SUCCESS [  
3.360 s]
[INFO] Module :: Rio Management User Interface  SUCCESS [ 
57.014 s]
[INFO] Rio Examples ... SUCCESS [  
0.007 s]
[INFO] Example :: Calculator .. SUCCESS [  
0.003 s]
[INFO] Example :: Calculator API .. SUCCESS [  
1.139 s]
[INFO] Example :: Calculator Service Implementation ... SUCCESS [ 
30.510 s]
[INFO] Example :: Events .. SUCCESS [  
0.001 s]
[INFO] Example :: Events API .. SUCCESS [  
0.866 s]
[INFO] Example :: Events Proxy  SUCCESS [  
0.881 s]
[INFO] Example :: Events Service Implementation ... SUCCESS [  
1.450 s]
[INFO] Example :: Events UI ... SUCCESS [  
1.032 s]
[INFO] Example :: Hospital  SUCCESS [  
0.003 s]
[INFO] Example :: Hospital API  SUCCESS [  
1.117 s]
[INFO] Example :: Hospital Rules .. SUCCESS [  
0.916 s]
[INFO] Example :: Hospital Service Implementation . SUCCESS [  
1.471 s]
[INFO] Example :: Hospital UI . SUCCESS [  
3.736 s]
[INFO] Example :: SpringBean .. SUCCESS [  
0.003 s]
[INFO] Example :: SpringBean API .. SUCCESS [  
0.747 s]
[INFO] Example :: SpringBean Service Implementation ... SUCCESS [  
8.693 s]
[INFO] Example :: Workflow  SUCCESS [  
0.002 s]
[INFO] Example :: Workflow API  SUCCESS [  
1.102 s]
[INFO] Example :: Workflow Service Implementation . SUCCESS [  
1.276 s]
[INFO] Example :: Tomcat .. SUCCESS [ 
54.039 s]
[INFO] Rio Distribution ... SUCCESS [  
0.003 s]
[INFO] 


[INFO] BUILD SUCCESS
[INFO] 


[INFO] Total time: 03:44 min
[INFO] Finished at: 2017-04-22T14:32:23+10:00
[INFO] Final Memory: 109M/247M
[INFO] 

River 3.0.1 release

[Peter Firmstone]

Anyone got some cycles to help out with the River 3.0.1 release?

There are some existing jira issues with patches or easy fixes we can include 
too.

I've also got a Jini compatibility library to assist people who want to migrate 
from pre 3.x versions that depend on common classes in the comsun. namespace.  
I can donate or make it available on github.

This has the potential to be the best release in the projects history IMHO.

Cheers,

Peter.

Sent from my Samsung device.
 

Roadmap proposal

[Peter Firmstone]

Proposed Release roadmap:

River 3.0.1 - thread leak fix
River 3.1 - Modular build restructure (& binary release)
River 3.2 - Input validation 4 Serialization, delayed unmarshalling & safe 
ServiceRegistrar lookup service.
River 3.3 - OSGi support

Changes in the modular build and delayed unmarshalling would set us up for 
later OSGi support.

I think this might allay any fears people have regarding the pace of change, in 
the past, latent race conditions prevented stabilisation, hence a significant 
amount of work was required leading up to River 3.0's release.

Thoughts?

Regards,

Peter.

Sent from my Samsung device.
 

object based annotations

[Peter Firmstone]

Mike, I recall the last time I looked at object based annotations, there was a 
backward compatibility issue because both ends of the Marshal streams expect 
string based annotations as does RMIClassLoader.

However if you are still keen to investigate object based annotations there's 
no reason you couldn't treat a string like a char array = byte array (beware 
signed byte) and have a RMIClassLoaderSPI deserialize the objects after they 
were sent in string form?

Regards,

Peter.

Sent from my Samsung device.

OSGi

[Peter Firmstone]

Any OSGi veterans willing to assist with JGDMS support for OSGi during the 
modular restructure?

I've added OSGi manifests to modules, but I also need to add classpath manifest 
entry's for non osgi application compatibility, I'm using the bnd-maven-plugin 
to generate the OSGi manifests.

I also want to enable using ServiceLoader  mediator manifest entry's for OSGi, 
as the use of service provider style abstractions within River are widespread.

River also has its own service provider lookup mechanism: 
org.apache.resources.Service

Then there's the use of context ClassLoader's throughout to consider.

Regards,

Peter.

Sent from my Samsung device.
 

Extensible MarshalledInstance

[Peter Firmstone]

Reference:  
https://github.com/pfirmstone/river-internet/blob/trunk/src/net/jini/io/MarshalledInstance.java

The River PMC is discouraging the use of JRMP RMI protocols, since the removal 
of support for JRMP in River 2.2.3.  MarshalledObject utilises JRMP, so 
discouraging (deprecating) its use and providing MarshalledInstance equivalent 
methods throughout River's api makes a lot of sense.

Since the community has also been discussing utilising other serialization 
protocols, I have created an extensible version of MarshalledInstance, at the 
above reference.

Thoughts?

Regards,

Peter.

Sent from my Samsung device.
 

Summary of my external work

[Peter Firmstone]

Forked from River trunk just before 3.0 release.

   * Security focused:

 o Supports updated modern cyphers, support for vulnerable
   cypers removed.
 o Reimplementation of serialization, includes input validation
   and defensive programming.
 o Additional SafeServiceRegistrar interface with lookup method
   that allows clients to authenticate services prior to
   downloading.
 o ServiceDiscoveryManager configured to use new lookup method,
   without changes to API.
 o proxy jar files can contain META-INF permissions.perm files
   with advisory permissions (same format as OSGi local
   permissions).
 o New secure multicast discovery providers that dynamically
   grant download and deserialization permission for
   authenticated lookup services.
 o Phoenix now supports using TLS sockets for Registry.
 o LookupLocator now supports https unicast discovery for
   firewall/proxy traversal.
   * IPv6 Multicast discovery support
   * Better support for Java 9, no longer accessing sun jvm
 implementation packages
 o Phoenix uses RMI Registry now instead of RegistrySunExporter
 o KerberosServerEndpoint reflectively accesses
   com.sun.security.jgss.GSSUtil.createSubject, but only when
   using client token delegation.  Delegation not supported on
   other vendors jvm's.
   * Deprecated:
 o RegistrySunExporter
 o SunJRMPExporter
 o ProxyTrust

Currently working on a Modular Maven based build.

   * Uniting packages and removing some dependencies to assist OSGi
 developers (like some package changes for non api org.apache.river
 namspaces).
 o org.apache.river.reggie.proxy
 o org.apache.river.reggie.service
   * OSGi bundles with package based dependencies.


Looking forward to donating this code to River.

[INFO] 


[INFO] Reactor Summary:
[INFO]
[INFO] JGDMS Project .. SUCCESS [  
0.712 s]
[INFO] Module :: JGDMS Collection . SUCCESS [ 
18.139 s]
[INFO] Module :: JGDMS Jini Platform .. SUCCESS 
[01:17 min]
[INFO] Module :: JGDMS Loader . SUCCESS [ 
18.179 s]
[INFO] Module :: JGDMS Extensible Remote Invocation ... SUCCESS [ 
45.867 s]
[INFO] Module :: JGDMS Resources .. SUCCESS [  
0.099 s]
[INFO] Module :: JGDMS URL providers and Integrity  SUCCESS [ 
17.804 s]
[INFO] Module :: JGDMS Activation Platform  SUCCESS [ 
23.309 s]
[INFO] Module :: JGDMS Service DL Library . SUCCESS [ 
18.373 s]
[INFO] Module :: JGDMS Lookup Discovery Providers . SUCCESS [ 
17.094 s]
[INFO] Module :: JGDMS Service Library  SUCCESS [ 
22.786 s]
[INFO] Module :: JGDMS Service Starter  SUCCESS [ 
27.622 s]
[INFO] Module :: JGDMS SharedGroup Destroy  SUCCESS [ 
22.416 s]
[INFO] Module :: JGDMS IIOP ... SUCCESS [ 
13.330 s]
[INFO] Module :: JGDMS JRMP ... SUCCESS [ 
13.663 s]
[INFO] Module :: JGDMS Service DL Library UI Factory .. SUCCESS [ 
21.488 s]
[INFO] Module :: Jini 2.1 compatibility ... SUCCESS [ 
16.572 s]
[INFO] Module :: Outrigger  SUCCESS [  
0.017 s]
[INFO] Module :: Outrigger Service Download classes ... SUCCESS [ 
15.488 s]
[INFO] Module :: Outrigger Service Implementation . SUCCESS [ 
27.113 s]
[INFO] Module :: Outrigger Snaplogstore ... SUCCESS [ 
16.609 s]
[INFO] Module :: Lookup Service ... SUCCESS [  
0.014 s]
[INFO] Module :: Reggie Service Download classes .. SUCCESS [ 
14.048 s]
[INFO] Module :: Reggie Service Implementation  SUCCESS [ 
23.095 s]
[INFO] Module :: Mahalo ... SUCCESS [  
0.014 s]
[INFO] Module :: Mahalo Service Download classes .. SUCCESS [ 
13.727 s]
[INFO] Module :: Mahalo Service Implementation  SUCCESS [ 
22.967 s]
[INFO] Module :: Mercury the Event Mailbox  SUCCESS [  
0.014 s]
[INFO] Module :: Mercury Service Download classes . SUCCESS [ 
21.990 s]
[INFO] Module :: Mercury Service Implementation ... SUCCESS [ 
16.790 s]
[INFO] Module :: Norm . SUCCESS [  
0.014 s]
[INFO] Module :: Norm Service Download classes  SUCCESS [ 
20.415 s]
[INFO] Module :: Norm Service Implementation .. SUCCESS [ 
23.257 s]
[INFO] Module :: Group  SUCCESS [  
0.018 s]
[INFO] Module :: Group Service Download classes ... SUCCESS [ 
12.737 s]
[INFO] Module :: Group Service Implementation . SUCCESS [ 
16.827 s]

River maven build - initial testing with Rio

[Peter Firmstone]

So far building Rio against the River maven build, there are 3 test 
failures.


I've added compatibility layer to River of com.sun.jini classes that 
extend theirorg.apache.rivercounterparts to provide a backward 
compatible subset of deprecated com.sun.jini namespace (only the parts 
of the com.sun.jini namespace used by Rio).  Over time this namespace 
would be removed, it seemed more friendly to provided it in deprecated 
form to encourage upgrading / adoption.


The River maven build test run is appended further below.  The maven 
build was constructed using a modified version of Dennis Reedy's groovy 
script, followed by breaking out some shared components into their own 
modules.  No classes are duplicated, everything has a defined module.


Note that the QA tests and jtreg tests remain an ant build at this 
stage, I haven't yet modified them to run against the River modules, 
which requires changing the tests classpath to include the new modules.


This code is currently on GitHub, I'd like it to be the basis for River 
3.1.0


Regards,

Peter.

[INFO] 


[INFO] Building Rio Distribution 5.2.3-SNAPSHOT
[INFO] 

[INFO] 


[INFO] Reactor Summary:
[INFO]
[INFO] Rio Project  SUCCESS [  
0.057 s]
[INFO] Module :: Rio Logging Support .. SUCCESS [  
1.049 s]
[INFO] Module :: Rio Platform . FAILURE [ 
16.956 s]
[INFO] Module :: Rio service starters . SUCCESS [  
0.944 s]
[INFO] Module :: Webster .. SUCCESS [  
4.383 s]
[INFO] Module :: Rio API .. SUCCESS [  
2.943 s]
[INFO] Module :: Rio Proxy  SUCCESS [  
0.061 s]
[INFO] Module :: Rio Library .. SUCCESS 
[02:58 min]
[INFO] Module :: Aether Resolver .. FAILURE 
[09:52 min]
[INFO] Module :: Project Module Resolver .. SUCCESS [  
6.578 s]
[INFO] Module :: Event Collector Service API .. SUCCESS [  
0.612 s]
[INFO] Module :: Event Collector Service Proxy  SUCCESS [  
0.096 s]
[INFO] Module :: Cybernode Service API  SUCCESS [  
0.058 s]
[INFO] Module :: Cybernode Service Proxy .. SUCCESS [  
0.097 s]
[INFO] Module :: Cybernode Service Implementation . SUCCESS [  
1.570 s]
[INFO] Module :: Monitor Service API .. SUCCESS [  
0.080 s]
[INFO] Module :: Rio Test Infrastructure .. SUCCESS [  
4.926 s]
[INFO] Module :: Event Collector Service Implementation ... SUCCESS [ 
32.481 s]
[INFO] Module :: Watch User Interface . SUCCESS [  
3.476 s]
[INFO] Module :: Cybernode Service UI . SUCCESS [  
0.086 s]
[INFO] Module :: Gnostic .. SUCCESS [  
0.003 s]
[INFO] Module :: Gnostic API .. SUCCESS [  
0.580 s]
[INFO] Module :: Gnostic Service Implementation ... SUCCESS [ 
13.398 s]
[INFO] Module :: Monitor Service Proxy  SUCCESS [  
0.106 s]
[INFO] Module :: Monitor Service Implementation ... SUCCESS [ 
24.122 s]
[INFO] Module :: Rio Command Line Interface ... SUCCESS [  
0.097 s]
[INFO] Module :: Rio Management User Interface  SUCCESS [  
0.319 s]
[INFO] Rio Examples ... SUCCESS [  
0.003 s]
[INFO] Example :: Calculator .. SUCCESS [  
0.002 s]
[INFO] Example :: Calculator API .. SUCCESS [  
0.317 s]
[INFO] Example :: Calculator Service Implementation ... SUCCESS [  
0.114 s]
[INFO] Example :: Events .. SUCCESS [  
0.002 s]
[INFO] Example :: Events API .. SUCCESS [  
0.047 s]
[INFO] Example :: Events Proxy  SUCCESS [  
0.045 s]
[INFO] Example :: Events Service Implementation ... SUCCESS [  
0.090 s]
[INFO] Example :: Events UI ... SUCCESS [  
0.043 s]
[INFO] Example :: Hospital  SUCCESS [  
0.003 s]
[INFO] Example :: Hospital API  SUCCESS [  
0.076 s]
[INFO] Example :: Hospital Rules .. SUCCESS [  
0.094 s]
[INFO] Example :: Hospital Service Implementation . SUCCESS [  
0.093 s]
[INFO] Example :: Hospital UI . SUCCESS [  
0.091 s]
[INFO] Example :: SpringBean .. SUCCESS [  
0.003 s]
[INFO] Example :: SpringBean API .. SUCCESS [  
0.044 s]
[INFO] Example :: SpringBean Service Implementation ... SUCCESS [  
0.113 s]
[INFO] Example :: Workflow 

River revamp

[Peter Firmstone]

A discussion recently ignited on river private about revamping the project.

For the benefit of the wider developer community can we restate the suggestions 
here, feel free to reword, correct, reject or suggest.  It was along the lines 
of:

* Website revamp
* Remove Jini focus, with a historical section...
* Focus on new security features.
* Make getting started simple, with just the bare bones basics, Extensible 
remote invocation with secure serialization. 
* Services, Javaspaces etc, become examples of what can be done with River, not 
what River  is.

Regards,

Peter.



Sent from my Samsung device.
 

ServiceDiscoveryManager

[Peter Firmstone]

I've finally got ServiceDiscoveryManager to a stage where I feel like it's been 
completely brought up to date.

Originally SDM's LookupCache had some latent race conditions that became 
evident after I created a non blocking DynamicPolicyProvider. 

I spent some time refactoring it, I separated LookupCacheImpl from SDM, into 
it's own file, but was never happy with remote method calls and filtering being 
performed whist synchronized on a monitor.

I've finally got it sorted nicely now, so it's much easier to understand.  I've 
used some Java 8 features like ConcurrentMap.computeIfPresent, but I'm finally 
happy with it.

Logging in SDM is now processed by a single threaded executor, so logging 
doesn’t interfere with concurrency, yep it's a complex class and logging helps 
you figure out what's going on.

The best feature though, is utilising the new ServiceRegistrar default lookup 
method.

SDM's and LookupCache's api is unchanged, you just configure it how you want it 
to behave.

So if you want the existing slow insecure behaviour where you download 
everything, authenticate and filter later you can keep doing that.

However, if you want security and performance, with delayed unmarshalling, 
don't want to download services you can't authenticate or your filter doesn’t 
match locally using logical comparisons of attributes, you can do that too.  
Because you only download what you want when you need it, performance is gr8.

I've written a bunch of new tests too.  I haven't committed the latest changes.

Oh and security doesn't require you to implement ProxyTrust (complex and 
confusing), because the service is authenticated prior to download, not after.

Cheers,

Peter.

Sent from my Samsung device.
 

Re: Level of interest and planned activity

[Peter Firmstone]

 
Thank you Zsolt, for your offer of help and support.  You're right about our 
need to evangelise, maybe an article on Dzone would be a good start?  Updating 
our website, awesome, I can apply your patches :). 

Once it's possible to do so securely, I'll be making a public service registrar 
available, discoverable via ipv6 multicast, or https unicast for ipv4 and once 
the underlying low level infrastructure is in place, then I'd like to focus on 
writing tools to assist developers to remove some of the pain and complexity of 
securing their services and clients.

People will be able to register their services on this public registrar.

New users can focus on client code first, followed by implementing their own 
services.

I'm currently working on adding support for a new ServiceRegistrar lookup 
method to ServiceDiscoveryManager that allows authentication prior to service 
and codebase downloads, service or attribute deserialization.  There are no 
changes to SDM's public api, just a configuration parameter to choose insecure 
lookup, should the user wish to use the slower insecure method of lookup (for 
compatibility with an earlier ServiceRegistrar).

The new lookup method allows local filtering to occur prior to service 
download, allowing clients to avoid unnecessary downloads.

I've written a number of tests for the new lookup method.

Regards,

Peter.

Sent from my Samsung device.
 
  Include original message
 Original message 
From: Zsolt Kúti 
Sent: 21/10/2016 12:39:09 am
To: dev@river.apache.org
Subject: Re: Level of interest and planned activity

See inlined. 


On Thu, Oct 20, 2016 at 1:02 AM, Patricia Shanahan  wrote: 

> Next month is a River board report month. Before drafting the report, I 
> would like to get an impression of the level of interest and planned 
> activity among the current contributors. 
> 

I am not yet a contributor, 


> 
> Are you planning to contribute to River in the next year, even if you are 
> currently too busy? 
> 

but planning. 


> Have you been following the "Future of River" thread? 
> 

Yes. 


> 
> If so, are you interested in working along the lines Peter has proposed? 
> 

I am. As a first step I would like to take the website revamp. We need 
something more eye catching and modern. 


> Do you have other ideas for future direction? 
> 
> 
Beside technical advancements attracting users is badly needed. Needs to 
find out how - like make it easy to find River (well placed tech articles, 
"evangelism", etc.), to understand its concepts, getting started and usage 
Example codes, small and larger... 

Exporters for other RPC frameworks

[Peter Firmstone]

Anyone interested in Exporters for other RPC Frameworks?

If so which and why?

Pete.

Sent from my Samsung device.
 

release artifacts

[Peter Firmstone]

Getting another set of release artifacts 4 River3 ready and have run all tests 
again, need to generate pgp signatures on weekend.

Decided not to use X500 release cert to sign jar files this release to prevent 
holding up progress, since I haven't worked out how others can verify release 
artifacts as the pgp signatures will be different when comparing artifacts 
containing signed jars with those that don't, then there's the issue of how to 
integrate it into the build process.

Regards,

Peter.

Sent from my Samsung device.
 

Hinkmond Wong 2014 blogs about jini and iot

[Peter Firmstone]

https://blogs.oracle.com/hinkmond/entry/easy_iot_sensor_on_boarding

Must have missed this earlier.

Sent from my Samsung device.
 

VOTE: Take Security seriously or my resignation.

[Peter Firmstone]

Option 1.  I propose that we take security seriously, no security patches are 
to be rejected prior to review, that we review and analyse them properly based 
on merit. That discussions about security issues be taken seriously.

Option 2.  Alternatively I resign my River committer status

Please cast your vote, the vote is open for 7 days.

Let the community decide.

Regards,

Peter

Sent from my Samsung device.
 

Release 3.0, package rename and ServiceProxyAccessor

[Peter Firmstone]

ServiceProxyAccesor is an interface in the start package, a similar interface 
called ProxyAccessor exists in the net.jini.export package.

The difference between these two interfaces is the former returns a smart 
proxy, which may not be a Remote object, while the latter returns the Remote 
proxy which is a reflective proxy or remote proxy stub.

Often, the former contains the latter, where the former implements service api, 
while the latter is used for remote communication with the service's server.

In a future version of River,  ServiceProxyAccessor could be used by clients to 
obtain the service proxy, from a bootstrap proxy, allowing authentication to 
occur prior to codebase download and unmarshalling.  Think security and delayed 
unmarshalling.

If this was to occur, we'd need to change the package for the 
ServiceProxyAccessor interface, to net.jini.export. 

Seeing as we're about to change the package of ServiceProxyAccessor in the 3.0 
release, it would have less impact on downstream code if this change was only 
made once.

If the security and performance improvements were not accepted, (I'm quite 
confident that we will agree on a solution once the benefits can be 
demonstrated) it still makes sense to move ServiceProxyAccessor to 
net.jini.exporter due to their related use and functionality.

Should I proceed with a vote to move ServiceProxyAccessor to net.jiniexporter 
now, prior to release, or should I leave it until River 3.1?

This wont cause a delay in the release process, but I'm happy to leave it if 
anyone is concerned it will.

Regards,

Peter.

Sent from my Samsung device.
 

River 3.0 release

[Peter Firmstone]

There's new api in org.apache.river.api.lookup and org.apache.river.api.util 
I'd like to remove before releasing.

This relates to
StreamServiceRegistrar, it doesn't have any implementation yet, also only 
net.jini.* is agreed upon as api.

I think the additional lookup method would be better if it was included with an 
implementation in River 3.1, perhaps as a java 8 default method in 
ServiceRegistrar for compatibility reasons (doesn't require an additional 
interface).

For now unless there's any disagreement I'll remove them before releasing.

This will be our best release yet, with over 200 less bugs than River 2.2.1, 
over 200 more unit tests and significant performance improvements while 
reducing our maintenance debt.

Although I've been mostly responsible for fixing JMM non compliance, this works 
is a result of the combined efforts of a number of committers and external 
contributors.

In following releases, I look forward to working with you, fixing performance 
issues identified by the original Jini team, documented on Jira, fixing 
security vulnerabilities, the modular build and support for ipv6 discovery.

Regards,

Peter.

eSent from my Samsung device.
 

committer keys for release

[Peter Firmstone]

Committers who have contributed to River, please append your pgp public key to 
the KEYS file in the trunk directory in preparation for release.

Thank you,

Peter.
 
Sent from my Samsung device.
 

research article on Jini /River security

[Peter Firmstone]

http://www.hindawi.com/journals/ijdsn/2015/205793/

Regards,

Peter

Re: Apache River hello world Success Run in Linux

[Peter Firmstone]

That's good news, well done!

On 18/02/2015 3:32 AM, amit batajoo wrote:

Hello Peter,

Thank you for your support and suggestion, finally I successfully run 
the apache-river and hello world program example on my linux 
environment with java version 1.8.0.

Here are the screenshot of my success run.
I am future doing research and study on apache river and distributed 
computing and hope same suggestion and support in future from you and 
your team.


Thank you
--
Er. BATAJOO, Amit
Researcher
Wakkanai Hokusei Gakuen University
Wakkanai, Hokkaido, Japan
---
Skype : abatajoo7
E-Mail :batajooseamu...@gmail.com 
mailto:batajooseamu...@gmail.com,batajooa...@gmail.com 
mailto:batajooa...@gmail.com, bataj007a...@gmail.com 
mailto:bataj007a...@gmail.com

Re: Security

[Peter Firmstone]

Continuing on ...

Lets say for example, we have a secure OS and we provide a service on a 
public port and we have a determined attacker attempting to use 
deserialization to take over our system or bring it to its knees with 
denial of service.


We know this is relatively easy with standard ObjectInputStream.

When we invoke a remote method, the return value is what you call a root 
object, that is it's the root of a serialized object tree, originating 
through the root object, via its fields.


Most Serializable objects have invariants.

We can consider the java serialization stream format as a string of 
commands that allows creation of any object, bypassing all levels of 
visibility.  That is an attacker can create package private classes or 
private internal classes, provided they implement Serializable.  So 
think of Serializable as a public constructor that lacks a context 
representing the attacker.  That's right, there is no context 
representing the remote endpoint in the thread call stack.


Now we can place restrictions on the stream, such as a requirement that 
the stream is reset before a limit on the number of objects deserialized 
is reached.  We can also place a limit on the count of all array 
elements read in from the stream, until the stream is reset.  These 
measures ensure the stream cannot go on forever, they also prevent stack 
overflow and out of memory errors.  At some point the caller will regain 
control of the stream without running out of resources.


But getting back to the previous problem, the attacker has command line 
access to create any Serializable object.


Now most objects have invariants that should be satisfied, for correct 
functional state, but a major problem with Serialization is it creates 
an instance, using the first zero arg constructor of the first non 
serializable superclass.  Yep that's right, that could be ClassLoader, 
you clever attacker you!  The poor old object hasn't even had a chance 
to check it's invariants and it's game over, that's not fair!


This will never do, I had to create a new contract for atomic object 
construction:


  1. The class must implement Serializable (can be inherited) AND one
 of the following conditions must be met:
  2. The class must be annotated with @AtomicSerial OR
  3. The class must be stateless and can be created by calling
 class.newInstance() OR
  4. The class must have DeSerializationPermission

Now if the class is annotated with @AtomicSerial, it must also have a 
constructor signature that has public or default visibility:


SomeClass(GetArg arg) throws IOException{
// The first thing I must do is to call a static method that checks 
invariants, before calling a superclass constructor, the static method 
should return the argument for another constructor.

}

Some simple rules for our object input stream:

  1. Though shalt not publish a reference to a partially constructed
 object.
  2. Though shalt not publish a reference if an object fails
 construction, where readObject, readObjectNoData and readResolve
 methods are considered to be constructors for the purpose of
 deserialization of conventional Serializable objects.
  3. Though shalt not attempt to construct a Serializable object that
 doesn't have an @AtomicSerial annotation, or has serial fields
 (state) and doesn't have DeSerializationPermission.
  4. Only call a protected constructor if the class doesn't implement
 @AtomicSerial and has DeSerializationPermission.
  5. Do not honour the standard java Serialization construction
 contract (not all Serializable classes can be constructed even if
 they have DeSerializationPermission).
  6. If a standard Serializable class has DeSerializationPermission for
 it to be constructed it must have a zero arg constructor or a
 constructor that accepts null object arguments and default
 primitive values.
  7. If an any object in a serialized object graph fails it's invariant
 checks, deserialization of the object graph fails at that point
 and control is returned to the caller, by way of an
 InvalidObjectException (a child class of IOException).
  8. Honour the standard java serialization stream protocol.
  9. Count number of objects received and throw a
 StreamCorruptedException if limit is exceeded.
 10. Count number of array elements received and throw
 StreamCorruptedException if limit is exceeded.
 11. readResolve() can be called on @AtomicSerial instances but,
 readObject() is never called and neither is readObjectNoData().

Obligations of our object output stream:

  1. Reset the stream before the limit is reached.
  2. Replace java collections and maps with safe @AtomicSerial
 implementations, it is the developers obligation to replace them
 with their preferred implementations during construction, these
 are functional but are only immutable containers for keys, values
 and comparators.
  3. Honour 

Re: Config File of config/start-reggie.config

[Peter Firmstone]

Hi Bishnu,

I have previously tested River 2.2.0 on ARM, but experienced a number of 
test failures.


You mentioned the host name isn't defined in your DNS server, if you 
configure host names with IP adrresses in your /etc/hostname files it 
still doesn't work with hostnames?


I spent a considerable amount of time fixing bugs (1 year if I remember 
correctly) and eventually succeeded with all tests passing with the 
qa-refactor branch on ARM.The qa-refactor branch has been tested on 
more architectures and OS's than earlier releases, even on IBM's J9 jvm, 
which the earlier releases won't even compile on.  Even so I haven't 
tested the latest changes on ARM, as I no longer have access to this 
architecture.


If your planning on ARM deployment, we might need to progress the 
qa-refactor branch.


I don't presently have a Unix test envronment set up, do you have the 
jtreg library installed on your Rasberry PI?


I can walk you through the build and test instructions.

I'd like to confirm your not experiencing a bug first, by checking all 
tests run properly.


Regards,

Peter.




On 18/02/2015 9:09 PM, Bishnu Gautam wrote:

Hi Peter

We are trying to deploy River in Raspberry pi. We were successful to 
deploy it however need to tweak few other things in order to utilize 
this platform more precisely.
Our problem is that we need to run the reggie and need to return IP 
address rather than domain name in order to run our client from other 
machine. Whenever we run our server, client, regggiee etc in the same 
machine it works fine. But when we run our client program from other 
machine, it could not resolve the domain name as the  server name 
(Where reggies runs)  is not recorded into our DNS records. So, we 
tweaked the hostname of the server and changed it to IP address, in 
that case it works successfully. However, we do not want our users 
change their hostname into ip address in order to make it workable.


So, we would like to tweak the config file of Reggie. The current 
config file does not return IP address. We tried to change it by 
changing ConfigUtil.getHostName() to ConfigUtil.getHostAddress(), but 
it still returns the hostname. I have attached 2 files. One of which 
started and showing IP address. In this case we changed our hostname 
(/etc/hostname file) to IP address. It works fine but as  I mentioned 
above, in this case, user needs to change his/her hostname to IP 
address which we do not want to demand.  And the other file is the 
hostname without change and this makes problem if we run our client 
from separate machine.


 Do you have any other methods so that it runs successfully and return 
IP address. I think the config file of start-reggie.config located at 
example/hello/config/start-reggie.config has following codes. Thank 
you very much for your help in this regards


*
import com.sun.jini.config.ConfigUtil;
import com.sun.jini.start.NonActivatableServiceDescriptor;
import com.sun.jini.start.ServiceDescriptor;

com.sun.jini.start {

private static codebase =
ConfigUtil.concat(
new Object[] {
http://;, ConfigUtil.getHostName(), 
:8080/reggie-dl.jar,

 ,
http://;, ConfigUtil.getHostName(), 
:8080/jsk-dl.jar } );

private static policy = config${/}reggie.policy;
private static classpath = ..${/}..${/}lib${/}reggie.jar;
private static config = config${/}jrmp-reggie.config;

static serviceDescriptors = new ServiceDescriptor[] {
new NonActivatableServiceDescriptor(
codebase, policy, classpath,
com.sun.jini.reggie.TransientRegistrarImpl,
new String[] { config })
};

}//end com.sun.jini.start

**

Re: Oracle Mobile application framework

[Peter Firmstone]

Some standard java se components are missing from compact 2:

Compiling 905 source files to 
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\build\classes

javac 1.8.0
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\discovery\kerberos\Client.java:36: 
error: KerberosPrincipal is not available in profile 'compact2'

import javax.security.auth.kerberos.KerberosPrincipal;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\discovery\kerberos\Client.java:118: 
error: KerberosPrincipal is not available in profile 'compact2'

private static KerberosPrincipal getKerberosPrincipal(
   ^
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:23: 
warning: [deprecation] JavaSpaceAdmin in com.sun.jini.outrigger has been 
deprecated

import com.sun.jini.outrigger.JavaSpaceAdmin;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:26: 
error: BorderLayout is not available in profile 'compact2'

import java.awt.BorderLayout;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:27: 
error: Component is not available in profile 'compact2'

import java.awt.Component;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:28: 
error: ActionEvent is not available in profile 'compact2'

import java.awt.event.ActionEvent;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:29: 
error: ActionListener is not available in profile 'compact2'

import java.awt.event.ActionListener;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:30: 
error: MouseAdapter is not available in profile 'compact2'

import java.awt.event.MouseAdapter;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:31: 
error: MouseEvent is not available in profile 'compact2'

import java.awt.event.MouseEvent;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:32: 
error: MouseListener is not available in profile 'compact2'

import java.awt.event.MouseListener;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:33: 
error: WindowAdapter is not available in profile 'compact2'

import java.awt.event.WindowAdapter;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:34: 
error: WindowEvent is not available in profile 'compact2'

import java.awt.event.WindowEvent;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:35: 
error: WindowListener is not available in profile 'compact2'

import java.awt.event.WindowListener;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:60: 
error: BorderFactory is not available in profile 'compact2'

import javax.swing.BorderFactory;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:61: 
error: DefaultListModel is not available in profile 'compact2'

import javax.swing.DefaultListModel;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:62: 
error: Icon is not available in profile 'compact2'

import javax.swing.Icon;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:63: 
error: JCheckBoxMenuItem is not available in profile 'compact2'

import javax.swing.JCheckBoxMenuItem;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:64: 
error: JFrame is not available in profile 'compact2'

import javax.swing.JFrame;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:65: 
error: JLabel is not available in profile 'compact2'

import javax.swing.JLabel;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:66: 
error: JList is not available in profile 'compact2'

import javax.swing.JList;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:67: 
error: JMenu is not available in profile 'compact2'

import javax.swing.JMenu;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:68: 
error: JMenuBar is not available in profile 'compact2'

import javax.swing.JMenuBar;
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\src\com\sun\jini\example\browser\Browser.java:69: 
error: JMenuItem is not available in 

Re: Build troubles

[Peter Firmstone]

On 14/02/2015 6:19 PM, Mike wrote:


Happy to hear that No ;) That said, I have been, and would like to
continue, hacking on the core. I'll talk about some of my reasons in a 
different thread than this one though.




I'd be interested to hear what you have in mind.

This build works on Java 8, it's also been tested on arm, sparc, OSX and 
IBM's J9:


http://svn.apache.org/viewvc/river/jtsk/skunk/qa_refactor/

There are numerous minor improvements internally, some of which include:

   * Multiple synchronization, concurrency and race condition fixes.
   * Fixes for bugs identified by FindBugs static analysis.
   * Runs at raw socket speed, all hotspots are native methods.
   * Elimination of unnecessary DNS calls.
   * Highly scalable security manager and policy providers. (Security
 impact on performance is 1%)

I've purchased a T5240 so I can perform more testing, there are some 
changes made in trunk I need to merge in before pre release.


I'm currently investigating securing ObjectInputStream against DOS attacks.

Regards,

Peter.

Re: Oracle Mobile application framework

[Peter Firmstone]

Still just pondering the possibilities.

Compact 2 profile doesn't include swing, awt, kerberos, IIOP or Corba.

But it is interesting to consider how we might support it, such as by 
providing a platform.jar that does.


On 14/02/2015 3:08 PM, Patricia Shanahan wrote:
We do need to understand why it does not build. That issue might 
affect other River activities. Ideally, it would be changed to make it 
work in MAF.


I feel the browser is a useful part of the examples. If it cannot be 
made compatible with the MAF I think it would be better to keep it in 
the examples and separate out the examples rather than to remove it 
from the examples.


Yes, I agree.



Patricia


On 2/13/2015 7:48 PM, Greg Trasuk wrote:

Hi Peter:

I’m not sure if supporting MAF is critical to River’s success, but 
even so I am in favour of removing the examples from the JTSK build.  
Anyone else have an opinion?


I’m curious though, why it’s the browser specifically that doesn’t 
build - what did you find?


Cheers,

Greg.

On Feb 13, 2015, at 6:59 PM, Peter j...@zeus.net.au wrote:


Maf 2.1.0 supports java 8 compact profile 2.

If I remove the browser example from qa-refactor it builds on this 
profile.


Greg, if your including the browser in the river examples project, I 
can delete it?


If so, we can support iphone and android.

Regards,

Peter.

Re: A public bootstrap proxy interface and an Entry

[Peter Firmstone]

Each nested array is created individually as an array object by the 
ObjectInputStream, first by creating an array, then by reading in each 
object from the stream, which can be another array.


So during creation, this structure will need to have each nested array's 
length subtracted from the limit, until the outer array returns, 
otherwise it can quickly consume all available memory with as little as 
four objects,  the jvm allocating space for all element references with 
each array object creation.


Peter.

On 12/02/2015 12:15 AM, Patricia Shanahan wrote:

How do the array length limits work? For example, consider:

int[][][] myArray = new int[1000][1000][1000];

Or the equivalent initialization done in loops in a constructor?

Patricia

On 2/11/2015 3:57 AM, Peter Firmstone wrote:
...

It appears that fixing ObjectInputStream and Serializable security
issues was much easier than expected, provided we're prepared to
implement atomic invariant validation and give up some functionality:

   1. Circular references
   2. Limits on object cache size and periodically calling reset()
   3. Limits on array lengths.
   4. Classes that don't implement Serializable's readObject() method
  safely.

...

A public bootstrap proxy interface and an Entry

[Peter Firmstone]

Our present security model relies on the safety of the java sandbox, but 
we know that model is flawed.


If DownloadPermission is not granted, we cannot lookup a service that 
uses a smart proxy and ask it for the bootstrap proxy.  We could 
however, lookup a bootstrap proxy, authenticate it, grant it 
DownloadPermission and ask it for the smart proxy.


Would someone like to propose an interface for a bootstrap proxy and an 
Entry that allows the bootstrap proxy to list the service interfaces 
that its smart proxy provides, in order to perform lookup?


It appears that fixing ObjectInputStream and Serializable security 
issues was much easier than expected, provided we're prepared to 
implement atomic invariant validation and give up some functionality:


  1. Circular references
  2. Limits on object cache size and periodically calling reset()
  3. Limits on array lengths.
  4. Classes that don't implement Serializable's readObject() method
 safely.

Despite placing limits on functionality, none of the tests in the 
qa-suite tested so far (lookup services and javaspaces) fail without it, 
or depend on it.


Would anyone like to assist construct some stream test cases that cause 
DOS?  Eg read in a stream that contains an array length of 
Integer.MAX_VALUE, or one that uses a known exploit, eg deserialization 
into privileged context to create a ClassLoader instance on an unpatched 
jvm?  I'm quite confident I can prevent them, anyone up for a challenge?


Regards,

Peter.

Re: Urgent Request for help

[Peter Firmstone]

Hi Amit,

At line 24:

# Shell script to run Reggie

set -x

java -Djava.security.policy=config/start.policy\
 -Djava.ext.dirs=../../lib-ext/\
 -jar ../../lib/start.jar\
 config/start-reggie.config


Make the following changes :

# Shell script to run Reggie

set -x

java -Djava.security.manager=java.rmi.RMISecurityManager   \
-Djava.security.policy=config/start.policy\
-Djava.rmi.server.useCodebaseOnly=false\
 -Djava.ext.dirs=../../lib-ext/\
 -jar ../../lib/start.jar\
 config/start-reggie.config


I can't promise you this will work, but it's worth trying.  The only 
build tested and known to work on ARM is qa-refactor which is an 
experimental unreleased build.  These are old examples, since Java 6, 
it's best to define the security manager as above.


let me know how you fare.

Regards,

Peter.

On 9/02/2015 5:00 PM, amit batajoo wrote:

hi peter

I am trying apache-river-2.2.2 in linux debian. I have successfully 
run the httpd.sh script but while i am trying to jrmp-reggie.sh I am 
getting give error, please help to solve this problem.


Command and error messgae is given below, please suggest me step by 
step and provide me files is i am missing any, or if you have some 
runnable open source code than please provide for sample example.


root@raspberrypi:/opt/jyaguchi/apache-river-2.2.2/examples/hello/scripts# 
./jrmp-reggie.sh
+ java -Djava.security.policy=config/start.policy 
-Djava.ext.dirs=/opt/jyaguchi/apache-river-2.2.2/lib-ext/ -jar 
/opt/jyaguchi/apache-river-2.2.2/lib/start.jar config/start-reggie.config

Exception in thread main java.lang.ExceptionInInitializerError
at 
net.jini.config.ConfigurationProvider.getInstance(ConfigurationProvider.java:192)
at 
net.jini.config.ConfigurationProvider.getInstance(ConfigurationProvider.java:137)

at com.sun.jini.start.ServiceStarter.main(ServiceStarter.java:475)
Caused by: java.security.AccessControlException: access denied 
(java.lang.RuntimePermission createSecurityManager)
at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
at 
java.security.AccessController.checkPermission(AccessController.java:884)
at 
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)

at java.lang.SecurityManager.init(SecurityManager.java:299)
at 
net.jini.security.Security$ClassContextAccess.init(Security.java:965)
at 
net.jini.security.Security$ClassContextAccess.init(Security.java:965)

at net.jini.security.Security$1.run(Security.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at net.jini.security.Security.clinit(Security.java:165)
... 3 more


Thank you in advance

--
Er. BATAJOO, Amit
Researcher
Wakkanai Hokusei Gakuen University
Wakkanai, Hokkaido, Japan
---
Skype : abatajoo7
E-Mail :batajooseamu...@gmail.com 
mailto:batajooseamu...@gmail.com,batajooa...@gmail.com 
mailto:batajooa...@gmail.com, bataj007a...@gmail.com 
mailto:bataj007a...@gmail.com

Re: River-examples project

[Peter Firmstone]

The qa-refactor build fully supports Windows file paths, so Window's 
users won't need cigwin when it's eventually released.


I was thinking it might help the experience for new users if we can move 
all examples into their own build and distribution, so that users aren't 
distracted by the main build?


Regards,

Peter.

On 9/02/2015 4:55 PM, Patricia Shanahan wrote:

On 2/8/2015 10:30 PM, Greg Trasuk wrote:


Auto-correct apparently doesn’t recognize “jvm” and changed it to 
“jam” below :-)


The only good auto-correct is a disabled auto-correct.



Greg Trasuk.

On Feb 9, 2015, at 1:22 AM, Greg Trasuk tras...@stratuscom.com wrote:



By the way, the command lines are just invoking the jam, so the 
exact same command line should work in Windows command shell.  But 
you’re right, we should have explicit Windows instructions in the 
tutorial as well.




I'll carry on using Cygwin, which now does start the hello Greeter 
service, and then go back and repeat the trail using a Windows command 
tool. Ideally, both will work and we can give new users their choice.


Thanks for the quick fix.

Patricia

Re: Security

[Peter Firmstone]

Ok, so here's where I discuss the how I've addressed serialization's 
security issues:


From the source code:
 // These two settings are to prevent DOS attacks.
private static final int MAX_ARRAY_LEN = 32768;
private static final int MAX_OBJECT_CACHE = 65664;

The output stream implementation automatically resets the cach, when it 
exceeds a certain size, while the ObjectInputStream throws a 
StreamCorruptedException if OBJECT_CACHE is exceeded.


@Override
public void writeObjectOverride(Object obj) throws IOException {
d.writeObject(obj);
// This is where we check the number of Object's cached and
// reset if we're getting towards half our limit.
// One day this limit will become too small.
if (d.numObjectsCached  32768) reset();
}

A new annotation

@AtomicSerial is provided for classes that implement Serializable and 
want to validate their invariants atomically.  That is if invariants 
aren't satisfied, the object is not created and doesn't exist so cannot 
be used as an attack vector.


An annotation was chosen since it is not inherited by subclasses.

Classes that implement Serializable, still continue to do so as usual, 
the serial form doesn't change.


Use of the new streams is determined by MethodConstraints.

Child classes in the stream that don't implement @AtomicSerial (apart 
from instances of Throwable, immutable Object versions of primitives and 
MarshalledObject) require DeSerializationPermission.


Because circular links are prohibited, when a conventional Serializable 
object is constructed, it is not published until after it's readObject, 
readObjectNoData or readResolve method has been called, so an attacker 
is prevented from obtaining a reference to it through the stream.  These 
object's are still vulnerable to finalizer attacks, however that 
requires downloaded code and the intent is for this to be used to 
establish proxy trust prior to granting DownloadPermission.


@AtomicSerial objects are constructed using a constructor:

SomeObject(GetArg arg) throws IOException{
super(check(arg));
Object somefield = arg.get(somefield, null);
}

Where GetArg extends GetFields, but provides caller sensitive methods, 
to ensure different classes can't see each others field namespaces.  To 
construct a GetArg instance requires 
SerializablePermission(enableSubclassImplementation).


The lowest extension class in the inheritance heirarchy is called first, 
it checks its invariants using a static method, each class in the 
inheritance heirarchy checks it's invariants before Object's default 
constructor is called, if any invariant checks fail the object is not 
constructed.


This is also generic parameter and final field friendly.

@ReadInput (used to annotate a static method that returns ReadObject (an 
interface) are both provided to allow classes to gain direct access to 
the stream, as they would in readObject(ObjectInputStream in), but 
without requiring their own object instance.


Conventional classes are deserialized using a best effort approach, by 
trying each constructor on the lowest extension class, using default 
values.  Not all Serializable object's can be constructed.


The intent is for services, proxy's and discovery to use DOS safe code 
to serialize their state while establishing trust.


Collection, List, Set, SortedSet, Map and SortedMap are replaced in the 
ObjectOutputStream implementation with DOS attack safe immutable 
versions backed by arrays, these are functional package private 
implementations, intended as parameters, so implementations of 
@AtomicSerial are required to pass them as parameters to their preferred 
collection implementation.  @AtomicSerial implementations are encouraged 
to use checked collections, see java.util.Collections.


Conventional serialization can be used for trusted connections.

@AtomicSerial is very easy to implement, and much easier to evolve than 
default serialization, some users may wish to use it anyway (it's also 
final field friendly), but it is definitely intended to be optional.


Regards,

Peter.



On 8/02/2015 6:11 PM, Peter Firmstone wrote:

Thanks Dan, hopefully I don't dissapoint.

... So continuing on, another benefit of secure Serialization, if 
you're a lookup service, you don't need to authenticate your clients, 
you can deal with anyone and your not subject to DOS attacks, other 
than more conventional attacks unrelated to java.


I've been investigating Serialization security, identifying issues and 
considering how to deal with them. I think everyone is aware 
Serialization has security pitfalls, if I fail to mention an issue 
you're aware of, please let me know.


At first I thought it was just an issue of limiting the classes that 
could be deserialized and that's relatively easily done, for example, 
ArrayList reads an integer from the stream, then uses it to create an 
array, without sanity checking it first.  Well that's easy, just 
prevent ArrayList and a bunch

Re: Security

[Peter Firmstone]

Thanks Dan, hopefully I don't dissapoint.

... So continuing on, another benefit of secure Serialization, if you're 
a lookup service, you don't need to authenticate your clients, you can 
deal with anyone and your not subject to DOS attacks, other than more 
conventional attacks unrelated to java.


I've been investigating Serialization security, identifying issues and 
considering how to deal with them. I think everyone is aware 
Serialization has security pitfalls, if I fail to mention an issue 
you're aware of, please let me know.


At first I thought it was just an issue of limiting the classes that 
could be deserialized and that's relatively easily done, for example, 
ArrayList reads an integer from the stream, then uses it to create an 
array, without sanity checking it first.  Well that's easy, just prevent 
ArrayList and a bunch of others from deserializing...


Not so fast, ObjectInputStream also creates arrays, without sanity 
checking, blockdata also has similar issues during byte array creation.


So you only need send an Integer.MAX_VALUE to bring the jvm to it's 
knees, and if that doesn't do it, send a multi dimension array with a 
few more.   It requires very little data from the sender, just a few 
bytes and a couple of integers.


In addition ObjectInputStream caches objects, so they don't have to be 
re-serialized, but if ObjectOutputStream  doesn't perform a reset, well 
you can figure that out without my help.


But wait there's more...

During deserialization, Serializable objects are instantiated by calling 
a zero arg constructor of the first non Serializable super class, this 
partially constructed Object, an instance of a Serializable child class, 
without any invariant checks or validation, is then allocated an integer 
handle and published, an attacker is now free to obtain a reference to 
the unconstructed object simply by inserting a reference handle into the 
stream.


At this time the ProtectionDomain's of the classes in the object's 
heirarchy are not present in the AccessControlContext, which is why 
attackers in the past have been able to creat ClassLoader instances and 
download code, when someone has deserialized into privileged context 
(that renders DownloadPermission useless, because the attacker can work 
around it).


After unsafe publication, the fields are read in from the stream from 
super class to child class and set.


Now I don't blame the developers back in the day, they had deadlines and 
targets to achieve, but  the market has changed significantly since then 
and the issue needs addressing.


The good news is, the Serialization stream protocol has all the 
components necessary to create a secure ObjectInputStream.


For example, the stream cache can be reset periodically, this also means 
we can place a limit on the number of objects cached, and require 
ObjectOuputStream to call reset.  If it doesn't and the object cache 
exceeds our limit, StreamCorruptedException.


For array length, again we can impose limits, and throw 
StreamCorruptedException if the limit is exceeded.


The collections themselves aren't hard to solve, simply replace all 
collection types with a safe replacement in ObjectOutputStream and 
require DeSerializationPermission for known insecure objects.


Circular links, can't be supported using existing deserialization 
mechanisms.


Does this last point matter?

My implementation of ObjectInputStream doesn't support circular links 
and passes all lookup service tests.  In this case circular links are 
replaced with null.  To support circular links safely would require some 
cooperation from the classes participating in deserialization.


Construction during deserialization is the last challenge, many existing 
Serializable classes don't have public constructors or zero arg 
constructors, even though implementing Serializable is equivalent to a 
public constructor.


... to be continued, until next time.

Cheers,

Peter.


On 5/02/2015 2:38 AM, Dan Rollo wrote:

Very interesting. Looking forward to the next episode.


On Feb 4, 2015, at 9:11 AM, dev-digest-h...@river.apache.org wrote:

to be continued...

Re: Security

[Peter Firmstone]

Thanks Dan, hopefully I don't dissapoint.

... So continuing on, another benefit of secure Serialization, if you're 
a lookup service, you don't need to authenticate your clients, you can 
deal with anyone and your not subject to DOS attacks, other than more 
conventional attacks unrelated to java.


I've been investigating Serialization security, identifying issues and 
considering how to deal with them. I think everyone is aware 
Serialization has security pitfalls, if I fail to mention an issue 
you're aware of, please let me know.


At first I thought it was just an issue of limiting the classes that 
could be deserialized and that's relatively easily done, for example, 
ArrayList reads an integer from the stream, then uses it to create an 
array, without sanity checking it first.  Well that's easy, just prevent 
ArrayList and a bunch of others from deserializing...


Not so fast, ObjectInputStream also creates arrays, without sanity 
checking, blockdata also has similar issues during byte array creation.


So you only need send an Integer.MAX_VALUE to bring the jvm to it's 
knees, and if that doesn't do it, send a multi dimension array with a 
few more.   It requires very little data from the sender, just a few 
bytes and a couple of integers.


In addition ObjectInputStream caches objects, so they don't have to be 
re-serialized, but if ObjectOutputStream  doesn't perform a reset, well 
you can figure that out without my help.


But wait there's more...

During deserialization, Serializable objects are instantiated by calling 
a zero arg constructor of the first non Serializable super class, this 
partially constructed Object, an instance of a Serializable child class, 
without any invariant checks or validation, is then allocated an integer 
handle and published, an attacker is now free to obtain a reference to 
the unconstructed object simply by inserting a reference handle into the 
stream.


At this time the ProtectionDomain's of the classes in the object's 
heirarchy are not present in the AccessControlContext, which is why 
attackers in the past have been able to creat ClassLoader instances and 
download code, when someone has deserialized into privileged context 
(that renders DownloadPermission useless, because the attacker can work 
around it).


After unsafe publication, the fields are read in from the stream from 
super class to child class and set.


Now I don't blame the developers back in the day, they had deadlines and 
targets to achieve, but  the market has changed significantly since then 
and the issue needs addressing.


The good news is, the Serialization stream protocol has all the 
components necessary to create a secure ObjectInputStream.


For example, the stream cache can be reset periodically, this also means 
we can place a limit on the number of objects cached, and require  
ObjectOuputStream to call reset.  If it doesn't and the object cache 
exceeds our limit, StreamCorruptedException.


For array length, again we can impose limits, and throw 
StreamCorruptedException if the limit is exceeded.


The collections themselves aren't hard to solve, simply replace all 
collection types with a safe replacement in ObjectOutputStream and 
require DeSerializationPermission for known insecure objects.


Circular links, can't be supported using existing deserialization 
mechanisms.


Does this last point matter?

My implementation of ObjectInputStream doesn't support circular links 
and passes all lookup service tests.  In this case circular links are 
replaced with null.  To support circular links safely would require some 
cooperation from the classes participating in deserialization.


Construction during deserialization is the last challenge, many existing 
Serializable classes don't have public constructors or zero arg 
constructors, even though implementing Serializable is equivalent to a 
public constructor.


... to be continued, until next time.

Cheers,

Peter.


On 5/02/2015 2:38 AM, Dan Rollo wrote:

Very interesting. Looking forward to the next episode.


On Feb 4, 2015, at 9:11 AM, dev-digest-h...@river.apache.org wrote:

to be continued...

Security

[Peter Firmstone]

There's a free certificate authority coming this year, I think privacy 
and security are hot topics these days: https://letsencrypt.org/


Just a quick note about something I'm currently exploring.

The good thing about River is it allows you to be mostly ignorant of 
security when developing services and clients and then later using 
configuration, secure services and clients.


River is secure for the following scenario:

   * One entity / company is reponsible for the lookup service,
 services and clients.
   * Secure Discovery v2 is used.
   * Codebase Integrity and TLS / SSL Endpoints.
   * Authentication of services and clients is required.

Where River is not secure:

   * More than two entites / companies interact using lookup services,
 services and clients.
   * Secure discovery v2 is used.
   * Codebase Integrity and TLS / SSL Endpoints.

Why isn't it secure, what's vulnerable?

Well we know the sandbox isn't secure against DOS, but what about 
Serialization ObjectInputStream and using only local code?


Well that's not secure either.

Lets for a moment pretend that it is, what are the benefits?

We could use simple proxy services from a trusted lookup service, for 
example, without code downloads as trust is easily established.


We could define an interface for obtaining smart proxy's from bootstrap 
proxy's, register the bootstrap proxy with entries on a lookup service.


We can prevent unauthorised code downloads with DownloadPermission using 
the right PreferredClassProvider.


This would allow clients to obtain the boostrap proxy first, 
authenticate it, grant DownloadPermission to it, then use the smart proxy.


Anyway out of time right now, to be continued...

I'm presently investigating deserialization security and trying to fix 
another annoying River concurrency bug, these always seem to pop up when 
you're in the middle of something, taking days off the actual project.


Regards,

Peter.

Re: My apologies for file replacement with Netbeans and SVN commit.

[Peter Firmstone]

Thanks Gregg,

That looks like the problem.

Cheers,

Peter.



On 27/10/2014 12:37 PM, Gregg Wonderly wrote:

Most Likely, you used the default settings on netbeans editor configuration 
which I believe is to replace all tabs with spaces.  A sad default…

Gregg Wonderly


On Oct 26, 2014, at 8:27 AM, Peter Firmstonej...@zeus.net.au  wrote:

I've just noticed that my last svn commit, called using netbeans on windows, 
entire files were replaced, even when only very minor changes were made (on 
line), I'm not sure if this is something to do with Windows txt files or a 
setting somewhere.

This has occurred on at least one other previous occassion.

I'm going to stop developing on River until I resolve this issue or set up a 
Unix computer for development.

Regards,

Peter.

Fwd: Jenkins build is back to normal : river-ServiceDiscoveryManagerTests #142

[Peter Firmstone]

FYI.

Regards,

Peter.

 Original Message 
Subject: 	Jenkins build is back to normal : 
river-ServiceDiscoveryManagerTests #142

Date:   Mon, 27 Oct 2014 11:44:55 + (UTC)
From:   Apache Jenkins Server jenk...@builds.apache.org
To: j...@zeus.net.au



Seehttps://builds.apache.org/job/river-ServiceDiscoveryManagerTests/142/changes

Taking qa_refactor build for a spin.

[Peter Firmstone]

Have you checked out and built qa_refactor recently?

Did you know, apart from JMM compliance and fixes for finalizer attacks 
and race conditions:


  1. All hot spots are native methods.
  2. The performance cost of security is 0%
  3. Tests pass on these architectures; arm, sparc, x64
  4. Tests pass on these OS'; Windows, Linux, OSX, Solaris
  5. It builds and passes on other vendors JVM's, eg IBM's J9
  6. The next frontier for performance improvement is reducing network
 communication and or faster sockets.
  7. Patience is rewarded.

Please check it out, build and test it and report any issues.

Cheers,

Peter.

Hot Spots - Method,Self time [%],Self time,Self time (CPU),Samples
java.net.SocketInputStream.socketRead0[native](),39.177204,106888.146 
ms,106888.146 ms,23
java.net.DualStackPlainSocketImpl.accept0[native](),30.980734,84525.51 
ms,84525.51 ms,4
java.net.TwoStacksPlainDatagramSocketImpl.peekData[native](),14.191044,38717.779 
ms,38717.779 ms,2
sun.management.ThreadImpl.dumpThreads0[native](),9.112191,24861.019 
ms,24861.019 ms,20
sun.misc.Unsafe.unpark[native](),2.7181048,7415.873 ms,7415.873 
ms,42
java.net.TwoStacksPlainDatagramSocketImpl.socketNativeSetOption[native](),2.546774,6948.427 
ms,6948.427 ms,1
java.net.DualStackPlainSocketImpl.connect0[native](),0.80895424,2207.09 
ms,2207.09 ms,2
java.lang.Thread.isInterrupted[native](),0.21507628,586.798 
ms,586.798 ms,6
java.util.concurrent.ThreadPoolExecutor.runWorker(),0.055594184,151.679 
ms,151.679 ms,26
sun.misc.Unsafe.park[native](),0.0527888,587889.131 ms,144.025 
ms,94
sun.misc.Unsafe.compareAndSwapObject[native](),0.04196494,114.494 
ms,114.494 ms,1
sun.management.ThreadImpl.getThreadInfo1[native](),0.028033318,76.484 
ms,76.484 ms,1
java.lang.Thread.holdsLock[native](),0.026751213,72.986 ms,72.986 
ms,1
java.net.Inet6AddressImpl.lookupAllHostAddr[native](),0.01836988,50.119 
ms,50.119 ms,1
java.util.concurrent.FutureTask.set(),0.015322588,41.805 
ms,41.805 ms,43
au.net.zeus.collection.ReferenceProcessor$CleanerTask.run(),0.011088489,30.253 
ms,30.253 ms,2

java.util.concurrent.FutureTask.run(),0.0,0.0 ms,0.0 ms,84
java.util.concurrent.ThreadPoolExecutor.getTask(),0.0,0.0 ms,0.0 
ms,69

java.util.concurrent.locks.LockSupport.park(),0.0,0.0 ms,0.0 ms,65
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(),0.0,0.0 
ms,0.0 ms,63

java.lang.reflect.Method.invoke(),0.0,0.0 ms,0.0 ms,61
sun.reflect.DelegatingMethodAccessorImpl.invoke(),0.0,0.0 ms,0.0 
ms,61
java.util.concurrent.LinkedBlockingQueue.take(),0.0,0.0 ms,0.0 
ms,58

java.net.SocketInputStream.read(),0.0,0.0 ms,0.0 ms,46
java.util.concurrent.Executors$RunnableAdapter.call(),0.0,0.0 
ms,0.0 ms,44

java.lang.Thread.run(),0.0,0.0 ms,0.0 ms,43
java.util.concurrent.FutureTask.finishCompletion(),0.0,0.0 ms,0.0 
ms,42
java.util.concurrent.locks.LockSupport.unpark(),0.0,0.0 ms,0.0 
ms,42
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(),0.0,0.0 
ms,0.0 ms,40
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(),0.0,0.0 
ms,0.0 ms,40

sun.rmi.transport.Transport$1.run(),0.0,0.0 ms,0.0 ms,38
java.util.concurrent.locks.LockSupport.parkNanos(),0.0,0.0 ms,0.0 
ms,29
java.security.AccessController.doPrivileged[native](),0.0,0.0 
ms,0.0 ms,27
java.util.concurrent.ThreadPoolExecutor$Worker.run(),0.0,0.0 
ms,0.0 ms,26
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(),0.0,0.0 
ms,0.0 ms,23
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(),0.0,0.0 
ms,0.0 ms,20

javax.management.StandardMBean.invoke(),0.0,0.0 ms,0.0 ms,20
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(),0.0,0.0 
ms,0.0 ms,20

sun.reflect.GeneratedMethodAccessor7.invoke(),0.0,0.0 ms,0.0 ms,20
sun.management.ThreadImpl.dumpAllThreads(),0.0,0.0 ms,0.0 ms,20
javax.management.remote.rmi.RMIConnectionImpl.invoke(),0.0,0.0 
ms,0.0 ms,20
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(),0.0,0.0 
ms,0.0 ms,20
javax.management.remote.rmi.RMIConnectionImpl.doOperation(),0.0,0.0 
ms,0.0 ms,20

sun.reflect.misc.MethodUtil.invoke(),0.0,0.0 ms,0.0 ms,20
javax.management.remote.rmi.RMIConnectionImpl.access$300(),0.0,0.0 
ms,0.0 ms,20
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(),0.0,0.0 ms,0.0 
ms,20
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(),0.0,0.0 
ms,0.0 ms,20

com.sun.jmx.mbeanserver.MBeanSupport.invoke(),0.0,0.0 ms,0.0 ms,20
com.sun.jmx.mbeanserver.PerInterface.invoke(),0.0,0.0 ms,0.0 ms,20
sun.reflect.misc.Trampoline.invoke(),0.0,0.0 ms,0.0 ms,20
java.io.BufferedInputStream.fill(),0.0,0.0 ms,0.0 ms,20
java.io.BufferedInputStream.read(),0.0,0.0 ms,0.0 ms,20
java.io.FilterInputStream.read(),0.0,0.0 ms,0.0 ms,20
java.lang.Object.wait[native](),0.0,296503.639 ms,0.0 ms,19
sun.reflect.GeneratedMethodAccessor34.invoke(),0.0,0.0 ms,0.0 
ms,19

sun.rmi.transport.Transport.serviceCall(),0.0,0.0 ms,0.0 ms,19
sun.rmi.server.UnicastServerRef.dispatch(),0.0,0.0 ms,0.0 ms,19

Re: River in Linux Help Required

[Peter Firmstone]

Hi Amit,

The class not found belongs is Reggie's proxy.  Basically the proxy 
codebase is either not being downloaded from your http codebase server, 
or your local client isn't allowing it to be downloaded.


One recent change to Java was to change a property to prevent codebase 
downloads.


http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/enhancements-7.html

Try setting the property *|java.rmi.server.useCodebaseOnly|* to false.

Regards,

Peter.

On 27/10/2014 10:20 PM, amit batajoo wrote:

Dear Peter.

Thank you very much for your previous instruction.
Recently I am facing a problem in jini2_1, I am java 
version jdk1.7.0_60, when I run my jini services, I face the error msg 
java.lang.ClassNotFoundException: 
com.sun.jini.reggie.ConstrainableRegistrarProxy and could not run the 
services .


Please can you tell me what is the actual problem with jini services, 
its a java version problem or some other library missing in jini servies.


Advance Thank you for help.

Thank you!

Amit Batajoo

On Tue, Oct 21, 2014 at 6:19 AM, amit batajoo 
batajooseam...@gmail.com mailto:batajooseam...@gmail.com wrote:


Dear Peter,

Thank you very much for links. I hope this will work me a lots on
my Research study.

Thank you.


On Tue, Oct 21, 2014 at 4:25 AM, Peter j...@zeus.net.au
mailto:j...@zeus.net.au wrote:

There isn't a jini2.1.jar. jsk-platform.jar, jsk-lib.jar
provide the Jini programming api. jsk-policy.jar is necessary
for security.

http://jan.newmarch.name/java/jini/tutorial/Jini.html

Will dig up some more links for you later.


Regards,

Peter.


- Original message -
 Dear Peter,

 Thank you very much for your instruction and I have already
tried many
 keywords related to Jini for jini tutorial installation and
start its
 services to my Linux environment, but still I am stuck. Your
instruction
 are some what similar to my previous implementation of
failure in my
 system and I could not download the proper jini2.1.jar file
packages to
 start the jini services in my system so that I can make deep
study in
 Apache River architecture that will support my project.
 Please can you provide me some tutorials links and
jini2.1.jar or
 upgraded version jar file, I will also do my research to
find these all
 resources.


 Waiting for reply

 Thank you!


 On Mon, Oct 20, 2014 at 7:01 PM, Peter j...@zeus.net.au
mailto:j...@zeus.net.au wrote:

  Hi Amit,
 
  River is a Java application, install jsk-platform.jar and
jsk-lib.jar
  in a directory that's visible to your applications classpath.
 
  Service implementations are like server applications, for
each service
  you'll also need to install the service jar (eg
reggie.jar) and
  start.jar on your classpath. You'll also need a webserver,
one is
  provided in tools.jar called ClassServer (from memory). If
you can,
  install your webserver and services on separate computers
to your
  clients.
 
  Clients will download proxy files on demand from the web
server.
 
  Clients typically use multicast discovery to discover your
lookup
  service (reggie), then they use the lookup service to find
other
  services (eg Javaspaces).
 
  You may either install jsk-policy.jar in your java
extensions directory
  (for all applications) or install it in a directory and
specify that
  directory as a java extension directory, using a jvm
command line
  argument.
 
  Most books are now quite dated, but are still useful.
Google Jan
  Newmarch's Jini tutorial for tips on configuration and
getting started.
 
  Regards,
 
  Peter.
 
 
 
  - Original message -
   Dear Apache Developers
  
   My name is Amit Batajoo, Research student at Wakkanai
University,
   Wakkanai Hokdaido Japan.
   My study project is related with jini technology
distributed system.
   I
  am
   trying to install Apache River in Linux. However, I
could not find
  proper
   documentation about how to install River in Linux. Could
you please
   let me know if there is any thread regarding this issue.
  
   My installation infrastructure is Debian 7.4, 32 bits.
  
   Waiting for reply
  
   Thank you!
  
   --
   

Re: Taking qa_refactor build for a spin.

[Peter Firmstone]

All concurrency bugs identified (with testing, static analysis or 
review) that matter have been fixed, meaning I have no plans to fix 
ClassDep or build bugs, since the build will be updated to a modular 
build at some point.


It's probably safe to move to a beta or preview release now, however I 
haven't had the opportunity to run jtreg tests yet (I need a Unix / 
Linux test environment to do so) and I'm time poor.


I've picked up a dual CPU (24 core 128 thread) Victoria falls sparc 
server that I plan to use for jtreg and concurrency testing, but given 
my current workload, if anyone has time to help, by running the jtreg 
tests, creating a beta or preview release, that would be greatly 
appreciated.


Regards,

Peter.

On 27/10/2014 10:29 PM, Bryan Thompson wrote:

Sounds great!

What are the plans to move to a beta or preview release?

Bryan


Bryan Thompson
Chief Scientist  Founder
SYSTAP, LLC
4501 Tower Road
Greensboro, NC 27410
br...@systap.com
http://bigdata.com
http://mapgraph.io

CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP. Any unauthorized review, use, disclosure,
dissemination or copying of this email or its contents or attachments is
prohibited. If you have received this communication in error, please notify
the sender by reply email and permanently delete all copies of the email
and its contents and attachments.

On Mon, Oct 27, 2014 at 8:20 AM, Peter Firmstonej...@zeus.net.au  wrote:


Have you checked out and built qa_refactor recently?

Did you know, apart from JMM compliance and fixes for finalizer attacks
and race conditions:

   1. All hot spots are native methods.
   2. The performance cost of security is 0%
   3. Tests pass on these architectures; arm, sparc, x64
   4. Tests pass on these OS'; Windows, Linux, OSX, Solaris
   5. It builds and passes on other vendors JVM's, eg IBM's J9
   6. The next frontier for performance improvement is reducing network
  communication and or faster sockets.
   7. Patience is rewarded.

Please check it out, build and test it and report any issues.

Cheers,

Peter.

Hot Spots - Method,Self time [%],Self time,Self time
(CPU),Samples
java.net.SocketInputStream.socketRead0[native](),39.177204,106888.146
ms,106888.146 ms,23
java.net.DualStackPlainSocketImpl.accept0[native](),30.980734,84525.51
ms,84525.51 ms,4
java.net.TwoStacksPlainDatagramSocketImpl.peekData[native](),14.191044,38717.779
ms,38717.779 ms,2
sun.management.ThreadImpl.dumpThreads0[native](),9.112191,24861.019
ms,24861.019 ms,20
sun.misc.Unsafe.unpark[native](),2.7181048,7415.873 ms,7415.873
ms,42
java.net.TwoStacksPlainDatagramSocketImpl.socketNativeSetOption[
native](),2.546774,6948.427 ms,6948.427 ms,1
java.net.DualStackPlainSocketImpl.connect0[native](),0.80895424,2207.09
ms,2207.09 ms,2
java.lang.Thread.isInterrupted[native](),0.21507628,586.798
ms,586.798 ms,6
java.util.concurrent.ThreadPoolExecutor.runWorker(),0.055594184,151.679
ms,151.679 ms,26
sun.misc.Unsafe.park[native](),0.0527888,587889.131 ms,144.025
ms,94
sun.misc.Unsafe.compareAndSwapObject[native](),0.04196494,114.494
ms,114.494 ms,1
sun.management.ThreadImpl.getThreadInfo1[native](),0.028033318,76.484
ms,76.484 ms,1
java.lang.Thread.holdsLock[native](),0.026751213,72.986 ms,72.986
ms,1
java.net.Inet6AddressImpl.lookupAllHostAddr[native](),0.01836988,50.119
ms,50.119 ms,1
java.util.concurrent.FutureTask.set(),0.015322588,41.805 ms,41.805
ms,43
au.net.zeus.collection.ReferenceProcessor$CleanerTask.run(),0.011088489,30.253
ms,30.253 ms,2
java.util.concurrent.FutureTask.run(),0.0,0.0 ms,0.0 ms,84
java.util.concurrent.ThreadPoolExecutor.getTask(),0.0,0.0 ms,0.0
ms,69
java.util.concurrent.locks.LockSupport.park(),0.0,0.0 ms,0.0
ms,65
java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.await(),0.0,0.0 ms,0.0 ms,63
java.lang.reflect.Method.invoke(),0.0,0.0 ms,0.0 ms,61
sun.reflect.DelegatingMethodAccessorImpl.invoke(),0.0,0.0 ms,0.0
ms,61
java.util.concurrent.LinkedBlockingQueue.take(),0.0,0.0 ms,0.0
ms,58
java.net.SocketInputStream.read(),0.0,0.0 ms,0.0 ms,46
java.util.concurrent.Executors$RunnableAdapter.call(),0.0,0.0
ms,0.0 ms,44
java.lang.Thread.run(),0.0,0.0 ms,0.0 ms,43
java.util.concurrent.FutureTask.finishCompletion(),0.0,0.0 ms,0.0
ms,42
java.util.concurrent.locks.LockSupport.unpark(),0.0,0.0 ms,0.0
ms,42
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(),0.0,0.0
ms,0.0 ms,40
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(),0.0,0.0
ms,0.0 ms,40
sun.rmi.transport.Transport$1.run(),0.0,0.0 ms,0.0 ms,38
java.util.concurrent.locks.LockSupport.parkNanos(),0.0,0.0 ms,0.0
ms,29
java.security.AccessController.doPrivileged[native](),0.0,0.0
ms,0.0 ms,27
java.util.concurrent.ThreadPoolExecutor$Worker.run(),0.0,0.0
ms,0.0 ms,26
java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.awaitNanos(),0.0,0.0 ms,0.0 ms,23

My apologies for file replacement with Netbeans and SVN commit.

[Peter Firmstone]

I've just noticed that my last svn commit, called using netbeans on 
windows, entire files were replaced, even when only very minor changes 
were made (on line), I'm not sure if this is something to do with 
Windows txt files or a setting somewhere.


This has occurred on at least one other previous occassion.

I'm going to stop developing on River until I resolve this issue or set 
up a Unix computer for development.


Regards,

Peter.

Re: Build failed in Jenkins: river-ServiceDiscoveryManagerTests #82

[Peter Firmstone]

I've finally solved the random test failures in ServiceDiscoveryManager, 
this has taken considerable time, I'll be performing some more test runs 
before committing.


Regards,

Peter.

On 28/08/2014 9:29 PM, Apache Jenkins Server wrote:

Seehttps://builds.apache.org/job/river-ServiceDiscoveryManagerTests/82/

--
[...truncated 9292 lines...]
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/DefaultDiscoverPublic.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/AddListenerNPE.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/CacheDiscard.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/CacheLookup.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/CacheLookupFilterFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/CacheLookupFilterNoFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/cache/CacheLookupNoFilterFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/discovery/Locator.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/discovery/MulticastAnnouncement.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/discovery/MulticastRequest.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/discovery/Permission.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/event/AddAttrServiceChanged.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/event/ModifyAttrServiceChanged.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/event/SetAttrServiceChanged.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/Lookup.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMax.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMaxFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinEqualsMax.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinEqualsMaxFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinLessMax.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinLessMaxFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinMaxNoBlock.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupMinMaxNoBlockFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupWait.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupWaitFilter.td
  [java]Adding test: 
com/sun/jini/test/spec/servicediscovery/lookup/LookupWaitNoBlock.td
  [java]
  [java] -
  [java] GENERAL HARNESS CONFIGURATION INFORMATION:
  [java]
  [java]Date started:
  [java]   Thu Aug 28 10:22:28 UTC 2014
  [java]Installation directory of the JSK:
  [java]   
com.sun.jini.jsk.home=https://builds.apache.org/job/river-ServiceDiscoveryManagerTests/ws/trunk
  [java]Installation directory of the harness:
  [java]   
com.sun.jini.qa.home=https://builds.apache.org/job/river-ServiceDiscoveryManagerTests/ws/trunk/qa
  [java]Categories being tested:
  [java]   categories=servicediscovery
  [java] -
  [java] ENVIRONMENT PROPERTIES:
  [java]
  [java]JVM information:
  [java]   Java HotSpot(TM) Server VM, 23.25-b01, 32 bit VM mode
  [java]   Oracle Corporation
  [java]OS information:
  [java]   Linux, 3.13.0-32-generic, i386
  [java]
  [java] -
  [java] STARTING TO RUN THE TESTS
  [java]
  [java]
  [java] SUMMARY =
  [java]
  [java] com/sun/jini/test/impl/servicediscovery/TerminateSemantics.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/impl/servicediscovery/cache/CacheTerminateSemantics.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] com/sun/jini/test/impl/servicediscovery/event/AddListenerEvent.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/impl/servicediscovery/event/DiscardDownReDiscover.td
  [java] Test Failed: Test Failed: com.sun.jini.qa.harness.TestException: # 
added expected = 24, # added received = 23, # removed expected = 12, # removed 
received = 12

Re: Missing jar?

[Peter Firmstone]

Hi Patricia,

It's in the main build directory under rc-libs, it's also available on 
sourceforge, a project called custard apple.


In case you're wondering, it's a library that wraps the ability to cache 
using timed, weak or soft references with any java collection 
implementation.


Peter.

On 28/07/2014 8:28 AM, Patricia Shanahan wrote:
Several programs in qa_refactor import classes from 
au.net.zeus.collection. For example, 
com/sun/jini/start/AggregatePolicyProvider.java contains:


import au.net.zeus.collection.RC;
import au.net.zeus.collection.Ref;
import au.net.zeus.collection.Referrer;

Does this represent a jar I need to compile River? If so, where do I 
get it?


Thanks,

Patricia


j

Re: Jobs tied to Jenkins Slaves

[Peter Firmstone]

Thanks Gavin.

On 13/07/2014 6:02 PM, Gavin McDonald wrote:

Good Day Devs,

Our Jenkins Instance has some slaves being replaced.

Specifically those labelled 'ubuntu[1,2,4,5,6]' are being replaced with
slaves already online called 'ubuntu-[1,2,4,5,6] (notice the dash -).

You are getting this email as at least one of your builds has been identified 
as having
been tied to at least one of the slaves due to go off-line next week - likely 
Tuesday 15th
July.

Apologies for the short notice on this. Feel free to ping the builds@ mailing 
list or create
an INFRA jira ticket (component Jenkins) should you have any issues switching 
to a new
slave.

It is preferred that instead of tieing your build(s) to a particular slave, 
that you choose a more
generic label (for Ubuntu slaves choose 'ubuntu or Ubuntu' but if not please do 
make sure
you change your config to tie to one of the newer slaves instead.

If by the time we switch off the old slaves and nothing has been done, we will 
automatically
change your job configs to point to the generic 'ubuntu' label instead.

Thanks for Listening.

Gav...

Fwd: Re: JEP 187 Serializable 2.0

[Peter Firmstone]

Any ideas for Serializable 2.0?

Peter.

 Original Message 
Subject:Re: JEP 187 Serializable 2.0
Date:   Sat, 12 Jul 2014 08:52:33 -0400
From:   Brian Goetz brian.go...@oracle.com
To: Peter Firmstone peter.firmst...@zeus.net.au




 Where should I post the writeup?


corelibs-dev would be reasonable.


 I've only recently been aware of efforts for Serialzation 2.0.

 What other options are being considered for Serialization 2.0?


We're still largely in the gathering ideas of what can be done phase.

JEP 187

[Peter Firmstone]

 not use Portable if you don't intend to honor this contract, use
 * Serializable instead.
 * p
 * Caveat:br
 * Portable Objects cannot be stored directly in a 
 * {@link java.rmi.MarshalledObject}, a {@link net.jini.io.MarshalledInstance}
 * must first be created and converted, also a Portable Object will be
 * returned as a {@link PortableFactory} when {@link java.rmi.MarshalledObject}
 * is un-marshaled, a {@link java.rmi.MarshalledObject} must first be
 * converted to {@link net.jini.io.MarshalledInstance} before un-marshaling.
 * p
 * @author Peter Firmstone.
 * @since 3.0.0
 */
public interface Portable {

/**
 * Prepare for transport in a PortableObjectOutputStream. 
 * ObjectInput uses PortableFactory to create the Portable Object at the 
 * remote end using a constructor, static method or object method.
 * 
 * @return A PortableFactory, PortableObjectInputStream uses PortableFactory
 * to create a Portable Object at the remote end using a constructor,
 * static method or an object method.
 */
PortableFactory factory();
}
/*
 *  Licensed to the Apache Software Foundation (ASF) under one or more
 *  contributor license agreements.  See the NOTICE file distributed with
 *  this work for additional information regarding copyright ownership.
 *  The ASF licenses this file to You under the Apache License, Version 2.0
 *  (the License); you may not use this file except in compliance with
 *  the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an AS IS BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 */
package org.apache.river.api.io;

import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;
import java.io.Serializable;
import java.io.StreamCorruptedException;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.net.ProtocolException;
import java.security.AccessControlContext;
import java.security.AccessController;
import java.security.CodeSource;
import java.security.Guard;
import java.security.PrivilegedActionException;
import java.security.PrivilegedExceptionAction;
import java.security.ProtectionDomain;
import java.security.cert.Certificate;
import java.util.Arrays;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 * Portable form, required to create objects during de-serialization, 
 * using a constructor, static method or an Object method.
 * 
 * This object must be Thread confined, it is not thread safe.  It should be
 * created on demand, it is primarily for use by {@link 
PortableObjectInputStream}
 * and {@link PortableObjectOutputStream}, it is created by {@link Portable}
 * implementations.
 * 
 * Internal state is guarded, arrays are not defensively copied.
 * 
 * This is compatible with Version 2 of the Java Serialization Protocol.
 * 
 * @author Peter Firmstone.
 * @see Portable
 * @see PortablePermission
 * @see Serializable
 * @see Externalizable
 * @since 3.0.0
 */
public final class PortableFactory implements Externalizable {
private static final long serialVersionUID = 1L;
/* Guard private state */
private static final Guard distributable = new PortablePermission();

/* Minimal protocol to write primitives directly to stream, only for 
parameters.
 * Strings are written as Objects, since they are a special case, identical
 * strings are sent by reference to the first and not duplicated.
 * Class is also a special case, it too is not duplicated.
 * 
 * Primitives are written separately to objects, the first value written 
 * to ObjectOutput is the length of the parameter array, each parameter is
 * proceeded by a byte header indicating its type.  For null values, only 
the
 * byte header is written to stream.
 * 
 */
private static byte VERSION = 1;
private static final byte BOOLEAN = 0;
private static final byte BYTE = 1;
private static final byte CHAR = 2;
private static final byte SHORT = 3;
private static final byte INT = 4;
private static final byte LONG = 5;
private static final byte FLOAT = 6;
private static final byte DOUBLE = 7;
private static final byte OBJECT = 8;
private static final byte NULL = 9;

// Serial Form
private Object classOrObject;
private String method;
private Class [] parameterTypes;
private Object [] parameters;

// Private local object state.
private int hash;
private boolean constructed; // default value is false.

/**
 * Public method provided for java serialization framework.
 */
public PortableFactory(){
constructed

Re: Getting re-started

[Peter Firmstone]

Hi Patricia,

Don't forget to define a build.properties file:

# To change this template, choose Tools | Templates
# and open the template in the editor
java.home=C:/PROGRA~1/Java/jdk1.8.0
debug=true
#profile=compact3
#target=8
#java.util.logging.config.file=
#run.categories=renewalmanager
#run.categories=eventmailbox,start,start_impl
#run.categories=lookupservice
#run.categories=loader
#run.categories=joinmanager
#run.categories=lookupdiscovery,locatordiscovery
#,discoveryservice,eventmailbox,renewalservice,joinmanager
#run.categories=servicediscovery
#run.categories=servicediscovery,lookupservice,lookupservice_impl
#run.categories=stress
#run.categories=lookupdiscovery,locatordiscovery,discoverymanager,eventmailbox
#run.categories=start,start_impl
#run.categories=loader,loader_spec
#run.categories=security,policyprovider
#run.categories=lookupdiscovery,locatordiscovery,joinmanager,servicediscovery
#run.categories=lookupservice,eventmailbox
run.categories=renewalservice,loader,security,policyprovider
#run.categories=eventmailbox
#run.categories=discoveryservice
#run.categories=discoveryservice,discoveryservice_impl
#run.categories=joinmanager,renewalmanager
#run.categories=joinmanager
#run.categories=lookupservice,lookupdiscovery,locatordiscovery,discoveryservice,joinmanager,discoverymanager
#run.categories=lookupservice,javaspace,javaspace_impl_leasing,txnmanager,txnmanager_impl,lookupdiscovery,locatordiscovery,servicediscovery
#run.categories=jeri
#run.categories=lookupdiscovery,locatordiscovery
#run.categories=javaspace,javaspace_impl_leasing
#run.categories=javaspace
#run.categories=activation,start,start_impl
#run.categories=servicediscovery
#run.categories=locatordiscovery
#run.categories=txnmanager,txnmanager_impl
#run.tests=\
#com/sun/jini/test/impl/start/aggregatepolicyprovider/GetContextTest.td
#run.tests=com/sun/jini/test/impl/locatordiscovery/BadLocatorDiscoveryListener.td
#com/sun/jini/test/spec/lookupservice/test_set03/SimpleModifyLookupAttributes.td
#run.tests=com/sun/jini/test/spec/discoveryservice/event/MulticastMonitorStop.td,\
#com/sun/jini/test/impl/start/ServiceStarterCreateBadTransientServiceTest.td

You don't need cigwin to build on Windows with qa-refactor.






On 8/07/2014 7:50 AM, Peter wrote:

I use netbeans but setup properties will be similar.

Set the following as source directories:

./src
./qa/src
./qa/harness

then set up your compile classpath inclides for each source directory,, before 
doing this, I tend to build from the command line first, so all classes are 
present.

I'm on the road presently,, no computer at hand, will help some more soon.

,Peter.

- Original message -

I've forgotten the recipe for building River :-(

I would like to use Eclipse.

Patricia

On 7/6/2014 8:11 PM, Patricia Shanahan wrote:

On 7/6/2014 7:10 PM, Peter wrote:

I could use some help with qa-refactor,:)   esp checking new code in
org.apache.river, making sure new public api makes sense. Some might
need delaying till a later release (such as new stream based
registrar interfaces).

I'll get started on that.


I've been working towards getting it ready for pre release, there are
changes in trunk that need to be merged, I'm currently testing on
Java 8, there are a number of policy file updates I need to commit
when ready (later this week).

After the next release, I'd like to work on DNS-SRV discovery, with
IPv6 deployment increasing, we should be able to get River services
online,, securing Serialization would be a priority.

Then there's the modularisation effort started by Dennis Reedy.

Further down the track, i'd like to do something with dynamic code
generation for remote Lambda expressions, using ASM.

Peter.

- Original message -

I'm now in a position to return to active development. What
sub-project most needs some extra programming effort? What version
should I check out, compile etc.?

Patricia

Re: SerialReflectionFactory - got a better name?

[Peter Firmstone]

How about Portable?


On 1/07/2014 12:55 AM, Gregg Wonderly wrote:

So, maybe transportable or transported or forwarded…

Gregg

On Jun 29, 2014, at 4:41 AM, Peter Firmstonej...@zeus.net.au  wrote:


Hi Gregg,

Thinking out loud:

Transferable, I think it's close, it works for TransferableObjectFactory, it's 
created on demand to transfer a non serializable object via a serialization 
stream and it transfers a factory from one jvm to another, in order to recrate 
the original object in another jvm.

As for Distributed, transfer would imply the original object is moved from one 
place to another (deleted and recreated), while that could occur, it's also 
possible for the originating object to be duplicated as well, so it works for 
what's currently called SerialReflectionFactory, but Transferable wouldn't be a 
candidate for the Distributed interface.

If the original object has state, then it would be a snapshot of the original 
object at some point in time, hence memento.

It could be called TransferableMemento or SerializableMemento, it's created 
within a ObjectOutputStream and replaced in an ObjectInputStream.

Then that leaves Distributed, perhaps Distributable is better?  Other words are 
Propagatable, Disseminatable.  Latin: disseminare scattering seeds

interface Distributable {
SerializableMemento distribute();
}

Dissemino - dis (in all directions) + semino (I plant, I sow).

Cheers,

Peter.

On 29/06/2014 1:32 PM, Gregg Wonderly wrote:

TransferableObjectFactory?

Gregg

Sent from my iPhone


On Jun 23, 2014, at 7:14 AM, Stefano Marianis.mari...@unibo.it   wrote:


Il giorno 23/giu/2014, alle ore 13:24, Peter 
Firmstonej...@zeus.net.aumailto:j...@zeus.net.au   ha scritto:

recreate themselves remotely

Why not

  *   RemoteRecreationFactory
  *   DistributedCloningFactory
  *   or a combination of the above?

This way the name is after the goal of the class, not its implementation...


Stefano Mariani
PhD student @ DISI - Alma Mater Studiorum, Bologna

s.mari...@unibo.itmailto:s.mari...@unibo.it
stefanomariani.apice.unibo.ithttp://apice.unibo.it/xwiki/bin/view/StefanoMariani/

Re: SerialReflectionFactory - got a better name?

[Peter Firmstone]

Hi Gregg,

Thinking out loud:

Transferable, I think it's close, it works for 
TransferableObjectFactory, it's created on demand to transfer a non 
serializable object via a serialization stream and it transfers a 
factory from one jvm to another, in order to recrate the original object 
in another jvm.


As for Distributed, transfer would imply the original object is moved 
from one place to another (deleted and recreated), while that could 
occur, it's also possible for the originating object to be duplicated as 
well, so it works for what's currently called SerialReflectionFactory, 
but Transferable wouldn't be a candidate for the Distributed interface.


If the original object has state, then it would be a snapshot of the 
original object at some point in time, hence memento.


It could be called TransferableMemento or SerializableMemento, it's 
created within a ObjectOutputStream and replaced in an ObjectInputStream.


Then that leaves Distributed, perhaps Distributable is better?  Other 
words are Propagatable, Disseminatable.  Latin: disseminare scattering 
seeds


interface Distributable {
SerializableMemento distribute();
}

Dissemino - dis (in all directions) + semino (I plant, I sow).

Cheers,

Peter.

On 29/06/2014 1:32 PM, Gregg Wonderly wrote:

TransferableObjectFactory?

Gregg

Sent from my iPhone


On Jun 23, 2014, at 7:14 AM, Stefano Marianis.mari...@unibo.it  wrote:


Il giorno 23/giu/2014, alle ore 13:24, Peter 
Firmstonej...@zeus.net.aumailto:j...@zeus.net.au  ha scritto:

recreate themselves remotely

Why not

  *   RemoteRecreationFactory
  *   DistributedCloningFactory
  *   or a combination of the above?

This way the name is after the goal of the class, not its implementation...


Stefano Mariani
PhD student @ DISI - Alma Mater Studiorum, Bologna

s.mari...@unibo.itmailto:s.mari...@unibo.it
stefanomariani.apice.unibo.ithttp://apice.unibo.it/xwiki/bin/view/StefanoMariani/

LookupLocator - deprecation of Discovery V1

[Peter Firmstone]

Presently, LookupLocator's method getRegistrar, discovers a lookup 
service, using Discovery V1 only.


ConstrainableLookupLocator overrides the getRegistrar method and can 
perform Discovery V1 or V2.


As a first step, in ensuring maximum compatibility, I'd like to propose 
changing LookupLocators implementation of getRegistrar to be identical 
to ConstrainableLookupLocator, but with InvocationConstraints.EMPTY, 
that is, without any constraints.


Secondly, I'd like to propose that Discovery V2 is used by default, and 
Discovery V1 can only be used by setting a system property, for 
migration purposes only.


Doing so will allow LookupLocator to function using either Discovery V1 
or V2.


This also reduces code duplication, presently there are two 
implementations of client side Discovery V1, this will reduce that to 
one implementation only.


Regards,

Peter.

Re: [concurrency-interest] Deadlock

[Peter Firmstone]

Interesting coincidence, I'll go over the recent discussion to familiarise 
myself.

Thanks,

Peter. 

- Original message -
 [+openjdk mailing lists]
 
 
 Yeah, I'm trying to excise all the networking code from TLR and
 SplittableRandom.
 
 There is also likely to be a missing wrapping in doPrivileged.
 
 
 On Tue, Jun 24, 2014 at 7:21 AM, Peter Levart peter.lev...@gmail.com
 wrote:
 
  On 06/24/2014 04:12 PM, Peter Levart wrote:
  
   On 06/24/2014 01:09 PM, David Holmes wrote:
   
Hi Peter,

What a strange coincidence - the fact that the initialization of
ThreadLocalRandom can lead to arbitrary code execution has just
been a topic
of discussion, and it looks like your deadlock is related to that.

   
   Uf, this time it's a combination of a custom SecurityManager being in
   effect when NetworkInterface.getHardwareAddress() is called. It looks
   like we should not use any networking code at all for TLR's
   initialization.
   
  
  Or make NetworkInterface.getHardwareAddress() a @CallerSensitive method
  and avoid SecurityManager invocation when called from one of system
  classes. That's still an alternative.
  
  Peter
  
  
  
   Regards, Peter
   
   
David Holmes

-Original Message-
 From: concurrency-interest-boun...@cs.oswego.edu
 [mailto:concurrency-interest-boun...@cs.oswego.edu]On Behalf Of
 Peter Firmstone
 Sent: Tuesday, 24 June 2014 8:25 PM
 To: concurrency-inter...@cs.oswego.edu
 Subject: [concurrency-interest] Deadlock
 
 
 This appears to be a ClassLoader deadlock in Java 7.
 
 The stack trace from the main thread is missing.
 
 Any ideas?
 
 Regards,
 
 Peter.
 
 Attaching to process ID 7124, please wait...
 Debugger attached successfully.
 Client compiler detected.
 JVM version is 25.0-b70
 Deadlock Detection:
 
 Found one Java-level deadlock:
 =
 
 main:
 waiting to lock Monitor@0x0094bb2c (Object@0x03d73c38, a
 java/lang/Object),
 which is held by Thread-1
 Thread-1:
 waiting to lock Monitor@0x0094c99c (Object@0x03f02e50, a [I),
 which is held by main
 
 Found a total of 1 deadlock.
 
 Thread 8: (state = BLOCKED)
 -
 au.net.zeus.collection.ReferenceFactory.create(java.lang.Object,
 au.net.zeus.collection.RefQueue, au.net.zeus.collection.Ref)
 @bci=229, line=60 (Interpreted frame) -
 au.net.zeus.collection.ReferenceProcessor.referenced(java.lang.Object,
 boolean, boolean) @bci=37, line=128 (Interpreted frame)
 -
 au.net.zeus.collection.ReferenceProcessor.referenced(java.lang.Object,
 boolean, boolean) @bci=4, line=44 (Interpreted frame)
 - au.net.zeus.collection.ReferenceMap.wrapKey(java.lang.Object,
 boolean, boolean) @bci=7, line=248 (Interpreted frame)
 -
 au.net.zeus.collection.ReferenceConcurrentMap.putIfAbsent(java.lan
 g.Object,
 java.lang.Object) @bci=8, line=67 (Interpreted frame)
 -
 org.apache.river.api.security.CombinerSecurityManager.checkPermiss
 ion(java.security.Permission,
 java.lang.Object) @bci=161, line=260 (Interpreted frame)
 -
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTr
 ustPermission(java.lang.Class)
 @bci=59, line=1848 (Interpreted frame)
 -
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBea
 n(java.lang.Object,
 javax.management.ObjectName) @bci=25, line=322 (Interpreted
 frame) - com.sun.jmx.mbeanserver.JmxMBeanServer$2.run() @bci=17,
 line=1225 (Interpreted frame)
 -
 java.security.AccessController.doPrivileged(java.security.Privileg
 edExceptionAction)
 @bci=0 (Interpreted frame)
 - com.sun.jmx.mbeanserver.JmxMBeanServer.initialize() @bci=25,
 line=1223 (Interpreted frame)
 - com.sun.jmx.mbeanserver.JmxMBeanServer.init(java.lang.String,
 javax.management.MBeanServer,
 javax.management.MBeanServerDelegate,
 com.sun.jmx.mbeanserver.MBeanInstantiator, boolean, boolean)
 @bci=133, line=255 (Interpreted frame) -
 com.sun.jmx.mbeanserver.JmxMBeanServer.newMBeanServer(java.lang.String,
 javax.management.MBeanServer,
 javax.management.MBeanServerDelegate, boolean) @bci=13,
 line=1437 (Interpreted frame) -
 javax.management.MBeanServerBuilder.newMBeanServer(java.lang.
 String, javax.management.MBeanServer,
 javax.management.MBeanServerDelegate) @bci=4, line=110
 (Interpreted frame) -
 javax.management.MBeanServerFactory.newMBeanServer(java.lang.
 String) @bci=36, line=329 (Interpreted frame)
 -
 javax.management.MBeanServerFactory.createMBeanServer(java.lang.String)
 @bci=6, line=231 (Interpreted frame)
 - javax.management.MBeanServerFactory.createMBeanServer() @bci=1,
 line=192 (Interpreted frame)
 - java.lang.management.ManagementFactory.getPlatformMBeanServer

SerialReflectionFactory - got a better name?

[Peter Firmstone]

Distributed object use SerialReflectionFactory to recreate themselves 
remotely using one of their public constructors, a static factory method 
or builder object, however one thing about SerialReflectionFactory 
bothers me.


SerialReflectionFactory is named after it's implementation, that is, it 
uses reflection.  At some point in time, we might decide we want to use 
something other than reflection, in that case, the name would be 
inappropriate.


How does DistributedObjectFactory sound?

Has anyone got a better name?

Regards,

Peter.

SerialReflectionFactory - got a better name?

[Peter Firmstone]

Distributed object use SerialReflectionFactory to recreate themselves 
remotely using one of their public constructors, a static factory method 
or builder object, however one thing about SerialReflectionFactory 
bothers me.


SerialReflectionFactory is named after it's implementation, that is, it 
uses reflection.  At some point in time, we might decide we want to use 
something other than reflection, in that case, the name would be 
inappropriate.


How does DistributedObjectFactory sound?

Has anyone got a better name?

Regards,

Peter.

email test results

[Peter Firmstone]

Due to the number of test results being emailed to the list, I've now 
diverted them to my own email address, any interesting failures will be 
relayed to dev@river.apache.org manually from now on.


Regards,

Peter.

Re: Investigating dynamic Lambda's for services

[Peter Firmstone]

In a word, no because the classes use standard java bytecode and methods.

Standard serializable lambda's are brittle, any change in the 
encapsulating class can cause breakage during deserialization.


What I'm looking at doing for lambda's that don't refer to their 
enclosing classes object methods, is removing this dependency and 
recreating a standard object, with a single static method, one object 
method and fields containing captured arguments.  Presently an 
encapsulating class will be serialized as well, with a lot of 
unnecessary code that may pull in other dependencies, so I'm looking at 
removing that.


SerializedLambda provides the names of the required methods, so by 
intercepting an ObjectOutputStream, before it's serialized and replacing 
the SerializedLambda in that stream (provided it implements our marker 
interface, that will extend Serializable) we can make lambda's less 
susceptible to breakage.


The point where breakage could occur is during serialization at runtime, 
if the lambda calls object methods, deserialization will be problem free.


Regards,

Peter.

On 28/05/2014 10:33 PM, Bryan Thompson wrote:

How stable is the lamba translation?  If this changes or is different for
different JVMs, will that break things?
Bryan


On Wed, May 28, 2014 at 8:29 AM, Peterj...@zeus.net.au  wrote:


So far, a subset of Lambda's appear very suitable for marshalling
remotely, where their code can execute in a remote jvm.

A lambda, that doesn't refer to object methods in its enclosing class,
results in the java compiller creating a static method that accepts
captured object arguments, containing the lambda's block code.

With ASM, I can very easily read a class and strip out all but this static
method.  A non instantiable class file with no class init.

This class can be given a temporary known name, unrelated to the original
client code (not intended for class loading, but identity).

A named identity of this class can then be created using a message digest
of the class bytes.

The class byte code can be packaged and serialized remotely.

Upon deserialization, the class can be rewritten again (if not loaded
previously) and named after its message digest, and given a new object
method to implement the functional interface, final fields added to hold
serialized captured arguments, and a constructor that accepts the captured
arguments.

It is now a simple basic object that calls a static method with arguments.

The class file would be verified, then loaded into a ClassLoader with no
permissions.

The class would be cached in a Map with weak values, the key being the
message digest, so idential deserialized lambda's wouldn't need to rewrite
and load their class bytecode.

These lambda's could be used in services to allow remote internal
iteration, over large data collections, like distributed hash tables, to
process and return results.

An argument captured by a lambda, could itself be a remote service, used
as a callback.  Lambda's could be stored by services, and used like queries
to notify remote listeners of changes in service state.

A bit like an Event notification version of Map Reduce.

It seems reasonable that distributed lambda's should provide a subset of
Java lambda expressions based on static compilled methods and that all
object arguments that lambda's capture should be or implement public api or
service api.

Thoughts?

Peter.

Playing around with ASMifier

[Peter Firmstone]

Take one very simple class, compile it and process it with ASMifier and 
guess what?


Classes that implement functional interfaces are compiled to contain 
synthetic bridge methods that check Generic type casts!


In other news I'm still waiting for a Jenkins run to complete without 
someone aborting it early, I've asked on infrastructure if they can 
limit who can abort running tests to those who created them or Jenkins 
administrators.


public class ConsumerTest1 implements ConsumerString {

@Override
public void accept(String t) {
System.out.println(t);
}

}

package asm.testing;
import java.util.*;
import org.objectweb.asm.*;
import org.objectweb.asm.attrs.*;
public class ConsumerTest1Dump implements Opcodes {

public static byte[] dump () throws Exception {

ClassWriter cw = new ClassWriter(0);
FieldVisitor fv;
MethodVisitor mv;
AnnotationVisitor av0;

cw.visit(52, ACC_PUBLIC + ACC_SUPER, testing/ConsumerTest1, 
Ljava/lang/Object;Ljava/util/function/ConsumerLjava/lang
/String;;, java/lang/Object, new String[] { 
java/util/function/Consumer });


{
mv = cw.visitMethod(ACC_PUBLIC, init, ()V, null, null);
mv.visitCode();
mv.visitVarInsn(ALOAD, 0);
mv.visitMethodInsn(INVOKESPECIAL, java/lang/Object, init, ()V, 
false);

mv.visitInsn(RETURN);
mv.visitMaxs(1, 1);
mv.visitEnd();
}
{
mv = cw.visitMethod(ACC_PUBLIC, accept, (Ljava/lang/String;)V, null, 
null);

mv.visitCode();
mv.visitFieldInsn(GETSTATIC, java/lang/System, out, 
Ljava/io/PrintStream;);

mv.visitVarInsn(ALOAD, 1);
mv.visitMethodInsn(INVOKEVIRTUAL, java/io/PrintStream, println, 
(Ljava/lang/String;)V, false);

mv.visitInsn(RETURN);
mv.visitMaxs(2, 2);
mv.visitEnd();
}
{
mv = cw.visitMethod(ACC_PUBLIC + ACC_BRIDGE + ACC_SYNTHETIC, accept, 
(Ljava/lang/Object;)V, null, null);

mv.visitCode();
mv.visitVarInsn(ALOAD, 0);
mv.visitVarInsn(ALOAD, 1);
mv.visitTypeInsn(CHECKCAST, java/lang/String);
mv.visitMethodInsn(INVOKEVIRTUAL, testing/ConsumerTest1, accept, 
(Ljava/lang/String;)V, false);

mv.visitInsn(RETURN);
mv.visitMaxs(2, 2);
mv.visitEnd();
}
cw.visitEnd();

return cw.toByteArray();
}
}

Re: Dependency on Sun internal API's

[Peter Firmstone]

On 25/05/2014 11:07 AM, Greg Trasuk wrote:

On May 24, 2014, at 8:42 PM, Peter Firmstonej...@zeus.net.au  wrote:


On 23/05/2014 9:53 PM, Simon IJskes - QCG wrote:

Yes, if possible we could sync up the trunk, by visual diffing trunk and 
qa_refactor, merging patches into trunk, commiting to trunk and release in 
small controlled steps. Netbeans is capable of doing that.

Gr. Simon



The original concern raised, was significant changes between the 2.2 branch and 
trunk.   Qa_refactor is branched off trunk although trunk has a couple of 
commits after the branch point, so it'd probably be easier to apply those 
changes to qa_refactor, rather than the other way around.

However I suspect what you meant was to apply the changes in qa_refactor to the 
2.2 branch?


I thought the plan was to merge qa_refactor over to trunk, and leave the 2.2. 
branch as-is.

Greg.



Yes I believe it is, but we already have an svn commit history with 
comments that explain changes from a recent trunk snapshot, we'd loose 
those comments, since trunk has been relatively static, it would be 
easier to merge trunk into qa_refactor and replace trunk.


My understanding is that we'd release qa_refactor and make incremental 
changes to it in multiple releases until it stabilised.


Trunk was branched because it became unstable.

Although qa refactor is quite stable, looking at Jenkins you could be 
forgiven for thinking otherwise, it's worth remembering the current 
snapshot hasn't run on Jenkins.  I have enabled a test that failed (as 
documented) during the Jini 2.1 release, that is now stable.


ServiceDiscoveryManager still has a race that presents occassionally, 
but I don't think its a reason not to release.


IIOP isn't running on IBM J9, it hasn't had the opportunity for a full 
test run, but results so far are better than expected.


For now I'm waiting for Jenkins to run the current snapshot on all 
architectures, probably a couple of weeks wait for the outcome.


Regards,

Peter.

Re: Dependency on Sun internal API's

[Peter Firmstone]

Jini has a small but loyal user base in financial services.

Looks like River is building on J9, real time java and IIOP seems to be 
working too.


I'm not expecting many tests to pass at this stage, since many 
permissions will be different, at least it's all compilling now.


Cheers,

Peter.

On 22/05/2014 6:53 PM, Dawid Loubser wrote:
Wow Peter, that sounds great! I imagine that a significant percentage 
of the (rather small) River user base might operate in environments 
requiring some of these other Java VMs with particular qualities 
around real-time processing, etc.


Dawid


On 22/05/2014 06:29, Peter Firmstone wrote:
Presently we are prevented from compilling and running on J9, JRockit 
or other Java VM's.


I've been able to modify Phoenix to use reflection at runtime to call 
Sun private implementations, meaning that Phoenix is strictly a Sun 
JVM only component, but would no longer prevent compilling and 
testing on other architectures.


There is one test that, 
com.sun.jini.test.impl.reggie.MultiHomedClientTest that uses the 
following internal sun API:


sun.net.spi.nameservice.NameService;
sun.net.spi.nameservice.NameServiceDescriptor;

If I was to change this test and associated classes to .txt 
extensions, we could run these manually on Sun's JVM, while allowing 
River to build on other architectures.


Regards,

Peter.

Re: Dependency on Sun internal API's

[Peter Firmstone]

   [java] -
 [java] GENERAL HARNESS CONFIGURATION INFORMATION:
 [java]
 [java]Date started:
 [java]   Thu May 22 11:23:39 UTC 2014
 [java]Installation directory of the JSK:
 [java]   
com.sun.jini.jsk.home=/home/hudson/jenkins-slave/workspace/river-qa-refactor-j9/trunk
 [java]Installation directory of the harness:
 [java]   
com.sun.jini.qa.home=/home/hudson/jenkins-slave/workspace/river-qa-refactor-j9/trunk/qa
 [java]Categories being tested:
 [java]   categories=id loader policyprovider locatordiscovery 
activation config discoverymanager joinmanager url iiop jrmp reliability thread 
renewalmanager constraint export lookupdiscovery servicediscovery io security 
lookupservice renewalservice eventmailbox jeri start discoveryservice 
discoveryproviders javaspace txnmanager
 [java] -
 [java] ENVIRONMENT PROPERTIES:
 [java]
 [java]JVM information:
 [java]   IBM J9 VM, 2.4, 64 bit VM mode
 [java]   IBM Corporation
 [java]OS information:
 [java]   Linux, 3.2.0-51-generic, amd64
 [java]
 [java] -
 [java] STARTING TO RUN THE TESTS
 [java]
 [java]



On 22/05/2014 9:10 PM, Peter Firmstone wrote:

Jini has a small but loyal user base in financial services.

Looks like River is building on J9, real time java and IIOP seems to 
be working too.


I'm not expecting many tests to pass at this stage, since many 
permissions will be different, at least it's all compilling now.


Cheers,

Peter.

On 22/05/2014 6:53 PM, Dawid Loubser wrote:
Wow Peter, that sounds great! I imagine that a significant percentage 
of the (rather small) River user base might operate in environments 
requiring some of these other Java VMs with particular qualities 
around real-time processing, etc.


Dawid


On 22/05/2014 06:29, Peter Firmstone wrote:
Presently we are prevented from compilling and running on J9, 
JRockit or other Java VM's.


I've been able to modify Phoenix to use reflection at runtime to 
call Sun private implementations, meaning that Phoenix is strictly a 
Sun JVM only component, but would no longer prevent compilling and 
testing on other architectures.


There is one test that, 
com.sun.jini.test.impl.reggie.MultiHomedClientTest that uses the 
following internal sun API:


sun.net.spi.nameservice.NameService;
sun.net.spi.nameservice.NameServiceDescriptor;

If I was to change this test and associated classes to .txt 
extensions, we could run these manually on Sun's JVM, while allowing 
River to build on other architectures.


Regards,

Peter.

Re: Dependency on Sun internal API's

[Peter Firmstone]

The build will be ready for release when test failures are low, once we 
reach that milestone, we can perform rapid iterative releases.


The recent fix to JERI appears to have increased stability, at least 
locally, although this isn't showing through on Jenkins yet.


You could say I took some time out, to do something easy, at least 
compared to the hair pulling concurrency bugs and race conditions I've 
been focused on.


Cheers,

Peter.

On 22/05/2014 9:31 PM, Bryan Thompson wrote:

This is all good, but I would personally be interested in getting to a
release based on the QA branch with less remaining development.  I would
suggest rapid iterations for releases with incremental change until a QA
branch based release is stable.


On Thu, May 22, 2014 at 7:24 AM, Peter Firmstonej...@zeus.net.au  wrote:


[java] -
  [java] GENERAL HARNESS CONFIGURATION INFORMATION:
  [java]
  [java]Date started:
  [java]   Thu May 22 11:23:39 UTC 2014
  [java]Installation directory of the JSK:
  [java]   com.sun.jini.jsk.home=/home/hudson/jenkins-slave/
workspace/river-qa-refactor-j9/trunk
  [java]Installation directory of the harness:
  [java]   com.sun.jini.qa.home=/home/hudson/jenkins-slave/
workspace/river-qa-refactor-j9/trunk/qa
  [java]Categories being tested:
  [java]   categories=id loader policyprovider locatordiscovery
activation config discoverymanager joinmanager url iiop jrmp reliability
thread renewalmanager constraint export lookupdiscovery servicediscovery io
security lookupservice renewalservice eventmailbox jeri start
discoveryservice discoveryproviders javaspace txnmanager
  [java] -
  [java] ENVIRONMENT PROPERTIES:
  [java]
  [java]JVM information:
  [java]   IBM J9 VM, 2.4, 64 bit VM mode
  [java]   IBM Corporation
  [java]OS information:
  [java]   Linux, 3.2.0-51-generic, amd64
  [java]
  [java] -
  [java] STARTING TO RUN THE TESTS
  [java]
  [java]




On 22/05/2014 9:10 PM, Peter Firmstone wrote:


Jini has a small but loyal user base in financial services.

Looks like River is building on J9, real time java and IIOP seems to be
working too.

I'm not expecting many tests to pass at this stage, since many
permissions will be different, at least it's all compilling now.

Cheers,

Peter.

On 22/05/2014 6:53 PM, Dawid Loubser wrote:


Wow Peter, that sounds great! I imagine that a significant percentage of
the (rather small) River user base might operate in environments requiring
some of these other Java VMs with particular qualities around real-time
processing, etc.

Dawid


On 22/05/2014 06:29, Peter Firmstone wrote:


Presently we are prevented from compilling and running on J9, JRockit
or other Java VM's.

I've been able to modify Phoenix to use reflection at runtime to call
Sun private implementations, meaning that Phoenix is strictly a Sun JVM
only component, but would no longer prevent compilling and testing on other
architectures.

There is one test that, com.sun.jini.test.impl.reggie.MultiHomedClientTest
that uses the following internal sun API:

sun.net.spi.nameservice.NameService;
sun.net.spi.nameservice.NameServiceDescriptor;

If I was to change this test and associated classes to .txt extensions,
we could run these manually on Sun's JVM, while allowing River to build on
other architectures.

Regards,

Peter.

Re: Dependency on Sun internal API's

[Peter Firmstone]

Well the news is someone aborted the build, this is quite common with 
River builds, I don't know whether it's because people think the build 
is hung, because it goes for so long or they need to shutdown a node for 
maintenance, who knows.


If people can perform test runs on their own hardware, it would 
definitely helpful.  Jenkins seems more suited to short test runs.


Regards,

Peter.

On 22/05/2014 9:24 PM, Peter Firmstone wrote:

   [java] -
 [java] GENERAL HARNESS CONFIGURATION INFORMATION:
 [java]
 [java]Date started:
 [java]   Thu May 22 11:23:39 UTC 2014
 [java]Installation directory of the JSK:
 [java]   
com.sun.jini.jsk.home=/home/hudson/jenkins-slave/workspace/river-qa-refactor-j9/trunk

 [java]Installation directory of the harness:
 [java]   
com.sun.jini.qa.home=/home/hudson/jenkins-slave/workspace/river-qa-refactor-j9/trunk/qa

 [java]Categories being tested:
 [java]   categories=id loader policyprovider locatordiscovery 
activation config discoverymanager joinmanager url iiop jrmp 
reliability thread renewalmanager constraint export lookupdiscovery 
servicediscovery io security lookupservice renewalservice eventmailbox 
jeri start discoveryservice discoveryproviders javaspace txnmanager

 [java] -
 [java] ENVIRONMENT PROPERTIES:
 [java]
 [java]JVM information:
 [java]   IBM J9 VM, 2.4, 64 bit VM mode
 [java]   IBM Corporation
 [java]OS information:
 [java]   Linux, 3.2.0-51-generic, amd64
 [java]
 [java] -
 [java] STARTING TO RUN THE TESTS
 [java]
 [java]



On 22/05/2014 9:10 PM, Peter Firmstone wrote:

Jini has a small but loyal user base in financial services.

Looks like River is building on J9, real time java and IIOP seems to 
be working too.


I'm not expecting many tests to pass at this stage, since many 
permissions will be different, at least it's all compilling now.


Cheers,

Peter.

On 22/05/2014 6:53 PM, Dawid Loubser wrote:
Wow Peter, that sounds great! I imagine that a significant 
percentage of the (rather small) River user base might operate in 
environments requiring some of these other Java VMs with particular 
qualities around real-time processing, etc.


Dawid


On 22/05/2014 06:29, Peter Firmstone wrote:
Presently we are prevented from compilling and running on J9, 
JRockit or other Java VM's.


I've been able to modify Phoenix to use reflection at runtime to 
call Sun private implementations, meaning that Phoenix is strictly 
a Sun JVM only component, but would no longer prevent compilling 
and testing on other architectures.


There is one test that, 
com.sun.jini.test.impl.reggie.MultiHomedClientTest that uses the 
following internal sun API:


sun.net.spi.nameservice.NameService;
sun.net.spi.nameservice.NameServiceDescriptor;

If I was to change this test and associated classes to .txt 
extensions, we could run these manually on Sun's JVM, while 
allowing River to build on other architectures.


Regards,

Peter.

Remaining test failures - suspect concurrency / race condition

[Peter Firmstone]

I've seen this test fail on Java 8 on Windows 32 bit and Java 7 on Linux 
(Jenkins).


This test relates to ServiceDiscoveryManager.

The test doesn't fail consistently.

Running com/sun/jini/test/impl/servicediscovery/event/ReRegisterBadEquals.td
Time is Wed Feb 26 14:39:57 UTC 2014
Starting test in separate process with command:
/home/jenkins/tools/java/jdk1.7.0_25-32/jre/bin/java 
-Djava.security.manager=org.apache.river.api.security.CombinerSecurityManager 
-Djava.security.policy=file:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/harness/policy/defaulttest.policy
 -Djava.rmi.server.codebase=http://minerva:9082/qa1-servicediscovery-dl.jar -cp 
/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/jiniharness.jar:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/jinitests.jar:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib/jsk-platform.jar:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib/jsk-lib.jar:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib/high-scale-lib.jar:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib/custard-apple-1.0.3.jar
 -ea -esa 
-Djava.ext.dirs=/home/jenkins/tools/java/jdk1.7.0_25-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib-ext
 -Dcom.sun.jini.jsk.port=9080 -Dcom.sun.jini.qa.port=9081 
-Dcom.sun.jini.jsk.home=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk
 
-Dcom.sun.jini.qa.home=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa
 
-Dcom.sun.jini.qa.harness.harnessJar=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/jiniharness.jar
 
-Dcom.sun.jini.qa.harness.testJar=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/jinitests.jar
 -Dcom.sun.jini.qa.harness.runjiniserver=true 
-Dcom.sun.jini.qa.harness.runkitserver=true 
-Djava.security.properties=file:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/harness/trust/dynamic-policy.properties
 -Dcom.sun.jini.qa.harness.testhosts= 
-Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/src/com/sun/jini/test/resources/qa1.logging
 -Djava.rmi.server.useCodebaseOnly=false 
-Dnet.jini.core.lookup.ServiceRegistrar.portAbitraryIfInUse=true 
-Dcom.sun.jini.test.home=/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa
 -Dcom.sun.jini.test.port=9082 
-Dcom.sun.jini.qa.harness.policies=file:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/src/com/sun/jini/test/resources/jinitest.policy
 
-Djava.ext.dirs=/home/jenkins/tools/java/jdk1.7.0_25-32/jre/lib/ext:/usr/java/packages/lib/ext:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib-ext:/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib-ext
 com.sun.jini.qa.harness.MasterTest 
com/sun/jini/test/impl/servicediscovery/event/ReRegisterBadEquals.td

TIME: 2:39:57 PM

MasterTest.doTest INFO:
== CALLING CONSTRUCT() 
==

Feb 26, 2014 2:39:58 PM com.sun.jini.tool.ClassServer run
INFO: ClassServer started 
[[/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/],
 port 9081]
Feb 26, 2014 2:39:58 PM com.sun.jini.tool.ClassServer run
INFO: ClassServer started 
[[/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/lib-dl/],
 port 9080]
Feb 26, 2014 2:39:58 PM com.sun.jini.tool.ClassServer run
INFO: ClassServer started 
[[/home/jenkins/jenkins-slave/workspace/river-qa-refactoring-jdk7/trunk/qa/lib/],
 port 9082]
MasterTest.doTest INFO:
=== CALLING RUN() ===

com.sun.jini.qa.harness.TestException:  -- failure -- nAdded = 3, 
nAddedExpected = 4, nRemoved = 1, nRemovedExpected = 2
at 
com.sun.jini.test.impl.servicediscovery.event.ReRegisterGoodEquals.applyTestDef(ReRegisterGoodEquals.java:131)
at 
com.sun.jini.test.spec.servicediscovery.AbstractBaseTest.run(AbstractBaseTest.java:552)
at com.sun.jini.qa.harness.MasterTest.doTest(MasterTest.java:256)
at com.sun.jini.qa.harness.MasterTest.main(MasterTest.java:144)

TIME: 2:41:11 PM

MasterTest.doTest INFO:
 CALLING TEARDOWN() =

Feb 26, 2014 2:41:21 PM com.sun.jini.tool.ClassServer terminate
INFO: ClassServer terminated [port 9082]
Feb 26, 2014 2:41:21 PM com.sun.jini.tool.ClassServer terminate
INFO: ClassServer terminated [port 9080]
Feb 26, 2014 2:41:21 PM com.sun.jini.tool.ClassServer terminate
INFO: ClassServer terminated [port 9082]
Feb 26, 2014 2:41:21 PM com.sun.jini.tool.ClassServer terminate
INFO: ClassServer terminated [port 9081]
Feb 

Re: Jini2.1 in Windows 8 and Java 1.8

[Peter Firmstone]

Thanks Bishnu,

Can you provide a patch?  I'll submit it to svn.

Our website could use some jazzing up too if you've got some cycles to 
spare?


http://svn.apache.org/viewvc/river/site

Peter.


On 20/05/2014 9:37 PM, Bishnu Gautam wrote:

HI All
I think there are still some old Jini fans who might like to play with jini and 
River. After a few while, I got back to play with Jini, however, it seems that 
the installer no more works in Windows 8. So, I got some trick to make it run 
in Windows 8. I would like to contribute this tutorial to River project. So, if 
you want to put it in the website, please let me know. Anyway, I am providing 
the link for temporary purpose in the following place.
URL:  http://wakhok.ac.jp/~gautam/jyaguchiProject/Jini-in-Windows-8.pdf
I think Jini/River is a great resource to the people who want to develop some 
distributed/cloud based application.
RegardsBishnu

Re: Jini2.1 in Windows 8 and Java 1.8

[Peter Firmstone]

Actually... I read the article and I'm interested in your upcoming River 
tutorial ;)


On 21/05/2014 7:38 PM, Bishnu Gautam wrote:

Hi Peter
Jini installer seems using an older version of LAX installer that is 7.0. It 
seems too old and I do not have patch for this installer. However, I have all 
required files of Jini2.1 as descrived below.
I have the extracted files of jini 2.1 in the following 
links:http://www.wakhok.ac.jp/~gautam/jyaguchiProject/jini2_1_all_files.rar
And also the steps of running Jini in the following pdf 
file.http://wakhok.ac.jp/~gautam/jyaguchiProject/Jini-in-Windows-8.pdf
You can put the files and also the tutorial of running Jini in River site. 
Certainly, I could share my time to upload my current/future works in River, 
for that could you provide me SVN login info of River SVN repository. Or let me 
know the way if I can edit above tutorial in River site by myself.
RegardsBishnu



Bishnu Prasad Gautam



Date: Wed, 21 May 2014 17:49:23 +1000
From: j...@zeus.net.au
To: dev@river.apache.org
Subject: Re: Jini2.1 in Windows 8 and Java 1.8

Thanks Bishnu,

Can you provide a patch?  I'll submit it to svn.

Our website could use some jazzing up too if you've got some cycles to
spare?

http://svn.apache.org/viewvc/river/site

Peter.


On 20/05/2014 9:37 PM, Bishnu Gautam wrote:

HI All
I think there are still some old Jini fans who might like to play with jini and 
River. After a few while, I got back to play with Jini, however, it seems that 
the installer no more works in Windows 8. So, I got some trick to make it run 
in Windows 8. I would like to contribute this tutorial to River project. So, if 
you want to put it in the website, please let me know. Anyway, I am providing 
the link for temporary purpose in the following place.
URL:  http://wakhok.ac.jp/~gautam/jyaguchiProject/Jini-in-Windows-8.pdf
I think Jini/River is a great resource to the people who want to develop some 
distributed/cloud based application.
RegardsBishnu

Dependency on Sun internal API's

[Peter Firmstone]

Presently we are prevented from compilling and running on J9, JRockit or 
other Java VM's.


I've been able to modify Phoenix to use reflection at runtime to call 
Sun private implementations, meaning that Phoenix is strictly a Sun JVM 
only component, but would no longer prevent compilling and testing on 
other architectures.


There is one test that, 
com.sun.jini.test.impl.reggie.MultiHomedClientTest that uses the 
following internal sun API:


sun.net.spi.nameservice.NameService;
sun.net.spi.nameservice.NameServiceDescriptor;

If I was to change this test and associated classes to .txt extensions, 
we could run these manually on Sun's JVM, while allowing River to build 
on other architectures.


Regards,

Peter.

[VOTE] Patricia Shanahan for PMC Chair

[Peter Firmstone]

Come on people, we need at least three votes to nominate a new Chair.

Lets give this project one last shot.

Tom  I have already voted in favour.

Peter.

Re: New Chair for Apache River PMC

[Peter Firmstone]

Perhaps we should postpone the rename for now.

On 14/05/2014 4:55 PM, Dawid Loubser wrote:

I agree strongly with Bryan here. I do like Rafał's suggested approach
of creating a namespace-compatibilty module if the package names will
change (which, agreed, is probably long overdue).

Making it as easy as possible for the existing (small) user community to
run with the new version will encourage verification of the quality - or
bugs - as soon as possible.

Dawid


On 13/05/2014 14:46, Bryan Thompson wrote:

I think that the ability to roll forward and backward production code between 
the current and the next release trumps everything else.

Re: The King of all Bugs

[Peter Firmstone]

Actually an alternative could be to use future send for all transfers, 
except for eof, which can be safely used for async send since the byte 
buffer doesn't change after eof.


On 14/05/2014 7:55 PM, Peter Firmstone wrote:
There are two methods of transfer, one, future send, where the 
original thread waits to be notified that the receiving thread has 
finished processing the ByteBuffer, this appears to be safely 
transferred.  This is for server mode.


Then there is async send, this is for client mode, here the calling 
thread doesn't wait for the receiving thread.


Have a look in com.sun.jini.jeri.internal.mux.Session, here the 
OutputStream uses a ByteBuffer which it shares via async send.


I think the answer is to defensively copy the async send ByteBuffer.





On 14/05/2014 7:14 PM, Patricia Shanahan wrote:
ByteBuffer is a subclass of Buffer, whose documentation says, under 
Thread Safety, Buffers are not safe for use by multiple concurrent 
threads. If a buffer is to be used by more than one thread then 
access to the buffer should be controlled by appropriate 
synchronization.


Is access controlled by appropriate synchronization? How are the 
buffers communicated between the threads?


Patricia

On 5/14/2014 2:03 AM, Peter Firmstone wrote:
One of the things I like about JERI is it's multiplexing and 
multithreaded.


What I don't like about JERI is, it passes ByteBuffers between calling
threads and pool threads.

Who can guess what's wrong with that?

Peter.

Re: The King of all Bugs

[Peter Firmstone]

Hmm, it already does that, I wonder if this is safe for direct ByteBuffer's?

Visibility is one issue, the second is mutual exclusion.  I think mutual 
exclusion is ok, not sure about visibility.


On 14/05/2014 8:01 PM, Peter Firmstone wrote:
Actually an alternative could be to use future send for all transfers, 
except for eof, which can be safely used for async send since the byte 
buffer doesn't change after eof.


On 14/05/2014 7:55 PM, Peter Firmstone wrote:
There are two methods of transfer, one, future send, where the 
original thread waits to be notified that the receiving thread has 
finished processing the ByteBuffer, this appears to be safely 
transferred.  This is for server mode.


Then there is async send, this is for client mode, here the calling 
thread doesn't wait for the receiving thread.


Have a look in com.sun.jini.jeri.internal.mux.Session, here the 
OutputStream uses a ByteBuffer which it shares via async send.


I think the answer is to defensively copy the async send ByteBuffer.





On 14/05/2014 7:14 PM, Patricia Shanahan wrote:
ByteBuffer is a subclass of Buffer, whose documentation says, under 
Thread Safety, Buffers are not safe for use by multiple 
concurrent threads. If a buffer is to be used by more than one 
thread then access to the buffer should be controlled by appropriate 
synchronization.


Is access controlled by appropriate synchronization? How are the 
buffers communicated between the threads?


Patricia

On 5/14/2014 2:03 AM, Peter Firmstone wrote:
One of the things I like about JERI is it's multiplexing and 
multithreaded.


What I don't like about JERI is, it passes ByteBuffers between calling
threads and pool threads.

Who can guess what's wrong with that?

Peter.

The King of all Bugs

[Peter Firmstone]

One of the things I like about JERI is it's multiplexing and multithreaded.

What I don't like about JERI is, it passes ByteBuffers between calling 
threads and pool threads.


Who can guess what's wrong with that?

Peter.

Re: The King of all Bugs

[Peter Firmstone]

Thanks Bryan,

That's done the trick, the OutputStream that contained the ByteBuffer 
was reliant on the mux output thread changing the state of position, 
limit and mark.


Now the duplicate has it's own state and is published safely, the 
OutputStream then waits for the mux output thread to finish if 
necessary, the original buffer is now cleared and is ready for more 
data, rather than relying on the mux output thread to reset it's state.


Fixed another data race while I was at it, a synchronized collection was 
shared outside of synchronization with a pool thread via a Runnable.


Regards,

Peter.

On 14/05/2014 8:15 PM, Bryan Thompson wrote:

All you need to do is dup the buffer. That gives you independent fields for 
position and limit, which is what is not thread safe.  You do need a 
synchronization section around that dup, but that is a very fast operation.  
You can also use the read only view of the buffer. Bryan


On May 14, 2014, at 6:06 AM, Peter Firmstonej...@zeus.net.au  wrote:

Hmm, it already does that, I wonder if this is safe for direct ByteBuffer's?

Visibility is one issue, the second is mutual exclusion.  I think mutual 
exclusion is ok, not sure about visibility.


On 14/05/2014 8:01 PM, Peter Firmstone wrote:
Actually an alternative could be to use future send for all transfers, except 
for eof, which can be safely used for async send since the byte buffer doesn't 
change after eof.


On 14/05/2014 7:55 PM, Peter Firmstone wrote:
There are two methods of transfer, one, future send, where the original thread 
waits to be notified that the receiving thread has finished processing the 
ByteBuffer, this appears to be safely transferred.  This is for server mode.

Then there is async send, this is for client mode, here the calling thread 
doesn't wait for the receiving thread.

Have a look in com.sun.jini.jeri.internal.mux.Session, here the OutputStream 
uses a ByteBuffer which it shares via async send.

I think the answer is to defensively copy the async send ByteBuffer.






On 14/05/2014 7:14 PM, Patricia Shanahan wrote:
ByteBuffer is a subclass of Buffer, whose documentation says, under Thread Safety, 
Buffers are not safe for use by multiple concurrent threads. If a buffer is to be used by 
more than one thread then access to the buffer should be controlled by appropriate 
synchronization.

Is access controlled by appropriate synchronization? How are the buffers 
communicated between the threads?

Patricia


On 5/14/2014 2:03 AM, Peter Firmstone wrote:
One of the things I like about JERI is it's multiplexing and multithreaded.

What I don't like about JERI is, it passes ByteBuffers between calling
threads and pool threads.

Who can guess what's wrong with that?

Peter.

Re: New Chair for Apache River PMC

[Peter Firmstone]

On 13/05/2014 9:59 AM, Dennis Reedy wrote:

Apologies for not chiming in earlier, I've been running around with my air
on fire for the past couple of weeks. As to whether River is dead, I don't
think it is, maybe mostly dead (in which case a visit to Miracle Max may be
in order). I think River is static, but not dead. The technology is so
worth at least maintaining, fixing bugs and continued care and feeding.

The issue to me is that the project has no direction, and River has no
community that participates and makes decisions as a community. There has
been tons of work in qa_refactor, is that the future for River? Or is it a
fork?


There are develpers who are concerned about the number of fixes made in 
qa-refactor, but no one yet has identified an issue I haven't been able 
to fix very quickly.  In any case the public api and serial form is 
backward compatible.


I encourage the community to test it, find out for themselves and report 
any issues.



Regards

Dennis


On Mon, May 12, 2014 at 9:59 AM, Greg Trasuktras...@stratuscom.com  wrote:


On May 11, 2014, at 12:30 AM, Peterj...@zeus.net.au  wrote:



Ultimately, if community involvement continues to decline, we may have

to send River to the attic.

Distributed computing is difficult and we often bump into the

shortcomings of the java platform, I think these difficulties are why
developers have trouble agreeing on solutions.

But I think more importantly we need increased user involvement.

Is there any advise or resources we can draw on from other Apache

projects?
It may be, ultimately, that the community has failed and River is headed
to the Attic.  The usual question is “Can the project round up the 3 ‘+1’
votes required to make an Apache release?”  Historically, we have been able
to do that, at least for maintenance releases, and I don’t see that
changing, at least for a while.

The problem is future development and the ongoing health of the project.
  On this point, we don’t seem to have consensus on where we want the
project to go, and there’s limited enthusiasm for user-focused
requirements.  Also, my calls to discuss the health of the project have had
no response (well, there was a tangent about the build system, but
personally I think that misses the point).

I will include in the board report the fact that no-one has expressed an
interest in taking over as PMC chair, and ask if there are any other expert
resources that can help.

Cheers,

Greg Trasuk.

JERI Scalability Testing

[Peter Firmstone]

Thought you may find these results interesting, just killed some more 
latent concurrency bugs in JERI.


   * Mahalo stress tests are now running with 0% contention at close to
 raw socket speed.
   * ClassLoading is thread confined for each classloader to avoid
 contention.
   * All underlying infrastructure takes advantage of modern Executors
 and concurrency utils.
   * TaskManager has been replaced and is no longer needed.
   * There are no unnecessary DNS calls, only those required to make
 connections.
   * Security checks have an imperceptible impact on performance.
   * RFC3986 compliant Uri class uses bitshift operations for ASCII
 string manipulation.

I need someone with access to decent hardware to really test this out.


Hot Spots - Method  Self time [%]   Self time   Self time (CPU) 
Samples
java.net.ServerSocket.accept()  59.933640   161513.736 ms   161513.736 ms   
6
java.net.DatagramSocket.receive()   29.966820   80756.868 ms
80756.868 ms3
java.io.ObjectInputStream.defaultReadObject() 	2.706319 	7293.194 ms 
7293.194 ms 	134

java.lang.Class.forName()   2.0821705611.19 ms  5611.19 ms  
9
com.sun.jini.start.AggregatePolicyProvider.getContext() 	0.925748 
2602.776 ms 	2494.776 ms 	7
java.util.concurrent.FutureTask.get() 	0.911941 	182311.234 ms 
2457.569 ms 	271

java.lang.ClassLoader.loadClass()   0.7322751973.389 ms 
1973.389 ms 7
java.util.Collections$UnmodifiableCollection.isEmpty() 	0.516405 
1391.647 ms 	1391.647 ms 	1
com.sun.jini.jeri.internal.mux.Session$MuxInputStream.read() 	0.516405 
182643.391 ms 	1391.647 ms 	246
java.io.ObjectOutputStream.writeObject() 	0.481305 	2885.993 ms 
1297.057 ms 	76
java.lang.System.identityHashCode[native]() 	0.459557 	1238.449 ms 
1238.449 ms 	1
java.lang.SecurityManager.checkConnect() 	0.115316 	310.762 ms 
310.762 ms 	3

java.lang.Long.toHexString()0.110310297.271 ms  297.271 ms  
1
java.io.ObjectOutputStream.defaultWriteObject() 	0.095964 	38327.99 ms 
258.61 ms 	71
com.sun.jini.jeri.internal.runtime.Util.unmarshalValue() 	0.091908 
247.68 ms 	247.68 ms 	133
au.net.zeus.collection.AbstractReferenceComparator.compare() 	0.079535 
214.336 ms 	214.336 ms 	2
java.lang.reflect.Proxy.getInvocationHandler() 	0.071852 	193.631 ms 
193.631 ms 	1
com.sun.jini.jeri.internal.runtime.Util.marshalValue() 	0.058676 
1049.831 ms 	158.124 ms 	78
java.util.concurrent.AbstractExecutorService.submit() 	0.040076 	108.0 
ms 	108.0 ms 	1

java.io.ObjectInputStream.init()0.02271861.223 ms   
61.223 ms   1
com.sun.jini.start.ActivateWrapper$ExportClassLoader.toString() 
0.022597 	60.895 ms 	60.895 ms 	2
java.security.AccessController.doPrivileged[native]() 	0.022204 
59.837 ms 	59.837 ms 	340

java.lang.reflect.Method.invoke()   0.02056931369.303 ms55.431 
ms   244
org.apache.river.api.net.RFC3986URLClassLoader$5.run() 	0.015687 
42.274 ms 	42.274 ms 	2
com.sun.jini.jeri.internal.runtime.Target.dispatch() 	0 	0.0 ms 	0.0 
ms 	474

Distributed Lambda Expressions

[Peter Firmstone]

I found this an interesting article about using ASM to wrap lambda 
methods for Java 7:


http://mkto-o0074.com/846PMW834HX00h0J900

My thoughts are that it's possible to parse lambda expressions, 
serialize them and capture any arguments, verify the lambda expression 
during unmarshalling then dynamically create a class remotely, while 
remaining compliant with the lambda spec.


Cheers,

Peter.

Re: New Chair for Apache River PMC

[Peter Firmstone]

I think I've been heading slowly in that general direction; I'm still 
fixing bugs.  Because there are ongoing concerns regarding the number of 
internal changes to modernise the code (public api compatible), there's 
no plan to make this a public release, but instead allow the community 
to decide what to do with the code, there is significant support to make 
these improvements, it's just the migration process is a source of 
contention.   I do plan to produce release candidates for wider testing 
in the near future however.


What I've worked on:

  1. Distributed objects, to replace Serializable, instead of using
 reflection to serialize an objects serial form, it serializes
 instructions to recreate objects remotely using method calls and
 arguments.  So now you can extend objects that aren't Serializable
 without a zero arg constructor and implement Distributed, and you
 can check invariants, copy arrays and Collections AND have final
 fields.
  2. Fixing Bugs, lots of bugs
  3. Eliminating unnecessary DNS calls.
  4. Making security scalable.
  5. We need access to decent hardware to perform real world testing.

What I'd like to do:

  1. UDT Sockets
  2. DNS-SRV Lookup Discovery
  3. Distributed Lambda Expressions
  4. Modular build

Good to hear River is still useful.

Regards,

Peter.

On 13/05/2014 12:54 AM, Bryan Thompson wrote:

We have been using river/jini since 2006.  While I have very little time for 
work on open source projects outside of our own (read - completely swamped), I 
am hugely in favor of its continued life.  The main points for me are not so 
much the evolution of the software as continuing to harden an already great 
platform such that it can continue to be used.

Thanks,
Bryan


On May 12, 2014, at 9:59 AM, Greg Trasuktras...@stratuscom.com  wrote:



On May 11, 2014, at 12:30 AM, Peterj...@zeus.net.au  wrote:



Ultimately, if community involvement continues to decline, we may have to send 
River to the attic.

Distributed computing is difficult and we often bump into the shortcomings of 
the java platform, I think these difficulties are why developers have trouble 
agreeing on solutions.

But I think more importantly we need increased user involvement.

Is there any advise or resources we can draw on from other Apache projects?

It may be, ultimately, that the community has failed and River is headed to the 
Attic.  The usual question is “Can the project round up the 3 ‘+1’ votes 
required to make an Apache release?”  Historically, we have been able to do 
that, at least for maintenance releases, and I don’t see that changing, at 
least for a while.

The problem is future development and the ongoing health of the project.  On 
this point, we don’t seem to have consensus on where we want the project to go, 
and there’s limited enthusiasm for user-focused requirements.  Also, my calls 
to discuss the health of the project have had no response (well, there was a 
tangent about the build system, but personally I think that misses the point).

I will include in the board report the fact that no-one has expressed an 
interest in taking over as PMC chair, and ask if there are any other expert 
resources that can help.

Cheers,

Greg Trasuk.

Re: New Chair for Apache River PMC

[Peter Firmstone]

I think it would be interesting to have a discussion about any 
shortcomings in the api and how things might be done differently with 
modern knowledge, to determine whether we need to redesign the api and 
if the extent required a full rewrite or just a backward compatiblity break.


So far I've managed to modernise the internals with promising results.  
I'm working on a bug in JERI at present.  The public API has remained 
compatible in keeping with general consensus.  I have a hard enough time 
changing private code, the list is very conservative with change.


Regards,

Peter.

On 13/05/2014 12:21 AM, Jeremy R. Easton-Marks wrote:
While I enjoy River I think that shelving it as is may be the best 
option. I think this project may have run its course in its current 
state and this doesn't encourage new development or interest in 
participating in the project.


However, I would like to present a strawman proposal to the group. The 
current committers put out a last maintenance release fixing any bugs 
that may have been been resolved but not yet released. After that the 
2.* branch is abandoned. At that point the River community decides if 
it is possible and worthwhile to start over from scratch. We begin 
this new project from day 1 with deciding what we want to accomplish, 
and how we accomplish. No code is written until a good set of 
requirements are written and voted upon. We keep the development 
community in mind and make sure that River 3.0 is approachable from 
scratch.


While this may take more time and at times probably be very tedious at 
times I think it gives the project a fresh start and not be beholden 
to old code, and requirements.


Just my 2 cents on the subject.

~Jeremy


On Mon, May 12, 2014 at 8:59 AM, Greg Trasuk tras...@stratuscom.com 
mailto:tras...@stratuscom.com wrote:



On May 11, 2014, at 12:30 AM, Peter j...@zeus.net.au
mailto:j...@zeus.net.au wrote:



 Ultimately, if community involvement continues to decline, we
may have to send River to the attic.

 Distributed computing is difficult and we often bump into the
shortcomings of the java platform, I think these difficulties are
why developers have trouble agreeing on solutions.

 But I think more importantly we need increased user involvement.

 Is there any advise or resources we can draw on from other
Apache projects?


It may be, ultimately, that the community has failed and River is
headed to the Attic.  The usual question is “Can the project round
up the 3 ‘+1’ votes required to make an Apache release?”
 Historically, we have been able to do that, at least for
maintenance releases, and I don’t see that changing, at least for
a while.

The problem is future development and the ongoing health of the
project.  On this point, we don’t seem to have consensus on where
we want the project to go, and there’s limited enthusiasm for
user-focused requirements.  Also, my calls to discuss the health
of the project have had no response (well, there was a tangent
about the build system, but personally I think that misses the point).

I will include in the board report the fact that no-one has
expressed an interest in taking over as PMC chair, and ask if
there are any other expert resources that can help.

Cheers,

Greg Trasuk.




--
Jeremy R. Easton-Marks

être fort pour être utile

Re: Build failed in Jenkins: river-qa-refactor-solaris #41

[Peter Firmstone]

Every now and again the build will fail with 
java.lang.NoClassDefFoundError some.arbitrary.class and that class will 
be the same class for all tests, even though the classes are present in 
jar files.


This seems to happen on Jenkins and it appears to occur regardless of 
which build version is being run, no matter what architecture (every 
time I check the jar files the classes are present, so it isn't a 
problem with the build tool).


This along with BindException port in use is very frustrating, as it 
makes the build appear to fail much more than it does on ordinary hardware.


Regards,

Peter.

On 13/04/2014 6:06 PM, Apache Jenkins Server wrote:

Seehttps://builds.apache.org/job/river-qa-refactor-solaris/41/changes

Changes:

[peter_firmstone] Minor improvements for ServiceDiscoveryManager

--
[...truncated 15905 lines...]
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeIfExistsTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeIfExistsWaitTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeNO_WAITTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeReadTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionTakeWaitTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteLeaseANYTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteLeaseFOREVERTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteNegativeLeaseTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeIfExistsNotifyTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeIfExistsTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeNotifyTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTakeTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotTransactionWriteTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteLeaseANYTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteLeaseFOREVERTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteNegativeLeaseTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 
com/sun/jini/test/spec/javaspace/conformance/snapshot/SnapshotWriteTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] com/sun/jini/test/impl/mahalo/AdminIFShutdownTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] com/sun/jini/test/impl/mahalo/AdminIFTest.td
  [java] Test Passed: OK
  [java]
  [java] -
  [java] 

Re: Health of the Apache River Project

[Peter Firmstone]

On 10/04/2014 10:42 PM, Rafał Krupiński wrote:

Dnia 2014-04-10, czw o godzinie 22:15 +1000, Peter Firmstone pisze:

Rafał,

If you're considering a new git hub project, I'd reccommend using the
qa_refactor branch, it contains a significant number of bug fixes.
ClassLoader and URI string handling perform very well also.

I'm only willing to start it if it's to be incorporated by the
community (I guess that would be Greg ;) and officially released at
some point, and I don't see a consensus on whether to replace current
development tree with qa_refactor.


No, my plans are to continue fixing bugs for now, then I'll leave the 
build for the River community to decide.


Dennis Reedy created a Maven build for river in skunk.


I was going to test our project with qa_refactor anyway. Is it a drop-in
replacement for 2.2.2?


Sure is, far fewer bugs and will completely blow away any other build 
performance wise, it's basically now limited by Java socket speed, so 
the next step would be to use faster network communications like UDT.


You won't see any contention, but beware, it will expose concurrency 
bugs in your software.


Regards,

Peter.

Re: Health of the Apache River Project

[Peter Firmstone]

It's not so much that ant is the problem, more so that classdep needs to 
be maintained for new java features to correctly determine 
dependencies.   But then it cannot determing Class.forName dependencies...


Tim Blackmann  I contributed the ClassDep Java 5 language support code 
based on ASM.


Regards,

Peter.


On 11/04/2014 5:40 AM, Gregg Wonderly wrote:

On Apr 10, 2014, at 2:35 PM, Rafał Krupińskirafal.krupin...@sorcersoft.com  
wrote:


Dnia 2014-04-10, czw o godzinie 14:40 -0400, Greg Trasuk pisze:

Hi Rafal:


On Apr 10, 2014, at 2:15 PM, Rafał Krupińskirafal.krupin...@sorcersoft.com  
wrote:


I think you missed the point.


Could be.  I guess the question is, what are you wanting to contribute?  If 
you’re going to debug or modify current code, then yes, the build system is an 
obstacle that you need to overcome.  In which case, maybe changing parts of it 
could be a great first contribution.  I’m just saying that’s going to be a 
pretty big job, no matter who does it.

If you want patches and committers it shouldn't be a problem to change a
few lines, or even half a class in the core River. But it's not, so you
get no patches nor new committers.

Maybe you can explain at this point.  Is the problem that  you can’t build, at 
all, to test your changes?  Is this because you don’t have ANT?  It seems it’s 
because  you don’t know how to use the ANT build system, which I can 
understand.  But also, you need to understand that there are people who have no 
idea how to use Maven either.

So, overall, how can we simplify things if there are always new and different 
build tools/standards that some people know and others don’t?

Gregg



  And it’s going to be a contentious subject (as it always has been in the 
past), because every developer has their favourite build system.

It's not the issue here.

(...)

Don’t get me wrong - I’m not defending the current project structure.

Then I guess I don't understand what are you doing.

Regards,
Rafał

Re: Health of the Apache River Project

[Peter Firmstone]

Rafał,

If you're considering a new git hub project, I'd reccommend using the 
qa_refactor branch, it contains a significant number of bug fixes.  
ClassLoader and URI string handling perform very well also.


Regards,

Peter.

On 10/04/2014 7:17 PM, Rafał Krupiński wrote:

Dnia 2014-04-09, śro o godzinie 23:33 -0400, Greg Trasuk pisze:

Hi all:

Hi Greg,

Now I feel bad for not contributing my extensions to River, that I have
implemented for SORCER (nb open source on github).

We have our own ServiceStarter, historically based on River's and I
think I could contribute some patches based on that..
I also have some preliminary work on downloadable URLStreamHandlers, and
I'd be happy to contribute it to River once it's done.

The only problem is, I don't know anything about River's build process,
ant scripts, classanddepjar tuning and @Betas. It would be much easier
for me to contribute if River was mavenized. There was a discussion on
that, but it's dead now and I don't see that materializing into code
(maybe I don't know where to look).

Maybe the problem is that I wait for others to do the work instead of
doing it? Shall I start a new github project for mavenized river and
initiate the migration?

Regards
Rafał

Re: River-436 Patch attached to Jira

[Peter Firmstone]

Michał,

Just though I'd mention that objects implementing Distributed don't need 
to implement Serializable.


Cheers,

Peter.


On 18/03/2014 11:18 PM, Michał Kłeczek wrote:

Thanks Peter,

There is no need to rush with it yet.
I need to run it locally first :-)

But it would be good to have branch for it so that it is easier to resolve
conflicts.

Regards,

On Tuesday 18 of March 2014 22:31:08 Peter wrote:

I'll get a branch build up and running on Hudson for this soon.

Regards,

Peter.


- Original message -


Folks,

I've attached a first patch to River-436.
IMHO the idea is really sound - fixing several issues with River and
creating   several new possibilities.

I am more than eager to discuss it and compare to alternatives.

Note that the code is not really production-grade (yet). I've attached
it for   review.

--
Michał Kłeczek
XPro Sp. z o. o.
ul. Borowskiego 2
03-475 Warszawa
Polska

Entry with final fields causes problems for ServiceDiscoveryManager

[Peter Firmstone]

I've noticed via experimentation with the test suite that 
ServiceDiscoveryManager doesn't detect attribute changes to a service if 
it uses Entry's with final fields.  I'm not sure of the root cause, but 
it could have something to do with Reggie's implementation of the Entry 
spec.


Thoughts?

Regards,

Peter.

Re: DistributedLambda

[Peter Firmstone]

To enable remote clients to invoke processing at the server with lambda 
expression invocation on remote objects, without code downloads.


Presently the enclosing class is serialized along with the lambda, 
because it contains the receipe generated by the static compiler. I'm 
investigating if it's possible to send only the receipe along with any 
parameter objects, for dynamic code generatation at the server.


I want to completely avoid code downloading, lambda's could potentially 
allow significant performance and latency benefits by filtering and 
processing at the server, prior to returning results to clients.


Cheers,

Peter.


On 9/03/2014 6:02 PM, Michał Kłeczek wrote:

Peter,

I'm still trying to grasp what you want to achieve...

Is it simply in-band code downloading?

Regards,

DistributedLambda

[Peter Firmstone]

The present serialized form of Lambda's don't access captured fields 
until after deserialization, making them dependant on the enclosing 
classes serial form, its deserialized object state and requiring the 
enclosing class to be Serializable.


This appears to be a fundamental mistake in the design of lambda 
serialization.


I don't believe this will be changed now, this decision is the result of 
an expert group and the release date for Java 8 draws near.


One possibility of fixing this is to create a marker interface, eg 
DistributedLambda.


By implementing a new serialization proxy for DistributedLambda that 
doesn't refer to the enclosing class, but instead captures fields at the 
time of serialization instead of after deserialization, we can provide 
independant short lived functional immutable objects, enabling lambda 
parameter arguments for remote objects that don't require codebase 
downloads at the exporting node.


My reasoning and consequences:

  1. Anonomous classes introduce brittleness in a distributed
 environment, they really should be avoided.
  2. Capturing state at the time of execution from deserialized
 encapsulating object state, invites mutability and side effects
 that diverge from the state of the original caller object and
 intent of functional programming.
  3. Storing a lambda into a @serialField within a Serializable object,
 would also mean that the lamdba parameters would be captured at
 the time of serialization, not at some later point of execution
 after deserialization, so a DistributedLambda wouldn't see updates
 to the encapsulating object after deserialization.
  4. Lamdba's are functional programming constructs and functional
 programming is founded on immutability, so DistributedLambda would
 honour that heritage.
  5. A DistributedLambda instance becomes an independant immutable
 implementation of a functional interface at the time of
 serialization, to avoid surprises, fields referred to by a lambda
 should be effectively final and side effects should be avoided.

Thoughts?

Regards,

Peter.

Re: DistributedLambda

[Peter Firmstone]

An update:

Lambda arguments are captured at serialization time, this is a good 
thing!  A static method is generated by the compiler on the enclosing 
class and this is called during deserialization.


I'm investigating how to capture the lambda receipe, so that it and it's 
arguments can be serialized, without requiring the enclosing class, so 
the lambda receipe can be used to create an object independant of it's 
original enclosing class during deserialization.


Will keep you posted.

Any thoughts appreciated.

Regards,

Peter.

On 9/03/2014 11:41 AM, Peter Firmstone wrote:
The present serialized form of Lambda's don't access captured fields 
until after deserialization, making them dependant on the enclosing 
classes serial form, its deserialized object state and requiring the 
enclosing class to be Serializable.


This appears to be a fundamental mistake in the design of lambda 
serialization.


I don't believe this will be changed now, this decision is the result 
of an expert group and the release date for Java 8 draws near.


One possibility of fixing this is to create a marker interface, eg 
DistributedLambda.


By implementing a new serialization proxy for DistributedLambda that 
doesn't refer to the enclosing class, but instead captures fields at 
the time of serialization instead of after deserialization, we can 
provide independant short lived functional immutable objects, enabling 
lambda parameter arguments for remote objects that don't require 
codebase downloads at the exporting node.


My reasoning and consequences:

  1. Anonomous classes introduce brittleness in a distributed
 environment, they really should be avoided.
  2. Capturing state at the time of execution from deserialized
 encapsulating object state, invites mutability and side effects
 that diverge from the state of the original caller object and
 intent of functional programming.
  3. Storing a lambda into a @serialField within a Serializable object,
 would also mean that the lamdba parameters would be captured at
 the time of serialization, not at some later point of execution
 after deserialization, so a DistributedLambda wouldn't see updates
 to the encapsulating object after deserialization.
  4. Lamdba's are functional programming constructs and functional
 programming is founded on immutability, so DistributedLambda would
 honour that heritage.
  5. A DistributedLambda instance becomes an independant immutable
 implementation of a functional interface at the time of
 serialization, to avoid surprises, fields referred to by a lambda
 should be effectively final and side effects should be avoided.

Thoughts?

Regards,

Peter.

Re: [jira] [Commented] (RIVER-435) Proposed Standard for Single-Archive Service Deployment Packaging

[Peter Firmstone]

It was River-362.

You're assuming the Java sandbox is secure, history tells us otherwise.  
Your Module provider is quite interesting however and it could still 
serve a purpose for River, but it's outside the scope of River-362.


The proposed solution is to restrict deserialization to a list of 
classes that form a trusted computing base (only code audited as secure 
would be included), this then allows two things:


   * Ability of clients to limit deserialization to a trusted computing
 base while authentication and proxy verification is performed. (In
 java anything that implements Serializable can be serialized, so
 this is a small trusted subset of what is currently Serializable).
   * Additional benefit of allowing untrusted parties to interact
 securely using no code downloads, (java.lang.reflect.Proxy) based
 services, with a limited trusted computing base.

So it limits plain old Serialization as well, not just codebase downloads.

This solution would also work with your module provider.

Can I suggest creating a new Jira for your module provider, with an 
explanation of its capabilities and benefits?


Hope this helps explain it.

Cheers,

Peter.


On 26/02/2014 10:09 PM, Michał Kłeczek wrote:

Peter,

I think you dont remember the idea. It's been a long time.
It DOES prevent untrusted code from executing because a codebase annotation
is verified BEFORE it has a chance to load code.

Regards,
Michal
26 lut 2014 12:31 Peterj...@zeus.net.au  napisał(a):


- Original message -

This. I like this. How would this work, would it be an Entry, an
attribute of the service (perhaps similar to the ServiceUI factory?).

My PoC is attached to one of the issues in Jira (I'll try to find it
tomorrow once I have some more time). It was discussed some time ago on
this list mainly with Peter.

In our previous discussion, the aim was to determine a way to ensure that
no untrusted objects could be deserialised and thus prevent DOS attacks.

Because ProxyTrustVerifier checks are performed after deserialisation,
this wasn't a good solution for that specific problem.  That doesn't mean
to say it wouldn't be useful for other purposes.

The solution to avoid downloading untrusted objects, is to prevent
deserialisation of untrusted classes, by checking the serialization byte
stream and limiting deserialisation to a specific list of local classes
until after trust can be established.

I think this combined with Dennis' method of provisioning codebases and
secure endpoints would be a very difficult to crack security layer.

That doesn't mean that object based annotations cannot be used, making the
annotation field a String doesn't mean you can't deserialise any Object, it
just means that after an object other than String is deserialised, a
ClassCastException will be thrown, by that time an attacker may have
already created a ClassLoader, if deserialisation is performed in
privileged context.

Once we depreciate RMIClassLoader and replace it with RiverClassLoading,
then we could allow annotations to be objects.  After all, codebase
annotations are also part of the serialised stream, so we can secure them
by limiting them to a known trusted subset of classes as stated above.

Regards,

Peter.



Basically the idea is to change codebase annotation from
java.lang.String which needs to be interpreted by the client to an
object implementing an interface.
This object can be verified in exactly the same way as normal proxies
are verified ( by a TrustVerifier - in particular the ProxyTrustVerifier
). All that happens during deserialization.
It does not have anything to do with Entries since it is implemented at
the layer below that - hence is available for _all_ downoladed code (for
RemoteEventListeners as well :-) )

Regards,
Michal

--
Michał Kłeczek
XPro Quality Matters
http://www.xpro.biz

DiscoveryAdmin, Reggie and Unicast discovery port.

[Peter Firmstone]

I'm currently getting a number of test failures where a port passed in 
by configuration is already in use.  If the port specified is 0, then an 
arbitrary port is selected if 4160 is not available.


However Reggie throws an exception during construction if a configured 
port is in use.


I'd like to change Reggie to use an arbitrary port if a configured port 
is in use, this then allows an admin to try a range of ports, using 
DiscoveryAdmin, so a port in use failure isn't terminal, but instead a 
minor inconvenience.


Regards,

Peter.

net.jini.discovery.DisoveryAdmin states:

/**
 * Changes the number of the port on which the lookup service is 
currently

 * listening for unicast discovery queries to the given port number.
 * If a value of zero is input, then the lookup service will first try
 * to listen on the standard unicast discovery port, but if that fails,
 * the lookup service will listen on an arbitrary port.
 *
 * @param port codeint/code representing the new port number on 
which

 * the lookup service should listen for unicast discovery
 * queries.
 *
 * @throws java.io.IOException because an invocation of this method 
will
 * result in the re-initiation of the unicast discovery 
process,

 * which can throw an codeIOException/code when socket
 * allocation occurs.
 *
 * @throws java.rmi.RemoteException typically, this exception 
occurs when

 * there is a communication failure between the client and the
 * server.
 */
public void setUnicastPort(int port) throws IOException, 
RemoteException;

}

Re: [Discuss] Please have a look at the River Container

[Peter Firmstone]

Haven't had time to participate in the latest conversation.

  1. Would like to see River adopt Rio's conventions at the very least
 as part of the new standard, also like to see Maven provisioning.
  2. Dennis, if you're interested, I'd be prepared to be one of your
 developers for RIO, if you decided to go the incubator route.
  3. Greg, for River container, I think we need to document whether
 certain classes are thread safe or not, using annotations.  Also
 I'd tend to favour constructors for dependency injection, rather
 than fields, since this allows for immutability.
  4. I've found, making old code thread safe is very hard, mutibility
 is very hard.  When you take advantage of immutability, you can
 use java.util.concurrent classes for controlling mutable state, or
 thread confine mutable state, then safely publish when you've
 finished mutating, never mutate again after publishing, but mutate
 a new object and replace the old one instead.

Regards,

Peter.


On 20/02/2014 9:24 AM, Greg Trasuk wrote:

The standard I proposed is what is currently implemented by River-Container.  
Rio’s convention is very much different, and relies on reading jar files from a 
Maven repository rather than from the local file system.  It represents a 
radical departure from the Service-Starter conventions, although it is 
compatible with the services.

This is false. Rio provides the capability to declare a service be loaded 
either by artifact resolution or by using declared jars. I have never moved 
away from the latter approach for the simple reason that there are deployments 
that require legacy support.

My mistake.  Thank you for the correction.


Using an artifact to annotate a codebase, or to resolve a service's classpath 
provides significant advancement in the build-deploy lifecycle for developers, 
and also provides performance benefits when accessing a service's codebase (as 
well as addressing perm-gen oome for containers).


Makes a lot of sense.  It’s something I’m intending to look into.


I'm thinking that way too. For now, I am withdrawing my offer of donating Rio 
to River. My intention was that it would greatly benefit River, by dramatically 
improving the out of box experience. I'll be happy if River would just mention 
it as a notable project that may be beneficial to developers getting to know 
River.

I'll also comment on your service archive standard, and if reasonable (and 
given time) I'll provide support for it in Rio.


I recognize your earlier concerns about the River project appearing to favour 
one container over another.  As such I’ll continue development of the 
River-Container outside the Apache River project.  I guess it’ll need a new 
name.

I’m not sure if we should leave River-435 open to discuss the service 
packaging.  Perhaps others can comment...


Regards

Dennis

Cheers,

Greg Trasuk

Re: [jira] [Commented] (RIVER-435) Proposed Standard for Single-Archive Service Deployment Packaging

[Peter Firmstone]

You could adopt the directory conventions api, impl and proxy, instead 
of lib and lib-dl?  That way you could make sure the api is loaded into 
the application class loader, while the implementation can be loaded 
into a child ClassLoader for maximum cooperation (in case the service 
implementation also uses other remote services) while avoiding name 
space visibility issues.


Regards,

Peter.

On 20/02/2014 12:58 PM, Greg Trasuk (JIRA) wrote:

 [ 
https://issues.apache.org/jira/browse/RIVER-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906531#comment-13906531
 ]

Greg Trasuk commented on RIVER-435:
---

Comments from the mailing list discussion...

Greg Trasuk
==
OK, so on the topic of the jar file naming conventions (hello-api.jar, 
hello-proxy.jar, hello-impl.jar, etc), I thought we had already adopted that as 
a recommended convention.  It follows common “good practices” that most of us 
have used for a long time, and it allows you to build without ‘classdepandjar’. 
 As well, it happens to dovetail nicely with a Maven build.

Having said that, I don’t believe that convention needs to be mentioned in the 
single-archive packaging spec (or at least not required - I suppose it could be 
referenced as good practice).

The spec differentiates between “class path” and “codebase” jars by having them 
in different folders inside the deployment archive (lib and lib-dl).  So, while 
the build that creates the archive may very well use the conventions to 
determine which dependent files go into which folder, from the container’s 
point of view, it doesn’t care about the naming conventions.  Basically, 
everything in the ‘lib’ dir gets included in the service’s class path, and 
everything in the ‘lib-dl’ dir gets published through the codebase server and 
included in the service’s codebase annotation.  In fact, a service may want to 
include other jar files in either its codebase or class path (for instance 
domain objects).  These other jar files shouldn’t be forced into a River 
convention, especially since that might require renaming or repackaging the jar 
files.

Jeff Ramsdale
===
No, services shouldn't be required to use this standard but the
River-provided services should model it as the best practice, as mentioned
in another thread.



Proposed Standard for Single-Archive Service Deployment Packaging
-

 Key: RIVER-435
 URL: https://issues.apache.org/jira/browse/RIVER-435
 Project: River
  Issue Type: Improvement
  Components: com_sun_jini_start
Reporter: Greg Trasuk
 Attachments: SingleArchiveServiceDeployment.docx, 
SingleArchiveServiceDeployment.pdf


The attached document proposes the layout and general requirements for an archive file to 
support simplified drag-and-drop deployment for services that adhere to the 
Service Starter conventions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Build system and Java 8 was: Re: [Vote] (RIVER-432) Jar files in svn and src distributions

[Peter Firmstone]

No, I fixed the build system last time to support Java 5 language 
features, I don't have the time or inclination to fix it again.  
ClassDep needs to be able to find dependencies by analysing byte code.  
So to support finding dependencies for Java 8 language features, someone 
will need to add that functionality to ClassDep.  Java 9 is due to be 
released 2 years later and so we have this maintenance task for writing 
new ClassDep code every time there are new language features.


The alternative is use conventions that avoid the need to support and 
maintain our own build tool.


I'm suggesting building without classdep.

On 13/02/2014 12:40 AM, Greg Trasuk wrote:

Hi Peter:

I’m still trying to understand what you’re suggesting.  Are you talking 
specifically about classDepAndJar?  When I try to run the build against 
JDK8-EA, I get an exception in ‘org.objectweb.asm.ClassReader’.  Which I 
suspect could be fixed by a more recent version of asm, although I haven’t 
tried yet.  Basically I suspect the existing build could be made to work 
without too much effort.  As far as supporting JDK8 features, isn’t that just a 
question of flipping the ‘1.8’ switch on the compiler?  And why won’t it work 
for client code, if the client code doesn’t use ‘classDepAndJar’ (I’ve never 
used it in client code, for example).

When you say “a new build system”, are you talking about fixing 
‘classDepAndJar’, or are you talking about changing to Maven, Gradle, or 
something else?  Are you talking about restructuring the source to make 
‘classDepAndJar’ unnecessary, or what?  Could you be more specific?

Regards,

Greg Trasuk.

On Feb 12, 2014, at 12:22 AM, Peterj...@zeus.net.au  wrote:


Well, River doesn't build, throws exceptions, it won't work for client code 
either and it doesn't support Java 8 features.

I wrote the java 5 language support code, I would rather spend time 
reorganising the build once, than maintain a build tool.

Just my thoughts.

- Original message -

Hi Peter:

I’m not familiar with Java 8 yet.   How is the current build not
supported?

Cheers,

Greg.

On Feb 11, 2014, at 5:35 AM, Peterj...@zeus.net.au  wrote:


+1 Peter.

Note that the current build system does not support java 8.

Maintaining a build system is a significant burden ( I know I had to
implement Java 5 support).

We will need a new build system for java 8, this looks like a step in
that direction.   In reality we need to adopt the build conventions
used by Rio.

Regards,

Peter.
- Original message -

As discussion has settled somewhat, I would like to call another
vote to accept the latest patch described in
https://issues.apache.org/jira/browse/RIVER-432

The patch removes the archived jar files for Velocity and ASM and
replaces them with Apache Ivy scripts that download the jars from
Maven Central the first time a build is done. From then on, the
jar files are in Ivy’s repository (for more info, see
http://ant.apache.org/ivy).

Voting will remain open at least until 2000 UTC Feb 13, 2014.

Cheers,

Greg.


On Jan 3, 2014, at 12:57 PM, Greg Trasuktras...@stratuscom.com
wrote:


On Jan 3, 2014, at 5:25 AM, Simon IJskes - QCGsi...@qcg.nl
wrote:


In order to gain some time to discuss this first i will vote -1.

First, we decided to NOT remove velocity builder.

When I read the email chain, my impression was that we wanted to
remove it (to quote you Sim, “To be honest, I hate it”), but there
was a dependency on it in the ‘extras’ folder that was added in
the trunk branch. As there is no ‘extras’ in the 2.2 branch, and
that is what this patch applies to, I thought it was fair to remove
VelocityConfigurationBuilder from the 2.2 branch. Perhaps we
should revisit the ConfigurationBuilder approach in another
thread. For now I’ll spin another patch that doesn’t remove
VelocityConfigurationBuilder.


Second, no need to remove the jars as specified in your own
comments on RIVER-432.

Pulling in external jars at compile time, shall we start here?

They are already in the svn. They are already in the build
scripts. What does this patch fix? No legal problems?


Apache policy is somewhat unclear on this point. One needs to
examine the mailing lists for clues on what we should really do.
I will argue that:

1 - The fundamental distribution model of Apache is source code,
not binaries. 2 - Distributing binaries is tolerated but not
encouraged.   Since the svn repository can be seen as a
distribution point, binaries in svn are also tolerated but not
encouraged. 3 - Downloading dependency binaries at build time is
technologically easy, provides the same guarantees as putting them
in cvs, and avoids the question of effectively distributing
someone else’s code.

All these together suggest that although we’re technically OK to
put dependency jars in a “-deps” package (note that the status quo
_is_ unacceptable - at the very least, we need to restructure the
dependencies into a “-deps” binary package), there is 

Re: [Vote] (RIVER-432) Jar files in svn and src distributions

[Peter Firmstone]

+1 Peter.

On 13/02/2014 2:09 AM, Simon IJskes - QCG wrote:

On 12-02-14 17:00, Greg Trasuk wrote:


OK, fair enough.  I’ll close this issue and open another one that
just makes sure the jars aren’t in the source distribution (that _is_
an Apache requirement) without adding Ivy.


+1


In general, though, as we move to the build structure discussion, are
you OK with using dependency management.  I’m not big on putting
other people’s software distributions into our source repository,
especially if we’re going to see more dependencies as time goes on.


Yes, i'm ok. Clearly we are not gaining critical mass where it comes 
to gaining interest and new blood for much needed innovations and 
improvements. Whe cannot let Peter do the work on his own, and the 
longer we let Peter work in isolation, without bringing his work into 
the main trunk, the worse we are of.


So if we need to change things, i see the adoptition of popular tools 
as a way to enlist others to work on river. Gradle i view as a bit 
much, maven should be general knowledge nowadays. I still claim we can 
do without it, but if we need a popular (and therefore recocnizable 
feel) to increase the adoption, i'm all for it. The way it is going 
right now, is an agony to most i think.


Gr. Simon

Fwd: Re: P2P Internet Services - no code downloads, lambda's

[Peter Firmstone]

 Original Message 
Subject:Re: P2P Internet Services - no code downloads, lambda's
Date:   Wed, 05 Feb 2014 22:44:54 +1000
From:   Peter Firmstone j...@zeus.net.au
To: Greg Trasuk tras...@stratuscom.com



I agree regarding SOAP and River needing to be easier to deploy and
use.  I think the conventions Dennis is using with Rio, makes deployment
much cleaner and easier to understand.   I'm doing what I can, but
deployment isn't my speciality, I think the changes Dennis proposes goes
a long way towards it and we also need to get the community more
involved in the River container work your doing too.

Not too sure how to get the community more engaged, I think it's a side
effect of a very specialised user base.

Regards,

Peter.

On 5/02/2014 12:59 PM, Greg Trasuk wrote:

 Hi Peter:

 I applaud your enthusiasm.  Having said that, I’m not sure what River brings 
to the internet space that isn’t already well-served by the http infrastructure 
and RESTful services.  I kind of think that race has run.  I also don’t foresee 
much uptake for mobile-code approaches (even dynamically-generated code) in the 
internet space.  Java’s reputation for insecurity on the browser has killed 
that idea.

 In the data centre and cloud, however, I think River/Jini is a great 
substitute for the XML/SOAP approach.  I’ve talked to hundreds of people over 
the past ten years who are implementing SOAP-based SOA.  The vast majority of 
them are using Java for all their development.  Assuming we can simplify 
deployment so that mere mortals can actually use it, it should be a no-brainer 
that writing Java-based services with Java interfaces is simpler than writing 
Java-based services with WSDL-based interfaces to get platform neutrality that 
hardly ever gets used.

 Plus, Jini’s inherent resilience, combined with JavaSpaces, should be an easy 
sell for scalable-on-demand systems (e.g. Amazon EC2).

 Cheers,

 Greg.


 On Feb 4, 2014, at 6:52 PM, Peterj...@zeus.net.au   wrote:


 Components:

 DNS-SRV Discovery
 UDT JERI Endpoints- Firewall traversal   superior WAN performance
 TLS Encryption
 Reflective Proxies - no code downloads
 Lambda expressions (for services) - remote dynamic code generation (this is 
big, don't download a codebase, the jvm generates code on demand)
 Secure serialization - limit classes to java.lang, and Jini trusted subsets.
 Size constraints on method invocation returns - anti dos measure

 A new internet lookup service, reflective proxy, with lambda expression based 
lookup.


 There's not a great deal of work required to do this and it should make a huge 
difference to our userbase, which at present appears limited.

 Thoughts?

 Peter.

RESULT: [VOTE] Theory based development

[Peter Firmstone]

Results:

+1 Peter Firmstone (PMC)
+1 Bishnu Prasad Gautam
+1 Luis Matta

Abstain:

Greg Trasuk

I would have liked to see more participation from PMC members, since 
this is a vote on a procedural matter and as there have been no 
objections after 72 hours, under Apache voting rules it passes.

River-344 - replacing TaskManager in SDM

[Peter Firmstone]

 Notes:



 ServiceDiscoveryManager


   NotifyEventTask


If the task list contains any RegisterListenerTasks

or LookupTasks associated with this task's lookup service

(ProxyReg), and if those tasks were queued prior to this

task (have lower sequence numbers), then run those tasks

before this task (return true).


Additionally, if the task list contains any other

ServiceIdTasks associated with this task's service ID

which were queued prior to this task, then run those

tasks before this task.


If the criteria outlined above is not satisfied, then this

task can be run immediately (return false).



   LookupTask


If the task list contains any RegisterListenerTasks,

other LookupTasks, or NotifyEventTasks associated with

this task's lookup service (ProxyReg), if those tasks

were queued prior to this task (have lower sequence

numbers), then run those tasks before this task (return

true). Otherwise this task can be run immediately

(return false).



   ServiceIdTask


If there is at least one task in the given task list that is

associated with the same serviceID as this task, and that task

has a sequence number less than the sequence number of this task,

then run this task *after* the task in the list (return true);

otherwise run this task now (return false).




   ServiceDiscoveryManager, CacheTask classes

  957:  private final class RegisterListenerTask extends CacheTask {

1117:  private final class ProxyRegDropTask extends CacheTask {

1005:  private final class LookupTask extends CacheTask implements Task {

  647:  private static abstract class ServiceIdTask extends CacheTask 
implements Task {

1149:  private final class DiscardServiceTask extends CacheTask {

1169:  private final class NotifyEventTask extends ServiceIdTask {

1416:  private final class NewOldServiceTask extends ServiceIdTask {

1498:  private final class UnmapProxyTask extends ServiceIdTask {





 RegisterListenerTask extends CacheTask

/** This task class, when executed, first registers to receive

* ServiceEvents from the given ServiceRegistrar. If the registration

* process succeeds (no RemoteExceptions), it then executes the

* LookupTask to query the given ServiceRegistrar for a snapshot

* of its current state with respect to services that match the

* given template.

*

* Note that the order of execution of the two tasks is important.

* That is, the LookupTask must be executed only after registration

* for events has completed. This is because when an entity registers

* with the event mechanism of a ServiceRegistrar, the entity will

* only receive notification of events that occur in the future,

* after the registration is made. The entity will not receive events

* about changes to the state of the ServiceRegistrar that may have

* occurred before or during the registration process.

*

* Thus, if the order of these tasks were reversed and the LookupTask

* were to be executed prior to the RegisterListenerTask, then the

* possibility exists for the occurrence of a change in the

* ServiceRegistrar's state between the time the LookupTask retrieves

* a snapshot of that state, and the time the event registration

* process has completed, resulting in an incorrect view of the

* current state of the ServiceRegistrar.

*/



 ProxyRegDropTask extends CacheTask

/** When the given registrar is discarded, this Task class is used to

* remove the registrar from the various maps maintained by this

* cache.

*/

/* For each itemReg in the serviceIdMap, disassociate the

* lookup service referenced here from the itemReg; and

* if the itemReg then has no more lookup services associated

* with it, remove the itemReg from the map and send a

* service removed event.

*/


 LookupTask extends CacheTask

/** This class requests a snapshot of the given registrar's state.*/



 ServiceIdTask extends CacheTask

/** Abstract base class for controlling the order-of-execution of tasks

* corresponding to a particular serviceID associated with a particular

* lookup service.

*/


 DiscardServiceTask extends CacheTask

/** Task class used to asynchronously notify service discard. */


 NotifyEventTask extends ServiceIdTask

/** Task class used to asynchronously notify all registered service

* discovery listeners of serviceAdded/serviceRemoved/serviceChanged

* events.

*/


 NewOldServiceTask extends ServiceIdTask

/** Task class used to asynchronously process the service state

* (snapshot), matching this cache's template, that was retrieved

* from the given lookup service.

*

* After retrieving the snapshot S, the LookupTask queues an instance

* of this task for each service referenced in S. This task determines

* if the given service is an already-discovered service (is currently

* in this cache's serviceIdMap), or is a new service. This task

* handles the service differently, depending on whether the service

* is a new or old.

*

* a. if the item is old, then this task will:

* - compare the given 

Final fields - discussion

[Peter Firmstone]

Rather than discuss specific instances where I've made changes to ensure 
an object reference doesn't escape during construction, I figure it 
would be more constructive to discuss final fields themselves.


Some of the arguments against using Startable were based on timing when 
references to partially constructed objects would be accessed, this is 
incorrect.  REF: JLS 17.5.1 Semantics of final fields, final paragraph 
For reads of |final| fields, the only writes that are deemed to come 
before the read of the |final| field are the ones derived through the 
|final| field semantics.  The problem was that the execution path from 
our service instances to the location where other threads read its 
reference, did not pass through a final field freeze at the end of a 
constructor; the compiler is under no obligation to reload these final 
fields.


The argument that ensued over Startable could have been avoided if 
someone simply proposed they preferred to use a public constructor that 
publishes and exports, after calling a private constructor that sets 
final fields, that would have also been correct.  REF: JLS 17.5.1 
Semantics of final fields, paragraph 2.


Does anyone here on the list still think it's acceptable to allow the 
reference of an object that has final fields to escape from the 
constructor that sets those final fields as our services in River 2.2 
and all earlier releases of Jini do?


Welcome to theoretical development.

The relevant section of the Java 7 Language Specification:


   17.5. |final| Field Semantics

Fields declared final are initialized once, but never changed under 
normal circumstances. The detailed semantics of |final| fields are 
somewhat different from those of normal fields. In particular, compilers 
have a great deal of freedom to move reads of |final| fields across 
synchronization barriers and calls to arbitrary or unknown methods. 
Correspondingly, compilers are allowed to keep the value of a |final| 
field cached in a register and not reload it from memory in situations 
where a non-|final| field would have to be reloaded.


|final| fields also allow programmers to implement thread-safe immutable 
objects without synchronization. A thread-safe immutable object is seen 
as immutable by all threads, even if a data race is used to pass 
references to the immutable object between threads. This can provide 
safety guarantees against misuse of an immutable class by incorrect or 
malicious code. |final| fields must be used correctly to provide a 
guarantee of immutability.


An object is considered to be /completely initialized/ when its 
constructor finishes. A thread that can only see a reference to an 
object after that object has been completely initialized is guaranteed 
to see the correctly initialized values for that object's |final| fields.


The usage model for |final| fields is a simple one: Set the |final| 
fields for an object in that object's constructor; and do not write a 
reference to the object being constructed in a place where another 
thread can see it before the object's constructor is finished. If this 
is followed, then when the object is seen by another thread, that thread 
will always see the correctly constructed version of that object's 
|final| fields. It will also see versions of any object or array 
referenced by those |final| fields that are at least as up-to-date as 
the |final| fields are.


*Example 17.5-1. |final| Fields In The Java Memory Model*

The program below illustrates how |final| fields compare to normal fields.

class FinalFieldExample {
final int x;
int y;
static FinalFieldExample f;

public FinalFieldExample() {
x = 3;
y = 4;
}

static void writer() {
f = new FinalFieldExample();
}

static void reader() {
if (f != null) {
int i = f.x;  // guaranteed to see 3
int j = f.y;  // could see 0
}
}
}

The class |FinalFieldExample| has a |final| |int| field |x| and a 
non-|final| |int| field |y|. One thread might execute the method 
|writer| and another might execute the method |reader|.


Because the |writer| method writes |f| /after/ the object's constructor 
finishes, the |reader| method will be guaranteed to see the properly 
initialized value for |f.x|: it will read the value |3|. However, |f.y| 
is not |final|; the |reader| method is therefore not guaranteed to see 
the value |4| for it.



*Example 17.5-2. |final| Fields For Security*

|final| fields are designed to allow for necessary security guarantees. 
Consider the following program. One thread (which we shall refer to as 
thread 1) executes:


Global.s = /tmp/usr.substring(4);

while another thread (thread 2) executes

String myS = Global.s;
if (myS.equals(/tmp))System.out.println(myS);

|String| objects are intended to be immutable and string operations do 
not perform synchronization. While the |String| implementation does not 
have any data races, other code could have 

RESULT: Re: VOTE: add Startable to Jini specification

[Peter Firmstone]

Vote results in chronical order, after 72 hours:

+1 Peter Firmstone
+1 Simon IJskes
+0 Greg Trasuk

According to our rules that's one vote short for inclusion into the Jini 
Specification.


On this occassion there was interest in developing a more comprehensive 
standard for starting services, I hope container providers in the River 
community are able to collaborate and provide such a solution.


Unless there are any objections (by lazy concensus), I propose we use 
Startable for River's service implementations and also allow River users 
to implement it, if they so choose, so they too may use final fields safely.


Regards,

Peter.

DISCUSS: Proposal for eliminating tensions; return to collaborative development

[Peter Firmstone]

Ongoing elimination of multithreaded issues with River code is causing 
division among remaining developers.


To make matters worse, due to the sheer number of issues found, the 
protacted duration required to fix them would likely make the situation 
worse, this work needed to be completed in a short period of time, 
however it has taken far longer than desired.


For this reason, I'm considering whether the code in qa_refactor should 
no longer form the basis of a release.


What I would expect this to do is remove tensions over perceived risks 
of change.


If this proposal is supported, I'd also reccommend that trunk be 
reverted back to the 2.2 River branch, with the exception of Sim's work 
on ClassLoading, which should be included.


Provided there is support, change trunk to review then commit without 
lazy concensus.


I would finish the work on qa_refactor and solve the remaining 
multithreaded issues (on a longer lower pressure time schedule), the 
River community can then decide whether it wants to use code from 
qa_refactor on an as needed basis.  I believe that the River community 
will find this code a useful reference for latent multithreaded bugs.


Regards,

Peter Firmstone.

[VOTE] Theory based development

[Peter Firmstone]

One of my first tasks on joining the Apache River project was to 
implement support for Java 5 language features in ClassDep .  I don't 
recall who requested Java 5 language feature support, however this 
seemed like a good place to start, based on an initial contribution from 
Tim Blackmann.  The ClassDep api remained the same, however the 
underlying implementation was completely rewritten.


This work has been included in all releases of Apache River since I 
joined and I'm yet to hear one complaint, without it, the transition to 
Java 5 may have been quite different.


Moving on to our current circumstance, most of River's code pre dates 
the Java Memory Model released with Java 5, it would be unreasonable to 
expect code pre dating a standard to be compliant with it.


My attempts to fix random intermittent test failures, unrelated to code 
changes, but affected by timing, has lead to a cascading series of 
failures, every time I fixed a problem a new problem would appear.


FindBugs no longer reports many multi-threaded issues in release code 
(FindBugs isn't instrumented to analyse classes that use ReadersWriter), 
although quite a few remain in the test suite.


I believe the majority of multi threaded issues are fixed, with some 
remaining hold outs based on TaskManager.Task.runAfter() and other 
remaining in the test suite.


The major problem that faces my development now however is not multi 
threaded bugs, instead it is the River development community remaining 
undecided on supporting or not supporting Theory based development.


My reasoning is this:

  1.

 To ameliorate issues reviewing change raised by other developers,
 I plan to document my changes on Jira (this will be a huge effort
 that will take months).

  2.

 Many of these issues are based on theory and analysis, many
 examples will be fields accessed from multiple threads without
 synchronisation, but there will be no test cases exposing bugs
 (most concurrency bugs are not repeatable).

  3.

 My recent attempt to create an interface Startable, met with much
 controversy.  I'm sure that if I had supplied a test case
 demonstrating a failure there would be no argument, however
 because my analysis was based on theory, the alternative solution
 posed was things were acceptable the way they were and that object
 construction would be complete before other threads could see the
 reference (without any evidence to prove otherwise either).

  4.

 There is a significant risk, with theoretical based issues raised
 on Jira, that the status quo will prevail and my efforts wasted.


The right thing to do is at least tell me, as your fellow developer, 
whether the community supports theory based development or not, to save 
me wasting time fixing any more bugs or documenting them, (if that is 
the case) and to allow me to focus on something the community does support.


If the community does support theory based development, then I suggest 
we pose the issue of publishing an object with final fields where other 
threads can see it, prior to its constructor completing to the 
concurrency interest list and see what the experts thoughts are, then 
I'd propose we also use this approach with other issues, by consulting 
experts in each field relating to a bug?


Do you support theory based development?

+1  Peter Firmstone.

ServiceDiscoveryManager Task Race

[Peter Firmstone]

With TaskManager.Task.runAfter, throughput wasn't significant enough for 
this race to occur.


If I make the ExecutorService single threaded, the error doesn't occur 
as the tasks are executed in correct dependency order, however, when the 
ExecutorService has a lot of threads ready, the tasks aren't able to be 
arranged in a queue as they're immediately handed off to a waiting thread.


TaskManager added sufficient latency to prevent this race from occurring.

I think the solution to this race condition (note that all accesses are 
synchronized) is to make the the operations atomic and separate out the 
event notifications.


Thoughts?

Peter.

com/sun/jini/test/impl/servicediscovery/event/LookupTaskServiceIdMapRace.td

**
 * This test attempts to simulate the following race condition that
 * can occur between an instance of UnmapProxyTask (created and queued
 * in LookupTask) and instances of NewOldServiceTask that are created
 * and queued by NotifyEventTask:
 *
 * - 1 LUS {L0}
 * - N (~250) services {s0, s1, ..., sN-1}, to be registered in L0
 * - M (~24) SDMs, each with 1 cache with template matching all si's
 *   {SDM_0/C0, SDM_1/C1, ... SDM_M-1/CM-1}
 *
 * Through the shear number of service registrations, caches, and events,
 * this test attempts to produce the conditions that cause the regular
 * occurrence of the race between an instance of UnmapProxyTask and
 * instances of NewOldServiceTask produced by NotifyEventTask when a
 * service event is received from L0.
 *
 * This test starts lookup L0 during construct. Then, when the test begins
 * running, half the services are registered with L0, followed by the
 * creation of half the SDMs and corresponding caches; which causes the
 * tasks being tested to be queued, and event generation to ultimately
 * begin. After registering the first half of the services and creating
 * the first half of the SDMs, the remaining services are registered and
 * the remaining SDMs and caches are created. As events are generated,
 * the number of serviceAdded and serviceRemoved events are tallied.
 *
 * When an SDM_i/cach_i pair is created, an instance of 
RegisterListenerTask

 * is queued and executed. RegisterListenerTask registers a remote event
 * listener with L0's event mechanism. When the services are registered 
with

 * L0, that listener receives service events; which causes NotifyEventTask
 * to be queued and executed. After RegisterListerTask registers for events
 * with L0, but before RegisterListenerTask exits, an instance of 
LookupTask

 * is queued and executed. LookupTask retrieves from L0 a snapshot of its
 * state. Thus, while events begin to arrive informing each cache of the
 * services that are registering with L0, LookupTask is querying L0 for
 * its current state.
 *
 * Upon receipt of a service event, NotifyEventTask queues a 
NewOldServiceTask

 * to determine if the service corresponding to the event represents a new
 * service that has been added, a change to a previously-registered 
service,

 * or the removal of a service from L0. If the event corresponds to a newly
 * registered service, the service is added to the cache's serviceIdMap and
 * a serviceAdded event is sent to any listeners registered with the cache.
 * That is,
 *
 * Service event received
 *
 *   NotifyEventTask {
 * if (service removed) {
 *   remove service from serviceIdMap
 *   send serviceRemoved
 * } else {
 *   NewOldServiceTask
 * if (service changed) {
 *   send serviceChanged
 * } else if (service is new) {
 *   add service to serviceIdMap
 *   send serviceAdded
 * }
 * }
 *   }
 *
 * While events are being received and processed by NotifyEventTask and
 * NewOldServiceTask, LookupTask is asynchronously requesting a snapshot
 * of L0's state and attempting to process that snapshot to populate
 * the same serviceIdMap that is being populated by instances of
 * NewOldServiceTask that are initiated by NotifyEventTask. LookupTask
 * first examines serviceIdMap, looking for services that are NOT in the
 * snapshot; that is, services that are not currently registered with L0.
 * Such a service is referred to as an, orphan. For each orphan service
 * that LookupTask finds, an instance of UnmapProxyTask is queued. That 
task

 * removes the service from the serviceIdMap and sends a serviceRemoved
 * event to any listeners registered with the cache. After processing
 * any orphans that it finds, LookupTask then queues an instance of
 * NewOldServiceTask for each service in the snapshot previously retrieved.
 * That is,
 *
 * LookupTask - retrieve snapshot {
 *
 *   for each service in serviceIdMap {
 * if (service is not in snapshot) { //orphan
 *   UnmapProxyTask {
 * remove service from serviceIdMap
 * send serviceRemoved
 *   }
 * }
 *   }
 *   for each service in snapshot {
 *   NewOldServiceTask
 * if (service changed) {
 *   send 

Replacing TaskManager with ExecutorService

[Peter Firmstone]

Two emails are worth reflecting on, as is River-344, this relates to 
replacing TaskManager with ExecutorService.


http://mail-archives.apache.org/mod_mbox/river-dev/201107.mbox/%3cbb4ad312-53c1-4ce6-9bff-01e5cc344...@sorcersoft.org%3e
http://mail-archives.apache.org/mod_mbox/river-dev/201105.mbox/%3c4dc17d85.5000...@acm.org%3e

TaskManager.Task.runAfter was slowly being eliminated by the original 
Jini team and has been a hot topic of discussion since Patricia began 
analysing its use.


One of the problems we have developing River, is a lack of access to big 
iron hardware, for this reason I've made no attempt to optimise any 
implementations.


The work I've done so far:

  1. Used time and sequence numbers and Comparable to allow ordering of
 tasks where necessary, the ExecutorServices for these will remain
 separate, so that comparable tasks are not mixed to replace
 Task.runAfter functionality
  2. Allowed users to implement their own ExecutorService
 implementations, to be optimised on their own hardware, since we
 are unable to.
  3. Used ThreadPoolExecutor with the same number of pool threads as
 specified for TaskManager, in a number of instances, because many
 queues are unbounded, corePoolthreads is set to the max threads
 setting of the TaskManager it replaces.  This is probably sub
 optimal as threads are not created on demand, they ramp up to
 corePoolthreads under relatively light load.


The following test is supposed to test compliance with the Jini 
specifications, however I can't find anywhere in the specification where 
services that are registered first, should have their events prioritised 
over services registered immediately after:


com/sun/jini/test/spec/lookupservice/test_set00/NotifyOnAttrAdd.td

Wouldn't it be more reasonable to expect that each ServiceEvent should 
arrive in order for each ServiceID, without regard for the order that 
the services (ServiceID's) themselves are registered?


Thoughts?

Regards,

Peter.

/** Executes the current QA test.
 *
 *  For each service instance created during construct, deletes the 
attribute

 *  belonging to that service by modifying the attribute with the
 *  the corresponding null attribute from the second set. Waits a
 *  configured amount of time to allow for all of the events to be
 *  generated and collected. Determines if all of the expected -- 
as well

 *  as no un-expected -- events have arrived. This test depends on the
 *  semantics of event-notification. That is, it uses the fact that
 *  if the events were generated for each service item in sequence
 *  (which they were), then the events will arrive in that same 
sequence.

 *  This means one can expect, when examining the event corresponding
 *  to index i, that the service ID returned in the ServiceEvent 
should
 *  correspond to the i_th service item registered. If it does not, 
then
 *  failure is declared. Thus, this test does the following:  1. 
Verifies
 *  that the number of expected events equals the number of events 
that

 *  have arrived. 2. Verifies that the transition returned in event[i]
 *  corresponds to the expected transition (MATCH_MATCH). 3. Verifies
 *  that the service ID returned in event[i] equals the service ID
 *  of the i_th service registered. 4. Verifies that the handback
 *  returned in the i_th event object equals the service ID of the
 *  i_th service.
 */

-


Running com/sun/jini/test/spec/lookupservice/test_set00/NotifyOnAttrAdd.td
Running com/sun/jini/test/spec/lookupservice/test_set00/NotifyOnAttrAdd.td
Time is Mon Jan 13 21:04:08 EST 2014
Time is Mon Jan 13 21:04:08 EST 2014
Starting test in separate process with command:
Starting test in separate process with command:
'C:\Program Files\Java\jdk1.7.0_45\jre\bin\java' 
-Djava.security.manager=org.apache.river.api.security.CombinerSecurityManager 
-Djava.security.policy=file:/C:/Users/peter/Documents/NetBeansProjects/peterConcurrentPolicy/qa/harness/policy/defaulttest.policy 
-Djava.rmi.server.codebase=http://medusa:9082/qa1-lookupservice-dl.jar 
-cp 
C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\qa\lib\jiniharness.jar;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\qa\lib\jinitests.jar;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\lib\jsk-platform.jar;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\lib\jsk-lib.jar;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\lib\high-scale-lib.jar;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\lib\custard-apple-1.0.3.jar 
-ea -esa '-Djava.ext.dirs=C:\Program 
Files\Java\jdk1.7.0_45\jre\lib\ext;C:\windows\Sun\Java\lib\ext;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\qa\lib-ext;C:\Users\peter\Documents\NetBeansProjects\peterConcurrentPolicy\lib-ext'