Agree with you. Waiting for JDK 7 is a good idea. On Mon, Oct 24, 2011 at 11:53 AM, Leo Neumeyer <[email protected]>wrote:
> Yes, we look a bit at osgi but seems overkill for what we need. I can > understand our simple implementation and seems to work for now. A more > complex framework will require more time and is not even clear that is the > right solution. Next JDK will support modular loading so it is probably a > good idea to wait if we can. Let's see if this works for what we need and we > can revisit later. However, if you have a good alternative to suggest > alternatives, please do so. > > Our requirements are very constrained because we only need to load > subclasses of App. This seems to simplify the problem quite a bit. Because > apps have their own CL, we can easily de-reference the loaded app and have > the CL + loaded classes get GCed. This seems to satisfies our > loading/unloading of apps requirement. For inter-app communication we only > use the EventSource API. > > -leo > > On Oct 24, 2011, at 11:35 AM, kishore g wrote: > > > I know you have tried lot of alternatives for dynamic app/module > deployment. > > Just wanted to confirm if we tried Apache Felix. > > > > On Mon, Oct 24, 2011 at 11:00 AM, Leo Neumeyer <[email protected] > >wrote: > > > >> Nice! > >> > >> any thoughts on what will it take to include the dependent jars in the > s4r > >> package? > >> > >> talk tomorrow, > >> -leo > >> > >> On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote: > >> > >>> Hi, > >>> > >>> I had a look into that, and the solution boils down to using the same > >>> classloader for classes loaded from s4r archives and for classes > >> dynamically > >>> generated. > >>> The JarLoader needs a small modification to handle that. > >>> It is now possible to load several apps concurrently, including apps > from > >>> the s4-example subproject. > >>> > >>> I'll share my changes tomorrow so we can discuss and see what the next > >> steps > >>> are. > >>> > >>> Matthieu > >>> > >>> > >>> On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <[email protected]> > >> wrote: > >>> > >>>> I made some progress getting the app to load. The problem was a > >>>> missing no-args constructor to get the newInstance method to work. > >>>> (Apps will be required to have a no-args constructor.) Once the app > >>>> starts, there is an exception because the CounterPE is not found. It > >>>> looks like a conflict between dynamic app loading and dynamic > >>>> dispatching. More debugging needed, here is the trace: > >>>> > >>>> comm.module=org.apache.s4.comm.Module > >>>> s4.logger_level=TRACE > >>>> comm.queue_emmiter_size=8000 > >>>> comm.queue_listener_size=8000 > >>>> cluster.hosts=localhost > >>>> cluster.ports=5077 > >>>> cluster.lock_dir=/tmp > >>>> cluster.isCluster=true > >>>> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read: > >>>> > >>>> > >> > file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r > >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.MyApp, resolveIt: true > >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: org.apache.s4.core.App, > >>>> resolveIt: true > >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder, > >>>> resolveIt: true > >>>> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.Throwable, > >>>> resolveIt: true > >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> java.lang.InterruptedException, resolveIt: true > >>>> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.303 [main] INFO org.apache.s4.comm.topology.Cluster - Added > >>>> cluster node: localhost:5077 > >>>> 19:43:53.336 [main] INFO o.a.s.c.topology.AssignmentFromFile - Host > >>>> Name: 10.0.1.97 > >>>> 19:43:53.336 [main] INFO o.a.s.c.topology.AssignmentFromFile - > >>>> Partition available: true > >>>> 19:43:53.352 [main] INFO o.a.s.c.topology.AssignmentFromFile - > >>>> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local Lock File > >>>> location: /tmp/s4-0.lock > >>>> 19:43:53.352 [main] INFO o.a.s.c.topology.AssignmentFromFile - > >>>> Acquire partition:success. > >>>> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.PrintPE, resolveIt: true > >>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.core.ProcessingElement, resolveIt: true > >>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.Object, resolveIt: > >>>> true > >>>> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: org.apache.s4.base.Event, > >>>> resolveIt: true > >>>> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true > >>>> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.util.List, resolveIt: > >>>> true > >>>> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.CounterPE, resolveIt: true > >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.CountEvent, resolveIt: true > >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: org.apache.s4.core.Stream, > >>>> resolveIt: true > >>>> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit, > >>>> resolveIt: true > >>>> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true > >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true > >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true > >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true > >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.core.SingletonPE, resolveIt: true > >>>> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: > >>>> org.apache.s4.example.counter.UserEvent, resolveIt: true > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a > >>>> system class. > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning newly loaded class. > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.String, resolveIt: > >>>> true > >>>> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.util.Random, resolveIt: > >>>> true > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.util.ArrayList, > >>>> resolveIt: true > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.Character, > >>>> resolveIt: true > >>>> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt: > >>>> true > >>>> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement > >>>> - OnCreateInternal > >>>> 19:43:53.423 [Gender Stream] TRACE > >>>> org.apache.s4.core.ProcessingElement - OnCreateInternal > >>>> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement - > >>>> OnCreateInternal > >>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt: > >>>> true > >>>> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:53.430 [Gender Stream] TRACE > >>>> org.apache.s4.core.ProcessingElement - Num PE instances: 0. > >>>> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement > >>>> - Num PE instances: 0. > >>>> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement - > >>>> Num PE instances: 0. > >>>> Exception in thread "Age Stream" java.lang.NoClassDefFoundError: > >>>> org/apache/s4/example/counter/CounterPE > >>>> at OverloadDispatcher847.dispatchEvent(Unknown Source) > >>>> at > >>>> > >> > org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) > >>>> at org.apache.s4.core.Stream.run(Stream.java:272) > >>>> at java.lang.Thread.run(Thread.java:680) > >>>> Caused by: java.lang.ClassNotFoundException: > >>>> org.apache.s4.example.counter.CounterPE > >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >>>> at java.security.AccessController.doPrivileged(Native Method) > >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > >>>> ... 4 more > >>>> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError: > >>>> org/apache/s4/example/counter/CounterPE > >>>> at OverloadDispatcher3022.dispatchEvent(Unknown Source) > >>>> at > >>>> > >> > org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) > >>>> at org.apache.s4.core.Stream.run(Stream.java:272) > >>>> at java.lang.Thread.run(Thread.java:680) > >>>> Caused by: java.lang.ClassNotFoundException: > >>>> org.apache.s4.example.counter.CounterPE > >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >>>> at java.security.AccessController.doPrivileged(Native Method) > >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > >>>> ... 4 more > >>>> Exception in thread "User Stream" java.lang.NoClassDefFoundError: > >>>> org/apache/s4/example/counter/CounterPE > >>>> at OverloadDispatcher2958.dispatchEvent(Unknown Source) > >>>> at > >>>> > >> > org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) > >>>> at org.apache.s4.core.Stream.run(Stream.java:272) > >>>> at java.lang.Thread.run(Thread.java:680) > >>>> Caused by: java.lang.ClassNotFoundException: > >>>> org.apache.s4.example.counter.CounterPE > >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >>>> at java.security.AccessController.doPrivileged(Native Method) > >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > >>>> ... 4 more > >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.lang.System, resolveIt: > >>>> true > >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> MultiClassLoader loadClass - className: java.io.PrintStream, > >>>> resolveIt: true > >>>> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - > >>>> Returning system class (in CLASSPATH). > >>>> Done. Closing... > >>>> Bye. > >>>> 19:43:58.430 [Age Count Stream] INFO org.apache.s4.core.Stream - > >>>> Closing stream Age Count Stream. > >>>> 19:43:58.430 [Gender Count Stream] INFO org.apache.s4.core.Stream - > >>>> Closing stream Gender Count Stream. > >>>> 19:43:58.430 [User Count Stream] INFO org.apache.s4.core.Stream - > >>>> Closing stream User Count Stream. > >>>> > >>>> > >>>> > >>>> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <[email protected] > > > >>>> wrote: > >>>>> Matthieu, please check my *modules* branch. > >>>>> I packaged the counter example in a *.s4r file (a jar with a > different > >>>>> extension name). Here is how: > >>>>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md > >>>>> I fixed the code so we can read the jar as a resource from the > >> classpath. > >>>>> The HelloApp example works fine but there is an exception when trying > >> to > >>>>> load the counter app. The MyApp class should load as is, I think. > You > >>>> will > >>>>> probably find the bug by the time I wake up :-) The good side of > being > >> 9 > >>>>> time zones apart. > >>>>> -leo > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> -leo > >>>> > >> > >> > >
