Nice! any thoughts on what will it take to include the dependent jars in the s4r package?
talk tomorrow, -leo On Oct 24, 2011, at 10:29 AM, Matthieu Morel wrote: > Hi, > > I had a look into that, and the solution boils down to using the same > classloader for classes loaded from s4r archives and for classes dynamically > generated. > The JarLoader needs a small modification to handle that. > It is now possible to load several apps concurrently, including apps from > the s4-example subproject. > > I'll share my changes tomorrow so we can discuss and see what the next steps > are. > > Matthieu > > > On Sun, Oct 23, 2011 at 4:53 AM, Leo Neumeyer <[email protected]> wrote: > >> I made some progress getting the app to load. The problem was a >> missing no-args constructor to get the newInstance method to work. >> (Apps will be required to have a no-args constructor.) Once the app >> starts, there is an exception because the CounterPE is not found. It >> looks like a conflict between dynamic app loading and dynamic >> dispatching. More debugging needed, here is the trace: >> >> comm.module=org.apache.s4.comm.Module >> s4.logger_level=TRACE >> comm.queue_emmiter_size=8000 >> comm.queue_listener_size=8000 >> cluster.hosts=localhost >> cluster.ports=5077 >> cluster.lock_dir=/tmp >> cluster.isCluster=true >> 19:43:53.281 [main] TRACE org.apache.s4.core.Server - Read: >> >> file:/Users/leo/Projects/s4project/s4-piper/subprojects/s4-core/bin/apps/CounterExample.s4r >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.MyApp, resolveIt: true >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.297 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: org.apache.s4.core.App, >> resolveIt: true >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: org.apache.s4.core.KeyFinder, >> resolveIt: true >> 19:43:53.298 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.Throwable, >> resolveIt: true >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> java.lang.InterruptedException, resolveIt: true >> 19:43:53.299 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.303 [main] INFO org.apache.s4.comm.topology.Cluster - Added >> cluster node: localhost:5077 >> 19:43:53.336 [main] INFO o.a.s.c.topology.AssignmentFromFile - Host >> Name: 10.0.1.97 >> 19:43:53.336 [main] INFO o.a.s.c.topology.AssignmentFromFile - >> Partition available: true >> 19:43:53.352 [main] INFO o.a.s.c.topology.AssignmentFromFile - >> Partition acquired by PID:16610 HOST:Leos-MacBook-Air.local Lock File >> location: /tmp/s4-0.lock >> 19:43:53.352 [main] INFO o.a.s.c.topology.AssignmentFromFile - >> Acquire partition:success. >> 19:43:53.354 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.PrintPE, resolveIt: true >> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.355 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.core.ProcessingElement, resolveIt: true >> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.356 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.Object, resolveIt: >> true >> 19:43:53.359 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: org.apache.s4.base.Event, >> resolveIt: true >> 19:43:53.360 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.CountKeyFinder, resolveIt: true >> 19:43:53.401 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.util.List, resolveIt: >> true >> 19:43:53.402 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.CounterPE, resolveIt: true >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.404 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.CountEvent, resolveIt: true >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: org.apache.s4.core.Stream, >> resolveIt: true >> 19:43:53.405 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.407 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.util.concurrent.TimeUnit, >> resolveIt: true >> 19:43:53.408 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.413 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.UserIDKeyFinder, resolveIt: true >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.414 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.GenderKeyFinder, resolveIt: true >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.415 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.AgeKeyFinder, resolveIt: true >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.416 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.GenerateUserEventPE, resolveIt: true >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.core.SingletonPE, resolveIt: true >> 19:43:53.417 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: >> org.apache.s4.example.counter.UserEvent, resolveIt: true >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - Not a >> system class. >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning newly loaded class. >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.String, resolveIt: >> true >> 19:43:53.418 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.util.Random, resolveIt: >> true >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.util.ArrayList, >> resolveIt: true >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.Character, >> resolveIt: true >> 19:43:53.420 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.Integer, resolveIt: >> true >> 19:43:53.421 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.423 [User Stream] TRACE org.apache.s4.core.ProcessingElement >> - OnCreateInternal >> 19:43:53.423 [Gender Stream] TRACE >> org.apache.s4.core.ProcessingElement - OnCreateInternal >> 19:43:53.423 [Age Stream] TRACE org.apache.s4.core.ProcessingElement - >> OnCreateInternal >> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.Thread, resolveIt: >> true >> 19:43:53.428 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:53.430 [Gender Stream] TRACE >> org.apache.s4.core.ProcessingElement - Num PE instances: 0. >> 19:43:53.430 [User Stream] TRACE org.apache.s4.core.ProcessingElement >> - Num PE instances: 0. >> 19:43:53.430 [Age Stream] TRACE org.apache.s4.core.ProcessingElement - >> Num PE instances: 0. >> Exception in thread "Age Stream" java.lang.NoClassDefFoundError: >> org/apache/s4/example/counter/CounterPE >> at OverloadDispatcher847.dispatchEvent(Unknown Source) >> at >> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) >> at org.apache.s4.core.Stream.run(Stream.java:272) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.s4.example.counter.CounterPE >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 4 more >> Exception in thread "Gender Stream" java.lang.NoClassDefFoundError: >> org/apache/s4/example/counter/CounterPE >> at OverloadDispatcher3022.dispatchEvent(Unknown Source) >> at >> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) >> at org.apache.s4.core.Stream.run(Stream.java:272) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.s4.example.counter.CounterPE >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 4 more >> Exception in thread "User Stream" java.lang.NoClassDefFoundError: >> org/apache/s4/example/counter/CounterPE >> at OverloadDispatcher2958.dispatchEvent(Unknown Source) >> at >> org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:376) >> at org.apache.s4.core.Stream.run(Stream.java:272) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.s4.example.counter.CounterPE >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 4 more >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.lang.System, resolveIt: >> true >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> MultiClassLoader loadClass - className: java.io.PrintStream, >> resolveIt: true >> 19:43:58.429 [main] DEBUG o.a.s4.base.util.MultiClassLoader - >> Returning system class (in CLASSPATH). >> Done. Closing... >> Bye. >> 19:43:58.430 [Age Count Stream] INFO org.apache.s4.core.Stream - >> Closing stream Age Count Stream. >> 19:43:58.430 [Gender Count Stream] INFO org.apache.s4.core.Stream - >> Closing stream Gender Count Stream. >> 19:43:58.430 [User Count Stream] INFO org.apache.s4.core.Stream - >> Closing stream User Count Stream. >> >> >> >> On Tue, Oct 18, 2011 at 11:00 PM, Leo Neumeyer <[email protected]> >> wrote: >>> Matthieu, please check my *modules* branch. >>> I packaged the counter example in a *.s4r file (a jar with a different >>> extension name). Here is how: >>> https://github.com/leoneu/s4-piper/blob/modules/NOTES.md >>> I fixed the code so we can read the jar as a resource from the classpath. >>> The HelloApp example works fine but there is an exception when trying to >>> load the counter app. The MyApp class should load as is, I think. You >> will >>> probably find the bug by the time I wake up :-) The good side of being 9 >>> time zones apart. >>> -leo >>> >> >> >> >> -- >> >> -leo >>
