When I tried jboss modules, I didn't have to do any of this. Maybe it is worth investing more times in getting jboss modules to work.
I started a jira for this topic here: https://issues.apache.org/jira/browse/S4-4 please add to this issue. -leo On Thu, Oct 13, 2011 at 3:23 AM, adam wojtuniak <[email protected]> wrote: > Hi Leo, > > We can't avoid class loading issue even if we want only to load apps > dynamically. Every app is going to have dependencies to s4 (plus s4 > dependencies) and also can happen that app can have their own dependencies, > all these dependencies must be visible to app classloader. One solution is > to expose all these dependencies as a system packages then they will be > visible to all bundles in the osgi container but that is a hack which are > going to lead to maintenance nightmare (every time is new dependency it must > be added to system package). Class.forName is another problem, for example > kryo lib is loading classes which in the container aren't visible to kryo > until kryo will be bundlized with DynamicImport-Package: *. These problems > can't be avoided and is no matter what solution is used (JBossModules or > Felix etc.). One solution is to find proper bundlized version of some s4 > dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put for > some of them DynamicImport-Package: *. > > Cheers, > Adam > > On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer > <[email protected]>wrote: > >> Hi Adam, >> >> very nice. >> >> We are only interested in loading apps dynamically. Is it possible in OSGI >> to just use it for the apps. The rest of the jars can be loaded at start >> time in the same way we do it now. >> >> Anyway, I will take a look. Can you post your comments to the dev list so >> everyone can see it and comment? >> >> [email protected]**che.org >> >> Thanks! >> -leo >> >> >> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote: >> >> Hi Leo, >> >> Thanks for description. >> I've implemented ideas which I sent you before. Now I can load the apps >> using embedded osgi container but there is still class loading issue >> (com.esotericsoftware.kryo.SerializationException: Unable to find class: >> io.s4.example.counter.UserEvent). To be able to run s4 I had to put plenty >> system packages in configuration file for felix. >> Around 15 jars defined in libraries are not osgi bundles (don't have >> proper manifest file). The biggest issue are dependencies and classloading >> (for example Kryo is trying to load UserEvent class but that class is not >> visible for him so Class.forName is not going to work).I will try to fix >> these problems, in the meantime you can have a look at my repo >> https://github.com/adamwojtuniak/s4-piper. I added task createRepo, it >> will create bundles directory with all dependencies and project jars. When >> you launch Main class from s4-core, it launches embedded felix which will >> auto deploy all bundles located in bundles directory. In the console you >> will see felix gogo shell ( >> http://www.packtpub.com/article/apache-felix-gogo ). >> >> Cheers, >> Adam >> >> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <[email protected]>wrote: >> >>> Responding to this: >>> >>> > Maybe you can write me some small description of s4 concept. >>> >>> Here is the very high level s4-piper (which is quite different than s4 >>> v0.3). We expect the next version of S4 to be based on s4-piper. This >>> is work in progress! >>> >>> API: >>> >>> * Event: all messages sent between PEs inherit from Event. >>> * ProcessingElement: The unit that processes an event. >>> * Stream: Streams are the edges between PEs. A source pushes events >>> into streams and streams push events into target PEs. Streams are >>> associated with a specific Key. >>> * Key: A value derived from the content of an event. Could be the >>> value of a field, a combination of fields, or a more complex >>> transformation of the fields. Each PE instance is associated with a >>> unique key value. Keys can also be used to map PEs to partitions in a >>> cluster. >>> * KeyFinder: A helper class that Implements the actual logic used by the >>> Key. >>> * App: A unit that encapsulates a complex task carried out by a graph >>> comprised of PEs and Streams. The App has a reference to all the PE >>> and Stream objects associated with the graph. A deployment may have >>> several Apps that may communicate with each other using EventSource. >>> Apps are associated with owners and units of work. A deployment may >>> include Apps that are or are not related to each other. Apps can also >>> be used to implement security controls. By default, PEs and Streams >>> are not exported outside an App. >>> * Streamable: classes that accept S4 events must implement streamable, >>> currently: Stream and EventSource. >>> * EventSource: use it to export events outside of an App. Any app in >>> the cluster can potentially subscribe to an event source at run time. >>> >>> Some internal classes: >>> >>> * Main: starts a node. >>> * Server: the server instance in a node. >>> * Sender: send an event to its destination in a cluster. >>> * Receiver: receive an event from somewhere in the cluster. >>> >>> Stories: >>> >>> PEs do all the application processing. PE use a prototype pattern >>> which is the key to the S4 design. The first PE object is called a PE >>> prototype which is configured for a specific task but will never be >>> used for actual processing. Events sent to PEs come with a specific >>> key. For example, key=user, key_value=joe. A PE instance, is >>> associated with a specific key, In this case the PE prototype will >>> check if PE for (user, joe) exists. If not, it will clone itself and >>> create a PE instance for (user, joe). The S4 framework guarantees that >>> all future events of the same type sent to a given PE prototype will >>> always arrive at the same PE instance. In this way, we can now do >>> computation for joe. The PE instance may emit an output event sent to >>> another prototype with another key. An App is a graph of PEs and >>> Streams. >>> >>> PEs put events into Stream{s). >>> >>> Stream(s) put events into PEs. >>> >>> Stream(s) are configured for a specific PE target and key finder. >>> >>> To send a user event with field "user" and value "joe" to a UserPE we >>> do the following: >>> >>> - Define UserEvent that includes "user" field. >>> - Define a KeyFinder that extracts the value of field "user" (will >>> return "joe" in the example.) >>> - Define UserPE that accepts events of type UserEvent and does >>> something with it, say count. >>> - Define a Stream whose target is UserPE configured with the KeyFinder >>> that gets the value of "user". >>> - Now put an object of type UserEvent into the stream and it will get >>> to the unique UserPE instance for (user, joe). The instance could be >>> anywhere in the cluster. >>> >>> Once a server is started, we expect it to run indefinitely until there >>> is a failure or we need to upgrade the framework software or we change >>> the cluster configuration. In S4, the cluster is symmetric, that is, >>> all the nodes are identical, with the same code base and the same >>> configuration. >>> >>> Applications are loaded dynamically after the cluster is started. Most >>> apps need an EventSource to receive events. App must be able to find >>> EventSource and subscribe to it during initialization. >>> >>> When an app is unloaded, it will close its EventSource(s). This in >>> turn, will close all the streams that are subscribed. The apps that >>> own the streams are responsible for taking action once their >>> dependencies are gone. >>> >>> -leo >>> >>> >>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <[email protected]> >>> wrote: >>> > [sending to the new s4-dev-subscribe list!] >>> > Thank you Adam, let me read the links you sent to understand this >>> better. >>> > BTW. I reorganized the project into subprojects to make it easier to >>> > separate examples from the framework. >>> > To implement inter-app communication I added EventSource which an app >>> can >>> > use to publish its output to other apps and other apps can subscribe to. >>> We >>> > can also use it to build adaptors to non-s4 events. We can use the osgi >>> > mechanisms for declaring dependencies such as app2 depends on app1 and >>> for >>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get events >>> from >>> > app1. Unless an app uses an EventSource, its events will not be >>> exported. >>> > This will create good isolation and make it easier to add a security >>> layer >>> > around apps. Both Stream and EventSource implement Streamable. >>> > My code is here until we set up the apache >>> > svn. https://github.com/leoneu/s4-piper >>> > thanks! >>> > -leo >>> > >>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote: >>> > >>> > Hi Leo, >>> > Yes I have a few ideas but I need couple days to implement them. >>> > First, is to build S4 and examples as proper OSGi bundles using gradle >>> osgi >>> > plugin http://gradle.org/osgi_plugin. >>> > Its using bnd tool http://www.aqute.biz/Bnd/Bnd. >>> > Then embed osgi container (using standard osgi api) in Controller >>> instead >>> > of JBoss modules >>> > see >>> http://felix.apache.org/site/apache-felix-framework-launching-and-embedding.html >>> . >>> > and configure directory from which felix should auto deploy our >>> > examples >>> http://felix.apache.org/site/apache-felix-framework-usage-documentation.html#ApacheFelixFrameworkUsageDocumentation-autodeploy >>> . >>> > For dynamic deployment we can use file install >>> > bundle http://felix.apache.org/site/apache-felix-file-install.html, it >>> will >>> > watch directory from which it loads new bundles. >>> > To configure properly examples we can use >>> > ipojo http://felix.apache.org/site/apache-felix-ipojo.html or >>> declarative >>> > services >>> http://felix.apache.org/site/apache-felix-service-component-runtime.html, >>> > they both using service component model and are not intrusive in the >>> code. >>> > That will be enough to load properly the apps. >>> > Later we have to think how we want to use osgi in s4, just to load apps >>> or >>> > properly use service management. Bundles are hiding the internals from >>> other >>> > bundles and should communicate through well defined services. And also I >>> > don't have clear picture how s4 is going to be distributed (have to look >>> > more at https://github.com/s4/s4). Maybe you can write me some small >>> > description of s4 concept. >>> > Cheers, >>> > Adam >>> > >>> > >>> >>> >>> >>> -- >>> >>> -leo >>> >> >> >> > -- -leo
