Hi, We have a distributed application that deals with huge volumes of telecom data based on Hadoop/HBase architecture. At the data ingest part, we want to be highly extensible w.r.t multiple transport support. In reality, data could be thrown at this system based on either JMS, SOAP, TCP, Files etc. So, we want to use Apache Camel to build this extensible transport adaptation layer. We have an O&M framework based on Apache Zookeeper and use ZK to store all configuration data. The O&M’s CLI (which is itself a ZK client) can be executed in any node in the cluster to view/modify configuration data and any changes are immediately picked up by interested applications (which are themselves ZK clients) in the cluster.
The architecture would be something like below: <http://camel.465427.n5.nabble.com/file/n5737497/cameld.png> I would like to build CLI commands similar to the following: CLI> creat-new-transport-endpoint --ip 10.184.17.1 –protocol jms –protocolSpecificArguments “broker=10.184.18.1:61616,type=queue,name=subdataq” –wireTo “bean-class-path” When this command is excecuted, I plan to update data in ZK. I plan to create a java daemon application “Camel Daemon”, which will be running in every node in the cluster. It will be a ZK client. So, when data is updated in ZK, the “cameld” will pick it up and I want it to dynamically create this new JMS consumer client, attached to the given broker, listening for the given queue and all the received messages should be sent to the given bean, which will be in the classpath. The bean will then do the needful. My questions are as follows: 1) Whats your feedback on this architecture? Are there better ways to achieve the stated requirements? 2) What is the best mechanism to implement this architecture? I see two options: a) Build a meta-data model for storing this config info in ZK, cameld reads the meta data, uses Camel API to internally create CamelContexts appropriately and start the contexts. Do we have all required API's in Camel to programmatically achieve this? Any pointers to equivalents would be helpful b) When the CLI command gets executed, dynamically create the appropriate spring XML route equivalent for Camel and store it inside ZK. This makes the CLI client logic a bit heavy but "cameld" would be very simple logic. It would just copy the new/changed spring XML file from ZK, place it inside itself and activate the route What are the pros & cons w.r.t extensibility & maintenance etc? Are there any better mechanisms to achieve this? Thanks in advance! MK -- View this message in context: http://camel.465427.n5.nabble.com/Camel-based-transport-adaptation-for-a-distributed-application-tp5737497.html Sent from the Camel - Users mailing list archive at Nabble.com.
