camel-cxfrs with custom http headers...

[boday]

I have a route that defines a CXFRS endpoint.  I need to retrieve custom HTTP
header values from the inbound request...

from(CXF_RS_ENDPOINT_URI)
  .process(new Processor() {
public void process(Exchange exchange) throws Exception
{
Message inMessage = exchange.getIn();
System.out.println("headers->" +
inMessage.getHeaders());
...

here is my unit test code to simulate a client request...

HttpGet get = new
HttpGet("http://localhost:9000/rest/customerservice/customers/126";);
get.addHeader("myHeader", "1234");
HttpClient httpclient = new DefaultHttpClient();
HttpResponse response = httpclient.execute(get);

The problem is that the custom header "myHeader" is unavailable from my
route.  Is there another way to get to it?  

>From looking at the camel-cxf source code, I can see that the default header
strategy/binding logic only sets up predefined HTTP headers.  Is there a
reason why other headers aren't passed through as well?

Either way, what is the best approach to override this behavior?  

thanks

-
Ben O'Day

-- 
View this message in context: 
http://camel.465427.n5.nabble.com/camel-cxfrs-with-custom-http-headers-tp2636965p2636965.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: camel-cxfrs with custom http headers...

[boday]

Jiang, our client is sending some needed parameters in the HTTP header of
each request and I'm trying to figure out how to retrieve them in my Camel
processor.  Are they accessible from the camel exchange/message now?  If so,
how can I retrieve them?

thanks...

-
Ben O'Day

-- 
View this message in context: 
http://camel.465427.n5.nabble.com/camel-cxfrs-with-custom-http-headers-tp2636965p2638222.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: camel-cxfrs with custom http headers...

[boday]

It works great...thanks Willem.  Any idea when this can be rolled into a Fuse 
release?  My client (Alcatel-Lucent) will be eager to get this fix.

thanks again...

 __
Ben O'Day
Senior IT Consultant






From: Willem.Jiang [via Camel] 
To: boday 
Sent: Wed, August 18, 2010 8:40:04 PM
Subject: Re: camel-cxfrs with custom http headers...

 FYI, I committed the patch into Camel trunk, please check out the latest 
Camel 2.5-SNAPSHOT to verify it. 

Willem 

boday wrote: 

> Jiang, our client is sending some needed parameters in the HTTP header of 
> each request and I'm trying to figure out how to retrieve them in my Camel 
> processor.  Are they accessible from the camel exchange/message now?  If so, 
> how can I retrieve them? 
> 
> thanks... 
> 
> - 
> Ben O'Day 
> 



 
View message @ 
http://camel.465427.n5.nabble.com/camel-cxfrs-with-custom-http-headers-tp2636965p2640408.html
 
To unsubscribe from Re: camel-cxfrs with custom http headers..., click here. 


-
Ben O'Day
IT Consultant (subcontractor for Progress)

-- 
View this message in context: 
http://camel.465427.n5.nabble.com/camel-cxfrs-with-custom-http-headers-tp2636965p2641014.html
Sent from the Camel - Users mailing list archive at Nabble.com.

guaranteed message processing question...

[boday]

I'm using Camel2 in SMX 3.3.1.  I have a simple Camel route as follows...

from("activemq:inboundMessages?maxConcurrentConsumers=50")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finishedQueue");

How do I guarentee that a message will not be removed from the
"inboundMessages" queue until it reaches the "finishedQueue"?  I need to
make sure that messages are not lost in the event of a server crash/restart,
etc.  Currently, this process is loosing messages if I restart the server
midway through a load test.

Do I have to use a transactional client approach (if so, how exactly)?  I
tried this briefly but haven't been able to get it to work.  Is there
another option like using the JMS acknowledgement mode or something?

thanks

-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26202573.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: guaranteed message processing question...

[boday]

thanks guys.  I added some parameters to the queue to make the route
"transactional"...

from("activemq:inboundMessages?maxConcurrentConsumers=50&lazyCreateTransactionManager=true&transacted=true")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finishedQueue"); 

Now I'm getting duplicates delivered.  My test pumped 10k messages into the
queue via an external client.  During the test, I restarted SMX 2 times. 
After the test concluded, I had 10,003 messages in the finishedQueue.  This
is better than losing messages, but still not the desired behavior...any
ideas?

One more question, what happens during exception handling?  Suppose I have
the following policy to retry 3 times, then go to an error queue...

onException(Exception.class).handled(true).maximumRedeliveries(3).to("activemq:errorQueue");

I assume the message is still in the inbound queue until either delivered to
the finished queue or error queue, correct?  So at any given time the sum of
the inbound, finished and error queues should equal to the number of
messages sent, correct?

I'm working on a unit test for this now, but wanted to make sure I
understood the intended behavior first...

thanks in advance


-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26206508.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: guaranteed message processing question...

[boday]

Claus, my questions are really specific to how Camel wraps the underlying AMQ
technologies.  In particular, how Camel handles transactions and client
acknowledgements when exceptions are raised and redelivery policies applied
(using Camel's onException clause, etc).

Also, I'm not running in DUPS_OK mode explicitly (using AUTO_ACK mode).  But
this seems to be the behavior that I'm seeing...

On another note, I did purchase the "Camel In Action" pre-release (which I
see you are an author of) but haven't found much on this specific topic yet. 
Will subsequent chapters will cover this?

thanks again...


Claus Ibsen-2 wrote:
> 
> Hi
> 
> Could you ask / search at the AMQ forum. The questions is now more
> related to messaging in general how to do reliable bridges.
> I remember there is also a QoS term (Quality of Service) which specified
> such as
> - duplicates allowed
> - at most one
> - etc.
> 
> Try to see if that can help, looks as if duplicates allowed is the
> kind of mode you run with.
> 
> 


-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26270070.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: guaranteed message processing question...

[boday]

I've been performance testing my newly transacted route, but have seen a
significant performance drop.  

I assumed there would be a bit of overhead with using transactions, but my
route goes from ~250 message/second to ~50 messages/second when I add the
"?transacted=true" to it (no other changes).

Has anyone else seen this type of drop in throughput?  I'm hoping it has
something to do with the Tx manager configuration or something.  I'm using
the default tx.xml file that shipped with Servicemix 3.3.1...

thanks...

-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/guaranteed-message-processing-question...-tp26202573p26376095.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Camel route with CLIENT_ACKNOWLEDGE?

[boday]

how do I explicitly acknowledge a message in a Camel route (using
CLIENT_ACKNOWLEGE)?

I'm using Camel2 in SMX 3.3.1.  I have a simple Camel route as follows...

from("activemq:start?maxConcurrentConsumers=10")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finished");

I'd like to guarantee that a message isn't removed from the start queue
until it has been received by the finished queue (in case of server crash,
etc).  I have another 
http://old.nabble.com/guaranteed-message-processing-question...-td26202573.html
thread  going about this in regards to using transactions, but I'm having
some performance issues and would like to try this instead...

I know that I need to setup my "activemq" factory's ack mode to
"CLIENT_ACKNOWLEDGE", but then what?  Is there an API that I can call at the
end of route to do this...

from("activemq:start?maxConcurrentConsumers=10")
.process(new Processor1())
.process(new Processor2())
.to("activemq:finished")
.acknowledge();   <

thanks in advance...




-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/Camel-route-with-CLIENT_ACKNOWLEDGE--tp26395347p26395347.html
Sent from the Camel - Users mailing list archive at Nabble.com.

performance issues with JMS transacted route...

[boday]

I have the following JMS transacted route...  

from("activemq:start?maxConcurrentConsumers=50&transacted=true")
.process(new ValidateMessage())
.process(new SaveToDatabase())
.to("activemq:finishedQueue"); 

I assumed there would be a bit of overhead with using transactions, but my
route goes from ~250 message/second to ~50 messages/second when I add the
"?transacted=true" to it (no other changes).  

I've seen similar performance hits when I added xpath (content based
routing, etc) to my route (so I worked around it).  My processor
"SaveToDatabase" does connect to a database (spring JDBC stored procedure
call), but is not transactional itself.  

Overall, my goal is to simply make my route guarantee messaging (server
crashes, restarts, etc) and maintain its high throughput.  Any other ideas?

Has anyone else seen this type of drop in throughput with transacted
messages?  I'm hoping it has something to do with the Tx manager
configuration or something.  I'm using the default tx.xml file that shipped
with Servicemix 3.3.1/Camel 2.0...

note: this is a followup to another thread
(http://old.nabble.com/guaranteed-message-processing-question...-to26202573.html#a26270332)

thanks...

-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/performance-issues-with-JMS-transacted-route...-tp26579725p26579725.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: performance issues with JMS transacted route...

[boday]

thanks Claus, that is exactly what I wanted to hear...

here is my setup...any advice would be appreciated...

JDK 1.6, ServiceMix 3.3.1, Camel 2.0 (patched the
/hotdeploy/servicemix-camel.zip)
AMQ 5.2 (persistent, small 1k messages, 250+ msg/second, no XA Tx)
Jenks AMQ pool 2.1, Spring 2.5.6

http://old.nabble.com/file/p26594255/tx.xml tx.xml 
http://old.nabble.com/file/p26594255/activemq.xml activemq.xml 
http://old.nabble.com/file/p26594255/servicemix.xml servicemix.xml 

// setup the ActiveMQ component
ActiveMQConnectionFactory queueConnectionFactory = new
ActiveMQConnectionFactory("tcp://localhost:61616");
PooledConnectionFactory pcf = new
PooledConnectionFactory(queueConnectionFactory);
pcf.setMaxConnections(50);

ActiveMQComponent amq =
ActiveMQComponent.activeMQComponent("tcp://localhost:61616");
amq.setConnectionFactory(pcf);
amq.setMaxMessagesPerTask(-1);
amq.setMaxConcurrentConsumers(50);
amq.setCacheLevelName("CACHE_CONSUMER");
amq.setAcknowledgementModeName("AUTO_ACKNOWLEDGE");
getContext().addComponent("activemq", amq);

My route does the following...

@Produce(uri = "activemq:inbound")
ProducerTemplate inboundProducer;

@Consume(uri = "jetty:http://0.0.0.0:9001/inboundMessage/";)
@InOnly
public void onMessage(String message)
{
String addressID = getAttributeValueFromXML(message,"addr");

//set the JMSXGroupID appropriately to enable AMQ MessageGroups
(guaranteed ordering)
inboundProducer.sendBodyAndHeader(message, "JMSXGroupID", addressID);
}

from("activemq:inbound?transacted=true&maxConcurrentConsumers=50")
.process(new ValidateMessage())
.process(new SaveToDatabase())
.to("activemq:topic:outbound"); 

Questions...

how do I set the prefetch buffer?
what Tx manager should I be using and how do I set it up?
should I be using newer versions of any components?

thanks again...




-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/performance-issues-with-JMS-transacted-route...-tp26579725p26594255.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: performance issues with JMS transacted route...

[boday]

Claus, I tried the following...

-disabled the message groups
-setup the Spring JmsTransactionManager as follows...

...
ActiveMQPrefetchPolicy prefetchPolicy = new ActiveMQPrefetchPolicy();
prefetchPolicy.setQueuePrefetch(50);   //tried 0, 10, 1000, 5000

ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory();
connFactory.setBrokerURL(brokerURL);
connFactory.setPrefetchPolicy(prefetchPolicy);

PooledConnectionFactory pcf = new PooledConnectionFactory(connFactory);
pcf.setMaxConnections(50);

JmsTransactionManager jmsTransactionManager = new JmsTransactionManager();
jmsTransactionManager.setConnectionFactory(pcf);

JmsComponent component = ActiveMQComponent.jmsComponentTransacted(pcf,
jmsTransactionManager);
component.getConfiguration().setConcurrentConsumers(50);
this.getContext().addComponent("activemq", component);
...

But, I'm still see the same result.  It appears to be throttling down my
HTTP endpoint (maybe including it in the transaction scope??) as I'm only
able to receive ~10 msg/second now (vs. ~250 w/o a transacted route) into my
inbound queue.  They are being processed from the queue as fast as they are
coming in...just coming in too slow.  

Any other thoughts on this?

thanks again...

-
Ben - Senior Consultant

-- 
View this message in context: 
http://old.nabble.com/performance-issues-with-JMS-transacted-route...-tp26579725p26618905.html
Sent from the Camel - Users mailing list archive at Nabble.com.

start/stop routes programmatically

[boday]

I see that Camel 2.1 now support starting/stopping routes via JMX/APIs.  But,
how do I identify routes properly at runtime...

For example, assume I have the following route...

from("activemq:queue1").to("activemq:queue2");
from("activemq:queue2").to("activemq:queue3");

They show up in JMX as "route1" and "route2".  The description/endpointURI
fields definitely help, but might be problematic to use when there are a lot
of routes.  

The same applies to finding a routes with Camel APIs.  I can get a list of
routes using this...

getContext().getRouteCollection().getRoutes()

I see that I can then call the following on a RouteDefinition to get to the
first endpoint name

routeDefinition.getInputs().get(0).getLabel()

I guess I could use this name to identify a route and then call
stop()/start() on it...but this seems like a very indirect approach to
identify it?  Am I missing something?

What I'd like to do is give a route an explicit name that can be referenced
later (in JMX/programmatically)...maybe the DescriptionDefinition class
could help, but am not sure how to use it...

something along these lines is what I'm looking for...

setRouteName("InboundRoute").from("activemq:queue1").to("activemq:queue2");
setRouteName("ProcessingRoute").from("activemq:queue2").to("activemq:queue3");

thanks in advance...






-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.0
-- 
View this message in context: 
http://old.nabble.com/start-stop-routes-programmatically-tp26781725p26781725.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: start/stop routes programmatically

[boday]

perfect, that is what I was looking for...thanks guys


willem.jiang wrote:
> 
> If you are using camel 2.0, you may need to use id() dsl,
> but for camel 2.1 you can use the routeId() dsl.
> 
> You can find more information here[1]
> 
> [1] https://issues.apache.org/activemq/browse/CAMEL-2109
> 
> Willem
> 
> 


-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/start-stop-routes-programmatically-tp26781725p26798471.html
Sent from the Camel - Users mailing list archive at Nabble.com.

DefaultManagementLifecycleStrategy - Could not register service

[boday]

I'm sending high volume/small messages to an AMQ topic using the following
API inside of a Camel (2.1) processor...

exchange.getContext().createProducerTemplate().sendBodyAndHeaders("activemq:topic:portal",
messageXML, headers);

I get this warning message every few minutes (stack trace below).  Am I now
using the ProducerTemplate incorrectly or something?  

I also tried the following to define the ProducerTemplate upfront and then
reuse it for each message...but I was getting NullPointerExceptions...why
doesn't this work?

@Produce(uri = "activemq:topic:portal")
ProducerTemplate portalTemplate;
...
portalTemplate.sendBodyAndHeaders(messageXML, headers);

Any other ideas?  

thanks in advance...

 WARN  - DefaultManagementLifecycleStrategy - Could not register service:
Producer[activemq://topic:portal] as Service MBean.
javax.management.InstanceAlreadyExistsException:
org.apache.camel:context=BensLaptop/camelContext,type=producers,name=JmsProducer(0xf8e153)
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at
org.apache.camel.management.DefaultManagementAgent.registerMBeanWithServer(DefaultManagementAgent.java:298)
at
org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:210)
at
org.apache.camel.management.DefaultManagementAgent.register(DefaultManagementAgent.java:193)
at
org.apache.camel.management.ManagedManagementStrategy.manageNamedObject(ManagedManagementStrategy.java:69)
at
org.apache.camel.management.ManagedManagementStrategy.manageObject(ManagedManagementStrategy.java:62)
at
org.apache.camel.management.DefaultManagementLifecycleStrategy.onServiceAdd(DefaultManagementLifecycleStrategy.java:249)
at
org.apache.camel.impl.DefaultCamelContext.addService(DefaultCamelContext.java:631)
at
org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:198)
at
org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:133)
at
org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:160)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:98)
at
org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:97)
at
org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:206)
at
org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:202)




-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/DefaultManagementLifecycleStrategy---Could-not-register-service-tp26915869p26915869.html
Sent from the Camel - Users mailing list archive at Nabble.com.

context.stopRoute() API hangs periodically

[boday]

I'm using Camel 2.1 on SMX 3.3.1.  I'm using a timer to periodically
reprocess messages that I've sent to an error queue.  While reprocessing I
want to stop the main message processing route as follows (for some specific
client requirements)...

from("direct:main").routeId("mainRoute").process(MessageProcessor());

from("timer://ErrorMessageProcessor?period=6")
.bean(blockedNodeManager, "processBlockedQueue");

then, in blockedNodeManager stops the route, acts as a polling consumer and
then restarts the route...

ServiceStatus routeStatus = context.getRouteStatus("mainRoute");
if(routeStatus.isStoppable())
{
logger.info("stopping route..." + routeId);
context.stopRoute(routeId);
logger.info("route stopped..." + routeId);
}

while (true) 
{
String msg = consumer.receiveBody("activemq:errorQueue", 100,
String.class);
if (msg == null)
{
break;
}
producer.sendBody("direct:main", msg);
}

routeStatus = context.getRouteStatus("mainRoute");
if(routeStatus.isStartable())
{
logger.info("starting route..." + routeId);
context.startRoute(routeId);
logger.info("route started..." + routeId);
}

This seems to work fine for a while (up to 12 hours sometimes), but will
periodically hang while trying to stop the main route.  The last message I
see in my log file is"stopping route mainRoute"...it never reaches the
"route stopped" debug statement.  At this point, I'm hosed...I can't even
use JMX to restart the route.  I restart the server and all works fine again
(for a while)...

Am I using these APIs incorrectly?  If there is a better way to accomplish
this, let me know...

thanks in advance...




-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/context.stopRoute%28%29-API-hangs-periodically-tp27026922p27026922.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Issue trying to browse a activemq queue

[boday]

I just use JMX to get these statistics...

here is some raw code to create a connection and iterate over the AMQ queues
to find a given queues size...

note: be careful to close/resuse JMX connections or they will cause
OutOfMemoryExceptions on the MBeanServer

...

JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" +
machineName + ":" + port
+ "/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
conn = jmxc.getMBeanServerConnection();

ObjectName activeMQ = new
ObjectName("org.apache.activemq:BrokerName=localhost,Type=Broker");
BrokerViewMBean mbean = (BrokerViewMBean)
MBeanServerInvocationHandler.newProxyInstance(mbsc, activeMQ,
BrokerViewMBean.class, true);

//find the queue in question and return the size
for (ObjectName name : mbean.getQueues())
{
QueueViewMBean queueMbean = (QueueViewMBean)
MBeanServerInvocationHandler.newProxyInstance(mbsc,
name,QueueViewMBean.class, true);
   if (queueMbean.getName().equals(queueName))
{
  return queueMbean.getQueueSize();
}
}


-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/Issue-trying-to-browse-a-activemq-queue-tp27014479p27026944.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: context.stopRoute() API hangs periodically

[boday]

thanks for the feedback guys...

I think I found the issue.  My ErrorMessageProcessor runs every 60 seconds
and was periodically taking longer than 60 seconds to run.  Then, I end up
with multiple threads trying to stop/start the same route (not a good thing
I imagine).

So, I increased the time between runs and synchronized the method to make
sure it is single threaded.  Since then, I haven't had any issues.  I'm
hoping this was it, if not...I'll try to get a thread dump, etc.

Also, Claus when will Camel 2.2 be available?

thanks again...


-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/context.stopRoute%28%29-API-hangs-periodically-tp27026922p27082410.html
Sent from the Camel - Users mailing list archive at Nabble.com.

clustering Camel with AMQ/ServiceMix

[boday]

I'm trying to determine my options for clustering my SMX 3.3.1/Camel 2.1/AMQ
5.3 application.  I'm performing high volume message processing and I need
to cluster for high availability and horizontal scalability.

Here is basically what my application
does...HTTP->QUEUE->PROCESS->DATABASE->TOPIC

from("jetty:http://0.0.0.0/inbound";)
.to("activemq:inboundQueue");

from("activemq:inboundQueue?maxConcurrentConsumers=50")
.process(decode())
.process(transform())
.process(validate())
.process(saveToDatabase())
.to("activemq:topic:ouboundTopic");

So, I've read all the ServiceMix and AcitveMQ clustering pages, but am still
not sure which way to go.  

I know I can use a Master/Slave setup for HA, but that doesn't help with
scalability.

I've read about network of brokers, but am not sure how this applies.  For
example, if I deploy identical Camel routes on multiple nodes in a cluster,
how will they "interact" exactly?  If I point my HTTP producer at one node
(NodeA), which messages will get sent to NodeB?  Will the queues/topics be
shared between Node A/B...if so how, are messages split or duplicated? 
Also, how would an external client subscribe to my "outboundTopic" exactly
(and get all messages, etc)?

Alternatively, I've been thinking that I should just share a broker between
multiple ServiceMix instances.  That would be cleaner in that there would
only be one set of queues/topics to manage and I could scale by adding more
instances.  But, now I'm limited to the scalability of a single broker and
I'm back to a single point of failure...

If anyone can clarify the trade-offs for me...I'd appreciate it.







-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/clustering-Camel-with-AMQ-ServiceMix-tp27462441p27462441.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: OnException and log level

[boday]

Julien, I added the following to suppress these messages...

DefaultErrorHandlerBuilder errHandler = defaultErrorHandler();
errHandler.loggingLevel(LoggingLevel.DEBUG);
errHandler.getRedeliveryPolicy().setRetriesExhaustedLogLevel(LoggingLevel.DEBUG);
errHandler.getRedeliveryPolicy().setRetryAttemptedLogLevel(LoggingLevel.DEBUG);
errorHandler(errHandler);


-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/OnException-and-log-level-tp27714424p27721142.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.

DirectProducer - no consumers available on endpoint...message is lost

[boday]

I have a polling consumer setup to periodically (using camel-timer) retry
messages from an error queue. 

like this...

//initialized in constructor of polling consumer
private static ProducerTemplate producer = context.createProducerTemplate();
...
Exchange exchange = consumer.receive("activemq:errorQueue", 100);
String msgBody = exchange.getIn().getBody(String.class);
producer.sendBody("direct::Decoder", msgBody);

Every now and then, I get the following warning in my logs...

WARN  - DirectProducer - No consumers available on endpoint:
Endpoint[direct://Decoder]

Whenever this happens, the message being sent to this endpoint is lost!!!  I
know the endpoint is active and started, so what would cause it to have "no
consumers available"?  

Also, is there a way for my polling consumer to detect/handle this case so
that the message isn't lost and simply waits for an unavailable consumer to
become available?

thanks

-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/DirectProducer---no-consumers-available-on-endpoint...message-is-lost-tp27828628p27828628.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: DirectProducer - no consumers available on endpoint...message is lost

[boday]

sorry, the '::' was a typo in my post only...

the route works fine during normal execution...basically the following

from('activemq:inbound')
.to('direct:decode');

from('direct:decode')
.convertBodyTo(DOMSource.class)
.process(new DecodeProcessor())
.process(new ValidateProcessor())
.process(new PersistProcessor());

I'm using a single Camel context and have 50 MaxConcurrentConsumers set on
the AMQ component.  I'm doing high volume (~200 msg/s) message processing.

My polling consumer routes traffic to the 'direct:decode' endpoint so it
will get processed immediately (rather than going to the 'inbound' queue). 
It seems like this might be the issue in that both the inbound queue route
and my polling consumer are competing for the same 'direct:decode'
consumers??



-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/DirectProducer---no-consumers-available-on-endpoint...message-is-lost-tp27828628p27829374.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: DirectProducer - no consumers available on endpoint...message is lost

[boday]

yes, the retry does work fine 99% of the time...randomly we see this error
after many successful retries of of other messages, etc...

-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/DirectProducer---no-consumers-available-on-endpoint...message-is-lost-tp27828628p27830046.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: DirectProducer - no consumers available on endpoint...message is lost

[boday]

Sorry, please ignore my typos in the route (I use constants for all
endpoints, but tried to simplify it for the post).

Does "no consumers available" mean the route isn't active or that there
aren't any available threads?  Also, where does the message go exactly?  Is
there a way to handle this case in my route and send it to an error queue?

As for the ordering, I don't explicitly define this now.  But, I don't think
its an issue because everything works fine on startup...the issue only shows
up after hours of normal operation.

I do stop/start this first route on certain errors (database down, etc)...

from(INBOUND_QUEUE).routeId(INBOUND_QUEUE)
.to(DECODE_DIRECT);

with this code...

ServiceStatus routeStatus = 
context.getRouteStatus(INBOUND_QUEUE);
if(routeStatus.isStoppable())
{
context.stopRoute(INBOUND_QUEUE);
}

...//retry database connection, if successful, restart

if(routeStatus.isStartable())
{
context.startRoute(INBOUND_QUEUE);
}

But, I don't ever stop the DECODE_DIRECT route itself.  If I were stopping
that route, then I'd expect to see an error saying that the route hasn't
been started or is unavailable.

thanks again...



-
Ben - Senior Consultant
using SMX 3.3.1/Camel 2.1
-- 
View this message in context: 
http://old.nabble.com/DirectProducer---no-consumers-available-on-endpoint...message-is-lost-tp27828628p27838518.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Super basic questions

[boday]

in short, Camel parses the from/to URIs to call a Java API that knows how to
speak that specific component's language (File I/O, HDFS, etc).  

So, yes...formatting an appropriate (see camel-hdfs examples) URI to your
HDFS server will tell Camel to poll for files from "://in" directory and
send them to your HDFS server/directory.

Also, for remote apps...Camel allow you to interface as a consumer ('from'
clause) or a producer ('to' clause) with other applications over various
remote protocols (file, http, SOAP, REST, JMS, etc) without any changes
necessary on the remote app side.

Otherwise, if you want any app to use Camel directly, then you generally
just need camel-core and optionally any camel- jars on the
classpath and to bootstrap the context (spring or programmatically, etc)...

see these links for more info:

http://camel.apache.org/what-are-the-dependencies.html
http://camel.apache.org/running-camel-standalone-and-have-it-keep-running.html

good luck...


mister blinky wrote
> Sorry for the probably very fundamental nature of these questions, but i
> have been reading stuff online all day and still am not sure.
> 
> I am messing with Camel with HDFS, but my questions generally relate to
> using Camel to integrate any two (or more) systems that are remote (not on
> same machine, not running in same JVM). Assume that I am not using an ESB,
> because for my larger goal, an ESB is not an option. I need to get
> disparate systems to work with one another with no ESB.
> 
> Here's a simplified scenario.  I have a hadoop cluster and I have a server
> with a directory of files (dir name is "in"). As a PoC, I want to transfer
> those files to HDFS on my cluster and back to my server.
> 
> To transfer the file to HDFS on the cluster, I assume I would use the
> Camel-hdfs component like this:
> 
> from("file://in").to("hdfs://
> 
> :9000/tmp/test"); 
> 
> First question: I was confused initially (maybe i still am!) about how
> Camel would know *where* to transfer the file. My assumption is that I
> identify this by putting the IP addr of the remote system into the URL. Is
> this the general pattern?
> 
> Second question: What is that URL in the to() supposed to be? Is it the
> URL of a web service -- in this case  of HDFS's HTTP API? Or does Camel
> provide some web-addressable wrapper around services that are local to a
> machine?
> 
> Third question: To get remote systems working together using Camel,
> generally do i have to install something Camel-related on each
> participating machine? If so, what? For example (a wild stab) would I have
> to put the Camel-hdfs compontent jar in the classpath? Or??
> 
> Thanks for wading into my confusion.





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Super-basic-questions-tp5746261p5746263.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Super basic questions

[boday]

the camel-hdfs producer will throw an error if the send operation
fails...internally it just uses a standard HdfsOutputStream...so any step in
the route following this send would only be triggered upon successful send
operation, and you can use standard Camel onException/doCatch routing
exception handling to deal with errors, etc.


mister blinky wrote
> Thanks for your response. I think i got it. Let me state it to confirm,
> then I have a follow up question. 
> 
> So the 
/
> .to(
> 
> :
> 
> )
/
>  establishes a contract with Camel that a) the 
/
> endpoint-type
/
>  defines the message format among other things required by the endpoint;
> and b) the 
/
> URL-of-endpoint-API
/
>  establishes where the service is located and any other arguments that the
> service generally requires.
> 
> As such, in this one-way case, there is no need for Camel on the HDFS
> side, since, from HDFS's point of view, it's just getting a normal web API
> call. Correct?
*
> Here's my follow up.
*
>  What if I want the HDFS server to reply back to the calling server to say
> "ok i'm done and i failed/succeeded"? Assuming for the moment that there
> is no such thing on the remote side, how would one use Camel to facilitate
> not just the sending of the message to the API, but a response from the
> API saying it's done? Is that a case, what role would Camel play on the
> remote server? Is this a typical scenario?  
> 
> thank you.





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Super-basic-questions-tp5746261p5746288.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apache camel guaranteed delivery - multiple endpoints

[boday]

might take a look at the idempotent consumer pattern to filter
duplicates...it can be used with a repository (hazelcast, file, etc) to
survive servers restarts, etc

http://camel.apache.org/idempotent-consumer.html


MichaelAtSAG wrote
> All,
> 
> Please provide best practices on supporting guaranteed delivery with
> routes that containing multiple endpoints. Our system writes to a ehCache
> with persistence on the producer, then routes to all endpoints. Once all
> endpoints receive the message and the route completes, the message is
> removed from the producer ehCache queue. In case of producer failure, like
> JVM crash, the in-process messages are restored from ehCache upon restart.
> 
> One concern is that this can result in duplicates being sent to certain
> endpoints. For example, route contains endpoints a,b,c. A message is sent
> to a and b, and then the producer jvm crashes. The message remains in the
> producer guaranteed delivery queue and will be resubmitted once the
> producer JVM restarts and sent to a,b,c. This results in duplicates to a
> and b. 
> 
> Our Camel routes support duplicate message detection in the consumer. We
> encourage consumers to implement this pattern in case duplicates are not
> acceptable, yet it is a performance hit.
> 
> Any best practices on this? 
> Any examples or techniques on suppressing the resend from routes to
> endpoints a and b on the producer-side?
> 
> Thanks,
> Michael





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Apache-camel-guaranteed-delivery-multiple-endpoints-tp5746503p5746607.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apollo support in Camel

[boday]

now that openwire supported has been added to Apollo...anyone using it with
Camel JMS yet? 

if so, any advice/lessons learned?


Claus Ibsen-2 wrote
> A future release of Apollo will support JMS, and thus you can then use
> the camel-jms component.
> 
> 
> On Mon, Feb 6, 2012 at 11:23 PM, MichaelAtSAG <

> mebevilacqua@

> > wrote:
>> I saw the recent release of Apollo 1.0 and wanted to understand if this
>> can
>> be used as a component in Camel.
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Apollo-support-in-Camel-tp5461514p5461514.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
> 
> 
> 
> -- 
> Claus Ibsen
> -
> FuseSource
> Email: 

> cibsen@

> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Apollo-support-in-Camel-tp5461514p5750965.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Apollo support in Camel

[boday]

thanks Christian...I'm looking into migrating a Camel 2.13 app from using an
external AMQ 5.9 broker to Apollo...just curious what (if any) client
code/config changes are required


I've played with it just to see if it works.. anything specific?





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Apollo-support-in-Camel-tp5461514p5751296.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Avoiding Serialization in DefaultProducerTemplate

[boday]

what endpoint are you producing to?  if you are using ActiveMQ with the VM
transport, then you can disable serialization by setting this property on
the connection factory...

ActiveMQConnectionFactory factory = new
ActiveMQConnectionFactory("vm://localhost");
factory.setObjectMessageSerializationDefered(true);


Klaus777 wrote
> Hi,
> 
> It seems DefaultProducerTemplate serializes my exchange body even if I
> want
> to simply send a message within the same VM.
> 
> Is there a way to avoid this serialization?
> 
> Thanks,
> Max.





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Avoiding-Serialization-in-DefaultProducerTemplate-tp5751784p5751785.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Broker definition - activemq.xml - Fuse 6

[boday]

sounds like you need to either disable security in the activemq.xml or setup
your Camel AMQ factory with the user/pw provided...



  



  




Happy User wrote
> I am wondering if anyone can help me with this.
> 
> I am accessing a topic using activeme:topic:TopicName in the
> blueprint.xml(camel) and I have defined the broker for activemq(bean) in
> activemq.xml in FUSE 6. But I am getting exception as username null or
> password is invalid.
> 
> So, when I defined the activemq bean in my blueprint itself, everything
> worked fine. Since I have around 40 blueprints in my projects which access
> the same broker, it doesnt sound good to repeat the activemq bean
> definition in all the bluprints. It would be ideal to define broker bean
> in
> activemq.xml for once and just access the topci in all the blueprints I
> need.
> 
> So what am I missing here?





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Broker-definition-activemq-xml-Fuse-6-tp5751782p5751786.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Hawtio - ActiveMQ Issue

[boday]

you should direct this to the hawtio forum instead: http://hawt.io/community/


Happy User wrote
> Could someone help me with the below issue?
> 
> 1. I gave credentials in preferences for ActiveMQ.
> 2. I created Queue in the default broker and it was successful.
> 3. I selected the queue and clicked on the Send button in the top menu.
> The
> message was sent successful and I was able to even see my message when I
> clicked on Browse.
> 4. But weird stuff is, I clicked on the Operations menu and selected the
> 'Send text message(java.lang.string)' and gave the body as 'Hi' and
> clicked
> on Execute. But unfortunately, I got javax.jms.JMSSecurityException: User
> name [null] or password is invalid. For topic also, when I select send
> menu, message was sent successful. But getting same JMS exception when
> trying to send message through Operations menu.
> 
> -Thanks





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Hawtio-ActiveMQ-Issue-tp5751790p5751862.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Green test, but my RouteBuilder dosent run...

[boday]

remove these lines from testSendMatchingMessage(), this is handled by
CamelTestSupport automatically and causing duplicate "direct:start" routes

context.addRoutes(createRouteBuilder());
context.start(); 


Frankiboy wrote
> What is wrong here
> 
> Green test, but notning happens i dont get into my AddIpBean with this
> signature.
> 
> package dk.bankconnect.hub.unittest;
> 
> import org.apache.camel.ProducerTemplate;
> import org.apache.camel.builder.RouteBuilder;
> import org.apache.camel.component.mock.MockEndpoint;
> import org.apache.camel.test.CamelTestSupport;
> import org.junit.Test;
> 
> import dk.bankconnect.hub.processor.AddIpBean;
> 
> public class FilterTest extends CamelTestSupport {
> 
> String header = "
> 
> 
> abc
> 
> 
> 20
> 
> 
> 
> 
> ";
> 
> protected MockEndpoint resultEndpoint;
> 
> protected ProducerTemplate template;
> 
> @Test
> public void testSendMatchingMessage() throws Exception {
> context.addRoutes(createRouteBuilder());
> context.start();
> resultEndpoint = context.getEndpoint("mock:result",
> MockEndpoint.class);
> template = context.createProducerTemplate();
> String expectedBody = "
> 
> ";
> resultEndpoint.expectedBodiesReceived(expectedBody);
> template.setDefaultEndpoint(resultEndpoint);
> template.sendBodyAndHeader(expectedBody, header, header);
> resultEndpoint.assertIsSatisfied();
> context.stop();
> }
> 
> @Override
> protected RouteBuilder createRouteBuilder() {
> return new RouteBuilder() {
> public void configure() {
>
> from("direct:start").bean(AddIpBean.class).to("mock:result");
> }
> };
> }
> 
> }
> 
> I get this exception:
> 
> org.apache.camel.FailedToStartRouteException: Failed to start route route2
> because of Multiple consumers for the same endpoint is not allowed:
> Endpoint[direct://start]
>   at
> org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2161)
>   at
> org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2118)
> 
> 
> Frank 





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Green-test-but-my-RouteBuilder-dosent-run-tp5752583p5752587.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Green test, but my RouteBuilder dosent run...

[boday]

the issue is also with your template/mock setup...you are mixing concepts a
bit...

remove the declarations outside of the method and just do something like
this...

MockEndpoint mock = getMockEndpoint("mock:result");
mock.expectedBodiesReceived("expected result");

template.sendBodyAndHeader("direct:start",
"message","headerName","headerValue");

assertMockEndpointsSatisfied();


Frankiboy wrote
> I have allerady tried this, i got green ligth but newer go into my
> AddIpBean.
> 
> Its like the Route newer startsup , thats why i tried.
> 
> context.addRoutes(createRouteBuilder()); 
> context.start();





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/Green-test-but-my-RouteBuilder-dosent-run-tp5752583p5752605.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: question about seda component

[boday]

yep, SEDA uses a BlockingQueue and is asynchronous from the producer (the
process that sends to your seda:trades endpoint)...so if you specify
concurrentConsumers=100 then it will use 100 threads to pull messages off
the queue and execute your route in parallel...


simafengyun wrote
> Hi All,
> 
> I have one question about the seda component. there is a parameter
> concurrentConsumers. I am confused about it. Sample code as below
> 
> 
> 
>  uri="seda:trades?size=100&concurrentConsumers=100
*
> "/>
> 
> 
> 
> 
> 
> 
> concurrentConsumers=100, does it means it the route publishData will use
> 100 threads to receive data from the seda?





-
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: 
http://camel.465427.n5.nabble.com/question-about-seda-component-tp5752720p5752721.html
Sent from the Camel - Users mailing list archive at Nabble.com.