Re: Camel - Making parallel GET requests and aggregating the results via Dynamic Routes using Java DSL
You can use JsonPathExpression in your split body and allow parallel
execution using executorServiceRef, following is a sample skeleton -
from("jetty:http://localhost:/hello";).split(new
JsonPathExpression("$.data[*]")).executorServiceRef(MY_EXECUTOR_REF).to(...)
This will run your individual json tokens in parallel which you can use
to send out independent GET requests to desired endpoints. You'll have
to use aggregationStrategy though at end to accumulate the results (if
you wish to combine some data) using strategyRef option. There are
number of examples at http://camel.apache.org/splitter.html - towards
the end of this documentation you'll see custome aggregationStrategy
example as well.
Just to clarify, JsonPathExpression comes from
org.apache.camel.model.language.JsonPathExpression class (You'll have to
import it in your route builder).
Thanks,
Avnish Pundir
On 18-10-2016 16:23, Debraj Manna wrote:
I am receiving a request on a Jetty Http Endpoint. The request body
contains some urls in the request body. I have to make a GET request to
those urls. Then aggregate the results from each of the GET request and
return it to the caller.
Request Body:-
{
"data" : [
{"name" : "Hello", "url" : "http://server1"}
{"name" : "Hello2", "url" : "http://server2"}
]
}
One way I can think of doing this is like below:-
from("jetty:http://localhost:/hello";).process(new Processor() {
public void process(Exchange exchange) throws Exception {
// 1. Make the GET request in parallel using ThreadPoolExecutor
// 2. Wait for all calls to finish. Collate the response
// 3. Write it to exchange.getOut().setBody
}
})
Can some one let me know if this can be achieved via Java DSL using camel
dynamic Routes, splitter & aggregator so that my Processor remains
relatively small?
I am using camel 2.16.3.
--
--
Disclaimer: The information contained in this communication is
confidential, private, proprietary, or otherwise privileged and is intended
only for the use of the addressee.Unauthorized use, disclosure,
distribution or copying is strictly prohibited and may be unlawful. If you
have received this communication in error, please delete this message and
notify the sender immediately - Samin TekMindz India Pvt.Ltd.
--
Re: apache camel returns statusCode:500
Looks like your web-service is expecting special headers (either content
type or something alike), you can try setting the content type and also
set the request type to POST explicitly (in one of the peculiar use case
I had to use it explicitly) using following code:
.setHeader(Exchange.HTTP_METHOD, constant("POST"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
I'll still recommend that you either look into your server logs to find
out exact root cause or enable the TRACE logging for apache http package
(assuming you are using http or http4 component) to see what's going on
the wire, compare it what postman is sending and adapt your route
accordingly.
Thanks,
Avnish Pundir
On 18-10-2016 07:06, meng wrote:
Hi All,
I'm the new of camel. I now use apache camel to call several our
webservices.
Here is my code:
public void configure() throws Exception{
from("direct:start")
.marshal()
.string(UTF_8)
.to(TARGET http)
.process(new Processor() {
@Override
public void process(Exchange exchange) throws
Exception {
String message =
exchange.getIn().getBody(String.class);
System.out.println(message);
}
})
.to("stream:out");
I'm sure there is no problem about the target endpoint, for I can get
correct response using postman.
But camel give me 500 error.
Caused by: org.apache.camel.http.common.HttpOperationFailedException: HTTP
operation failed invoking http:// with statusCode: 500
at
org.apache.camel.component.http.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:240)
~[camel-http-2.16.1.jar:2.16.1]
at
org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:162)
~[camel-http-2.16.1.jar:2.16.1]
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
~[camel-core-2.16.1.jar:2.16.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
~[camel-core-2.16.1.jar:2.16.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:68)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:412)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:380)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:270)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:380)
~[camel-core-2.16.1.jar:2.16.1]
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:238)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:128)
~[camel-core-2.16.1.jar:2.16.1]
at
org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:132)
~[camel-core-2.16.1.jar:2.16.1]
... 16 common frames omitted
Could you help what's my problem or how can I debug it?
Thanks a lot!
--
View this message in context:
http://camel.465427.n5.nabble.com/apache-camel-returns-statusCode-500-tp578.html
Sent from the Camel - Users mailing list archive at Nabble.com.
--
--
Disclaimer: The information contained in this communication is
confidential, private, proprietary, or otherwise privileged and is intended
only for the use of the addressee.Unauthorized use, disclosure,
distribution or copying is strictly prohibited
netty4(-http) producer workerGroup and producerPool
Hi Everyone, I am trying to use netty4-http (in producer mode) to connect to remote services. Apparently in my use case there are number of different remote services (more than 30 in total) I have to talk to. To keep the threads management under direct control, I was hoping to use same thread pool for all these endpoints. At a glance it looked like sharing workerGroup will do the trick. However from the documentation it feels like this option is for consumer (at least from the examples in http://camel.apache.org/netty4.html) whereas for producer we should be using producerPool* properties. I need guidance on following from the community experts - 1. Is the above understanding correct, i.e. workerGroup can be used ONLY with consumer? If not what impact will it have if used with producer and more importantly how does it impact/influence producerPool. 2. Is it possible to use single executor across all my netty4-http producers? I don't see usual option of setting executorServiceRef option with either netty4 or nett4-http. -- Thanks, *Avnish Pundir* -- -- Disclaimer: The information contained in this communication is confidential, private, proprietary, or otherwise privileged and is intended only for the use of the addressee.Unauthorized use, disclosure, distribution or copying is strictly prohibited and may be unlawful. If you have received this communication in error, please delete this message and notify the sender immediately - Samin TekMindz India Pvt.Ltd. --
Re: netty4(-http) producer workerGroup and producerPool
Hi Willem, I just tried sharing the workerpool as per docs of netty4 (http://camel.apache.org/netty4.html) however the class mentioned in example (org.jboss.netty.channel.socket.nio.WorkerPool) for sharing of worker pools doesn't exist in netty4. I have camel-netty4 as a dependency in maven and "netty-all(4.0.34.FINAL)" is present on my project classpath as a transitive dependency. The mentioned class exists in netty3 as far as I can tell from googling around. Looks to me docs for netty4 are still pointing to camel-netty documentation for example section at least, as the example mention "boss and worker thread pools" whereas the options mention bossGroup and workerGroup (not pools). You did mentioned that you have updated wiki doc, if it's relevant can you please share updated doc link or point me in right direction for sharing workerGroup with netty4? Thanks, Avnish Pundir On 03-11-2016 15:14, Willem Jiang wrote: Hi, I just checked the code of camel-netty4, the workerGroup can be share between the consumers and producers if the option is specified otherwise camel creates a new workerGroup per consumer or producer. The wiki doc is updated, I will commit a quick fix to the camel netty component. -- Willem Jiang Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On November 3, 2016 at 12:55:34 PM, Pundir Avnish (avnish.pun...@tekmindz.com) wrote: Hi Everyone, I am trying to use netty4-http (in producer mode) to connect to remote services. Apparently in my use case there are number of different remote services (more than 30 in total) I have to talk to. To keep the threads management under direct control, I was hoping to use same thread pool for all these endpoints. At a glance it looked like sharing workerGroup will do the trick. However from the documentation it feels like this option is for consumer (at least from the examples in http://camel.apache.org/netty4.html) whereas for producer we should be using producerPool* properties. I need guidance on following from the community experts - 1. Is the above understanding correct, i.e. workerGroup can be used ONLY with consumer? If not what impact will it have if used with producer and more importantly how does it impact/influence producerPool. 2. Is it possible to use single executor across all my netty4-http producers? I don't see usual option of setting executorServiceRef option with either netty4 or nett4-http. -- Thanks, *Avnish Pundir* -- -- -- Disclaimer: The information contained in this communication is confidential, private, proprietary, or otherwise privileged and is intended only for the use of the addressee.Unauthorized use, disclosure, distribution or copying is strictly prohibited and may be unlawful. If you have received this communication in error, please delete this message and notify the sender immediately - Samin TekMindz India Pvt.Ltd. --
Re: netty4(-http) producer workerGroup and producerPool
Found it from source code, the example for netty4 should be following: class="org.apache.camel.component.netty4.NettyWorkerPoolBuilder"> factory-bean="poolBuilder" factory-method="build" destroy-method="destroy"> Then in the Camel routes we can refer to this worker pools by configuring the workerGroup option in the URI as shown below: ... Thanks, Avnish Pundir On 07-11-2016 14:44, Avnish Pundir wrote: Hi Willem, I just tried sharing the workerpool as per docs of netty4 (http://camel.apache.org/netty4.html) however the class mentioned in example (org.jboss.netty.channel.socket.nio.WorkerPool) for sharing of worker pools doesn't exist in netty4. I have camel-netty4 as a dependency in maven and "netty-all(4.0.34.FINAL)" is present on my project classpath as a transitive dependency. The mentioned class exists in netty3 as far as I can tell from googling around. Looks to me docs for netty4 are still pointing to camel-netty documentation for example section at least, as the example mention "boss and worker thread pools" whereas the options mention bossGroup and workerGroup (not pools). You did mentioned that you have updated wiki doc, if it's relevant can you please share updated doc link or point me in right direction for sharing workerGroup with netty4? Thanks, Avnish Pundir On 03-11-2016 15:14, Willem Jiang wrote: Hi, I just checked the code of camel-netty4, the workerGroup can be share between the consumers and producers if the option is specified otherwise camel creates a new workerGroup per consumer or producer. The wiki doc is updated, I will commit a quick fix to the camel netty component. -- Willem Jiang Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On November 3, 2016 at 12:55:34 PM, Pundir Avnish (avnish.pun...@tekmindz.com) wrote: Hi Everyone, I am trying to use netty4-http (in producer mode) to connect to remote services. Apparently in my use case there are number of different remote services (more than 30 in total) I have to talk to. To keep the threads management under direct control, I was hoping to use same thread pool for all these endpoints. At a glance it looked like sharing workerGroup will do the trick. However from the documentation it feels like this option is for consumer (at least from the examples in http://camel.apache.org/netty4.html) whereas for producer we should be using producerPool* properties. I need guidance on following from the community experts - 1. Is the above understanding correct, i.e. workerGroup can be used ONLY with consumer? If not what impact will it have if used with producer and more importantly how does it impact/influence producerPool. 2. Is it possible to use single executor across all my netty4-http producers? I don't see usual option of setting executorServiceRef option with either netty4 or nett4-http. -- Thanks, *Avnish Pundir* -- -- -- Disclaimer: The information contained in this communication is confidential, private, proprietary, or otherwise privileged and is intended only for the use of the addressee.Unauthorized use, disclosure, distribution or copying is strictly prohibited and may be unlawful. If you have received this communication in error, please delete this message and notify the sender immediately - Samin TekMindz India Pvt.Ltd. --
sjms creating too many unnecessary threadpools and proposal for fixing it
Hi, In my spring+camel integration application we are using SJMS extensively. There are more than 100 producers and more than 50 consumers that we have in our application. Recently I started discovering the usage of threads and noticed from JMX console there are way t many threadpools most of which are SJMS. All our SJMS consumers are transacted thus we are processing all messages synchronously using the artemis consumer thread. Further digging down in the SjmsConsumer I found that the executor threadpool is created even if this endpoint (thus consumer) is configured for synchronous processing. Further dig down in the code confirms that SjmsConsumer itself doesn't uses this executor and it's passed to InOnlyMessageHandler or InOutMessageHandler -- both of which do not use this executor if the processing is to have transaction support (isTransacted()) or configured for synchronous processing (isSynchronous()). This essentially means that with each new SjmsConsumer, a new ThreadPoolExecutor object is created which is never used for transacted or synchronous consumers. Although all these ThreadPoolExecutor do not have any threads in their pool since this is never used (PoolSize is confirmed to 0 all the times from JMX for these pools) but seeing so many threadpool objects itself made me uncomfortable in first spot. It's only after this much digging that I figured out that it's only ThreadPoolExecutor object without any active thread which we have in our application. IMHO, it makes sense to enhance SjmsConsumer to skip creation of this executor in doStart method if it's transacted or synchronous consumer. I can see "if (this.executor != null)" checks in SjmsConsumer already (e.g. doStop method) thus this should not impact the flow in any way (may be we'll have to add few more checks like this). If core developer agrees, I can take up this enhancements with a new pull request. (My another objective is to dynamically increase/decrease consumer count for SJMS consumer, however that I'll take up in a separate thread). -- Thanks, *Avnish Pundir* -- -- Disclaimer: The information contained in this communication is confidential, private, proprietary, or otherwise privileged and is intended only for the use of the addressee.Unauthorized use, disclosure, distribution or copying is strictly prohibited and may be unlawful. If you have received this communication in error, please delete this message and notify the sender immediately - Samin TekMindz India Pvt.Ltd. --
Re: mutually-authenticated SSL with websockets
I recently did something very similar (not with websocket, instead with
netty4-http though). However looking at websocket docs, its very much
similar. You'll have to use sslContextParameters endpoint option to
specify the contextparameter. In my case I created it as a (spring) bean
and provided the bean name in this parameter. The way I created this
bean is following:
public SSLContextParameters myMutualAuthParams() {
// setup keystore having public/private key.
// This can be used as trust store (for trusting external certs)
//and/or can be used for our own certificate for mutual certauth
KeyStoreParameters ksp = new KeyStoreParameters();
ksp.setResource("/ssl/clientcert.pfx"); // classpath resource
ksp.setPassword("pfxpassword"); // change as per your setup
ksp.setType("pkcs12");
// supply private key pass that shall be used for mutual auth
KeyManagersParameters pkParams = new KeyManagersParameters();
pkParams.setKeyPassword("pfxpassword"); // change as per your setup
pkParams.setKeyStore(ksp);
SSLContextParameters scp = new SSLContextParameters();
scp.setKeyManagers(pkParams);
// this is NOT needed for pkcs12 but you may need it for
// jks keystore types having multiple private keys
// scp.setCertAlias("");
// this SSLContextParameters will use JRE default keystore
//for trusting incoming certificates
// If you want to override trust store, use following:
// TrustManagersParameters tmp = new TrustManagersParameters();
// tmp.setKeyStore(ksp);
// scp.setTrustManagers(tmp);
// Ofcourse you can use *NEW* KeyStoreParameters as your TrustManager
// with this SSLContextParameters
return scp;
}
Hope it saves time for someone as I ended up spending more than a day to
get it right after going through docs again and again and trying hit and
trials before I get it right.
Thanks,
*Avnish Pundir*
On 13-12-2016 23:00, Zoran Regvart wrote:
Hi Mark,
there is the `clientAuthentication` parameter in `serverParameters` of
`SSLContextParameters` that you can set to `REQUIRE`. I wager adding
SSLContextParameters instance to registry and referencing in the
component/endpoint configuration should do the trick.
zoran
[1]http://camel.apache.org/camel-configuration-utilities.html#CamelConfigurationUtilities-SSLContextServerParameters
On Tue, Dec 13, 2016 at 6:24 PM, Mark wrote:
I have a requirement for mutually-authenticated SSL with a websocket. Does
the Camel-Websocket component support this functionality? According to the
camel websocket page, the SSL params are for consumers only which makes me
think that 2-way SSL isn't supported.
Cheers,
Mark
--
--
Disclaimer: The information contained in this communication is
confidential, private, proprietary, or otherwise privileged and is intended
only for the use of the addressee.Unauthorized use, disclosure,
distribution or copying is strictly prohibited and may be unlawful. If you
have received this communication in error, please delete this message and
notify the sender immediately - Samin TekMindz India Pvt.Ltd.
--
