Concurrent modification while ingesting to elasticsearch
I have a route which will ingest data to elasticsearch. But it is throwing
concurrentModificationException while ingesting to elasticsearch
from("file://somefile")
.multicast()
.to("seda:someroute?concurrentConsumers=8","seda:elasticsearchStore?concurrentConsumers=8");
from("seda:elasticsearchStore?concurrentConsumers=8")
.log("Ingesting to elasticsearch")
.to("elasticsearch://cluster?operation=index&ip=localhost&port=9300"");
from("seda:someroute?concurrentConsumers=8")
.to("someroute")
Below is the stacktrace for concurrent modification Exception
elasticSearch:159 - java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
at
java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
at
java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
at java.util.AbstractMap.toString(AbstractMap.java:518)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at
org.apache.camel.builder.BinaryPredicateSupport.matchesReturningFailureMessage(BinaryPredicateSupport.java:63)
at
org.apache.camel.builder.BinaryPredicateSupport.matches(BinaryPredicateSupport.java:50)
at
org.apache.camel.processor.FilterProcessor.matches(FilterProcessor.java:65)
at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:93)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:827)
at
org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:85)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:320)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context:
http://camel.465427.n5.nabble.com/Concurrent-modification-while-ingesting-to-elasticsearch-tp5787540.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Fuse Blueprint properties override
Hi, In Fuse it looks as though if you have a property set in xml in cm:property-placeholder then you cannot override it in persistent.id.cfg file in /etc. I could see the properties in there but they appeared not to be used by the route, it just used the ones set in the xml. Is this true, if so I couldn't see it documented anywhere? O. -- View this message in context: http://camel.465427.n5.nabble.com/Fuse-Blueprint-properties-override-tp5787541.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Fuse Blueprint properties override
Hello Did you use update-strategy="reload" in ? See technical explanation here: http://ggrzybek.blogspot.com/2015/12/camel-blueprint-test-support.html regards Grzegorz Grzybek 2016-09-14 11:47 GMT+02:00 owain : > Hi, > > In Fuse it looks as though if you have a property set in xml in > cm:property-placeholder then you cannot override it in persistent.id.cfg > file in /etc. I could see the properties in there but they appeared not to > be used by the route, it just used the ones set in the xml. > > Is this true, if so I couldn't see it documented anywhere? > > O. > > > > -- > View this message in context: http://camel.465427.n5.nabble. > com/Fuse-Blueprint-properties-override-tp5787541.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Issue with Request-Reply pattern in JMS camel 2.10
Hi All,
Iam trying to achieve the below scenario.
I have a switchyard application with soap binding which puts the request in
request Queue and waits for the response from response queue,request-reply
pattern.i have another route In-only mep which picks the message from a
queue process it and put the response in response queue both are separate
threads.I have set the message ID as correlation ID in the route.But iam
getting the response same as the request when i trigger soap request.Can you
please tell me what is wrong?
Switchyard.xml
http://docs.oasis-open.org/ns/opencsa/sca/200912";
xmlns:soap="urn:switchyard-component-soap:config:1.1"
xmlns:sy="urn:switchyard-config:switchyard:1.1"
name="switchyard-example-Test-Inout"
targetNamespace="urn:com.example.switchyard:switchyard-example-Test-Inout:1.0">
wsdl/CoreInsuranceSearchInterface.wsdl
switchyard-example-Test-Inout
Test.in.queue
#ConnectionFactory
Test.out.queue.debug
5000
Test.in.queue
#ConnectionFactory
Test.out.queue.debug
#ConnectionFactory
route.xml:
http://camel.apache.org/schema/spring";>
routeTwo.xml 2:
http://camel.apache.org/schema/spring";>
You get some trailing whitespace
characters.
${headers.JMSMessageID}
--
View this message in context:
http://camel.465427.n5.nabble.com/Issue-with-Request-Reply-pattern-in-JMS-camel-2-10-tp5787542.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Fuse Blueprint properties override
As Grzegorz pointed out you have to have the reload flag in to hot swap values. And it's important that you make sure you specify the 1.1.0 of cm in your header as the reload doesn't work earlier. Having said that, if you change the value in the .cfg file and then start you Fuse instance it should preferentially use the .cfg values. Is that much happening? Also, when you look at the Header to make sure the reload works. http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd";> //May be obvious but make sure the .cfg is not part of the PID. Otherwise if the PID below were foo.bar.cfg it would be looking in the etc directory for a file called foo.bar.cfg.cfg. On Wed, Sep 14, 2016 at 6:28 AM, Grzegorz Grzybek wrote: > Hello > > Did you use update-strategy="reload" in ? > See technical explanation here: > http://ggrzybek.blogspot.com/2015/12/camel-blueprint-test-support.html > > regards > Grzegorz Grzybek > > 2016-09-14 11:47 GMT+02:00 owain : > > > Hi, > > > > In Fuse it looks as though if you have a property set in xml in > > cm:property-placeholder then you cannot override it in persistent.id.cfg > > file in /etc. I could see the properties in there but they appeared not > to > > be used by the route, it just used the ones set in the xml. > > > > Is this true, if so I couldn't see it documented anywhere? > > > > O. > > > > > > > > -- > > View this message in context: http://camel.465427.n5.nabble. > > com/Fuse-Blueprint-properties-override-tp5787541.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > >
camel seda queue log
I am using simple seda route and seeing that seda is using shared queue in log. Endpoint Endpoint[seda://elasticSearchStore?concurrentConsumers=5&queueFactory=%23elasticsearchStoreLinkedQueue] is using shared queue: seda://elasticSearchStore with size: 2147483647 2016-09-14 21:33:51,952 INFO SedaEndpoint:170 - Endpoint Endpoint[seda://splunkStore?concurrentConsumers=10&queueFactory=%23splunkStoreLinkedQueue] is using shared queue: seda://splunkStore with size: 2147483647 Why it is using shared queue, even though i have defined the queue. Any suggestions will be more helpfule. -- View this message in context: http://camel.465427.n5.nabble.com/camel-seda-queue-log-tp5787556.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel seda queue log
I'm not sure what is happening inside your queueFactory but is there a particular reason you need a queue factory and not just use the default queue? There could be, obviously, if you need to create some sort of priority queue or other mechanism but it isn't necessary otherwise. Brad On Wed, Sep 14, 2016 at 3:40 PM, Akram wrote: > I am using simple seda route and seeing that seda is using shared queue in > log. > > Endpoint > Endpoint[seda://elasticSearchStore?concurrentConsumers=5&queueFactory=% > 23elasticsearchStoreLinkedQueue] > is using shared queue: seda://elasticSearchStore with size: 2147483647 > 2016-09-14 21:33:51,952 INFO SedaEndpoint:170 - Endpoint > Endpoint[seda://splunkStore?concurrentConsumers=10&queueFactory=% > 23splunkStoreLinkedQueue] > is using shared queue: seda://splunkStore with size: 2147483647 > > Why it is using shared queue, even though i have defined the queue. Any > suggestions will be more helpfule. > > > > -- > View this message in context: http://camel.465427.n5.nabble. > com/camel-seda-queue-log-tp5787556.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: camel seda queue log
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/component/seda/SedaEndpoint.java By the way if you look in there you can see where the log statement comes from in the createQueue method. On Wed, Sep 14, 2016 at 3:40 PM, Akram wrote: > I am using simple seda route and seeing that seda is using shared queue in > log. > > Endpoint > Endpoint[seda://elasticSearchStore?concurrentConsumers=5&queueFactory=% > 23elasticsearchStoreLinkedQueue] > is using shared queue: seda://elasticSearchStore with size: 2147483647 > 2016-09-14 21:33:51,952 INFO SedaEndpoint:170 - Endpoint > Endpoint[seda://splunkStore?concurrentConsumers=10&queueFactory=% > 23splunkStoreLinkedQueue] > is using shared queue: seda://splunkStore with size: 2147483647 > > Why it is using shared queue, even though i have defined the queue. Any > suggestions will be more helpfule. > > > > -- > View this message in context: http://camel.465427.n5.nabble. > com/camel-seda-queue-log-tp5787556.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: camel seda queue log
I have tried using default queue mechanism. There also i am seeing the same
log "using shared queue". So trying to understand what it actually does
mean.
Also I am sending same object to two different seda routes using multicast.
I wants to understand whether both routes use same object reference, as it
is using the shared queue.
Here is my actual route.
from("file://somefile")
.split(bodyAs(String.class).tokenize("\n"))
.multicast().to("seda:splunkstore","seda:elasticsearchstore");
--
View this message in context:
http://camel.465427.n5.nabble.com/camel-seda-queue-log-tp5787556p5787559.html
Sent from the Camel - Users mailing list archive at Nabble.com.
How to launch
Hi, I investigated and I would like to know how I sould do the next EIP I have 20 scheduler task that launch a task, each with different data but using the same component. The component is a custom component that connects to a remote service to read data. I would like to instantiate only one component for this read tasks. Example schedule1->data1->readcomponent->persist schedule2->data2->readcomponent->persist schedule3->data3->readcomponent->persist The readcomponent supports SchedulerPoooling but I don't want to use one instance by schedule because "readcomponent" is too heavy. Is it correct if I develop "readcomponent" for using an external service (Spring @Service) that manages only one connection and do the heavy work. Regards -- Alberto
Re: camel seda queue log
Yes the multicast does a shallow copy so its the same reference.
You can use onPrepareRef where you can write some code that performs a
deep copy of the message.
http://camel.apache.org/multicast
On Wed, Sep 14, 2016 at 11:31 PM, Akram wrote:
> I have tried using default queue mechanism. There also i am seeing the same
> log "using shared queue". So trying to understand what it actually does
> mean.
>
> Also I am sending same object to two different seda routes using multicast.
> I wants to understand whether both routes use same object reference, as it
> is using the shared queue.
>
> Here is my actual route.
>
> from("file://somefile")
> .split(bodyAs(String.class).tokenize("\n"))
> .multicast().to("seda:splunkstore","seda:elasticsearchstore");
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/camel-seda-queue-log-tp5787556p5787559.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
