Thanks Scott, for the info.
Indeed - it looks like I have run into CAMEL-7501.
Only one election policy- so one route is actually causing another route
to be shut down.
On 6/27/14, 2:53 PM, "Scott Stults" <sstu...@opensourceconnections.com>
wrote:
>I was doing something similar, and what I noticed in JConsole was that
>there was only ever one zookeeper election route created even when I had
>multiple route policies. There are two similar Jira issues related to
>this: CAMEL-7501 and CAMEL-5627. They might be the source of your trouble.
>
>Hope that helps. If it really is a bug in the zk routing policy then that
>might save you some hair-pulling in your own code.
>
>-Scott
>
>
>On Fri, Jun 27, 2014 at 5:41 PM, Michael Moores <mmoo...@realnetworks.com>
>wrote:
>
>> I tried an experiment that seems to point to a race condition in the
>>Camel
>> ZooKeeper component.
>>
>> I initialized the routes with autoStart=false, then started each route
>> manually.
>> When I do this everything works fine - no immediate shutdown of one of
>>the
>> routes.. I think it might have something to do with
>> ZooKeeperElection.awaitElectionResults(), but not sure.
>>
>>
>>
>> On 6/27/14, 2:00 PM, "Michael Moores" <mmoo...@realnetworks.com> wrote:
>>
>> >I have two routes that I integrated with ZooKeeperRoutePolicy to
>> >establish basic distributed route election.
>> >(Using camel 2.13.1)
>> >
>> >When used with one route instance, election/failover works fine.
>> >When I set up multiple routes with multiple instances of
>> >ZooKeeperRoutePolicy, one route almost always gets immediately shut
>>down
>> >but NOT 100% of the time.
>> >The behavior seems nondeterministic- like a race condition.
>> >
>> >Each RouteBuilder instance has a separate instance of a
>> >ZooKeeperRoutePolicy that uses the same server but a different
>>zookeeper
>> >znode:
>> >Like this:
>> >Route 1: zookeeper URI: localhost:2181/scheduler-node-election/test
>> >Route 2: zookeeper URI:
>>localhost:2181/scheduler-node-election/googleplay
>> >
>> >In the zookeeper CLI:
>> >[zk: localhost:2181(CONNECTED) 6] ls /scheduler-node-election
>> >[googleplay, test]
>> >
>> >(I have one zookeeper node running)
>> >
>> >
>> >I¹m seeing that most of the time one route (either one) just gets
>> >shutdown during Spring initialization, while the other one runs. But
>> >once in a while, both routes will run.
>> >In this case the ³googleplay² route is shut down and the associated
>> >session is closed.
>> >
>> >Here¹s the Camel/Zookeeper logging:
>> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Initiating
>> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@733ed393
>> >[//googleplay-#1] org.apache.zookeeper.ZooKeeper : Initiating
>> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@21b49301
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt
>>to
>> >authenticate using SASL (unknown error)
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt
>>to
>> >authenticate using SASL (unknown error)
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>> >0x146de3b813e0032, negotiated timeout = 5000
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>> >0x146de3b813e0031, negotiated timeout = 5000
>> >[timer://test-#2] o.a.c.c.zookeeper.ZookeeperProducer : Node
>>
>>>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a26
>>>23
>> >690' did not exist, creating it.
>> >[//googleplay-#1] o.a.c.c.zookeeper.ZookeeperProducer : Node
>>
>>>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b
>>>44
>> >e008854db' did not exist, creating it.
>> >[timer://test-#2] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>> >node
>>
>>>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a26
>>>23
>> >690' has been created
>> >[//googleplay-#1] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>> >node
>>
>>>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b
>>>44
>> >e008854db' has been created
>> >[//googleplay-#1] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos started and consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Starting
>>to
>> >graceful shutdown 1 routes (timeout 300 seconds)
>> >[ShutdownTask-#4] o.a.camel.impl.DefaultShutdownStrategy : Route:
>> >election-route-localhos shutdown complete, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Graceful
>> >shutdown of 1 routes completed in 0 seconds
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos is stopped, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[-#1-EventThread] org.apache.zookeeper.ClientCnxn :
>>EventThread
>> >shut down
>> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Session:
>> >0x146de3b813e0031 closed
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos is shutdown and removed, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos started and consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/test]
>> >
>> >
>> >
>> >My route builder code is below. I call it twice, once for vendor =
>> >³test², and once for vendor = ³googleplay².
>> >
>> >private RouteBuilder scheduledEventBatchExecutionTimerRoute(
>> > final String vendor,
>> > final int batchSize,
>> > final int timerPeriodSeconds,
>> > final boolean autoStart) {
>> >
>> > log.info("scheduledEventBatchExecutionTimerRoute: vendor={},
>> >doStart={}, queryInterval={}s, batchSize={}",
>> > vendor, autoStart, timerPeriodSeconds, batchSize);
>> >
>> > return new RouteBuilder() {
>> > @Override
>> > public void configure() throws Exception {
>> >
>> > from("timer://" + vendor + "?period=" +
>> >timerPeriodSeconds + "s")
>> > .autoStartup(autoStart)
>> > .routePolicy(electionRoutePolicy(vendor))
>> > .setHeader("vendor",
>> >constant(vendor)).setHeader("batchSize", constant(batchSize))
>> > .beanRef("scheduleEventRepository",
>> >"findExecutableEvents(entitlement.create.${in.header.vendor},
>> >${in.header.batchSize})")
>> >
>> > .choice()
>> > .when().simple("${body.size} == 0")
>> > .log(LoggingLevel.DEBUG, "Route:
>> >vendor=${in.header.vendor}: No items to process")
>> > .when().simple("${body.size} > 0")
>> > .to(ROUTE_DIRECT_EVENT_BATCH_EXECUTION + "-" +
>> >vendor)
>> >
>> > .endChoice()
>> > .routeId(vendor + "-timer-execution-route");
>> >
>> > }
>> > };
>> >
>> > }
>> >
>> > private RoutePolicy electionRoutePolicy(String vendor) {
>> >
>> > RoutePolicy routePolicy;
>> >
>> > if (doUseDistributedRouteElection) {
>> > log.info("Creating zookeeper distributed route election
>> >policy for {}", vendor);
>> > // Ensure that only one node is processing work for the
>> >specified vendor.
>> > // See http://camel.apache.org/zookeeper.html - distributed
>> >route policy.
>> > routePolicy = new ZooKeeperRoutePolicy(zookeeperUri +
>> >"scheduler-node-election/" + vendor, 1);
>> >
>> > } else {
>> > // Just return a default route policy that does nothing.
>> > log.info("No-op route policy for {}", vendor);
>> > routePolicy = new RoutePolicySupport() {
>> > @Override
>> > public void onInit(Route route) {
>> > super.onInit(route);
>> > }
>> > };
>> > }
>> >
>> > return routePolicy;
>> > }
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>
>
>--
>Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC
>| 434.409.2780
>http://www.opensourceconnections.com
