Thanks guys. This is the information I needed. I will pass along Mike's AWS references to the Ops people to see if we can optimize them (ELBs are a long-running bone of contention on this project). And I’ll run CORB with the JVM setting for HTTP compatibility (insert grumble here about that not being the default).
Thanks. > On Jul 1, 2016, at 4:56 PM, David Lee <david....@marklogic.com> wrote: > > For ELB's on AWS there are different types of affinity > http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-sticky-sessions.html > > Try using the Duration based stickiness -- that may help isolate the issue. > > http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-sticky-sessions.html#enable-sticky-sessions-duration > > > > -----Original Message----- > From: general-boun...@developer.marklogic.com > [mailto:general-boun...@developer.marklogic.com] On Behalf Of Jason Hunter > Sent: Friday, July 01, 2016 11:21 AM > To: MarkLogic Developer Discussion <general@developer.marklogic.com> > Subject: Re: [MarkLogic Dev General] XCC (CORB) over Amazon ELB > > Hi Ron, > > Did you tell XCC to be HTTP compliant? It's not by default. > > http://docs.marklogic.com/guide/xcc/concepts#id_28335 > > Enable HTTP 1.1 Compliance > > Enabling HTTP compliant mode guarantees the traffic between your XDBC App > Server and your XCC client is compliant with the HTTP 1.1 protocol. This > enables properly configured load balancers to detect the SessionID cookie > generated by MarkLogic Server and use it to enforce session affinity. > > To enable this mode for a Java application, set the xcc.httpcompliant system > property to true on the Java command line. For example: > > java -Dxcc.httpcompliant=true ... > To enable this mode for a .NET application, set the xcc.httpcompliant > environment variable to True. > > Setting xcc.httpcompliant to true is incompatible with enabling content > entity resolution using ContentCreateOptions.setResolveEntities (Java) or the > ResolveEntities property (.NET). > > If xcc.httpcompliant is not set explicitly, xcc.httpcompliant is false. > > You must also configure your load balancer to use the value in the SessionID > cookie for session affinity. Some routers or load balancers may need to have > xcc.httpcompliant enabled to allow any traffic through, regardless of session > affinity issues. > > Configure the Load Balancer > > In addition to setting xcc.httpcompliant to true, you must configure your > load balancer to use the SessionID cookie generated by MarkLogic Server for > session affinity. You might also need to enable session affinity or sticky > sessions in your load balancer. The exact configuration steps depend on the > load balancer; see your load balancer documentation for details. > > Your load balancer must be HTTP 1.1 compliant and support cookie-based > session affinity to use this feature of XCC. > > A SessionID cookie looks similar to the following: > > SessionID=25b877c32807aa9f > > -jh- > >> On Jul 1, 2016, at 10:10 PM, Ron Hitchens <r...@ronsoft.com> wrote: >> >> >> We’re having quite a bit of trouble with ELBs in AWS. There is an ELB in >> front of the service/app nodes, and another ELB between them and the three >> node MarkLogic cluster. >> >> We’ve had problems with the ELB in front of ML marking nodes down if they >> are not responsive enough, which results in 504 responses on calls to ML >> after that. We’ve had to crank up the timeouts to fairly high levels to >> avoid one slow query from causing subsequent queries to fail. >> >> That’s annoying but manageable. What’s really problematic is apparent data >> corruption when using XCC, specifically CORB. >> >> For most calls to MarkLogic, we’re using our bespoke REST interface using >> URL rewriting on an HTTP appserver. We don’t use XCC normally. But we’re >> doing some one time batch reprocessing with CORB. Pointing CORB at one of >> the E-nodes works fine. But if we point it at the ELB (which works fine for >> the REST calls) then we randomly get failures like the one below. >> >> Since I originally wrote XCC, I know that this error message is the result >> of the data stream being out of sync with what XCC is expecting to see. >> It’s looking for a sub-header in the partitioned response that describes the >> XDM type of the next result item. It’s seeing something else instead. >> >> Are there known issues with ELBs and XCC/XDBC and/or CORB? Is there an ELB >> setting get around this? I have a feeling that the ELB may be munging the >> response in some way that is tripping up XCC. >> >> Any help appreciated. >> >> ======= >> >> INFO: monitoring 1,171,599 tasks >> Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Manager >> populateQueue >> INFO: received first uri: >> urn:cambridge.org:id:content-bundle:S0360966900011993:version:1 >> 2016-06-30 15:35:23.454 SEVERE [1] (StreamingResultSequence.next): >> IOException instantiating ResultItem 240: Nodes of type 'rsion' are >> not supported in XCC result sequences >> java.io.IOException: Nodes of type 'rsion' are not supported in XCC result >> sequences >> >> com.marklogic.xcc.impl.AbstractResultSequence.nodeFactory(AbstractResultSequence.java:248) >> >> com.marklogic.xcc.impl.AbstractResultSequence.instantiateResultItem(AbstractResultSequence.java:202) >> >> com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:147) >> >> com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:166) >> >> com.marklogic.developer.corb.QueryUrisLoader.next(QueryUrisLoader.java:158) >> com.marklogic.developer.corb.Manager.populateQueue(Manager.java:719) >> com.marklogic.developer.corb.Manager.run(Manager.java:472) >> com.marklogic.developer.corb.Manager.main(Manager.java:162) >> Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Manager stop >> INFO: cleaning up >> Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.QueryUrisLoader >> close >> INFO: closing uris session >> Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Monitor run >> SEVERE: interrupted: exiting >> java.lang.InterruptedException >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) >> at >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) >> at >> java.util.concurrent.ExecutorCompletionService.poll(ExecutorCompletionService.java:202) >> at com.marklogic.developer.corb.Monitor.monitorResults(Monitor.java:96) >> at com.marklogic.developer.corb.Monitor.run(Monitor.java:75) >> at java.lang.Thread.run(Thread.java:745) >> >> 2016-06-30 15:35:23.475 WARNING [10] >> (AbstractRequestController.runRequest): Cannot obtain connection: null >> java.nio.channels.ClosedByInterruptException >> >> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) >> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659) >> java.nio.channels.SocketChannel.open(SocketChannel.java:189) >> >> com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:93) >> >> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:85) >> >> com.marklogic.xcc.impl.SessionImpl.submitRequestInternal(SessionImpl.java:437) >> com.marklogic.xcc.impl.SessionImpl.submitRequest(SessionImpl.java:432) >> >> com.marklogic.developer.corb.AbstractTask.invokeModule(AbstractTask.java:233) >> com.marklogic.developer.corb.AbstractTask.call(AbstractTask.java:148) >> com.marklogic.developer.corb.AbstractTask.call(AbstractTask.java:56) >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> java.lang.Thread.run(Thread.java:745) >> Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.AbstractTask >> handleRequestException >> WARNING: Encountered ServerConnectionException from Marklogic Server. >> Retrying attempt 1 after 60 seconds..: null at URI: >> urn:cambridge.org:id:content-bundle:S0360966900011993:version:1 >> >> --- >> Ron Hitchens {r...@overstory.co.uk} +44 7879 358212 >> >> _______________________________________________ >> General mailing list >> General@developer.marklogic.com >> Manage your subscription at: >> http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > General@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > General@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
