You may be filling up your OS's file table. When a socket
is closed, the OS holds onto it for a while (default usually
about two minutes) to reliably detect any straggler packets.
When you cycle a lot of connection quickly, this can max out
internal data structures in the OS. If you do a netstat and
see zillions of connections in TIME_WAIT state, that's probably
what's happening.
If you're connecting across a LAN, this delay is not really
needed, because it's hard for packet to get rerouted anywhere else.
You can tune the socket wait time down to 5 seconds or so and that
will allow file table slots to be re-used more quickly.
You can also insert multiple documents per request, all of
which will be transferred together and result in fewer low-level
sockets being opened.
On Mar 14, 2010, at 9:28 PM, Lee, David wrote:
> FYI, here's a stack trace from the same program but in this case its the
> query component under load.
> This is very consistent as well after about 10 -20k requests
>
>
> com.marklogic.xcc.exceptions.ServerConnectionException: Error parsing HTTP
> headers: Premature EOF, partial header line read: ''
> [Session: user=DLEE, cb={default} [ContentSource: user=DLEE, cb={none}
> [provider: address=home/192.168.1.10:8011, pool=0/64]]]
> at
> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:99)
> at
> com.marklogic.xcc.impl.SessionImpl.submitRequest(SessionImpl.java:280)
> at org.xmlsh.marklogic.put.setChecksum(put.java:341)
> at org.xmlsh.marklogic.put.flushContent(put.java:315)
> at org.xmlsh.marklogic.put.putContent(put.java:288)
> at org.xmlsh.marklogic.put.load(put.java:272)
> at org.xmlsh.marklogic.put.load(put.java:266)
> at org.xmlsh.marklogic.put.run(put.java:126)
> at org.xmlsh.core.XCommand.run(XCommand.java:86)
> at org.xmlsh.core.XCommand.run(XCommand.java:63)
> at
> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:121)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:124)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.shell.Shell.runScript(Shell.java:362)
> at org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:75)
> at
> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:121)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:124)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.shell.Shell.runScript(Shell.java:362)
> at org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:75)
> at
> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:121)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:124)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.shell.Shell.interactive(Shell.java:461)
> at org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:82)
> at org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54)
> at org.xmlsh.sh.shell.Shell.main(Shell.java:690)
> Caused by: java.io.IOException: Error parsing HTTP headers: Premature EOF,
> partial header line read: ''
> at
> com.marklogic.http.HttpHeaders.nextHeaderLine(HttpHeaders.java:326)
> at
> com.marklogic.http.HttpHeaders.parseResponseHeaders(HttpHeaders.java:287)
> at
> com.marklogic.http.HttpChannel.parseHeaders(HttpChannel.java:323)
> at
> com.marklogic.http.HttpChannel.receiveMode(HttpChannel.java:293)
> at
> com.marklogic.http.HttpChannel.getResponseCode(HttpChannel.java:187)
> at
> com.marklogic.xcc.impl.handlers.EvalRequestController.issueRequest(EvalRequestController.java:111)
> at
> com.marklogic.xcc.impl.handlers.EvalRequestController.serverDialog(EvalRequestController.java:62)
> at
> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:72)
> ... 29 more
>
>
>
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Lee, David
> Sent: Saturday, March 13, 2010 7:42 PM
> To: General Mark Logic Developer Discussion
> Subject: RE: [MarkLogic Dev General] ServerConnectionException-consistantly
> after about 20, 000 files
>
> Here's a full stack trace, including my code in the stack.
> by "opening connections" I mean calling
>
> URI serverUri = new URI (connect);
> ContentSource cs = ContentSourceFactory.newContentSource (serverUri);
>
> for ever file instead of reusing the ContentSource for all files.
> Although that may be a red-herring ... when I do it that way (new Content
> Source for each file) I'm not aborting the push operation if one file fails
> so I may be missing these errors in that case.
>
> --------- Stack Trace
>
>
>
> 2010-03-13 16:17:13,748 12310138 ERROR [main] core.SimpleCommand - Exception
> running command: ml:put
> com.marklogic.xcc.exceptions.ServerConnectionException: An established
> connection was aborted by the software in your host machine
> [Session: user=DLEE, cb={default} [ContentSource: user=DLEE, cb={none}
> [provider: address=home/192.168.1.10:8011, pool=0/64]]]
> at
> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:99)
> at
> com.marklogic.xcc.impl.SessionImpl.insertContent(SessionImpl.java:204)
> at org.xmlsh.marklogic.put.load(put.java:180)
> at org.xmlsh.marklogic.put.load(put.java:171)
> at org.xmlsh.marklogic.put.run(put.java:99)
> at org.xmlsh.core.XCommand.run(XCommand.java:86)
> at org.xmlsh.core.XCommand.run(XCommand.java:63)
> at
> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:121)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:124)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.shell.Shell.runScript(Shell.java:362)
> at org.xmlsh.core.ScriptCommand.run(ScriptCommand.java:75)
> at
> org.xmlsh.sh.core.SimpleCommand.exec(SimpleCommand.java:121)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.core.Pipeline.exec(Pipeline.java:124)
> at org.xmlsh.sh.shell.Shell.exec(Shell.java:560)
> at org.xmlsh.sh.shell.Shell.interactive(Shell.java:461)
> at org.xmlsh.commands.builtin.xmlsh.run(xmlsh.java:82)
> at org.xmlsh.core.BuiltinCommand.run(BuiltinCommand.java:54)
> at org.xmlsh.sh.shell.Shell.main(Shell.java:690)
> Caused by: java.io.IOException: An established connection was aborted by the
> software in your host machine
> at sun.nio.ch.SocketDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:33)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
> at sun.nio.ch.IOUtil.write(IOUtil.java:60)
> at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at
> com.marklogic.http.HttpChannel.writeBuffer(HttpChannel.java:373)
> at
> com.marklogic.http.HttpChannel.writeBody(HttpChannel.java:353)
> at
> com.marklogic.http.HttpChannel.flushRequest(HttpChannel.java:346)
> at com.marklogic.http.HttpChannel.write(HttpChannel.java:134)
> at
> com.marklogic.xcc.impl.handlers.ContentInsertController.writeChunkHeader(ContentInsertController.java:299)
> at
> com.marklogic.xcc.impl.handlers.ContentInsertController.issueRequest(ContentInsertController.java:210)
> at
> com.marklogic.xcc.impl.handlers.ContentInsertController.serverDialog(ContentInsertController.java:112)
> at
> com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:72)
> ... 20 more
>
>
>
>
>
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Sam Neth
> Sent: Saturday, March 13, 2010 6:08 PM
> To: General Mark Logic Developer Discussion
> Subject: Re: [MarkLogic Dev General] ServerConnectionException -consistantly
> after about 20, 000 files
>
> Could you post a stack trace?
>
> What version of XCC are you using?
>
> What specifically are you referring to when you talk about "opening
> connections"?
>
> On Mar 13, 2010, at 2:33 PM, Lee, David wrote:
>
>
> If I use XCC to iteratively insert a large set of documents I consistently
> get this error
>
> com.marklogic.xcc.exceptions.ServerConnectionException: An established
> connectin was aborted by the software in your host machine [Session:
> user=DLEE, cb={default} [ContentSource: user=DLEE, cb={none} [providr:
> address=home/192.168.1.10:8011, pool=0/64]]]
>
>
> This occurs after about 20,000 files and aborts the program.
> I'm thinking of implementing a exception handler to retry but I dont want to
> be retrying after more serious errors.
> The server log doesnt show any problems, and this is on a dedicated 1GB wired
> LAN so I dont think its internet problems.
>
> If instead of using the same connection I open the connection for each file
> it often gets around this problem, but not always,
> I think its getting around it because I'm not aborting on error in that case
> (just going to the next file).
>
> I'm using this code snippet to create the content in bulks of 1-20 ( files in
> a directory )
>
> Content content= ContentFactory.newContent (uri, file, mCreateOptions);
> contents.add(content);
> ...
>
> if( ! contents.isEmpty() )
> session.insertContent (contents.toArray(new Content[ contents.size()]));
>
>
>
> Any suggestions ?
>
>
>
> ----------------------------------------
> David A. Lee
> Senior Principal Software Engineer
> Epocrates, Inc.
> [email protected]
> 812-482-5224
>
> _______________________________________________
> General mailing list
> [email protected]
> http://xqzone.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> [email protected]
> http://xqzone.com/mailman/listinfo/general
---
Ron Hitchens {mailto:[email protected]} Ronsoft Technologies
(650) 766-2355 (Home Office) http://www.ronsoft.com
(707) 924-3878 (fax) Bit Twiddling At Its Finest
"No amount of belief establishes any fact." -Unknown
_______________________________________________
General mailing list
[email protected]
http://xqzone.com/mailman/listinfo/general
