Very interesting... thanks Thom. I haven't given HttpClient a shot yet, but
will be soon.
-S
On 7/31/06, Thom Nelson <[EMAIL PROTECTED]> wrote:
I had a similar problem and was able to fix it in Solr by manually
buffering the responses to a StringWriter before sending it to Tomcat.
Essentially, Tomcat's buffer will only hold so much and at that point
it blocks (thus it always hangs at a constant number of documents).
However, a better solution (to be implemented) is to use more
intelligent code on the client to read the response at the same time
that it is sending input -- not too difficult to do, though best to do
with two threads (i.e. fire off a thread to read the response before
you send any data). Seeing as the HttpClient code probably does this
already, I'll most likely end up using that.
On 7/31/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
> Those are some great ideas Chris... I'm going to try some of them
out. I'll
> post the results when I get a chance to do more testing. Thanks.
>
> At this point I can work around the problem by ignoring Solr's response
but
> this is obviously not ideal. I would feel better knowing what is causing
the
> issue as well.
>
> -Sangraal
>
>
>
> On 7/29/06, Chris Hostetter <[EMAIL PROTECTED]> wrote:
> >
> >
> > : Sure, the method that does all the work updating Solr is the
> > doUpdate(String
> > : s) method in the GanjaUpdate class I'm pasting below. It's hanging
when
> > I
> > : try to read the response... the last output I receive in my log is
Got
> > : Reader...
> >
> > I don't have the means to try out this code right now ... but i can't
see
> > any obvious problems with it (there may be somewhere that you are
opening
> > a stream or reader and not closing it, but i didn't see one) ... i
notice
> > you are running this client on the same machine as Solr (hence the
> > localhost URLs) did you by any chance try running the client on a
seperate
> > machine to see if hte number of updates before it hangs changes?
> >
> > my money is still on a filehandle resource limit somwhere ... if you
are
> > running on a system that has "lsof" (on some Unix/Linux installations
you
> > need sudo/su root permissions to run it) you can use "lsof -p ####" to
> > look up what files/network connections are open for a given
process. You
> > can try running that on both the client pid and the Solr server pid
once
> > it's hung -- You'll probably see a lot of Jar files in use for both,
but
> > if you see more then a few XML files open by the client, or more then
a
> > 1 TCP connection open by either the client or the server, there's your
> > culprit.
> >
> > I'm not sure what Windows equivilent of lsof may exist.
> >
> > Wait ... i just had another thought....
> >
> > You are using InputStreamReader to deal with the InputStreams of your
> > remote XML files -- but you aren't specifying a charset, so it's using
> > your system default which may be differnet from the charset of the
> > orriginal XML files you are pulling from the URL -- which (i *think*)
> > means that your InputStreamReader may in some cases fail to read all
of
> > the bytes of the stream, which might some dangling filehandles (i'm
just
> > guessing on that part ... i'm not acctually sure whta happens in that
> > case).
> >
> > What if you simplify your code (for the purposes of testing) and just
put
> > the post-transform version ganja-full.xml in a big ass String variable
in
> > your java app and just call GanjaUpdate.doUpdate(bigAssString) over
and
> > over again ... does that cause the same problem?
> >
> >
> > :
> > : ----------
> > :
> > : package com.iceninetech.solr.update;
> > :
> > : import com.iceninetech.xml.XMLTransformer;
> > :
> > : import java.io.*;
> > : import java.net.HttpURLConnection;
> > : import java.net.URL;
> > : import java.util.logging.Logger;
> > :
> > : public class GanjaUpdate {
> > :
> > : private String updateSite = "";
> > : private String XSL_URL = "http://localhost:8080/xsl/ganja.xsl";;
> > :
> > : private static final File xmlStorageDir = new
> > : File("/source/solr/xml-dls/");
> > :
> > : final Logger log = Logger.getLogger(GanjaUpdate.class.getName());
> > :
> > : public GanjaUpdate(String siteName) {
> > : this.updateSite = siteName;
> > : log.info("GanjaUpdate is primed and ready to update " +
siteName);
> > : }
> > :
> > : public void update() {
> > : StringWriter sw = new StringWriter();
> > :
> > : try {
> > : // transform gawkerInput XML to SOLR update XML
> > : XMLTransformer transform = new XMLTransformer();
> > : log.info("About to transform ganjaInput XML to Solr Update
XML");
> > : transform.transform(getXML(), sw, getXSL());
> > : log.info("Completed ganjaInput/SolrUpdate XML transform");
> > :
> > : // Write transformed XML to Disk.
> > : File transformedXML = new File(xmlStorageDir,
updateSite+".sml");
> > : FileWriter fw = new FileWriter(transformedXML);
> > : fw.write(sw.toString());
> > : fw.close();
> > :
> > : // post to Solr
> > : log.info("About to update Solr for site " + updateSite);
> > : String result = this.doUpdate(sw.toString());
> > : log.info("Solr says: " + result);
> > : sw.close();
> > : } catch (Exception e) {
> > : e.printStackTrace();
> > : }
> > : }
> > :
> > : public File getXML() {
> > : String XML_URL = "http://localhost:8080/"; + updateSite +
"/ganja-
> > : full.xml";
> > :
> > : // check for file
> > : File localXML = new File(xmlStorageDir, updateSite + ".xml");
> > :
> > : try {
> > : if (localXML.createNewFile() && localXML.canWrite()) {
> > : // open connection
> > : log.info("Downloading: " + XML_URL);
> > : URL url = new URL(XML_URL);
> > : HttpURLConnection conn = (HttpURLConnection)
url.openConnection
> > ();
> > : conn.setRequestMethod("GET");
> > :
> > : // Read response to File
> > : log.info("Storing XML to File" + localXML.getCanonicalPath
());
> > : FileOutputStream fos = new FileOutputStream(new
> > File(xmlStorageDir,
> > : updateSite + ".xml"));
> > :
> > : BufferedReader rd = new BufferedReader(new
InputStreamReader(
> > : conn.getInputStream()));
> > : String line;
> > : while ((line = rd.readLine()) != null) {
> > : line = line + '\n'; // add break after each line. It
preserves
> > : formatting.
> > : fos.write(line.getBytes("UTF8"));
> > : }
> > :
> > : // close connections
> > : rd.close();
> > : fos.close();
> > : conn.disconnect();
> > : log.info("Got the XML... File saved.");
> > : }
> > : } catch (Exception e) {
> > : e.printStackTrace();
> > : }
> > :
> > : return localXML;
> > : }
> > :
> > : public File getXSL() {
> > : StringBuffer retVal = new StringBuffer();
> > :
> > : // check for file
> > : File localXSL = new File(xmlStorageDir, "ganja.xsl");
> > :
> > : try {
> > : if (localXSL.createNewFile() && localXSL.canWrite()) {
> > : // open connection
> > : log.info("Downloading: " + XSL_URL);
> > : URL url = new URL(XSL_URL);
> > : HttpURLConnection conn = (HttpURLConnection)
url.openConnection
> > ();
> > : conn.setRequestMethod("GET");
> > : // Read response
> > : BufferedReader rd = new BufferedReader(new
InputStreamReader(
> > : conn.getInputStream()));
> > : String line;
> > : while ((line = rd.readLine()) != null) {
> > : line = line + '\n';
> > : retVal.append(line);
> > : }
> > : // close connections
> > : rd.close();
> > : conn.disconnect();
> > :
> > : log.info("Got the XSLT.");
> > :
> > : // output file
> > : log.info("Storing XSL to File" + localXSL.getCanonicalPath
());
> > : FileOutputStream fos = new FileOutputStream(new
> > File(xmlStorageDir,
> > : "ganja.xsl"));
> > : fos.write(retVal.toString().getBytes());
> > : fos.close();
> > : log.info("File saved.");
> > : }
> > : } catch (Exception e) {
> > : e.printStackTrace();
> > : }
> > : return localXSL;
> > : }
> > :
> > : private String doUpdate(String sw) {
> > : StringBuffer updateResult = new StringBuffer();
> > : try {
> > : // open connection
> > : log.info("Connecting to and preparing to post to SolrUpdate
> > : servlet.");
> > : URL url = new URL("http://localhost:8080/update";);
> > : HttpURLConnection conn = (HttpURLConnection)
url.openConnection();
> > : conn.setRequestMethod("POST");
> > : conn.setRequestProperty("Content-Type",
> > "application/octet-stream");
> > : conn.setDoOutput(true);
> > : conn.setDoInput(true);
> > : conn.setUseCaches(false);
> > :
> > : // Write to server
> > : log.info("About to post to SolrUpdate servlet.");
> > : DataOutputStream output = new DataOutputStream(
> > conn.getOutputStream
> > : ());
> > : output.writeBytes(sw);
> > : output.flush();
> > : output.close();
> > : log.info("Finished posting to SolrUpdate servlet.");
> > :
> > : // Read response
> > : log.info("Ready to read response.");
> > : BufferedReader rd = new BufferedReader(new InputStreamReader(
> > : conn.getInputStream()));
> > : log.info("Got reader....");
> > : String line;
> > : while ((line = rd.readLine()) != null) {
> > : log.info("Writing to result...");
> > : updateResult.append(line);
> > : }
> > : rd.close();
> > :
> > : // close connections
> > : conn.disconnect();
> > :
> > : log.info("Done updating Solr for site" + updateSite);
> > : } catch (Exception e) {
> > : e.printStackTrace();
> > : }
> > :
> > : return updateResult.toString();
> > : }
> > : }
> > :
> > :
> > : On 7/28/06, Chris Hostetter <[EMAIL PROTECTED]> wrote:
> > : >
> > : >
> > : > : I'm sure... it seems like solr is having trouble writing to a
tomcat
> > : > : response that's been inactive for a bit. It's only 30 seconds
> > though, so
> > : > I'm
> > : > : not entirely sure why that would happen.
> > : >
> > : > but didn't you say you don't have this problem when you use curl
--
> > just
> > : > your java client code?
> > : >
> > : > Did you try Yonik's python test client? or the java client in
Jira?
> > : >
> > : > looking over the java clinet codey you sent, it's not clear if you
are
> > : > reading the response back, or closing the connections ... can you
post
> > a
> > : > more complete sample app thatexhibits the problem for you?
> > : >
> > : >
> > : >
> > : > -Hoss
> > : >
> > : >
> > :
> >
> >
> >
> > -Hoss
> >
> >
>
>
