Mikhail & Ludovic, Thanks for both your replies, very helpful indeed!
Ludovic, I was actually looking into just that and did some tests with SolrJ, it does work well but needs some changes on the Solr server if we want to send out individual documents a various times. This could be done with a write() and flush() to the FastOutputStream (daos) in JavBinCodec. I therefore think that a combination of this and Mikhail's solution would work best! Mikhail, you mention that your solution doesn't currently work and not sure why this is the case, but could it be that you haven't flushed the data (os.flush()) you've written in the collect method of DocSetStreamer? I think placing the output stream into the SolrQueryRequest is the way to go, so that we can access it and write to it how we intend. However, I think using the JavaBinCodec would be ideal so that we can work with SolrJ directly, and not mess around with the encoding of the docs/data etc... At the moment the entry point to JavaBinCodec is through the BinaryResponseWriter which calls the highest level marshal() method which decodes and sends out the entire SolrQueryResponse (line 49 @ BinaryResponseWriter). What would be ideal is to be able to break up the response and call the JavaBinCodec for pieces of it with a flush after each call. Did a few tests with a simple Thread.sleep and a flush to see if this would actually work and looks like it's working out perfectly. Just trying to figure out the best way to actually do it now :) any ideas? An another note, for a solution to work with the chunked transfer encoding (and therefore web browsers), a lot more development is going to be needed. Not sure if it's worth trying yet but might look into it later down the line. Nick On Fri, 16 Mar 2012 07:29:20 +0300, Mikhail Khludnev <mkhlud...@griddynamics.com> wrote: > Ludovic, > > I looked through. First of all, it seems to me you don't amend regular > "servlet" solr server, but the only embedded one. > Anyway, the difference is that you stream DocList via callback, but it > means that you've instantiated it in memory and keep it there until it will > be completely consumed. Think about a billion numfound. Core idea of my > approach is keep almost zero memory for response. > > Regards > > On Fri, Mar 16, 2012 at 12:12 AM, lboutros <boutr...@gmail.com> wrote: > >> Hi, >> >> I was looking for something similar. >> >> I tried this patch : >> >> https://issues.apache.org/jira/browse/SOLR-2112 >> >> it's working quite well (I've back-ported the code in Solr 3.5.0...). >> >> Is it really different from what you are trying to achieve ? >> >> Ludovic. >> >> ----- >> Jouve >> France. >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Responding-to-Requests-with-Chunks-Streaming-tp3827316p3829909.html >> Sent from the Solr - User mailing list archive at Nabble.com. >>
