Re: SolrJ addField with Reader

2010-10-22 Thread Bojan Vukojevic 
Is there an example of how to use ContentStreamBase.FileStream from SolrJ
during indexing to reduce memory footprint? Using addField is requiring a
string. The only example I could find in JUnits is below and does not show
indexing...

thx!

*public* *void* testFileStream() *throws* IOException {
46:*File*
http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/io-nio/java/io/File.java.htm
file = *new* *File*
http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/io-nio/java/io/File.java.htm(README);
47:assertTrue(file.exists()); // make sure you are
running from: solr\src\test\test-files
48:
49:*ContentStreamBase*
http://www.java2s.com/Open-Source/Java-Document/Search-Engine/apache-solr-1.2.0/org/apache/solr/util/ContentStreamBase.java.htm
stream = *new* *ContentStreamBase.FileStream*
http://www.java2s.com/Open-Source/Java-Document/Search-Engine/apache-solr-1.2.0/org/apache/solr/util/ContentStreamBase.java.htm(
50:file);
51:assertEquals(file.length(), stream.getSize().intValue());
52:assertTrue(IOUtils.contentEquals(*new*
*FileInputStream*
http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/io-nio/java/io/FileInputStream.java.htm(file),
53:stream.getStream()));
54:assertTrue(IOUtils.contentEquals(*new* *FileReader*
http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/io-nio/java/io/FileReader.java.htm(file),
stream
55:.getReader()));
56:}

Re: SolrJ addField with Reader

2010-08-25 Thread Shalin Shekhar Mangar 
On Tue, Aug 24, 2010 at 10:37 AM, Bojan Vukojevic email...@gmail.comwrote:

 I am using SolrJ with embedded  Solr server and some documents have a lot
 of
 text. Solr will be running on a small device with very limited memory. In
 my
 tests I cannot process more than 3MB of text (in a body) with 64MB heap.
 According to Java there is about 30MB free memory before I call server.add
 and with 5MB of text it runs out of memory.

 Is there a way around this?

 Is there a plan to enhance SolrJ to allow a reader to be passed in instead
 of a string?


Can you please open a Jira issue?

-- 
Regards,
Shalin Shekhar Mangar.

Re: SolrJ addField with Reader

2010-08-25 Thread Lance Norskog 
There are a couple of options here. Solr can fetch text from a file or
from HTTP given an url. Look at the stream.file and stream.url
parameters. You can use these from EmbeddedSolr.

Also, there are 'ContentStream' objects in the SolrJ API which you can
also use. Look at
http://lucene.apache.org/solr/api/org/apache/solr/common/util/ContentStreamBase.FileStream.html.
The unit tests have a few examples of how to use it.

Lance

On Wed, Aug 25, 2010 at 12:43 AM, Shalin Shekhar Mangar
shalinman...@gmail.com wrote:
 On Tue, Aug 24, 2010 at 10:37 AM, Bojan Vukojevic email...@gmail.comwrote:

 I am using SolrJ with embedded  Solr server and some documents have a lot
 of
 text. Solr will be running on a small device with very limited memory. In
 my
 tests I cannot process more than 3MB of text (in a body) with 64MB heap.
 According to Java there is about 30MB free memory before I call server.add
 and with 5MB of text it runs out of memory.

 Is there a way around this?

 Is there a plan to enhance SolrJ to allow a reader to be passed in instead
 of a string?


 Can you please open a Jira issue?

 --
 Regards,
 Shalin Shekhar Mangar.




-- 
Lance Norskog
goks...@gmail.com