Throwing a multi-gigabyte file at Solr and expecting it to index it is asking for a bit too much. You either have to stream it up and break it apart or something similar.
And consider what happens if you just index the log as a single document. How do you search it? Do you return several G as the result? Most applications break the log file up into individual documents and index each event individually to enable searches like "all OOM errors between 12:00 and 13:00 yesterday" or similar. How do you expect to do such a thing if it's one big document? I may be completely off base here, but I think you need to define the problem you're solving more clearly. I can flat guarantee that trying to index a large log file as one document will be unsatisfactory to search, even if you can get it into the index. Best, Erick On Wed, Oct 30, 2013 at 12:47 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > Hm, sorry for not helping with this particular issue directly, but it > looks like you are *uploading* your logs and indexing that way? > Wouldn't pushing them be a better fit when it comes to log indexing? > We recently contributed a Logstash output that can index logs to Solr, > which may be of interest - have a look at > https://twitter.com/otisg/status/395563043045638144 -- includes a > little diagram that shows how this fits into the picture. > > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Wed, Oct 30, 2013 at 9:55 AM, keshari.prerna > <keshari.pre...@gmail.com> wrote: > > Hello, > > > > As suggested by Chris, now I am accessing the files using java program > and > > creating SolrInputDocument, but i ran into this exception while doing > > server.add(document). When i tried to increase "ramBufferSizeMB", it > doesn't > > let me make it more than 2 gig. > > > > org.apache.solr.client.solrj.SolrServerException: Server at > > http://localhost:8983/solr/logsIndexing returned non ok status:500, > > message:the request was rejected because its size (2097454) exceeds the > > configured maximum (2097152) > > org.apache.commons.fileupload.FileUploadBase$SizeLimitExceededException: > the > > request was rejected because its size (2097454) exceeds the configured > > maximum (2097152) at > > > org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl$1.raiseError(FileUploadBase.java:902) > > at > > > org.apache.commons.fileupload.util.LimitedInputStream.checkLimit(LimitedInputStream.java:71) > > at > > > org.apache.commons.fileupload.util.LimitedInputStream.read(LimitedInputStream.java:128) > > at > > > org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) > > at > > > org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) > > at java.io.InputStream.read(Unknown Source) at > > org.apache.commons.fileupload.util.Streams.copy(Streams.java:94) > at > > org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) > at > > > org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:362) > > at > > > org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) > > at > > > org.apache.solr.servlet.MultipartRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:344) > > at > > > org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:397) > > at > > > org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:115) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:244) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > > at org.mortbay.jetty.handler.ContextHand > > at > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:328) > > at > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) > > at > > > org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) > > at > org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:121) > > at > org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:106) > > at Filewalker.walk(LogsIndexer.java:48) > > at Filewalker.main(LogsIndexer.java:69) > > > > How do I get rid of this? > > > > Thanks, > > Prerna > > > > > > > > -- > > View this message in context: > http://lucene.472066.n3.nabble.com/Indexing-logs-files-of-thousands-of-GBs-tp4097073p4098438.html > > Sent from the Solr - User mailing list archive at Nabble.com. >