Have you considered storing your indexes server-side? I haven't used compression but usually the trade-off of compression is CPU usage which will also be a drain on battery life. Or maybe consider how important the highlighter is to your users - is it worth the trade-off of either disk space or battery life? If it's more of a nice-to-have then maybe hold off on the feature for a later release until you've had some feedback and some more time to figure out the best solution. Of course I don't know much about your application, so take my advice with a grain of salt.
On Mon, Sep 16, 2013 at 2:22 AM, Jebarlin Robertson <jebar...@gmail.com>wrote: > I am using Apache Lucene in Android. I have around 1 GB of Text documents > (Logs). When I Index these text documents using this > *new Field(ContentIndex.KEY_TEXTCONTENT, contents, Field.Store.YES, > Field.Index.ANALYZED,TermVector.WITH_POSITIONS_OFFSETS)*, the index > directory is consuming 1.59GB memory size. > But without Field Store it will be adound 0.59 GB indexed size. If the > Lucene indexing is taking this much space to create index and to store the > original text just to use hightlight feature, it will be big problem for > mobile devices. So I just want some help that, is there any alternative > ways to do this without occupying more space to use highligh feature in > Android powered devices. > > > On Sun, Sep 15, 2013 at 3:26 AM, Erick Erickson <erickerick...@gmail.com > >wrote: > > > bq: I thought that I can use the CompressionTool to minimize the memory > > size. > > > > This doesn't make a lot of sense. Highlighting needs the raw data to > > figure out what to highlight, so I don't see how the CompressionTool > > will help you there. > > > > And unless you have a huge document and only a very few of them, then > > the memory occupied by the uncompressed data should be trivial > > compared to the various low-level caches. This really is seeming like > > an XY problem. Perhaps if you backed up and explained _why_ this > > seems important to do people could be more helpful. > > > > > > Best, > > Erick > > > > > > On Sat, Sep 14, 2013 at 12:21 PM, Jebarlin Robertson <jebar...@gmail.com > > >wrote: > > > > > Thank you very much Erick. Actually I was using Highlighter tool, that > > > needs the entire data to be stored to get the relevant searched > sentence. > > > But when I use that, It was consuming more memory (Indexed data size + > > > Store.YES - the entire content) than the actual documents size. > > > I thought that I can use the CompressionTool to minimize the memory > size. > > > You can help, if there is any possiblities or way to store the entire > > > content and to use the highlighter feature. > > > > > > Thankyou > > > > > > > > > On Fri, Sep 13, 2013 at 6:54 PM, Erick Erickson < > erickerick...@gmail.com > > > >wrote: > > > > > > > Compression is for the _stored_ data, which is not searched. Ignore > > > > the compression and insure that you index the data. > > > > > > > > The compressing/decompressing for looking at stored > > > > values is, I believe, done at a very low level that you don't > > > > need to care about at all. > > > > > > > > If you index the data in the field, you shouldn't have to do > > > > anything special to search it. > > > > > > > > Best, > > > > Erick > > > > > > > > > > > > On Fri, Sep 13, 2013 at 1:19 AM, Jebarlin Robertson < > > jebar...@gmail.com > > > > >wrote: > > > > > > > > > Hi, > > > > > > > > > > I am trying to store all the Field values using CompressionTool, > But > > > > When I > > > > > search for any content, it is not finding any results. > > > > > > > > > > Can you help me, how to create the Field with CompressionTool to > add > > to > > > > the > > > > > Document and how to decompress it when searching for any content in > > it. > > > > > > > > > > -- > > > > > Thanks & Regards, > > > > > Jebarlin Robertson.R > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks & Regards, > > > Jebarlin Robertson.R > > > GSM: 91-9538106181. > > > > > > > > > -- > Thanks & Regards, > Jebarlin Robertson.R > GSM: 91-9538106181. > -- Mark J. Miller Blog: http://www.developmentalmadness.com LinkedIn: http://www.linkedin.com/in/developmentalmadness
