Can you please elaborate more about in memory index? What kind of software did you used to implement this ? Regards On Sat, May 16, 2009 at 8:55 PM, tim robertson <[email protected]>wrote:
> Perhaps some kind of in memory index would be better than iterating an > array? Binary tree or so. > I did similar with polygon indexes and point data. It requires > careful memory planning on the nodes if the indexes are large (mine > were several GB). > > Just a thought, > > Tim > > On Sat, May 16, 2009 at 1:56 PM, PORTO aLET <[email protected]> wrote: > > Hi, > > > > I am trying to include the stop words into hadoop map reduce, and later > on, > > into hive. > > What is the accepted solution regarding the stop words in hadoop? > > > > All I can think is to load all the stop words into an array in the > mapper, > > and then check each token against the stop words..(this would be O(n^2) ) > > > > Regards > > >
