[appengine-java] Re: blobstore & lucene search index
My search engine is a full-text proximity based engine which you can check out in action in an Android add called NewsDroid. The indexer part indexes Reuters News semi-continuously and a user searching on Android will be shown query hits in matching news where as many keywords as possible were found together. This is, btw, an XML search engine -- in addition to keywords you need to specify in which elements to search for text passages. I am currently working on another application of the engine, beyond news. The original version of the search engine used a number of files to store indexes. As I "ported" the system to GAE I had to figure out ways to store and use indexes stored as blobs in the Datastore. Basically, any file can be split into chunks of up to 1,000,000 bytes. Things work OK, but not great on GAE. Loading indexes into memory is taking time when servlets are restarted. On Dec 25, 7:51 am, Toby wrote: > Hallo Jacek, > > Thank you for your mail. Can you give a bit more details about that > search engine that you implemented? > I would like to use lucene because I like to use their analyzers and > things like levenshtein distance result ordering etc. > Currently I use the compass-lucen framework that comes with a special > adapter to store the index in the > datastore:http://www.kimchy.org/searchable-google-appengine-with-compass/ > I wonder if now the blob-store could be a better way to achieve the > same goal. > > Cheers, > Tobi > > On Dec 23, 3:18 pm, "jacek.ambroziak" > wrote: > > > > > I have been using a search engine of my own design. > > I had first to adapt it to GAE by generalizing the index > > storage mechanism so that it can be *implemented* > > using files (originally the only way) or with Datastore. > > It was a good couple of days of work but it is not hard. > > > On Dec 22, 6:33 pm, Toby wrote: > > > > Hello, > > > > I think a sophisticated search api is what a lot of us are missing. I > > > use currently the compass-lucene project but I am not completely > > > satisfied with the performance. I think there is also the GAELucene > > > project that uses a similar aproach. > > > I guess the main issue was so far that you could not write the index > > > files to a "regular" file system. I wonder if the new blob-store API > > > could solve that issue and if anyone has done some experimentation > > > with it. > > > > Cheers, > > > Tobi -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: blobstore & lucene search index
My search engine is a full-text proximity based engine which you can check out in action in an Android add called NewsDroid. The indexer part indexes Reuters News semi-continuously and a user searching on Android will be shown query hits in matching news where as many keywords as possible were found together. This is, btw, an XML search engine -- in addition to keywords you need to specify in which elements to search for text passages. I am currently working on another application of the engine, beyond news. The original version of the search engine used a number of files to store indexes. As I "ported" the system to GAE I had to figure out ways to store and use indexes stored as blobs in the Datastore. Basically, any file can be split into chunks of up to 1,000,000 bytes. Things work OK, but not great on GAE. Loading indexes into memory is taking time when servlets are restarted. On Dec 25, 7:51 am, Toby wrote: > Hallo Jacek, > > Thank you for your mail. Can you give a bit more details about that > search engine that you implemented? > I would like to use lucene because I like to use their analyzers and > things like levenshtein distance result ordering etc. > Currently I use the compass-lucen framework that comes with a special > adapter to store the index in the > datastore:http://www.kimchy.org/searchable-google-appengine-with-compass/ > I wonder if now the blob-store could be a better way to achieve the > same goal. > > Cheers, > Tobi > > On Dec 23, 3:18 pm, "jacek.ambroziak" > wrote: > > > > > I have been using a search engine of my own design. > > I had first to adapt it to GAE by generalizing the index > > storage mechanism so that it can be *implemented* > > using files (originally the only way) or with Datastore. > > It was a good couple of days of work but it is not hard. > > > On Dec 22, 6:33 pm, Toby wrote: > > > > Hello, > > > > I think a sophisticated search api is what a lot of us are missing. I > > > use currently the compass-lucene project but I am not completely > > > satisfied with the performance. I think there is also the GAELucene > > > project that uses a similar aproach. > > > I guess the main issue was so far that you could not write the index > > > files to a "regular" file system. I wonder if the new blob-store API > > > could solve that issue and if anyone has done some experimentation > > > with it. > > > > Cheers, > > > Tobi -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: blobstore & lucene search index
Hallo Jacek, Thank you for your mail. Can you give a bit more details about that search engine that you implemented? I would like to use lucene because I like to use their analyzers and things like levenshtein distance result ordering etc. Currently I use the compass-lucen framework that comes with a special adapter to store the index in the datastore: http://www.kimchy.org/searchable-google-appengine-with-compass/ I wonder if now the blob-store could be a better way to achieve the same goal. Cheers, Tobi On Dec 23, 3:18 pm, "jacek.ambroziak" wrote: > I have been using a search engine of my own design. > I had first to adapt it to GAE by generalizing the index > storage mechanism so that it can be *implemented* > using files (originally the only way) or with Datastore. > It was a good couple of days of work but it is not hard. > > On Dec 22, 6:33 pm, Toby wrote: > > > Hello, > > > I think a sophisticated search api is what a lot of us are missing. I > > use currently the compass-lucene project but I am not completely > > satisfied with the performance. I think there is also the GAELucene > > project that uses a similar aproach. > > I guess the main issue was so far that you could not write the index > > files to a "regular" file system. I wonder if the new blob-store API > > could solve that issue and if anyone has done some experimentation > > with it. > > > Cheers, > > Tobi > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: blobstore & lucene search index
I have been using a search engine of my own design. I had first to adapt it to GAE by generalizing the index storage mechanism so that it can be *implemented* using files (originally the only way) or with Datastore. It was a good couple of days of work but it is not hard. On Dec 22, 6:33 pm, Toby wrote: > Hello, > > I think a sophisticated search api is what a lot of us are missing. I > use currently the compass-lucene project but I am not completely > satisfied with the performance. I think there is also the GAELucene > project that uses a similar aproach. > I guess the main issue was so far that you could not write the index > files to a "regular" file system. I wonder if the new blob-store API > could solve that issue and if anyone has done some experimentation > with it. > > Cheers, > Tobi -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
