Hi All, Has anyone used rsync or similar utilities on Windows OS to replicate Lucene index across multiple machines?
Any pointers on it will be very useful? Regards, Rajesh --- Rajesh parab <[EMAIL PROTECTED]> wrote: > Hi Michael, > > Thanks a lot for your suggestions. > > I was looking at rsync; as per this link > (http://samba.anu.edu.au/rsync/features.html), rsync > is a file transfer program for UNIX. Is there rsync > support for Windows as well? I found few rsync > programs that works for Windows, but I am not sure > if > they will server the purpose. Has anyone using rsync > on Windows? > > Regards, > Rajesh > > --- Michael McCandless <[EMAIL PROTECTED]> > wrote: > > > > > Rajesh parab wrote: > > > Hi, > > > > > > We are currently using Lucene 2.0 for full-text > > > searches within our enterprise application, > which > > can > > > be deployed in clustered environment. We > generate > > > Lucene index for data stored inside relational > > > database. > > > > > > As Lucene 2.0 did not have solid NFS support and > > as we > > > wanted Lucene based searches to work properly in > > > Clustered environment, we had decided on > following > > > approach: > > > 1. The index generation happens on a machine > > (could be > > > one of the cluster nodes or a separate machine) > > and > > > once the Lucene index is generated, we copy all > > the > > > index files to the database. > > > > Note that you can do also incremental replication: > > often, the Lucene > > index changes in minor ways (eg a single new > segment > > is flushed and a > > new segments_N and segments.gen is written) so you > > should only sync > > the files that are new (and remove the ones that > are > > now gone). > > Lucene's write-once approach makes this very > simple > > (you just have to > > compare file names, not the contents of each > file). > > > > It's also possible to replicate without using a > DB. > > EG rsync does a > > great job. > > > > > 2. The index search request on each cluster node > > > retrieves the index files from database (during > > first > > > search or after index update), copies to the > file > > > system and use it for searches. > > > 3. Thus, each cluster node has its own copy of > the > > > index and it keeps on picking up latest version > if > > it > > > is available inside database. > > > > > > This has worked fine for us till now, though we > > will > > > not be able continue with this model in future > as > > we > > > want to support Lucene based searches across our > > > application and also want to index large > > components > > > inside our application like Wiki, forums, etc. > As > > the > > > index will grow, storing and retrieving index > > files > > > from database will not be an efficient > operation. > > > > > > My questions are: > > > - Will we be able to use NFS if we move to > Lucene > > > 2.3.0? > > > > Make sure you update to 2.3.1, not 2.3.0. > > > > > - Will there be any significant performance > impact > > on > > > index generation and searches if we move to NFS? > > > - Is Lucene + NFS combination supported for all > > > operating systems? (We support Windows, Solaris, > > AIX, > > > HP-UX, Red Hat Linux) > > > > NFS *should* work, however: > > > > * It's not widely used, so, test thoroughly in > > your particular setup. > > > > * Most likely to work is if you use a single > > machine writing to > > the index, and many readers. > > > > * Performance is likely not great, especially > on > > searching, but > > you should test in your specific situation. > > > > > - Is there any other alternative available other > > than > > > NFS? > > > > > > > It's also possible to replicate without using a > DB. > > EG rsync does > > agreat job. > > > > You should look at Solr, since it already has all > > the infrastructure > > toaccept updates, replicate index changes to > remote > > machines, etc. > > > > > I will really appreciate your comments/thoughts > on > > > this topic. > > > > > > Regards, > > > Rajesh > > > > > > > > > > > > > > > ______________________________________________________________________ > > > > > ______________ > > > You rock. That's why Blockbuster's offering you > > one month of > > > Blockbuster Total Access, No Cost. > > > > http://tc.deals.yahoo.com/tc/blockbuster/text5.com > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > ____________________________________________________________________________________ > You rock. That's why Blockbuster's offering you one > month of Blockbuster Total Access, No Cost. > http://tc.deals.yahoo.com/tc/blockbuster/text5.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
