Clustering IndexWriter?

2006-09-20 Thread Steve Harris
Warning, I'm a vendor dude but this isn't really a vendor message. My IT guy had mentioned to me that a bunch of the open source products we use (JIRA, JForum etc) have Lucene inside and in the name of eating our own dog food I tried to cluster IndexWriter (with a RAMDirectory) using our (terraco

Re: Clustering IndexWriter?

2006-09-21 Thread adasal
Don't be coy, what's your comapany? Adam On 21/09/06, Steve Harris <[EMAIL PROTECTED]> wrote: Warning, I'm a vendor dude but this isn't really a vendor message. My IT guy had mentioned to me that a bunch of the open source products we use (JIRA, JForum etc) have Lucene inside and in the name o

Re: Clustering IndexWriter?

2006-09-21 Thread Vic Bancroft
adasal wrote: Don't be coy, what's your comapany? This URL is derivable from the text, with a little search ening help . . . ** http://www.terracottatech.com/terracotta_spring.shtml more, l8r, v On 21/09/06, Steve Harris <[EMAIL PROTECTED]> wrote: Warning, I'm a vendor dude but this isn

Re: Clustering IndexWriter?

2006-09-21 Thread Yonik Seeley
On 9/20/06, Steve Harris <[EMAIL PROTECTED]> wrote: Is clustering the IndexWriter really all I need to do? Hi Steve, Could you explain the details of what "clustering" really means in this context? -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server

Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Yep, that's us. No secret, just didn't want to make my question an billboard :-). Just needed a bit of info from the people who know best. Cheers, steve On 9/21/06, Vic Bancroft <[EMAIL PROTECTED]> wrote: adasal wrote: > Don't be coy, what's your comapany? This URL is derivable from the text,

Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Sure, I'm fairly new to Lucene but what I was trying to do was make it so that an index could be shared among multiple nodes. If an index is updated in any way it would be updated across the cluster coherently. In my first version I was really only taking advantage of the fact that we detect fine

Re: Re: Clustering IndexWriter?

2006-09-21 Thread Otis Gospodnetic
it? Solr has a different approach. There, only the master index is modified, while slave servers copy the master index periodically. Otis - Original Message From: Steve Harris <[EMAIL PROTECTED]> To: java-dev@lucene.apache.org Sent: Thursday, September 21, 2006 11:18:43

Re: Re: Clustering IndexWriter?

2006-09-21 Thread Chris Hostetter
: Questions: : Is this useful in the real world : Would it be possible to get that one small thing changed. I'm not really clear on what the "small thing" is that you are asking about ... you mentioned SegmentInfos subclassing Vector, are you proposing an alternative? If you've got a patch that

Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Fair question. All I did/need was take SegmentInfos and instead of subclassing Vector I made it contain a Vector. Went from subclassing to aggregation. As far as I could tell from reading the code it would make no difference to anyone and should have no performance impact (good or bad). It just a

Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Yonik Seeley
While automatically clustering java objects sure sounds cool, I have to wonder what the performance ends up being. Every small change to the clustered objects is broadcast to all the nodes, correct? Have you done any performance comparisons to see if this is a practical approach for Lucene? -Yo

Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Good question. May or may not be performant enough. Only time (and testing) will tell. My guess is that it will depend heavily on the rate in which the data changes (or read write ratio). Believe me, I'm not proposing that everyone go out and cluster lucene with terracotta dso. I'm really just pl

Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Yonik Seeley
On 9/21/06, Steve Harris <[EMAIL PROTECTED]> wrote: My guess is that some segment of the world cares a lot about realtime coherent updates and some segment of the world needs blinding speed. Part of my research is to gather the expertise of this group on these issues. I hear ya... There is ano

Re: Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Interesting. I wonder, I have a notification mechanism at my disposal as well. I wonder if it could be worked out that, much like a mvc, an IndexReader could be notified when the underlying Directory has changed so that the reader can adjust itself? Cheers, Steve On 9/21/06, Yonik Seeley <[EMAI

Re: Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Yonik Seeley
On 9/21/06, Steve Harris <[EMAIL PROTECTED]> wrote: Interesting. I wonder, I have a notification mechanism at my disposal as well. I wonder if it could be worked out that, much like a mvc, an IndexReader could be notified when the underlying Directory has changed so that the reader can adjust its

Re: Re: Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
I'm don't know list servers rules but I figured I would just include the text of the file I changed. If that is bad form give me a heads up and I won't do it again. Would this change break anything or bother anyone? package org.apache.lucene.index; /** * Copyright 2004 The Apache Software Found

Re: Re: Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
So I clustered this app: So I switched to clustering the RAMDirectory instead of the IndexWriter and it worked in my experiments. What I did was create a new IndexWriter on Document Adds and a new IndexSearcher on document queries. What I want to know is. How non-standard is this approach? Chee

Re: Re: Re: Re: Re: Re: Re: Clustering IndexWriter?

2006-09-21 Thread Steve Harris
Oops, I made a change and didn't test it. Doh, This should work better: package org.apache.lucene.index; /** * Copyright 2004 The Apache Software Foundation * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * Yo