[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hinegardner updated SOLR-433: Attachment: SOLR-433.patch I've updated the patch to apply cleanly against trunk. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication (scripts), spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.5 > > Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch, > SOLR-433.patch, SOLR-433.patch, SOLR-433.patch, SOLR-433.patch, > solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Haggstrom updated SOLR-433: - Attachment: SOLR-433.patch I've been using the patch submitted by Jonathan Lee on 10-02-08 for replicating a spelling directory in addition to the index, and it works very well for that purpose. I'm attaching a slightly modified patch that allows the snapshooter "-c" option to work with an index or spelling directory that is not named "index". > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication (scripts), spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.4 > > Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch, > SOLR-433.patch, SOLR-433.patch, SOLR-433.patch, solr-433.patch, > SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lee updated SOLR-433: -- Attachment: SOLR-433.patch Includes Stephane's fixes for snappuller & snapinstaller and some minor edits > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.4 > > Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch, > SOLR-433.patch, SOLR-433.patch, solr-433.patch, SOLR-433_unified.patch, > spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[
https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Lalevée updated SOLR-433:
-
Attachment: SOLR-433-r698590.patch
The patch that works the best for me is the last one, Jonathan's one, as I run
Solr with only one core but with two spellchecker indexes.
* I have fixed in that patch the snapcleaner when called to explicitely clean
the main index ({{./snapcleaner -i index}}).
* I also fixed every "USAGE" printing.
* and I have included the patch on {{RunExecutableListener}} and a little
simplified it (it can now access to the solr core instance).
> MultiCore and SpellChecker replication
> --
>
> Key: SOLR-433
> URL: https://issues.apache.org/jira/browse/SOLR-433
> Project: Solr
> Issue Type: Improvement
> Components: replication, spellchecker
>Affects Versions: 1.3
>Reporter: Otis Gospodnetic
> Fix For: 1.4
>
> Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch,
> SOLR-433.patch, solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch
>
>
> With MultiCore functionality coming along, it looks like we'll need to be
> able to:
> A) snapshot each core's index directory, and
> B) replicate any and all cores' complete data directories, not just their
> index directories.
> Pulled from the "spellchecker and multi-core index replication" thread -
> http://markmail.org/message/pj2rjzegifd6zm7m
> Otis:
> I think that makes sense - distribute everything for a given core, not just
> its index. And the spellchecker could then also have its data dir (and only
> index/ underneath really) and be replicated in the same fashion.
> Right?
> Ryan:
> Yes, that was my thought. If an arbitrary directory could be distributed,
> then you could have
> /path/to/dist/index/...
> /path/to/dist/spelling-index/...
> /path/to/dist/foo
> and that would all get put into a snapshot. This would also let you put
> multiple cores within a single distribution:
> /path/to/dist/core0/index/...
> /path/to/dist/core0/spelling-index/...
> /path/to/dist/core0/foo
> /path/to/dist/core1/index/...
> /path/to/dist/core1/spelling-index/...
> /path/to/dist/core1/foo
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic updated SOLR-433: -- Fix Version/s: 1.4 > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.4 > > Attachments: RunExecutableListener.patch, SOLR-433.patch, > solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lee updated SOLR-433: -- Attachment: SOLR-433.patch (patch from comment https://issues.apache.org/jira/browse/SOLR-433?focusedCommentId=12620430#action_12620430) > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Attachments: RunExecutableListener.patch, SOLR-433.patch, > solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hinegardner updated SOLR-433: Attachment: SOLR-433_unified.patch A single patch file that combines the original solr-433.patch and RunExecutableListener.patch into a single patch against svn trunk > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Attachments: RunExecutableListener.patch, solr-433.patch, > SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Klaas updated SOLR-433: Fix Version/s: (was: 1.3) > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Attachments: RunExecutableListener.patch, solr-433.patch, > spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Steigerwald updated SOLR-433: -- Attachment: spellindexfix.patch This may be better suited as another bug, or even posted in the lucene project, but I also made a small patch for the lucene spellchecker. Hitting the spellcheck request handler with a reopen command wasn't working after we installed a new spell index snapshot. Searcher was being created for the new index, but not a reader. Also, if you rebuild the spell index after it has already been built, it cleans out the index. You then have to send it a rebuild again to actually rebuild the index. Frequency of words in the spell index seemed to remain constant when rebuilding the spell index multiple times. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > Attachments: RunExecutableListener.patch, solr-433.patch, > spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Steigerwald updated SOLR-433: -- Attachment: RunExecutableListener.patch Should work. Had to pull changes out of a separate class I write to create this patch. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > Attachments: RunExecutableListener.patch, solr-433.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-433) MultiCore and SpellChecker replication
[
https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Steigerwald updated SOLR-433:
--
Attachment: solr-433.patch
We have a ctl script that controls all of the functions and makes sure we don't
run some things on the slaves (ie, snappuller, snapinstaller, rsyncd stuff).
We pass it a core and an index type:
{noformat}
./snapctl -a [rsyncd-enable/snappuller/snapinstaller/etc] -c core0 -i spell
{noformat}
spell is the name of our spellcheck index. Default index is 'index'.
This is testing well in QA right now. Hopefully this will help others out and
maybe we'll have something similar committed soon.
Here's the basics of our snapctl script. It's really tailored to our
environment, so I probably won't post it as is.
{noformat}
$SOLR_BIN=/home/dsteiger/apps/solr/solr/bin
$CORES_PATH=/home/dsteiger/local/solr/cores
$CORE=core0 # from -c arg, default is 'null'
$INDEX=spell # from -i arg, default is 'index'
$SOLR_LOGS=/home/dsteiger/apps/solr/logs # symlinked to somewhere else
# $MASTER_HOST is determined based on environment (devel/qa/prod) from the
scripts.conf
$SOLR_BIN/rsyncd-enable
$SOLR_BIN/rsyncd-disable
$SOLR_BIN/rsyncd-start -d $CORES_PATH
$SOLR_BIN/rsyncd-stop
$SOLR_BIN/snappuller-enable
$SOLR_BIN/snappuller-disable
$SOLR_BIN/snapshooter -d $CORES_PATH/$CORE/data -i $INDEX
$SOLR_BIN/snappuller -M $MASTER_HOST -S $SOLR_LOGS/clients -D
$CORES_PATH/$CORE/data -d $CORES_PATH/$CORE/data -z -c $CORE -i $INDEX
$SOLR_BIN/snapinstaller -M $MASTER_HOST -S $SOLR_LOGS/clients -d
$CORES_PATH/$CORE/data -c $CORE -i $INDEX
$SOLR_BIN/snapcleaner -D 1 -d $CORES_PATH/$CORE/data -i $INDEX
{noformat}
We also modified core.RunExecutableListener to be able to pass the core name to
out snapctl script.
{code:xml}
./solr/bin/snapctl
.
true
true
-a snapshooter-i index
./solr/bin/snapctl
.
true
true
-a snapshooter
{code}
I think this is all I changed for RunExecutableListener:
{code:title=core.RunExecutableListener.java|borderStyle=solid}
// in init()
// Add the core name to the command.
if ("true".equals(args.get("coreName")) ||
Boolean.TRUE.equals(args.get("coreName"))) {
cmdlist.add("-c " + core.getName());
}
{code}
> MultiCore and SpellChecker replication
> --
>
> Key: SOLR-433
> URL: https://issues.apache.org/jira/browse/SOLR-433
> Project: Solr
> Issue Type: Improvement
> Components: replication, spellchecker
>Affects Versions: 1.3
>Reporter: Otis Gospodnetic
> Fix For: 1.3
>
> Attachments: solr-433.patch
>
>
> With MultiCore functionality coming along, it looks like we'll need to be
> able to:
> A) snapshot each core's index directory, and
> B) replicate any and all cores' complete data directories, not just their
> index directories.
> Pulled from the "spellchecker and multi-core index replication" thread -
> http://markmail.org/message/pj2rjzegifd6zm7m
> Otis:
> I think that makes sense - distribute everything for a given core, not just
> its index. And the spellchecker could then also have its data dir (and only
> index/ underneath really) and be replicated in the same fashion.
> Right?
> Ryan:
> Yes, that was my thought. If an arbitrary directory could be distributed,
> then you could have
> /path/to/dist/index/...
> /path/to/dist/spelling-index/...
> /path/to/dist/foo
> and that would all get put into a snapshot. This would also let you put
> multiple cores within a single distribution:
> /path/to/dist/core0/index/...
> /path/to/dist/core0/spelling-index/...
> /path/to/dist/core0/foo
> /path/to/dist/core1/index/...
> /path/to/dist/core1/spelling-index/...
> /path/to/dist/core1/foo
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
