[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558031#action_12558031 ] Doug Steigerwald commented on SOLR-433: --- Is anyone looking into this? > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558068#action_12558068 ] Doug Steigerwald commented on SOLR-433: --- It's on our TODO list here if no one was working on it yet. I'll see what we can get done based on the comments in the description. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558064#action_12558064 ] Otis Gospodnetic commented on SOLR-433: --- I don't think so, Doug. If you are asking if you should start working on this and contribute a patch, by all means, please! :) > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[
https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558528#action_12558528
]
Hoss Man commented on SOLR-433:
---
as i recall, most of the scripts currently take a "-d data_dir" option ... and
then use makethe following assumptions...
1) ${data_dir}/index is what gets snapshooted/snapinstalled
2) ${data_dir}/snapshot* is the pattern for naming snapshots.
a good way to evolve the scripts would probably be to have an alternate set of
options ... maybe "-D dir" and "-S snapshots" that assumes ${dir} is where
*all* the data to be snapshooted/snapinstalled lives, and ${snapshots} is where
*all* snapshoots live.
(except i think some of the scripts already have a -D .. snappuller or
snapcleaner maybe?)
alternate idea: we don't have to add a command line option at all ... support
for something like this could require special scripts.conf options.
> MultiCore and SpellChecker replication
> --
>
> Key: SOLR-433
> URL: https://issues.apache.org/jira/browse/SOLR-433
> Project: Solr
> Issue Type: Improvement
> Components: replication, spellchecker
>Affects Versions: 1.3
>Reporter: Otis Gospodnetic
> Fix For: 1.3
>
>
> With MultiCore functionality coming along, it looks like we'll need to be
> able to:
> A) snapshot each core's index directory, and
> B) replicate any and all cores' complete data directories, not just their
> index directories.
> Pulled from the "spellchecker and multi-core index replication" thread -
> http://markmail.org/message/pj2rjzegifd6zm7m
> Otis:
> I think that makes sense - distribute everything for a given core, not just
> its index. And the spellchecker could then also have its data dir (and only
> index/ underneath really) and be replicated in the same fashion.
> Right?
> Ryan:
> Yes, that was my thought. If an arbitrary directory could be distributed,
> then you could have
> /path/to/dist/index/...
> /path/to/dist/spelling-index/...
> /path/to/dist/foo
> and that would all get put into a snapshot. This would also let you put
> multiple cores within a single distribution:
> /path/to/dist/core0/index/...
> /path/to/dist/core0/spelling-index/...
> /path/to/dist/core0/foo
> /path/to/dist/core1/index/...
> /path/to/dist/core1/spelling-index/...
> /path/to/dist/core1/foo
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571501#action_12571501 ] Otis Gospodnetic commented on SOLR-433: --- You definitely want to put this in a separate JIRA issue and in Lucene's JIRA where you'll find a few similar issues already. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.3 > > Attachments: RunExecutableListener.patch, solr-433.patch, > spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617694#action_12617694 ] Jeremy Hinegardner commented on SOLR-433: - If this patch works for folks, I would like to see it committed and put in the nightly snapshot. Or at least the RunExecutableListener.patch and solr-433.patch. If there is any more work here, I'd be happy to work on it. > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication, spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Attachments: RunExecutableListener.patch, solr-433.patch, > spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[
https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620430#action_12620430
]
Jonathan Lee commented on SOLR-433:
---
I'm interested in getting using this patch to replicate the spell index, but I
am not using multiple cores. However, the scripts in the patch do not work for
single core setups since the assume that ${core} contains a valid value. Even
specifying -c "" will not work, since the variable is used in paths.
> MultiCore and SpellChecker replication
> --
>
> Key: SOLR-433
> URL: https://issues.apache.org/jira/browse/SOLR-433
> Project: Solr
> Issue Type: Improvement
> Components: replication, spellchecker
>Affects Versions: 1.3
>Reporter: Otis Gospodnetic
> Attachments: RunExecutableListener.patch, solr-433.patch,
> SOLR-433_unified.patch, spellindexfix.patch
>
>
> With MultiCore functionality coming along, it looks like we'll need to be
> able to:
> A) snapshot each core's index directory, and
> B) replicate any and all cores' complete data directories, not just their
> index directories.
> Pulled from the "spellchecker and multi-core index replication" thread -
> http://markmail.org/message/pj2rjzegifd6zm7m
> Otis:
> I think that makes sense - distribute everything for a given core, not just
> its index. And the spellchecker could then also have its data dir (and only
> index/ underneath really) and be replicated in the same fashion.
> Right?
> Ryan:
> Yes, that was my thought. If an arbitrary directory could be distributed,
> then you could have
> /path/to/dist/index/...
> /path/to/dist/spelling-index/...
> /path/to/dist/foo
> and that would all get put into a snapshot. This would also let you put
> multiple cores within a single distribution:
> /path/to/dist/core0/index/...
> /path/to/dist/core0/spelling-index/...
> /path/to/dist/core0/foo
> /path/to/dist/core1/index/...
> /path/to/dist/core1/spelling-index/...
> /path/to/dist/core1/foo
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[
https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636273#action_12636273
]
Stephane Bailliez commented on SOLR-433:
Note that on the last patches, the grep to retrieve the snapshot is incorrect:
{noformat}
ls ${data_dir}|grep "${snap_prefix}\."|grep -v wip|sort -r|head -1
{noformat}
would always retrieve the latest one on the ls, it needs to be with an anchor
in the grep for the prefix otherwise it will never update the index snapshot
(since 'snapshot' is present in every snapshot of index)
{noformat}
ls ${data_dir}|grep "^${snap_prefix}\."|grep -v wip|sort -r|head -1
{noformat}
should be changed in snappuller and snapinstaller
> MultiCore and SpellChecker replication
> --
>
> Key: SOLR-433
> URL: https://issues.apache.org/jira/browse/SOLR-433
> Project: Solr
> Issue Type: Improvement
> Components: replication, spellchecker
>Affects Versions: 1.3
>Reporter: Otis Gospodnetic
> Fix For: 1.4
>
> Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch,
> SOLR-433.patch, solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch
>
>
> With MultiCore functionality coming along, it looks like we'll need to be
> able to:
> A) snapshot each core's index directory, and
> B) replicate any and all cores' complete data directories, not just their
> index directories.
> Pulled from the "spellchecker and multi-core index replication" thread -
> http://markmail.org/message/pj2rjzegifd6zm7m
> Otis:
> I think that makes sense - distribute everything for a given core, not just
> its index. And the spellchecker could then also have its data dir (and only
> index/ underneath really) and be replicated in the same fashion.
> Right?
> Ryan:
> Yes, that was my thought. If an arbitrary directory could be distributed,
> then you could have
> /path/to/dist/index/...
> /path/to/dist/spelling-index/...
> /path/to/dist/foo
> and that would all get put into a snapshot. This would also let you put
> multiple cores within a single distribution:
> /path/to/dist/core0/index/...
> /path/to/dist/core0/spelling-index/...
> /path/to/dist/core0/foo
> /path/to/dist/core1/index/...
> /path/to/dist/core1/spelling-index/...
> /path/to/dist/core1/foo
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-433) MultiCore and SpellChecker replication
[ https://issues.apache.org/jira/browse/SOLR-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787155#action_12787155 ] Jason Rutherglen commented on SOLR-433: --- Are the existing patches for multiple cores or only for spellchecking? > MultiCore and SpellChecker replication > -- > > Key: SOLR-433 > URL: https://issues.apache.org/jira/browse/SOLR-433 > Project: Solr > Issue Type: Improvement > Components: replication (scripts), spellchecker >Affects Versions: 1.3 >Reporter: Otis Gospodnetic > Fix For: 1.5 > > Attachments: RunExecutableListener.patch, SOLR-433-r698590.patch, > SOLR-433.patch, SOLR-433.patch, SOLR-433.patch, SOLR-433.patch, > solr-433.patch, SOLR-433_unified.patch, spellindexfix.patch > > > With MultiCore functionality coming along, it looks like we'll need to be > able to: > A) snapshot each core's index directory, and > B) replicate any and all cores' complete data directories, not just their > index directories. > Pulled from the "spellchecker and multi-core index replication" thread - > http://markmail.org/message/pj2rjzegifd6zm7m > Otis: > I think that makes sense - distribute everything for a given core, not just > its index. And the spellchecker could then also have its data dir (and only > index/ underneath really) and be replicated in the same fashion. > Right? > Ryan: > Yes, that was my thought. If an arbitrary directory could be distributed, > then you could have > /path/to/dist/index/... > /path/to/dist/spelling-index/... > /path/to/dist/foo > and that would all get put into a snapshot. This would also let you put > multiple cores within a single distribution: > /path/to/dist/core0/index/... > /path/to/dist/core0/spelling-index/... > /path/to/dist/core0/foo > /path/to/dist/core1/index/... > /path/to/dist/core1/spelling-index/... > /path/to/dist/core1/foo -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
