[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Sekiguchi updated SOLR-561: Component/s: (was: replication (scripts)) replication (java) change component from scripts to java > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication (java) >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-fixes.patch, SOLR-561-fixes.patch, SOLR-561-fixes.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch comments only > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-fixes.patch, SOLR-561-fixes.patch, SOLR-561-fixes.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated SOLR-561:
--
Attachment: SOLR-561-fixes.patch
Here's an update to the "fixes" patch that fixes an issue with
setReserveDuration when called with different reserveTimes. Previously, the
new value overwrites the old, regardless of it's value. The approach to fix is
a basic spin loop (see below). Anyone see issues with this approach?
{code}
public void setReserveDuration(Long indexVersion, long reserveTime) {
long timeToSet = System.currentTimeMillis() + reserveTime;
for(;;) {
Long previousTime = reserves.put(indexVersion, timeToSet);
// this is the common success case: the older time didn't exist, or
// came before the new time.
if (previousTime == null || previousTime <= timeToSet) break;
// At this point, we overwrote a longer reservation, so we want to
restore the older one.
// the problem is that an even longer reservation may come in concurrently
// and we don't want to overwrite that one too. We simply keep retrying
in a loop
// with the maximum time value we have seen.
timeToSet = previousTime;
}
}
{code}
I think this is also a great example of where comments explaining how things
work are really needed.
> Solr replication by Solr (for windows also)
> ---
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
>Affects Versions: 1.4
> Environment: All
>Reporter: Noble Paul
>Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: deletion_policy.patch, SOLR-561-core.patch,
> SOLR-561-fixes.patch, SOLR-561-fixes.patch, SOLR-561-fixes.patch,
> SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch,
> SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch, SOLR-561.patch
>
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> * From a development perspective, we can unit test it
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-561: -- Attachment: SOLR-561-fixes.patch Updated the fixes patch with more thread safety fixes. Q: what is ReplicationHandler.getIndexVersion() supposed to return, and why? It currently returns the version of the visible index (registered). Should it be the most recent version of the index we have? Any reason it isn't using ReplicationHandler.indexCommitPoint? Also, I think we should all work at adding more comments to code as it is written. Lack of comments made this patch harder to review. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-fixes.patch, SOLR-561-fixes.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-561: -- Attachment: SOLR-561-fixes.patch Attaching some little thread safety fixes (mostly adding volatile to values modified and read from different threads). > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-fixes.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch silly me. The packetsWriitten variable was not incremented > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch patch contains changes for reserve being set for 10secs by default after every 5 packets (5 MB) are written. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: SOLR-561.patch Again a minor fix in replication admin page > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-561: --- Attachment: SOLR-561.patch Another iteration over Akshay's patch. # Made the collections used for keeping statistics synchronized to avoid concurrent modification exceptions. # Removed @author tags and put @version and @since 1.4 tags > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: SOLR-561.patch Patch with minor fixes related to the admin page. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-561: --- Attachment: SOLR-561.patch Updated patch with a couple of bug fixes related to closing connections and refcounted index searcher. Other cosmetic changes include code formatting and javadocs. Noble has put up a wiki page at http://wiki.apache.org/solr/SolrReplication detailing the features and configuration. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: SOLR-561.patch Patch with following changes: # segments_ file moved in the end. # Some minor changes in replication admin page # Test case for index and config files replication. # Some minor bug fixes. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: SOLR-561-full.patch Patch includes changes for SOLR-658 and SOLR-561. Changes: Abort command implementation in ReplicationHandler to abort ongoing replication. More information displayed in the replication admin page, such as file being copied, time elapsed, time remaining, size of file downloaded and so on. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Comment: was deleted > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: (was: SOLR-561-full.patch) > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshay K. Ukey updated SOLR-561:
Attachment: SOLR-561-full.patch
Full patch with:
1. Support for reserving commit point. Configurable with commitReserveDuration
configuration in ReplicationHandler section.
{code:xml}
commit
schema.xml,stopwords.txt,elevate.xml
01:00:00
{code}
2. Admin page for displaying replication details.
3. Combines changes for SOLR 617 and SOLR 658.
> Solr replication by Solr (for windows also)
> ---
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
>Affects Versions: 1.4
> Environment: All
>Reporter: Noble Paul
>Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: deletion_policy.patch, SOLR-561-core.patch,
> SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch,
> SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch, SOLR-561.patch
>
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> * From a development perspective, we can unit test it
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshay K. Ukey updated SOLR-561:
Attachment: SOLR-561-full.patch
Full patch with:
1. Support for reserving commit point. Configurable with commitReserveDuration
configuration in ReplicationHandler section.
{code:xml}
commit
schema.xml,stopwords.txt,elevate.xml
01:00:00
{code}
2. Admin page for displaying replication details.
3. Combines changes for SOLR 617 and SOLR 658.
> Solr replication by Solr (for windows also)
> ---
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
>Affects Versions: 1.4
> Environment: All
>Reporter: Noble Paul
>Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: deletion_policy.patch, SOLR-561-core.patch,
> SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561-full.patch,
> SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch, SOLR-561.patch
>
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> * From a development perspective, we can unit test it
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay K. Ukey updated SOLR-561: Attachment: SOLR-561-full.patch Patch in sync with the trunk, includes changes for [SOLR-617|https://issues.apache.org/jira/browse/SOLR-617] and [SOLR-658|https://issues.apache.org/jira/browse/SOLR-658]. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-561: --- Attachment: SOLR-561-full.patch This patch contains all the replication related changes as well as changes in Solr itself. It contains the changes in SOLR-617 and SOLR-658. Un-tested version. I just compiled all the various patches and brought them up to trunk. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561-full.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-561: --- Attachment: SOLR-561.patch This patch contains only replication related changes. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Fix For: 1.4 > > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, > SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-561: --- Description: The current replication strategy in solr involves shell scripts . The following are the drawbacks with the approach * It does not work with windows * Replication works as a separate piece not integrated with solr. * Cannot control replication from solr admin/JMX * Each operation requires manual telnet to the host Doing the replication in java has the following advantages * Platform independence * Manual steps can be completely eliminated. Everything can be driven from solrconfig.xml . ** Adding the url of the master in the slaves should be good enough to enable replication. Other things like frequency of snapshoot/snappull can also be configured . All other information can be automatically obtained. * Start/stop can be triggered from solr/admin or JMX * Can get the status/progress while replication is going on. It can also abort an ongoing replication * No need to have a login into the machine * From a development perspective, we can unit test it This issue can track the implementation of solr replication in java was: The current replication strategy in solr involves shell scripts . The following are the drawbacks with the approach * It does not work with windows * Replication works as a separate piece not integrated with solr. * Cannot control replication from solr admin/JMX * Each operation requires manual telnet to the host Doing the replication in java has the following advantages * Platform independence * Manual steps can be completely eliminated. Everything can be driven from solrconfig.xml . ** Adding the url of the master in the slaves should be good enough to enable replication. Other things like frequency of snapshoot/snappull can also be configured . All other information can be automatically obtained. * Start/stop can be triggered from solr/admin or JMX * Can get the status/progress while replication is going on. It can also abort an ongoing replication * No need to have a login into the machine This issue can track the implementation of solr replication in java Affects Version/s: (was: 1.3) 1.4 > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.4 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > * From a development perspective, we can unit test it > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch This patch is to * sync with the trunk (SOLR-638 changes) * uses java1.4 _ScheduledExcecutorService_ instead of timer * _SolrCore.close()_ is done in a refcounted way > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch new patch that takes care of the refcount. this is a complete patch > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561-core.patch Just the changes required to the core > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul > Attachments: deletion_policy.patch, SOLR-561-core.patch, > SOLR-561.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
The latest patch makes some serious changes to the core Solr classes .
I wish somebody (who knows the core well) to take a look at the
changes and comment on those changes.
The major changes are
* index directory can change
* Core reload
* IndexDeletionPolicy (This patch is Yonik's)
It is important for us to know that our approach is right/acceptable
because we will be rolling this out into production as soon as our QA
team clears the patch.
On Wed, Jul 9, 2008 at 5:01 PM, Noble Paul (JIRA) <[EMAIL PROTECTED]> wrote:
>
> [
> https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Noble Paul updated SOLR-561:
>
>
>Attachment: SOLR-561.patch
>
> This patch relies on the IndexDeletionPolicy to identify files to be
> replicated. It also supposrts replication of conf files.
>
> The configuration is as follows
> on master
> {code:title=solrconfig.xml}
>
>
>
>commit
>
> schema.xml,stopwords.txt,elevate.xml
>
>
> {code}
> on slave
> {code:title=solrconfig.xml}
>
>
> name="masterUrl">http://localhost:port/solr/corename/replication
>00:00:20
>
>
> {code}
>
> The Replication strategy is changed as follows
> * CMD_INDEX_VERSION: gets the version of the current _IndexCommit _ to be
> replicated from the master. if the version is same, no need to replicate. If
> it is different
> * CMD_FILE_LIST : Get the list of file names for the current _IndexCommit_ .
> Checks with the local index and identifies modified files by comparing names
> an sizes.
> * CMD-FILE_CONTENT : For each files to be downloaded, issue this command an
> download the content to a temp folder. After successful completion copy them
> to the index folder and isse a commit
> * If the current index is stale, or not able to synchronize, copy all the
> files . An index,properties file is written, which has the location of the
> new index directory
> * _CoreDescriptor_ has a new method to reload core.
> * If conf files are modified they are copied to the conf folder after taking
> a backup of the old. Then the core is reloaded
>
>
>
>
>
>
>
>
>> Solr replication by Solr (for windows also)
>> ---
>>
>> Key: SOLR-561
>> URL: https://issues.apache.org/jira/browse/SOLR-561
>> Project: Solr
>> Issue Type: New Feature
>> Components: replication
>>Affects Versions: 1.3
>> Environment: All
>>Reporter: Noble Paul
>> Attachments: deletion_policy.patch, SOLR-561.patch, SOLR-561.patch,
>> SOLR-561.patch
>>
>>
>> The current replication strategy in solr involves shell scripts . The
>> following are the drawbacks with the approach
>> * It does not work with windows
>> * Replication works as a separate piece not integrated with solr.
>> * Cannot control replication from solr admin/JMX
>> * Each operation requires manual telnet to the host
>> Doing the replication in java has the following advantages
>> * Platform independence
>> * Manual steps can be completely eliminated. Everything can be driven from
>> solrconfig.xml .
>> ** Adding the url of the master in the slaves should be good enough to
>> enable replication. Other things like frequency of
>> snapshoot/snappull can also be configured . All other information can be
>> automatically obtained.
>> * Start/stop can be triggered from solr/admin or JMX
>> * Can get the status/progress while replication is going on. It can also
>> abort an ongoing replication
>> * No need to have a login into the machine
>> This issue can track the implementation of solr replication in java
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
--Noble Paul
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-561:
Attachment: SOLR-561.patch
This patch relies on the IndexDeletionPolicy to identify files to be
replicated. It also supposrts replication of conf files.
The configuration is as follows
on master
{code:title=solrconfig.xml}
commit
schema.xml,stopwords.txt,elevate.xml
{code}
on slave
{code:title=solrconfig.xml}
http://localhost:port/solr/corename/replication
00:00:20
{code}
The Replication strategy is changed as follows
* CMD_INDEX_VERSION: gets the version of the current _IndexCommit _ to be
replicated from the master. if the version is same, no need to replicate. If it
is different
* CMD_FILE_LIST : Get the list of file names for the current _IndexCommit_ .
Checks with the local index and identifies modified files by comparing names an
sizes.
* CMD-FILE_CONTENT : For each files to be downloaded, issue this command an
download the content to a temp folder. After successful completion copy them to
the index folder and isse a commit
* If the current index is stale, or not able to synchronize, copy all the files
. An index,properties file is written, which has the location of the new index
directory
* _CoreDescriptor_ has a new method to reload core.
* If conf files are modified they are copied to the conf folder after taking a
backup of the old. Then the core is reloaded
> Solr replication by Solr (for windows also)
> ---
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
>Affects Versions: 1.3
> Environment: All
>Reporter: Noble Paul
> Attachments: deletion_policy.patch, SOLR-561.patch, SOLR-561.patch,
> SOLR-561.patch
>
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-561: -- Attachment: deletion_policy.patch Attaching deletion_policy.patch This exports a SolrDeletionPolicy via UpdateHandler.getDeletionPolicy() It can be used to get the latest SolrIndexCommit, which lists the files that are part of the commit, and can be used to reserve/lease the commit point for a certain amount of time. This could be used to enable replication directly out of the index directory and avoid copying on systems like Windows. Each SolrIndexCommit has an id, which can be used by a client as a correlation id. Since a single file can be part of multiple commit points, a replication client should specify what commit point it is copying. The server can then look up that commit point and extend the lease. > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul > Attachments: deletion_policy.patch, SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-561: Attachment: SOLR-561.patch This patch includes * a new method in ReplicationHandler _filechecksum_ , which can return the checksums of a given list of files in a snapshot * The snappuller will request for the checksums of the files if the name and size are same (compared to the current index) * Only if the checksums are different the file is downloaded * Other files are copied from the current index > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul > Attachments: SOLR-561.patch, SOLR-561.patch > > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-561:
Attachment: SOLR-561.patch
The first cut. very crude , but worx .No OS specific commands
The design for snapshoot , snappull is same as described in the design overview
snapinstall is done the following way
* write out a file index.properties in dataDirectory.
* call commit command
* The SolrCore is modified. A new method getNewIndexDir() is added. It loads
the properties file (if exists, else fall back to the old behavior), read a
property 'index' and return
* Any new SolrIndexWriter, SolrIndexSearcher is loaded from the new index dir
* Old ones continue to use the old index dir . getindexDir() returns the index
dir used by the current SolrIndexReader/SolrIndexWriter
Use the following configuration
in master
# register snapshooter
{code:xml}
true
{code}
# register replication Handler
{code:xml}
{code}
# register a new ResponseWriter
{code:xml}
{code}
In the Slave
# register the replication handler
{code:xml}
http://localhost:8080/solr/replication
00:00:30
{code}
> Solr replication by Solr (for windows also)
> ---
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
>Affects Versions: 1.3
> Environment: All
>Reporter: Noble Paul
> Attachments: SOLR-561.patch
>
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-561) Solr replication by Solr (for windows also)
[ https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Klaas updated SOLR-561: Fix Version/s: (was: 1.3) > Solr replication by Solr (for windows also) > --- > > Key: SOLR-561 > URL: https://issues.apache.org/jira/browse/SOLR-561 > Project: Solr > Issue Type: New Feature > Components: replication >Affects Versions: 1.3 > Environment: All >Reporter: Noble Paul > > The current replication strategy in solr involves shell scripts . The > following are the drawbacks with the approach > * It does not work with windows > * Replication works as a separate piece not integrated with solr. > * Cannot control replication from solr admin/JMX > * Each operation requires manual telnet to the host > Doing the replication in java has the following advantages > * Platform independence > * Manual steps can be completely eliminated. Everything can be driven from > solrconfig.xml . > ** Adding the url of the master in the slaves should be good enough to enable > replication. Other things like frequency of > snapshoot/snappull can also be configured . All other information can be > automatically obtained. > * Start/stop can be triggered from solr/admin or JMX > * Can get the status/progress while replication is going on. It can also > abort an ongoing replication > * No need to have a login into the machine > This issue can track the implementation of solr replication in java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
