[
https://issues.apache.org/jira/browse/SOLR-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599559#action_12599559
]
noble.paul edited comment on SOLR-561 at 5/25/08 9:48 PM:
----------------------------------------------------------
We shall post a patch in the next few days
The design is as follows:
* SnapShooter.java : registered as a listener on _postCommit/postOptimize_ .
It makes a copy of the latest index to a new snapshot folder (same as it is
today). Only in master. It can optionally take in a 'snapDir' as configuration
if the snapshot is to be created ina folder other than the data directory.
* ReplicationHandler: A requesthandler. This is registered in master & slave.
It takes in the following config in the slave. Master node just needs an empty
requesthandler registration.
{code:xml|title=solrconfig.xml}
<requestHandler name="replication" class="solr.ReplicationHandler">
<str name="masterUrl">http://<host>:<port>/solr/<corename>/replication</str>
<str name="pollInterVal">HH:MM:SS</str>
</requestHandler>
{code}
ReplicationHandler Implements the following methods. Every method is invoked
over *http GET*. These methods are usually trigerred from the slave (over http)
or timer (for snappull). Admin can provide means to invoke some methods like
snappull,snapshoot .
* CMD_GET_FILE: _(command=filecontent&snapshhot=<snapshotname>&file=<
filename>&offset=<fileOffset>&len=<length-ofchunk>&checksum=<true|false>)_ .
This is invoked by a slave only to fetch a file or a part of it . This uses a
custom format (described later)
* CMD_LATEST_SNAP: _(command=latestsnap)_. Returns the name of the latest
snapshot (a namedlist response)
* CMD_GET_SNAPSHOTS: _(command=snaplist)_. Returns a list of all snapshot names
(a namedlist response)
* CMD_GET_FILE_LIST: _(command=filelist&snap=<snapshotname>)_ . A list of all
the files in the snapshot .conains name, lastmodified,size. (a namedlist
response)
* CMD_SNAP_SHOOT: _(command=snapshoot)_. Do a force snapshoot.
* CMD_DISABLE_SNAPPOLL: _(command=disablesnappoll)_. For stopping the timer task
* CMD_SNAP_PULL : _(command=snappull)_. Does the following operations (done in
slave). It is mostly triggered from a timertask based on the pollInterval value.
** calls a CMD_LATEST_SNAP to the master and get the latest snapshot name
** checks if it has the same (or if a snappull is going on)
** if it is to be pulled, call CMD_GET_FILE_LIST to the master
** for each file in the list make a call CMD_GET_FILE to the master. This
command works in the following way
*** the server reads the file stream
*** It uses a CustomStreamResponseWriter _(wt=filestream)_ to write the
content. It has a packetSize (say 1mb)
*** It writes an int for length and another long for Adler32 checksum (if
checksum=true). The packets are written one after another till EOF or an
Exception.
*** SnapPuller.java In the client reads the packet length and checksum and
tries to read the packet.If it is unable to read the given packet or the
checksum does not match or there is an exception , it closes the connection and
makes a new CMD_GET_FILE command with the offset = (totalbytesReceived). If
everything is fine the packets are read till the __bytesDownloaded == fileSize__
*** This is continued till all the files are downloaded.
** creates a folder index.tmp
** for each file in the copied snapshot , try to create a hardlink in the
index.tmp folder.(runs an OS specific command)
** If hardlink creation fails use a copy
** rename _index.tmp_ to _index_
** calls a commit on the updatehandler
**note: The download tries to use the same stream to download the complete file
.
Please comment on the design
was (Author: noble.paul):
We shall post a patch in the next few days
The design is as follows:
* SnapShooter.java : registered as a listener on _postCommit/postOptimize_ .
It makes a copy of the latest index to a new snapshot folder (same as it is
today). Only in master
* ReplicationHandler: A requesthandler. This is registered in master & slave.
It takes in the following config in the slave
{code:xml}
<str name="masterUrl">http://<host>:<port>/solr/<corename>/replication</str>
<str name="pollInterVal">HH:MM:SS</str>
{code}
Implements the following methods
* CMD_GET_FILE: _(command=filecontent&snapshhot=<snapshotname>&file=<
filename>&offset=<fileOffset>&len=<length-ofchunk>&checksum=<true|false>)_
Would
* CMD_LATEST_SNAP: _(command=latestsnap)_. Returns the name of the latest
snapshot (a namedlist response)
* CMD_GET_SNAPSHOTS: _(command=snaps)_. Returns a list of all snapshotnames (a
namedlist response)
* CMD_GET_FILE_LIST: _(command=filelist&snap=<snapshotname>)_ . A list of all
the files in the snapshot .conains name, lastmodified,size. (a namedlist
response)
* CMD_SNAP_SHOOT: _(command=snapshoot)_. Do a force snapshoot.
* CMD_DISABLE_SNAPPOLL: _(command=disablesnappoll)_. For stopping the timer task
* CMD_SNAP_PULL : _(command=snappull)_. Does the following operations (done in
slave). It is mostly triggered from a timertask based on the pollInterval value.
** calls a CMD_LATEST_SNAP to the master and get the latest snapshot name
** checks if it has the same (or if a snappull is going on)
** if it is to be pulled, call CMD_GET_FILE_LIST to the master
** for each file in the list make a call CMD_GET_FILE to the master. This
command works in the following way
*** the server reads the file stream
*** It uses a FileStreamresponseWriter _(wt=filestream)_ to write the content.
It has a packetSize (say 1mb)
*** It writes an int for length and another long for Adler32 checksum (if
checksum=true). The packets are written one after another till EOF or an
Exception.
*** SnapPuller.java In the client reads the packet length and checksum and
tries to read the packet.If it is unable to read the given packet or the
checksum does not match or there is an exception , it closes the connection and
makes a new CMD_GET_FILE command with the offset = (totalbytesReceived+1). If
everything is fine the packets are read till the __bytesDownloaded == fileSize__
*** This is continued till all the files are downloaded.
** creates a folder index.tmp
** for each file in the copied snapshhot , try to create a hardlink in the
index.tmp folder.(runs an OS specific command)
** If hardlink creation fails use a copy
** rename _index.tmp_ to _index_
** calls a commit on the updatehandler
**note: The download tries to use the same stream to download the complete file
.
Please comment on the design
> Solr replication by Solr (for windows also)
> -------------------------------------------
>
> Key: SOLR-561
> URL: https://issues.apache.org/jira/browse/SOLR-561
> Project: Solr
> Issue Type: New Feature
> Components: replication
> Affects Versions: 1.3
> Environment: All
> Reporter: Noble Paul
>
> The current replication strategy in solr involves shell scripts . The
> following are the drawbacks with the approach
> * It does not work with windows
> * Replication works as a separate piece not integrated with solr.
> * Cannot control replication from solr admin/JMX
> * Each operation requires manual telnet to the host
> Doing the replication in java has the following advantages
> * Platform independence
> * Manual steps can be completely eliminated. Everything can be driven from
> solrconfig.xml .
> ** Adding the url of the master in the slaves should be good enough to enable
> replication. Other things like frequency of
> snapshoot/snappull can also be configured . All other information can be
> automatically obtained.
> * Start/stop can be triggered from solr/admin or JMX
> * Can get the status/progress while replication is going on. It can also
> abort an ongoing replication
> * No need to have a login into the machine
> This issue can track the implementation of solr replication in java
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
