[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747857#action_12747857
]
Noble Paul commented on SOLR-1383:
--
The files are reserved for replication (10 secs) . if you do commits very
frequently, say 5 times in 10 secs you end up keeping all the old index files
on the disk. if you restart solr or do a commit again after sometime (10 secs
or more) they will get cleaned up.
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/master/solr/data/index/_b.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 80 ->
> /index/master/solr/data/index/_c.tis
> lr-x
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748139#action_12748139
]
Lance Norskog commented on SOLR-1383:
-
The files are reserved for replication for X amount of time. This makes sense,
but- shouldn't they go away at the end of the timer, or after the replication
is finished? After all, they are not part of a snapshot. Shouldn't the snapshot
client say "I'm done with them?".
I understand about the problems involved in this; it seems like there's a hole
in the state sequences.
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/mast
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748257#action_12748257
]
Noble Paul commented on SOLR-1383:
--
bq.his makes sense, but- shouldn't they go away at the end of the timer, or
after the replication is finished
ReplicationHandler does not take care of cleaning up the index. That is left to
Lucene. Whenever the next IndexWriter#open happens it should cleanup all the
unreserved indexcommits.
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/master/solr/data/index/_b.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 80 ->
> /index/master/solr/
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748682#action_12748682
]
Lance Norskog commented on SOLR-1383:
-
I checked again- the files do not go away. Not after another commit, not after
restarting solr.
The replication commit reservation code definitely has a bug.
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/master/solr/data/index/_b.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 80 ->
> /index/master/solr/data/index/_c.tis
> lr-x-- 1 root root 64 Aug 25 22:52 81 ->
> /index/master/solr/data/index/_c.frq
> lr-x-- 1 root roo
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748688#action_12748688
]
Noble Paul commented on SOLR-1383:
--
bq.he files do not go away. Not after another commit, not after restarting solr.
All the old files are necessary for the index to work. The latest commit is not
the only one that is used. You do an optimize and the old files will go away
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/master/solr/data/index/_b.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 80 ->
> /index/master/solr/data/index/_c.tis
> lr-x-- 1 root root 64 Aug 25 22:
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748964#action_12748964
]
Lance Norskog commented on SOLR-1383:
-
The old files did go away after an optimize. Thank you.
Restarting did not remove them. I suggest that old index files should be
removed after all runtime requirements for them disappear. They should
definitely be removed by restarting. Restarting Solr should cure all runtime
problems; this includes extra files.
There are a lot of Solr sites that want continuous propagation from data source
to indexing to query. If they use Java replication to poll continuously for
updates, it will leave vast amounts of junk files behind.
The current functionality is fine for a Solr 1.4 release, but this issue should
be fixed after that. Please reopen it and mark it for 1.5.
Thanks.
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root r
[jira] Commented: (SOLR-1383) Replication causes master to fail to delete old index files
[
https://issues.apache.org/jira/browse/SOLR-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748968#action_12748968
]
Yonik Seeley commented on SOLR-1383:
bq. I suggest that old index files should be removed after all runtime
requirements for them disappear.
How do you know this isn't happening?
> Replication causes master to fail to delete old index files
> ---
>
> Key: SOLR-1383
> URL: https://issues.apache.org/jira/browse/SOLR-1383
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Environment: Linux CentOS - latest Solr 1.4 trunk - Java 1.6
>Reporter: Lance Norskog
> Fix For: 1.4
>
>
> I have developed a way to make replication leave old index files in the
> master's data/index directory. It is timing-dependent. A sequence of commands
> runs correctly or fails, depending on the timing between the commands.
> Here is the test scenario:
> Start a master and slave version of the Solr distributed example. I used 8080
> for the slave. (See example/etc/jetty.xml)
> Be sure to start with empty solr/data/index files on both master and slave.
> Open the replication administration jsp on the slave (
> http://localhost:8080/solr/admin/replication/index.jsp )
> Disable polling.
> In a text window, go to the example/exampledocs directory and run this script
> {code}
> for x in *.xml
> do
> echo $x
> sh post.sh $x
> sleep 15
> curl "http://localhost:8080/solr/replication?command=fetchindex";
> done
> {code}
> This prints each example file, indexes it, and does a replication command. At
> the end of this exercise, the master and slave solr/data/index files will be
> identical.
> Now, kill master & slave, remove the solr/index/data directories, and start
> over. This time, remove the sleep command from the script. In my
> environment, old Lucene index files were left in the master's data/index.
> Here is what is left in the master data/index.
> The segments_? files are random across runs, but the index files left over
> are consistent.
> Note (courtesy of the Linux 'ls -l /proc/PID/fd' command) that the old files
> are not kept open by the master solr; they are merely left behind.
> In the master server:
> {code}
> % ls solr/data/index
> _0.fdt _1.prx _2.tvx _4.nrm _5.tii _7.frq _8.tvd _a.tvx _c.nrm
> _0.fdx _1.tii _3.fdt _4.prx _5.tis _7.nrm _8.tvf _b.fdt _c.prx
> _0.fnm _1.tis _3.fdx _4.tii _6.fdt _7.prx _8.tvx _b.fdx _c.tii
> _0.frq _2.fdt _3.fnm _4.tis _6.fdx _7.tii _a.fdt _b.fnm _c.tis
> _0.nrm _2.fdx _3.frq _4.tvd _6.fnm _7.tis _a.fdx _b.frq segments.gen
> _0.prx _2.fnm _3.nrm _4.tvf _6.frq _8.fdt _a.fnm _b.nrm segments_8
> _0.tii _2.frq _3.prx _4.tvx _6.nrm _8.fdx _a.frq _b.prx segments_9
> _0.tis _2.nrm _3.tii _5.fdt _6.prx _8.fnm _a.nrm _b.tii segments_a
> _1.fdt _2.prx _3.tis _5.fdx _6.tii _8.frq _a.prx _b.tis segments_b
> _1.fdx _2.tii _4.fdt _5.fnm _6.tis _8.nrm _a.tii _c.fdt segments_c
> _1.fnm _2.tis _4.fdx _5.frq _7.fdt _8.prx _a.tis _c.fdx segments_d
> _1.frq _2.tvd _4.fnm _5.nrm _7.fdx _8.tii _a.tvd _c.fnm
> _1.nrm _2.tvf _4.frq _5.prx _7.fnm _8.tis _a.tvf _c.frq
> {code}
> {code}
> % ls -l /proc/PID/fd
> lr-x-- 1 root root 64 Aug 25 22:52 137 ->
> /index/master/solr/data/index/_a.tis
> lr-x-- 1 root root 64 Aug 25 22:52 138 ->
> /index/master/solr/data/index/_a.frq
> lr-x-- 1 root root 64 Aug 25 22:52 139 ->
> /index/master/solr/data/index/_a.prx
> lr-x-- 1 root root 64 Aug 25 22:52 140 ->
> /index/master/solr/data/index/_a.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 141 ->
> /index/master/solr/data/index/_a.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 142 ->
> /index/master/solr/data/index/_a.tvx
> lr-x-- 1 root root 64 Aug 25 22:52 143 ->
> /index/master/solr/data/index/_a.tvd
> lr-x-- 1 root root 64 Aug 25 22:52 144 ->
> /index/master/solr/data/index/_a.tvf
> lr-x-- 1 root root 64 Aug 25 22:52 145 ->
> /index/master/solr/data/index/_a.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 72 ->
> /index/master/solr/data/index/_b.tis
> lr-x-- 1 root root 64 Aug 25 22:52 73 ->
> /index/master/solr/data/index/_b.frq
> lr-x-- 1 root root 64 Aug 25 22:52 74 ->
> /index/master/solr/data/index/_b.prx
> lr-x-- 1 root root 64 Aug 25 22:52 76 ->
> /index/master/solr/data/index/_b.fdt
> lr-x-- 1 root root 64 Aug 25 22:52 78 ->
> /index/master/solr/data/index/_b.fdx
> lr-x-- 1 root root 64 Aug 25 22:52 79 ->
> /index/master/solr/data/index/_b.nrm
> lr-x-- 1 root root 64 Aug 25 22:52 80 ->
> /index/master/solr/data/index/_c.tis
> lr-x-- 1 root root 64 Aug 25 22:52 81 ->
> /index/master/solr/data/index/_c.frq
> lr-x-- 1 root root 64 Aug 25 22:52 82
