[
https://issues.apache.org/jira/browse/SOLR-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436567#comment-16436567
]
Steve Rowe edited comment on SOLR-11920 at 4/13/18 12:38 AM:
-------------------------------------------------------------
{{TestReplicationHandler.doTestStressReplication()}} has failed recently on
Jenkins in new code committed under this issue - hard linking fails AFAICT
because the source file can't be found, e.g. from
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1675/] - I'll attach the
full log:
{noformat}
[junit4] 2> 756661 INFO (explicit-fetchindex-cmd) [ ]
o.a.s.h.IndexFetcher Starting download (fullCopy=true) to
MockDirectoryWrapper(NIOFSDirectory@/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/index-NIOFSDirectory-040
lockFactory=org.a
pache.lucene.store.NativeFSLockFactory@cd4307)
[junit4] 2> 756661 INFO (explicit-fetchindex-cmd) [ ]
o.a.s.h.IndexFetcher Don't need to download this file. Local file's path is:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.2018040903324047
5/_0.si, checksum is: 4156999003
[junit4] 2> 756662 ERROR (explicit-fetchindex-cmd) [ ]
o.a.s.h.ReplicationHandler Index fetch failed
:org.apache.solr.common.SolrException: Index fetch failed :
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:699)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:369)
[junit4] 2> at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
[junit4] 2> at
org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:298)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.nio.file.NoSuchFileException:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.20180409033240691/_0.si
-> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core
/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.20180409033240475/_0.si
[junit4] 2> at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
[junit4] 2> at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[junit4] 2> at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
[junit4] 2> at java.nio.file.Files.createLink(Files.java:1086)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1046)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:592)
[junit4] 2> ... 4 more
[...]
[junit4] 2> 786815 INFO
(TEST-TestReplicationHandler.doTestStressReplication-seed#[9FAEE8E64FDC4AB6]) [
] o.a.s.h.TestReplicationHandler Waiting for 162 docs
[junit4] 2> 786915 INFO (qtp18488161-9808) [ x:collection1]
o.a.s.c.S.Request [collection1] webapp=/solr path=/select
params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=107 status=0 QTime=0
[junit4] 2> 786915 INFO
(TEST-TestReplicationHandler.doTestStressReplication-seed#[9FAEE8E64FDC4AB6]) [
] o.a.s.h.TestReplicationHandler Waited for 30000ms and found 107 docs
[...]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestReplicationHandler -Dtests.method=doTestStressReplication
-Dtests.seed=9FAEE8E64FDC4AB6 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=et-EE -Dtests.timezone=Africa/Tripoli -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 31.0s J1 | TestReplicationHandler.doTestStressReplication
<<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<162> but
was:<107>
[junit4] > at
__randomizedtesting.SeedInfo.seed([9FAEE8E64FDC4AB6:4405E8204AF42305]:0)
[junit4] > at
org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:904)
[junit4] > at java.lang.Thread.run(Thread.java:748)
{noformat}
was (Author: steve_rowe):
{{TestReplicationHandler.testStressReplication()}} has failed recently on
Jenkins in new code committed under this issue - hard linking fails AFAICT
because the source file can't be found, e.g. from
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1675/] - I'll attach the
full log:
{noformat}
[junit4] 2> 756661 INFO (explicit-fetchindex-cmd) [ ]
o.a.s.h.IndexFetcher Starting download (fullCopy=true) to
MockDirectoryWrapper(NIOFSDirectory@/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/index-NIOFSDirectory-040
lockFactory=org.a
pache.lucene.store.NativeFSLockFactory@cd4307)
[junit4] 2> 756661 INFO (explicit-fetchindex-cmd) [ ]
o.a.s.h.IndexFetcher Don't need to download this file. Local file's path is:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.2018040903324047
5/_0.si, checksum is: 4156999003
[junit4] 2> 756662 ERROR (explicit-fetchindex-cmd) [ ]
o.a.s.h.ReplicationHandler Index fetch failed
:org.apache.solr.common.SolrException: Index fetch failed :
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:699)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:369)
[junit4] 2> at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
[junit4] 2> at
org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:298)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.nio.file.NoSuchFileException:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.20180409033240691/_0.si
-> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core
/test/J1/temp/solr.handler.TestReplicationHandler_9FAEE8E64FDC4AB6-001/solr-instance-017/./collection1/data/index.20180409033240475/_0.si
[junit4] 2> at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
[junit4] 2> at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[junit4] 2> at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
[junit4] 2> at java.nio.file.Files.createLink(Files.java:1086)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1046)
[junit4] 2> at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:592)
[junit4] 2> ... 4 more
[...]
[junit4] 2> 786815 INFO
(TEST-TestReplicationHandler.doTestStressReplication-seed#[9FAEE8E64FDC4AB6]) [
] o.a.s.h.TestReplicationHandler Waiting for 162 docs
[junit4] 2> 786915 INFO (qtp18488161-9808) [ x:collection1]
o.a.s.c.S.Request [collection1] webapp=/solr path=/select
params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=107 status=0 QTime=0
[junit4] 2> 786915 INFO
(TEST-TestReplicationHandler.doTestStressReplication-seed#[9FAEE8E64FDC4AB6]) [
] o.a.s.h.TestReplicationHandler Waited for 30000ms and found 107 docs
[...]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestReplicationHandler -Dtests.method=doTestStressReplication
-Dtests.seed=9FAEE8E64FDC4AB6 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=et-EE -Dtests.timezone=Africa/Tripoli -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 31.0s J1 | TestReplicationHandler.doTestStressReplication
<<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<162> but
was:<107>
[junit4] > at
__randomizedtesting.SeedInfo.seed([9FAEE8E64FDC4AB6:4405E8204AF42305]:0)
[junit4] > at
org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:904)
[junit4] > at java.lang.Thread.run(Thread.java:748)
{noformat}
> Differential file copy for IndexFetcher
> ---------------------------------------
>
> Key: SOLR-11920
> URL: https://issues.apache.org/jira/browse/SOLR-11920
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Ishan Chattopadhyaya
> Assignee: Ishan Chattopadhyaya
> Priority: Major
> Attachments: SOLR-11920.patch, SOLR-11920.patch,
> thetaphi.Lucene.Solr.7.x.Linux.1675.log.txt.gz
>
>
> In the case of fullCopy=true, all files are copied over from the
> leader/master irrespective of whether or not that exact file exists with the
> replica/slave. This is wasteful, esp. in tlog replicas or pull replicas, when
> only a fraction of the total files are different.
> This stems from SOLR-11815.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
