[
https://issues.apache.org/jira/browse/SOLR-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148468#comment-16148468
]
Amrit Sarkar commented on SOLR-11278:
-------------------------------------
Shalin, I am onto this.
CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster is failing
intermittently when Bootstrap gets failed due to unannounced SolrCore shutdown.
This SolrCore shutdown may not be the root issue, but fetchIndex from source.
Please see below:
{code}
[beaster] 1> Adding 10 docs with commit=true, numDocs=1100
[beaster] 2> 62998 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 62998 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63003 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=10
[beaster] 2> 63010 INFO (qtp1081974557-457) [n:127.0.0.1:36154_solr
c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.c.S.Request [cdcr-target_shard1_replica1] webapp=/solr path=/cdcr
params={qt=/cdcr&masterUrl=http://127.0.0.1:36872/solr/cdcr-source_shard1_replica1/&action=BOOTSTRAP&wt=javabin&version=2}
status=0 QTime=32
[beaster] 2> 63012 INFO (qtp1081974557-459) [n:127.0.0.1:36154_solr
c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.c.S.Request [cdcr-target_shard1_replica1] webapp=/solr path=/cdcr
params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[beaster] 2> 63013 INFO
(cdcr-bootstrap-status-136-thread-1-processing-n:127.0.0.1:36872_solr
x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1)
[n:127.0.0.1:36872_solr c:cdcr-source s:shard1 r:core_node1
x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager CDCR bootstrap
running for 1 seconds, sleeping for 2000 ms
[beaster] 2> 63171 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@7cd86816[cdcr-source_shard1_replica1] realtime]
[beaster] 2> 63171 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[beaster] 2> 63171 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 63171 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63172 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2}
status=0 QTime=174
[beaster] 2> 63175 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[beaster] 2> 63176 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Master's generation: 12
[beaster] 2> 63176 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Master's version:
1504153213504
[beaster] 2> 63176 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Slave's generation: 1
[beaster] 2> 63176 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Slave's version: 0
[beaster] 2> 63176 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting replication process
[beaster] 2> 63205 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.h.ReplicationHandler Adding tlog files to list: [{size=4649,
name=tlog.0000000000000000000.1577218955048648704}, {size=4770,
name=tlog.0000000000000000001.1577218955134631936}, {size=4770,
name=tlog.0000000000000000002.1577218955245780992}, {size=4770,
name=tlog.0000000000000000003.1577218955347492864}, {size=4770,
name=tlog.0000000000000000004.1577218955448156160}, {size=4770,
name=tlog.0000000000000000005.1577218955573985280}, {size=4770,
name=tlog.0000000000000000006.1577218955690377216}, {size=4770,
name=tlog.0000000000000000007.1577218955795234816}, {size=4770,
name=tlog.0000000000000000008.1577218955899043840}, {size=4770,
name=tlog.0000000000000000009.1577218956028018688}]
[beaster] 2> 63206 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&wt=javabin&version=2&command=filelist}
status=0 QTime=28
[beaster] 2> 63217 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Number of files in latest
index in master: 39
[beaster] 2> 63217 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Number of tlog files in
master: 10
[beaster] 2> 63228 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is
ready to be used.
[beaster] 2> 63229 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting download
(fullCopy=false) to MockDirectoryWrapper(RAMDirectory@799f4d8d
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@633c5d00)
[beaster] 2> 63240 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_LuceneVarGapDocFreqInterval_0.tiv&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63246 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_Lucene50_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63247 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@3fed3b93[cdcr-source_shard1_replica1] main]
[beaster] 2> 63248 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[beaster] 2> 63248 INFO
(searcherExecutor-170-thread-1-processing-n:127.0.0.1:36872_solr
x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1)
[n:127.0.0.1:36872_solr c:cdcr-source s:shard1 r:core_node1
x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1]
Registered new searcher Searcher@3fed3b93[cdcr-source_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1) Uninverting(_c(6.6.1):C99)))}
[beaster] 2> 63249 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.fdt&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63254 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_Lucene50_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63257 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_Lucene50_0.tim&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63260 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.si&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63265 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.fdx&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63269 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_LuceneVarGapDocFreqInterval_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63273 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_Lucene50_0.tip&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63276 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_TestBloomFilteredLucenePostings_0.blm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63279 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.nvm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63282 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.fnm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63286 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_TestBloomFilteredLucenePostings_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63289 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_TestBloomFilteredLucenePostings_0.tim&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63293 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_LuceneVarGapDocFreqInterval_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63296 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_TestBloomFilteredLucenePostings_0.tip&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63299 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a.nvd&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63302 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_LuceneVarGapDocFreqInterval_0.tib&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63306 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_a_TestBloomFilteredLucenePostings_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63309 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_LuceneVarGapDocFreqInterval_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63312 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.nvd&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63316 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_TestBloomFilteredLucenePostings_0.blm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63320 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.si&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63324 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.nvm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63327 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.fnm&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63330 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_LuceneVarGapDocFreqInterval_0.tiv&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63334 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_TestBloomFilteredLucenePostings_0.tip&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63337 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_Lucene50_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63340 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_Lucene50_0.tip&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63343 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_TestBloomFilteredLucenePostings_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63347 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_TestBloomFilteredLucenePostings_0.tim&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63348 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
status=0 QTime=343
[beaster] 1> Adding 10 docs with commit=true, numDocs=1200
[beaster] 2> 63350 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_Lucene50_0.doc&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63354 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_Lucene50_0.tim&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63356 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=5
[beaster] 2> 63357 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.fdx&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63358 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 63358 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63361 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_TestBloomFilteredLucenePostings_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63410 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_LuceneVarGapDocFreqInterval_0.tib&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63416 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b.fdt&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63420 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=_b_LuceneVarGapDocFreqInterval_0.pos&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63424 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&file=segments_c&checksum=true&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63425 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting download of tlog
files from master: [{size=4649,
name=tlog.0000000000000000000.1577218955048648704}, {size=4770,
name=tlog.0000000000000000001.1577218955134631936}, {size=4770,
name=tlog.0000000000000000002.1577218955245780992}, {size=4770,
name=tlog.0000000000000000003.1577218955347492864}, {size=4770,
name=tlog.0000000000000000004.1577218955448156160}, {size=4770,
name=tlog.0000000000000000005.1577218955573985280}, {size=4770,
name=tlog.0000000000000000006.1577218955690377216}, {size=4770,
name=tlog.0000000000000000007.1577218955795234816}, {size=4770,
name=tlog.0000000000000000008.1577218955899043840}, {size=4770,
name=tlog.0000000000000000009.1577218956028018688}]
[beaster] 2> 63428 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000000.1577218955048648704&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63433 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000001.1577218955134631936&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63437 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000002.1577218955245780992&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63442 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000003.1577218955347492864&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63447 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000004.1577218955448156160&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63451 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000005.1577218955573985280&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63468 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000006.1577218955690377216&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63472 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000007.1577218955795234816&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63479 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000008.1577218955899043840&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63483 INFO (qtp358530981-498) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/replication
params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000009.1577218956028018688&wt=filestream&command=filecontent}
status=0 QTime=0
[beaster] 2> 63484 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Total time taken for
download (fullCopy=false,bytesDownloaded=101357) : 0 secs (null bytes/sec) to
MockDirectoryWrapper(RAMDirectory@799f4d8d
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@633c5d00)
[beaster] 2> 63506 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@4db46823[cdcr-source_shard1_replica1] main]
[beaster] 2> 63506 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[beaster] 2> 63508 INFO
(searcherExecutor-170-thread-1-processing-n:127.0.0.1:36872_solr
x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1)
[n:127.0.0.1:36872_solr c:cdcr-source s:shard1 r:core_node1
x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1]
Registered new searcher Searcher@4db46823[cdcr-source_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1) Uninverting(_c(6.6.1):C99)
Uninverting(_d(6.6.1):C100)))}
[beaster] 2> 63508 INFO (qtp358530981-500) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
status=0 QTime=150
[beaster] 1> Adding 10 docs with commit=true, numDocs=1300
[beaster] 2> 63510 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is
ready to be used.
[beaster] 2> 63510 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Reloading SolrCore
cdcr-target_shard1_replica1
[beaster] 2> 63517 INFO (qtp358530981-501) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=5
[beaster] 2> 63520 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 63520 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63577 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrConfig Using
Lucene MatchVersion: 6.6.1
[beaster] 2> 63614 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema
[cdcr-target_shard1_replica1] Schema name=minimal
[beaster] 2> 63621 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema Loaded
schema minimal/1.1 with uniqueid field id
[beaster] 2> 63621 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.CoreContainer
Reloading SolrCore 'cdcr-target_shard1_replica1' using configuration from
collection cdcr-target
[beaster] 2> 63621 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@5e7f193f[cdcr-source_shard1_replica1] main]
[beaster] 2> 63622 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[beaster] 2> 63624 INFO
(searcherExecutor-170-thread-1-processing-n:127.0.0.1:36872_solr
x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1)
[n:127.0.0.1:36872_solr c:cdcr-source s:shard1 r:core_node1
x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1]
Registered new searcher Searcher@5e7f193f[cdcr-source_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1) Uninverting(_c(6.6.1):C99)
Uninverting(_d(6.6.1):C100) Uninverting(_e(6.6.1):C100)))}
[beaster] 2> 63625 INFO (qtp358530981-503) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
status=0 QTime=104
[beaster] 1> Adding 10 docs with commit=true, numDocs=1400
[beaster] 2> 63629 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore
[[cdcr-target_shard1_replica1] ] Opening new SolrCore at
[/home/jenkins/lucene-solr/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_20C3001E01E3B41B-001/cdcr-target-004/node1/cdcr-target_shard1_replica1],
dataDir=[/home/jenkins/lucene-solr/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_20C3001E01E3B41B-001/cdcr-target-004/node1/./cdcr-target_shard1_replica1/data/]
[beaster] 2> 63634 INFO (qtp358530981-502) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=6
[beaster] 2> 63636 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 63636 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63806 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@48e60ebb[cdcr-source_shard1_replica1] main]
[beaster] 2> 63806 INFO
(searcherExecutor-170-thread-1-processing-n:127.0.0.1:36872_solr
x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1)
[n:127.0.0.1:36872_solr c:cdcr-source s:shard1 r:core_node1
x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1]
Registered new searcher Searcher@48e60ebb[cdcr-source_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1) Uninverting(_c(6.6.1):C99)
Uninverting(_d(6.6.1):C100) Uninverting(_e(6.6.1):C100)
Uninverting(_f(6.6.1):C100)))}
[beaster] 2> 63807 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Hard
AutoCommit: disabled
[beaster] 2> 63807 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Soft
AutoCommit: disabled
[beaster] 2> 63807 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[beaster] 2> 63808 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
status=0 QTime=171
[beaster] 1> Adding 10 docs with commit=true, numDocs=1500
[beaster] 2> 63810 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher
Opening [Searcher@28386b90[cdcr-target_shard1_replica1] main]
[beaster] 2> 63813 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/cdcr-target
[beaster] 2> 63814 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/cdcr-target
[beaster] 2> 63814 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at
/configs/cdcr-target/managed-schema
[beaster] 2> 63815 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[beaster] 2> 63815 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.ReplicationHandler
Commits will be reserved for 10000
[beaster] 2> 63827 INFO (qtp358530981-504) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.c.S.Request [cdcr-source_shard1_replica1] webapp=/solr path=/update
params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=15
[beaster] 2> 63832 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[beaster] 2> 63832 INFO (qtp358530981-497) [n:127.0.0.1:36872_solr
c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@10111fe6
[beaster] 2> 63853 INFO
(searcherExecutor-188-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1]
Registered new searcher Searcher@28386b90[cdcr-target_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1)))}
[beaster] 2> 63860 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
[beaster] 2> 63865 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher
Opening [Searcher@4c22895c[cdcr-target_shard1_replica1] main]
[beaster] 2> 63866 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore
[cdcr-target_shard1_replica1] CLOSING SolrCore
org.apache.solr.core.SolrCore@ebbd9dc
[beaster] 2> 63866 INFO
(searcherExecutor-188-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1]
Registered new searcher Searcher@4c22895c[cdcr-target_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1)))}
[beaster] 2> 63866 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler
Solr core is being closed - shutting down CDCR handler @ cdcr-target:shard1
[beaster] 2> 63868 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.core.cdcr-target.shard1.replica1
[beaster] 2> 63869 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler No replay needed.
[beaster] 2> 63869 ERROR
(updateExecutor-109-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Bootstrap operation
failed
[beaster] 2> java.util.concurrent.ExecutionException:
java.lang.AssertionError
[beaster] 2> at
java.util.concurrent.FutureTask.report(FutureTask.java:122)
[beaster] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:192)
[beaster] 2> at
org.apache.solr.handler.CdcrRequestHandler.lambda$handleBootstrapAction$0(CdcrRequestHandler.java:644)
[beaster] 2> at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[beaster] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[beaster] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
[beaster] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[beaster] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[beaster] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[beaster] 2> at java.lang.Thread.run(Thread.java:748)
[beaster] 2> Caused by: java.lang.AssertionError
[beaster] 2> at
org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:787)
[beaster] 2> at
org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:714)
[beaster] 2> at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
[beaster] 2> ... 5 more
{code}
Replication takes place successfully once:
{code}
[beaster] 2> 63860 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1]
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
[beaster] 2> 63865 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher
Opening [Searcher@4c22895c[cdcr-target_shard1_replica1] main]
[beaster] 2> 63866 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore
[cdcr-target_shard1_replica1] CLOSING SolrCore
org.apache.solr.core.SolrCore@ebbd9dc
[beaster] 2> 63866 INFO
(searcherExecutor-188-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1]
Registered new searcher Searcher@4c22895c[cdcr-target_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.6.1):C1000)
Uninverting(_b(6.6.1):C1)))}
[beaster] 2> 63484 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Total time taken for
download (fullCopy=false,bytesDownloaded=101357) : 0 secs (null bytes/sec) to
MockDirectoryWrapper(RAMDirectory@799f4d8d
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@633c5d00)
{code}
then {{cdcr-target}} abruptly closes, with bootstrap operation failed, giving
status "cancelled":
{code}
[beaster] 2> 63866 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler
Solr core is being closed - shutting down CDCR handler @ cdcr-target:shard1
[beaster] 2> 63868 INFO (Thread-98) [n:127.0.0.1:36154_solr c:cdcr-target
s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.core.cdcr-target.shard1.replica1
[beaster] 2> 63869 INFO
(recoveryExecutor-110-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler No replay needed.
[beaster] 2> 63869 ERROR
(updateExecutor-109-thread-1-processing-n:127.0.0.1:36154_solr
x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1)
[n:127.0.0.1:36154_solr c:cdcr-target s:shard1 r:core_node1
x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Bootstrap operation
failed
[beaster] 2> java.util.concurrent.ExecutionException:
java.lang.AssertionError
.......
{code}
I am adding addition logging in CdcrRequesHandler to catch and print the root
problem of failed bootstrap operation. will post the results soon.
> CdcrBootstrapTest failing in branch_6_6
> ---------------------------------------
>
> Key: SOLR-11278
> URL: https://issues.apache.org/jira/browse/SOLR-11278
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
> Reporter: Amrit Sarkar
> Assignee: Varun Thacker
> Attachments: SOLR-11278-cancel-bootstrap-on-stop.patch,
> SOLR-11278.patch, test_results
>
>
> I ran beast for 10 rounds:
> ant beast -Dtestcase=CdcrBootstrapTest -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=vi -Dtests.timezone=Asia/Yekaterinburg -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII -Dbeast.iters=10
> and seeing following failure:
> {code}
> [beaster] [01:37:16.282] FAILURE 153s |
> CdcrBootstrapTest.testBootstrapWithSourceCluster <<<
> [beaster] > Throwable #1: java.lang.AssertionError: Document mismatch on
> target after sync expected:<2000> but was:<1000>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
