Hi all
1. Setup simple 1 node solrcloud test setup using docker-compose,
solr:8.5.2, zookeeper:3.5.8.
2. Upload a configset
3. Create two collections, one standard collection, one CRA, both
using the same configset
legacy:
action=CREATE&name=products_old&collection.configName=products&autoAddReplicas=true&numShards=1&maxShardsPerNode=-1
CRA:
{
"create-alias": {
"name": "products_20200609",
"router": {
"name": "category",
"field": "date_published.year",
"maxCardinality": 30,
"mustMatch": "(199[6-9]|20[0,1,2][0-9])"
},
"create-collection": {
"config": "products",
"numShards": 1,
"nrtReplicas": 1,
"tlogReplicas": 0,
"maxShardsPerNode": 1,
"autoAddReplicas": true
}
}
}
Post a small selection of docs in JSON format using curl to non-CRA
collection -> OK
> $ docker-compose exec -T solr curl -H 'Content-Type: application/json'
> -d@/resources/product-json/products-12381742.json
> http://solr:8983/solr/products_old/update/json/docs
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.6M 100 71 100 11.6M 5 950k 0:00:14 0:00:12 0:00:02 687k
{
"responseHeader":{
"rf":1,
"status":0,
"QTime":12541}}
The same documents, sent to the CRA -> boom
> $ docker-compose exec -T solr curl -H 'Content-Type: application/json'
> -d@/resources/product-json/products-12381742.json
> http://solr:8983/solr/products_20200609/update/json/docs
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.6M 100 888 100 11.6M 366 4913k 0:00:02 0:00:02 --:--:-- 4914k
{
"responseHeader":{
"status":400,
"QTime":2422},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException",
"error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
"root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
"msg":"Async exception during distributed update: Error from
server at
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/:
null\n\n\n\nrequest:
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/\nRemote
error message: Cannot parse provided JSON: JSON Parse Error:
char=\u0002,position=0 AFTER='\u0002'
BEFORE='��¶ms��2update.contentType0applicat'",
"code":400}}
Repeating the request again to the CRA -> OK
> $ docker-compose exec -T solr curl -H 'Content-Type: application/json'
> -d@/resources/product-json/products-12381742.json
> http://solr:8983/solr/products_20200609/update/json/docs
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.6M 100 71 100 11.6M 6 1041k 0:00:11 0:00:11 --:--:-- 706k
{
"responseHeader":{
"rf":1,
"status":0,
"QTime":11446}}
It seems to be related to when a new collection is needed to be
created by the CRA.
The relevant logs:
2020-06-09 02:12:56.107 INFO
(OverseerThreadFactory-9-thread-3-processing-n:10.20.36.130:8983_solr)
[ ] o.a.s.c.a.c.CreateCollectionCmd Create collection
products_20200609__CRA__2005
2020-06-09 02:12:56.232 INFO
(OverseerStateUpdate-72169202568593409-10.20.36.130:8983_solr-n_0000000000)[
] o.a.s.c.o.SliceMutator createReplica() {
"operation":"ADDREPLICA",
"collection":"products_20200609__CRA__2005",
"shard":"shard1",
"core":"products_20200609__CRA__2005_shard1_replica_n1",
"state":"down",
"base_url":"http://10.20.36.130:8983/solr";,
"node_name":"10.20.36.130:8983_solr",
"type":"NRT",
"waitForFinalState":"false"}
2020-06-09 02:12:56.444 INFO (qtp90045638-25) [
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&coreNodeName=core_node2&collection.configName=products&newCollection=true&name=products_20200609__CRA__2005_shard1_replica_n1&action=CREATE&numShards=1&collection=products_20200609__CRA__2005&shard=shard1&wt=javabin&version=2&replicaType=NRT
2020-06-09 02:12:56.476 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.5.1
2020-06-09 02:12:56.512 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.s.IndexSchema
[products_20200609__CRA__2005_shard1_replica_n1] Schema name=variants
2020-06-09 02:12:56.543 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Registered ManagedResource impl
org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager
for path /schema/analysis/synonyms/default
2020-06-09 02:12:56.543 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Registered ManagedResource impl
org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path
/schema/analysis/stopwords/verbatim
2020-06-09 02:12:56.543 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Added observer of type
org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory to
existing ManagedResource /schema/analysis/synonyms/default
2020-06-09 02:12:56.544 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.s.IndexSchema
Loaded schema variants/1.5 with uniqueidfield id
2020-06-09 02:12:56.544 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Added observer of type
org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory to
existing ManagedResource /schema/analysis/synonyms/default
2020-06-09 02:12:56.545 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Added observer of type
org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory to
existing ManagedResource /schema/analysis/synonyms/default
2020-06-09 02:12:56.545 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.r.RestManager
Added observer of type
org.apache.solr.rest.schema.analysis.ManagedStopFilterFactory to
existing ManagedResource /schema/analysis/stopwords/verbatim
2020-06-09 02:12:56.545 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.CoreContainer Creating SolrCore
'products_20200609__CRA__2005_shard1_replica_n1' using configuration
from configset products, trusted=true
2020-06-09 02:12:56.545 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.products_20200609__CRA__2005.shard1.replica_n1' (registry
'solr.core.products_20200609__CRA__2005.shard1.replica_n1') enabled at
server: com.sun.jmx.mbeanserver.JmxMBeanServer@39d9314d
2020-06-09 02:12:56.545 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SolrCore
[[products_20200609__CRA__2005_shard1_replica_n1] ] Opening new
SolrCore at [/var/solr/data/products_20200609__CRA__2005_shard1_replica_n1],
dataDir=[/var/solr/data/products_20200609__CRA__2005_shard1_replica_n1/data/]
2020-06-09 02:12:56.594 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
2020-06-09 02:12:56.594 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
2020-06-09 02:12:56.596 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2020-06-09 02:12:56.596 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-06-09 02:12:56.597 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@13e63f8f[products_20200609__CRA__2005_shard1_replica_n1]
main]
2020-06-09 02:12:56.599 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIOwith
znodeBase: /configs/products
2020-06-09 02:12:56.600 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json
using ZooKeeperStorageIO:path=/configs/products
2020-06-09 02:12:56.600 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path
_schema_analysis_stopwords_verbatim.json using
ZooKeeperStorageIO:path=/configs/products
2020-06-09 02:12:56.600 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResource Loaded initArgs {ignoreCase=false} for
/schema/analysis/stopwords/verbatim
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.s.a.ManagedWordSetResource Loaded 0 words for
/schema/analysis/stopwords/verbatim
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResource Notified 1 observers of
/schema/analysis/stopwords/verbatim
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded null at path
_schema_analysis_synonyms_default.json using
ZooKeeperStorageIO:path=/configs/products
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.s.a.ManagedSynonymFilterFactory Loaded 0 synonym mappings for
/schema/analysis/synonyms/default
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.r.ManagedResource Notified 2 observers of
/schema/analysis/synonyms/default
2020-06-09 02:12:56.601 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for themanaged
schema at /configs/products/managed-schema
2020-06-09 02:12:56.602 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the
latest
2020-06-09 02:12:56.603 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
2020-06-09 02:12:56.604 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1668985679096315904
2020-06-09 02:12:56.607 INFO
(searcherExecutor-24-thread-1-processing-n:10.20.36.130:8983_solr
x:products_20200609__CRA__2005_shard1_replica_n1
c:products_20200609__CRA__2005 s:shard1 r:core_node2)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
Searcher@13e63f8f[products_20200609__CRA__2005_shard1_r
eplica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
2020-06-09 02:12:56.607 INFO
(searcherExecutor-24-thread-1-processing-n:10.20.36.130:8983_solr
x:products_20200609__CRA__2005_shard1_replica_n1
c:products_20200609__CRA__2005 s:shard1 r:core_node2)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.S.Request
[products_20200609__CRA__2005_shard1_replica_n1] webapp=null
path=null params={q=*:*&distrib=false&sort=
date_published+desc,+id+asc&rows=100&event=firstSearcher} hits=0
status=0 QTime=0
2020-06-09 02:12:56.607 INFO
(searcherExecutor-24-thread-1-processing-n:10.20.36.130:8983_solr
x:products_20200609__CRA__2005_shard1_replica_n1
c:products_20200609__CRA__2005 s:shard1 r:core_node2)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-06-09 02:12:56.610 INFO
(searcherExecutor-24-thread-1-processing-n:10.20.36.130:8983_solr
x:products_20200609__CRA__2005_shard1_replica_n1
c:products_20200609__CRA__2005 s:shard1 r:core_node2)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SolrCore
[products_20200609__CRA__2005_shard1_replica_n1] Registered new
searcher Searcher@13e63f8f[products_20200$
09__CRA__2005_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
2020-06-09 02:12:56.617 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.ZkShardTerms
Successful update of terms at
/collections/products_20200609__CRA__2005/terms/shard1 to
Terms{values={core_node2=0}, version=0}
2020-06-09 02:12:56.618 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContextBase make sure parent is created
/collections/products_20200609__CRA__2005/leaders/shard1
2020-06-09 02:12:56.631 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2020-06-09 02:12:56.632 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader -try and
sync
2020-06-09 02:12:56.632 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SyncStrategy
Sync replicas to
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
2020-06-09 02:12:56.632 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SyncStrategy
Sync Success - now sync replicas to me
2020-06-09 02:12:56.633 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.SyncStrategy
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
has no replicas
2020-06-09 02:12:56.633 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration
node /collections/products_20200609__CRA__2005/leaders/shard1/leader
after winning as
/collections/products_20200609__CRA__2005/leader_elect/shard1/election/72169202568593409-core_node2-n_0000000000
2020-06-09 02:12:56.639 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
shard1
2020-06-09 02:12:56.745 INFO (zkCallback-8-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/products_20200609__CRA__2005/state.json] for
collection [products_20200609__CRA__2005] has occurred - updating...
(live nodes size: [1])
2020-06-09 02:12:56.745 INFO (zkCallback-8-thread-5) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/products_20200609__CRA__2005/state.json] for
collection [products_20200609__CRA__2005] has occurred - updating...
(live nodes size: [1])
2020-06-09 02:12:56.747 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.c.ZkController
I am the leader, no recovery necessary
2020-06-09 02:12:56.751 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=products&newCollection=true&name=products_20200609__CRA__2005_shard1_replica_n1&action=CREATE&numShards=1&collection=products_20200609__CRA__2005$
shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=308
2020-06-09 02:12:56.751 INFO
(OverseerThreadFactory-9-thread-3-processing-n:10.20.36.130:8983_solr)
[ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be
active for at most 45 seconds. Check all shard replicas
2020-06-09 02:12:56.857 INFO (zkCallback-8-thread-5) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/products_20200609__CRA__2005/state.json] for
collection [products_20200609__CRA__2005] has occurred - updating...
(live nodes size: [1])
2020-06-09 02:12:56.857 INFO (zkCallback-8-thread-6) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/products_20200609__CRA__2005/state.json] for
collection [products_20200609__CRA__2005] has occurred - updating...
(live nodes size: [1])
2020-06-09 02:12:56.857 INFO (zkCallback-8-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/products_20200609__CRA__2005/state.json] for
collection [products_20200609__CRA__2005] has occurred - updating...
(live nodes size: [1])
2020-06-09 02:12:57.013 INFO (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.u.p.LogUpdateProcessorFactory
[products_20200609__CRA__2005_shard1_replica_n1] webapp=/solr
path=/update
params={update.distrib=NONE&df=stemmed_cosmetics_packaging_fulltext&json.command=false&alias.update.distrib=TOLEADER&update.contentType=application/json&distrib.f$
om=http://10.20.36.130:8983/solr/products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1/&wt=javabin&version=2}{}
0 2
2020-06-09 02:12:57.013 ERROR (qtp90045638-25)
[c:products_20200609__CRA__2005 s:shard1 r:core_node2
x:products_20200609__CRA__2005_shard1_replica_n1]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
Cannot parse provided JSON: JSON Parse Error: char=,position=0
AFTER='' BEFORE='��¶ms��2update.contentType0applicat'
at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:123)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:84)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Unknown Source)
2020-06-09 02:12:58.116 INFO
(OverseerCollectionConfigSetProcessor-72169202568593409-10.20.36.130:8983_solr-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000004 doesn't exist.
Requestor may have disconnected from ZooKeeper
2020-06-09 02:12:58.505 ERROR
(updateExecutor-5-thread-1-processing-x:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1
r:core_node2 null n:10.20.36.130:8983_solr
c:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP
s:shard1)
[c:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP
s:shard1 r:core_node2 x:products_20200609__CRA__NEW_CATEGORY_ROU
TED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1]
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{,id=553496-388005}; node=ForwardNode:
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
to http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
=> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error
from server at
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/:
null
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/:
null
request:
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
Remote error message: Cannot parse provided JSON: JSON Parse Error:
char=,position=0 AFTER=''
BEFORE='��¶ms��2update.contentType0applicat'
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:274)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2020-06-09 02:12:58.506 INFO (qtp90045638-16)
[c:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP
s:shard1 r:core_node2
x:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1]
o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 1 errors
2020-06-09 02:12:58.507 ERROR (qtp90045638-16)
[c:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP
s:shard1 r:core_node2
x:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1]
o.a.s.h.RequestHandlerBase
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at http
://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/: null
request:
http://10.20.36.130:8983/solr/products_20200609__CRA__2005_shard1_replica_n1/
Remote error message: Cannot parse provided JSON: JSON Parse Error:
char=,position=0 AFTER=''
BEFORE='��¶ms��2update.contentType0applicat'
at
org.apache.solr.update.processor.RoutedAliasUpdateProcessor.finish(RoutedAliasUpdateProcessor.java:217)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Unknown Source)
2020-06-09 02:12:58.507 INFO (qtp90045638-16)
[c:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP
s:shard1 r:core_node2
x:products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1]
o.a.s.c.S.Request
[products_20200609__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_shard1_replica_n1]
webapp=/solr path=/update/json/docs params={} status=400 QTime=2422
Cheers
Tom
