[
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062793#comment-14062793
]
Nathan Neulinger edited comment on SOLR-6251 at 7/15/14 10:52 PM:
------------------------------------------------------------------
{noformat}
16.24 = POD SRV
16.204 = SOLR 1
16.207 = SOLR 2
16.24 ⇒ 16.204
CAP 1
11344 14:29:49.299883
POST /solr/d-_v22/update/json?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 8677c2fb-8b92-4220-bb73-1e4c610d95be 2057
User-Agent: HivePoint (Factory JSON client:null:2056)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 1555
Connection: keep-alive
{ "add": { "commitWithin" : 5000, "doc" :
{"hive":"vdates","at":"2014-07-10T21:28:41Z","timestamp":1405027721000,"type":"MESSAGE","channel":["dev"],"from":"pr...@sevogle.com","to":["a...@sevogle.com","vi...@sevogle.com","d...@sevogle.com","s...@hive.sevogle.com"],"subject":"Re:
Deployments - B and then C","body":"eve.....SNIP.......stem.
","id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1"}
} }
16.204 ⇒ 16.207
CAP 1
POST
/solr/d-_v22_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.220.16.204%3A8983%2Fsolr%2Fd-_v22_shard1_replica1%2F&wt=javabin&version=2
HTTP/1.1
User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
Content-Type: application/javabin
Transfer-Encoding: chunked
Host: 10.220.16.207:8983
Connection: Keep-Alive
64c
...¶ms...update.distrib(TOLEADER.,distrib.from?.http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/.&delByQ..'docsMap.....?....$hive&vdates."at42014-07-10T21:28:41Z.)timestampx.......$type'MESSAGE.'channel.#dev.$from1pr...@sevogle.com."to.0adam@sevogle.com1vi...@sevogle.com/dev@sevogle.com4...@hive.sevogle.com.'subject>Re:
Deployments - B and then C.$body?#eve.....SNIP.......tem.
."id?.4b2c4d09-31e2-4fe2-b767-3868efbdcda1.*message_id?.2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1
.."ow.."cwX...
0
16.207 ⇒ 16.204
CAP 1
11368 14:29:49.495301
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 40
....responseHeader..&status......%QTimeK
16.24 ⇒ 16.204
CAP 1
11371 14:29:49.496308
INDEX COMPLETE
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2C
{"responseHeader":{"status":0,"QTime":195}}
0
16.24 ⇒ 16.207
CAP 2
9218 14:29:57.065156
9232 14:29”57.099274
Search (two different search results to two servers?) that show the timestamp
is set.
POST /solr/d-_v22/select?indent=on&wt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 957d1ca5-7200-4058-9c70-16a17fc64c19 2069
User-Agent: HivePoint (Factory JSON client:null:2068)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 244
Connection: keep-alive
q=%2B%28*%29&fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A27%3A56Z+TO+*%5D&start=0&rows=300&sort=at+desc%2C+id+desc&fl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2BB
{
"responseHeader":{
"status":0,
"QTime":3,
"params":{
"fl":"id,hive,timestamp,type,message_id,file_instance_id,score",
"sort":"at desc, id desc",
"indent":"on",
"start":"0",
"q":"+(*)",
"wt":"json",
"fq":"+hive:vdates AND +channel:(adam bethany dev notifications preet
share) AND at:[2014-07-10T21:27:56Z TO *]",
"rows":"300"}},
"response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
{
"hive":"vdates",
"timestamp":1405027721000,
"type":"MESSAGE",
"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1",
"message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1",
"score":1.0}]
}}
0
16.24 ⇒ 16.207
CAP 2
9415 14:30:00.310995
Update Channel
POST /solr/d-_v22/update?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 92fa6c11-78d8-44cc-a143-9ff3e4c132f4 2115
User-Agent: HivePoint (Factory JSON client:null:2114)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 102
Connection: keep-alive
[{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
"preet"},"channel": {"add": "adam"}}]HTTP/1.1 400 Bad Request
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
96
{"responseHeader":{"status":400,"QTime":1},"error":{"msg":"[doc=4b2c4d09-31e2-4fe2-b767-3868efbdcda1]
missing required field: timestamp","code":400}}
0
CAP 2
9602 14:30:08.082758
Subsequent search, after update
POST /solr/d-_v22/select?indent=on&wt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 196bee69-e79c-455e-b0cb-6ad6ecdab4e0 1813
User-Agent: HivePoint (Factory JSON client:null:1811)
X-Forwarded-For: 10.220.16.230
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 261
Connection: keep-alive
q=%2B%28*+-%28%28foo%29%29%29&fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A28%3A07Z+TO+*%5D&start=0&rows=300&sort=at+desc%2C+id+desc&fl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2C4
{
"responseHeader":{
"status":0,
"QTime":3,
"params":{
"fl":"id,hive,timestamp,type,message_id,file_instance_id,score",
"sort":"at desc, id desc",
"indent":"on",
"start":"0",
"q":"+(* -((foo)))",
"wt":"json",
"fq":"+hive:vdates AND +channel:(adam bethany dev notifications preet
share) AND at:[2014-07-10T21:28:07Z TO *]",
"rows":"300"}},
"response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
{
"hive":"vdates",
"timestamp":1405027721000,
"type":"MESSAGE",
"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1",
"message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1",
"score":1.0}]
}}
0
{noformat}
was (Author: nneul):
16.24 = POD SRV
16.204 = SOLR 1
16.207 = SOLR 2
16.24 ⇒ 16.204
CAP 1
11344 14:29:49.299883
POST /solr/d-_v22/update/json?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 8677c2fb-8b92-4220-bb73-1e4c610d95be 2057
User-Agent: HivePoint (Factory JSON client:null:2056)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 1555
Connection: keep-alive
{ "add": { "commitWithin" : 5000, "doc" :
{"hive":"vdates","at":"2014-07-10T21:28:41Z","timestamp":1405027721000,"type":"MESSAGE","channel":["dev"],"from":"pr...@sevogle.com","to":["a...@sevogle.com","vi...@sevogle.com","d...@sevogle.com","s...@hive.sevogle.com"],"subject":"Re:
Deployments - B and then C","body":"eve.....SNIP.......stem.
","id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1"}
} }
16.204 ⇒ 16.207
CAP 1
POST
/solr/d-_v22_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.220.16.204%3A8983%2Fsolr%2Fd-_v22_shard1_replica1%2F&wt=javabin&version=2
HTTP/1.1
User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
Content-Type: application/javabin
Transfer-Encoding: chunked
Host: 10.220.16.207:8983
Connection: Keep-Alive
64c
...¶ms...update.distrib(TOLEADER.,distrib.from?.http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/.&delByQ..'docsMap.....?....$hive&vdates."at42014-07-10T21:28:41Z.)timestampx.......$type'MESSAGE.'channel.#dev.$from1pr...@sevogle.com."to.0adam@sevogle.com1vi...@sevogle.com/dev@sevogle.com4...@hive.sevogle.com.'subject>Re:
Deployments - B and then C.$body?#eve.....SNIP.......tem.
."id?.4b2c4d09-31e2-4fe2-b767-3868efbdcda1.*message_id?.2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1
.."ow.."cwX...
0
16.207 ⇒ 16.204
CAP 1
11368 14:29:49.495301
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 40
....responseHeader..&status......%QTimeK
16.24 ⇒ 16.204
CAP 1
11371 14:29:49.496308
INDEX COMPLETE
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2C
{"responseHeader":{"status":0,"QTime":195}}
0
16.24 ⇒ 16.207
CAP 2
9218 14:29:57.065156
9232 14:29”57.099274
Search (two different search results to two servers?) that show the timestamp
is set.
POST /solr/d-_v22/select?indent=on&wt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 957d1ca5-7200-4058-9c70-16a17fc64c19 2069
User-Agent: HivePoint (Factory JSON client:null:2068)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 244
Connection: keep-alive
q=%2B%28*%29&fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A27%3A56Z+TO+*%5D&start=0&rows=300&sort=at+desc%2C+id+desc&fl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2BB
{
"responseHeader":{
"status":0,
"QTime":3,
"params":{
"fl":"id,hive,timestamp,type,message_id,file_instance_id,score",
"sort":"at desc, id desc",
"indent":"on",
"start":"0",
"q":"+(*)",
"wt":"json",
"fq":"+hive:vdates AND +channel:(adam bethany dev notifications preet
share) AND at:[2014-07-10T21:27:56Z TO *]",
"rows":"300"}},
"response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
{
"hive":"vdates",
"timestamp":1405027721000,
"type":"MESSAGE",
"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1",
"message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1",
"score":1.0}]
}}
0
16.24 ⇒ 16.207
CAP 2
9415 14:30:00.310995
Update Channel
POST /solr/d-_v22/update?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 92fa6c11-78d8-44cc-a143-9ff3e4c132f4 2115
User-Agent: HivePoint (Factory JSON client:null:2114)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 102
Connection: keep-alive
[{"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1","channel": {"add":
"preet"},"channel": {"add": "adam"}}]HTTP/1.1 400 Bad Request
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
96
{"responseHeader":{"status":400,"QTime":1},"error":{"msg":"[doc=4b2c4d09-31e2-4fe2-b767-3868efbdcda1]
missing required field: timestamp","code":400}}
0
CAP 2
9602 14:30:08.082758
Subsequent search, after update
POST /solr/d-_v22/select?indent=on&wt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 196bee69-e79c-455e-b0cb-6ad6ecdab4e0 1813
User-Agent: HivePoint (Factory JSON client:null:1811)
X-Forwarded-For: 10.220.16.230
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 261
Connection: keep-alive
q=%2B%28*+-%28%28foo%29%29%29&fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A28%3A07Z+TO+*%5D&start=0&rows=300&sort=at+desc%2C+id+desc&fl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2C4
{
"responseHeader":{
"status":0,
"QTime":3,
"params":{
"fl":"id,hive,timestamp,type,message_id,file_instance_id,score",
"sort":"at desc, id desc",
"indent":"on",
"start":"0",
"q":"+(* -((foo)))",
"wt":"json",
"fq":"+hive:vdates AND +channel:(adam bethany dev notifications preet
share) AND at:[2014-07-10T21:28:07Z TO *]",
"rows":"300"}},
"response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
{
"hive":"vdates",
"timestamp":1405027721000,
"type":"MESSAGE",
"id":"4b2c4d09-31e2-4fe2-b767-3868efbdcda1",
"message_id":"2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1",
"score":1.0}]
}}
0
> incorrect 'missing required field' during update - document definitely has it
> -----------------------------------------------------------------------------
>
> Key: SOLR-6251
> URL: https://issues.apache.org/jira/browse/SOLR-6251
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.8
> Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All
> on EC2. The two hosts are round-robin'd behind an ELB.
> Reporter: Nathan Neulinger
> Labels: replication
>
> Document added on solr1. We can see the distribute take place from solr1 to
> solr2 and returning a success. Subsequent searches returning document,
> clearly showing the field as being there. Later on, an update is done to add
> to an element of the document - and the update fails. The update was sent to
> solr2 instance.
> Schema marks the 'timestamp' field as required, so the initial insert should
> not work if the field isn't present.
> Symptom is intermittent - we're seeing this randomly, with no warning or
> triggering that we can see, but in all cases, it's getting the error in
> response to an update when the instance tries to distribute the change to the
> other node.
> Searches that were run AFTER the update also show the field as being present
> in the document.
> Will add full trace of operations in the comments shortly. pcap captures of
> ALL traffic for the two nodes on 8983 is available if requested.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
