[jira] [Comment Edited] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it

2014-07-15 Thread Nathan Neulinger (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062800#comment-14062800
 ] 

Nathan Neulinger edited comment on SOLR-6251 at 7/15/14 10:51 PM:
--

and here's an update in that same debug log from shortly before the error (the 
distribute from the insert of the document on solr1):

{noformat}
2014-07-10 21:29:49,313 INFO  qtp1599863753-30844 
[solr.update.processor.LogUpdateProcessor]  - [d-_v22_shard1_replica2] 
webapp=/solr path=/update 
params={distrib.from=http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/update.distrib=TOLEADERwt=javabinversion=2}
 {add=[4b2c4d09-31e2-4fe2-b767-3868efbdcda1 (1473278419196182528)]} 0 11
2014-07-10 21:29:49,416 INFO  qtp1599863753-30844 
[org.apache.solr.update.UpdateHandler]  - start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
{noformat}


was (Author: nneul):
and here's an update in that same debug log from shortly before the error (the 
distribute from the insert of the document on solr1):

2014-07-10 21:29:49,313 INFO  qtp1599863753-30844 
[solr.update.processor.LogUpdateProcessor]  - [d-_v22_shard1_replica2] 
webapp=/solr path=/update 
params={distrib.from=http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/update.distrib=TOLEADERwt=javabinversion=2}
 {add=[4b2c4d09-31e2-4fe2-b767-3868efbdcda1 (1473278419196182528)]} 0 11
2014-07-10 21:29:49,416 INFO  qtp1599863753-30844 
[org.apache.solr.update.UpdateHandler]  - start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}


 incorrect 'missing required field' during update - document definitely has it
 -

 Key: SOLR-6251
 URL: https://issues.apache.org/jira/browse/SOLR-6251
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.8
 Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All 
 on EC2. The two hosts are round-robin'd behind an ELB.
Reporter: Nathan Neulinger
  Labels: replication

 Document added on solr1. We can see the distribute take place from solr1 to 
 solr2 and returning a success. Subsequent searches returning document, 
 clearly showing the field as being there. Later on, an update is done to add 
 to an element of the document - and the update fails. The update was sent to 
 solr2 instance. 
 Schema marks the 'timestamp' field as required, so the initial insert should 
 not work if the field isn't present.
 Symptom is intermittent - we're seeing this randomly, with no warning or 
 triggering that we can see, but in all cases, it's getting the error in 
 response to an update when the instance tries to distribute the change to the 
 other node. 
 Searches that were run AFTER the update also show the field as being present 
 in the document. 
 Will add full trace of operations in the comments shortly. pcap captures of 
 ALL traffic for the two nodes on 8983 is available if requested. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it

2014-07-15 Thread Nathan Neulinger (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062793#comment-14062793
 ] 

Nathan Neulinger edited comment on SOLR-6251 at 7/15/14 10:52 PM:
--

{noformat}
16.24 = POD SRV
16.204 = SOLR 1
16.207 = SOLR 2

16.24 ⇒ 16.204 
CAP 1
11344 14:29:49.299883

POST /solr/d-_v22/update/json?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 8677c2fb-8b92-4220-bb73-1e4c610d95be 2057
User-Agent: HivePoint (Factory JSON client:null:2056)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 1555
Connection: keep-alive

{ add: { commitWithin : 5000, doc : 
{hive:vdates,at:2014-07-10T21:28:41Z,timestamp:1405027721000,type:MESSAGE,channel:[dev],from:pr...@sevogle.com,to:[a...@sevogle.com,vi...@sevogle.com,d...@sevogle.com,s...@hive.sevogle.com],subject:Re:
 Deployments - B and then C,body:eve.SNIP...stem. 
,id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,message_id:2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1}
 } }


16.204 ⇒ 16.207
CAP 1


POST 
/solr/d-_v22_shard1_replica2/update?update.distrib=TOLEADERdistrib.from=http%3A%2F%2F10.220.16.204%3A8983%2Fsolr%2Fd-_v22_shard1_replica1%2Fwt=javabinversion=2
 HTTP/1.1
User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
Content-Type: application/javabin
Transfer-Encoding: chunked
Host: 10.220.16.207:8983
Connection: Keep-Alive

64c
...params...update.distrib(TOLEADER.,distrib.from?.http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/.delByQ..'docsMap.?$hivevdates.at42014-07-10T21:28:41Z.)timestampx...$type'MESSAGE.'channel.#dev.$from1pr...@sevogle.com.to.0adam@sevogle.com1vi...@sevogle.com/dev@sevogle.com4...@hive.sevogle.com.'subjectRe:
 Deployments - B and then C.$body?#eve.SNIP...tem. 
.id?.4b2c4d09-31e2-4fe2-b767-3868efbdcda1.*message_id?.2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1
..ow..cwX...
0



16.207 ⇒ 16.204 
CAP 1
11368 14:29:49.495301
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 40

responseHeader..status..%QTimeK



16.24 ⇒ 16.204 
CAP 1
11371 14:29:49.496308

INDEX COMPLETE

HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked

2C
{responseHeader:{status:0,QTime:195}}

0


16.24 ⇒ 16.207 
CAP 2
9218 14:29:57.065156
9232 14:29”57.099274

Search (two different search results to two servers?) that show the timestamp 
is set.

POST /solr/d-_v22/select?indent=onwt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 957d1ca5-7200-4058-9c70-16a17fc64c19 2069
User-Agent: HivePoint (Factory JSON client:null:2068)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 244
Connection: keep-alive

q=%2B%28*%29fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A27%3A56Z+TO+*%5Dstart=0rows=300sort=at+desc%2C+id+descfl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
 200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked

2BB
{
  responseHeader:{
status:0,
QTime:3,
params:{
  fl:id,hive,timestamp,type,message_id,file_instance_id,score,
  sort:at desc, id desc,
  indent:on,
  start:0,
  q:+(*),
  wt:json,
  fq:+hive:vdates AND +channel:(adam bethany dev notifications preet 
share) AND at:[2014-07-10T21:27:56Z TO *],
  rows:300}},
  response:{numFound:1,start:0,maxScore:1.0,docs:[
  {
hive:vdates,
timestamp:1405027721000,
type:MESSAGE,
id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,
message_id:2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1,
score:1.0}]
  }}

0




16.24 ⇒ 16.207 
CAP 2
9415 14:30:00.310995

Update Channel

POST /solr/d-_v22/update?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 92fa6c11-78d8-44cc-a143-9ff3e4c132f4 2115
User-Agent: HivePoint (Factory JSON client:null:2114)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 102
Connection: keep-alive

[{id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,channel: {add: 
preet},channel: {add: adam}}]HTTP/1.1 400 Bad Request
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked

96
{responseHeader:{status:400,QTime:1},error:{msg:[doc=4b2c4d09-31e2-4fe2-b767-3868efbdcda1]
 missing required field: timestamp,code:400}}

0


CAP 2
9602 14:30:08.082758

Subsequent search, after update

POST /solr/d-_v22/select?indent=onwt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8