[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064477#comment-14064477
]
Nathan Neulinger commented on SOLR-6251:
FYI. We finally tracked down the problem at least 99.9% sure at this point,
and it was staring me in the face the whole time - just never noticed:
{noformat}
[{id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,channel: {add:
preet},channel: {add: adam}}]
{noformat}
Look at the JSON... It's trying to add two channels... Should have been:
{noformat}
[{id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,channel: {add: preet}},
{id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,channel: {add: adam}}]
{noformat}
I half wonder how it chose to interpret that particular chunk of json, but
either way, I think the origin of our issue is resolved.
incorrect 'missing required field' during update - document definitely has it
-
Key: SOLR-6251
URL: https://issues.apache.org/jira/browse/SOLR-6251
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.8
Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All
on EC2. The two hosts are round-robin'd behind an ELB.
Reporter: Nathan Neulinger
Labels: replication
Attachments: schema.xml
Document added on solr1. We can see the distribute take place from solr1 to
solr2 and returning a success. Subsequent searches returning document,
clearly showing the field as being there. Later on, an update is done to add
to an element of the document - and the update fails. The update was sent to
solr2 instance.
Schema marks the 'timestamp' field as required, so the initial insert should
not work if the field isn't present.
Symptom is intermittent - we're seeing this randomly, with no warning or
triggering that we can see, but in all cases, it's getting the error in
response to an update when the instance tries to distribute the change to the
other node.
Searches that were run AFTER the update also show the field as being present
in the document.
Will add full trace of operations in the comments shortly. pcap captures of
ALL traffic for the two nodes on 8983 is available if requested.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062793#comment-14062793
]
Nathan Neulinger commented on SOLR-6251:
16.24 = POD SRV
16.204 = SOLR 1
16.207 = SOLR 2
16.24 ⇒ 16.204
CAP 1
11344 14:29:49.299883
POST /solr/d-_v22/update/json?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 8677c2fb-8b92-4220-bb73-1e4c610d95be 2057
User-Agent: HivePoint (Factory JSON client:null:2056)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 1555
Connection: keep-alive
{ add: { commitWithin : 5000, doc :
{hive:vdates,at:2014-07-10T21:28:41Z,timestamp:1405027721000,type:MESSAGE,channel:[dev],from:pr...@sevogle.com,to:[a...@sevogle.com,vi...@sevogle.com,d...@sevogle.com,s...@hive.sevogle.com],subject:Re:
Deployments - B and then C,body:eve.SNIP...stem.
,id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,message_id:2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1}
} }
16.204 ⇒ 16.207
CAP 1
POST
/solr/d-_v22_shard1_replica2/update?update.distrib=TOLEADERdistrib.from=http%3A%2F%2F10.220.16.204%3A8983%2Fsolr%2Fd-_v22_shard1_replica1%2Fwt=javabinversion=2
HTTP/1.1
User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0
Content-Type: application/javabin
Transfer-Encoding: chunked
Host: 10.220.16.207:8983
Connection: Keep-Alive
64c
...params...update.distrib(TOLEADER.,distrib.from?.http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/.delByQ..'docsMap.?$hivevdates.at42014-07-10T21:28:41Z.)timestampx...$type'MESSAGE.'channel.#dev.$from1pr...@sevogle.com.to.0adam@sevogle.com1vi...@sevogle.com/dev@sevogle.com4...@hive.sevogle.com.'subjectRe:
Deployments - B and then C.$body?#eve.SNIP...tem.
.id?.4b2c4d09-31e2-4fe2-b767-3868efbdcda1.*message_id?.2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1
..ow..cwX...
0
16.207 ⇒ 16.204
CAP 1
11368 14:29:49.495301
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 40
responseHeader..status..%QTimeK
16.24 ⇒ 16.204
CAP 1
11371 14:29:49.496308
INDEX COMPLETE
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2C
{responseHeader:{status:0,QTime:195}}
0
16.24 ⇒ 16.207
CAP 2
9218 14:29:57.065156
9232 14:29”57.099274
Search (two different search results to two servers?) that show the timestamp
is set.
POST /solr/d-_v22/select?indent=onwt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 957d1ca5-7200-4058-9c70-16a17fc64c19 2069
User-Agent: HivePoint (Factory JSON client:null:2068)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 244
Connection: keep-alive
q=%2B%28*%29fq=%2Bhive%3Avdates+AND+%2Bchannel%3A%28adam+bethany+dev+notifications+preet+share%29+AND+at%3A%5B2014-07-10T21%3A27%3A56Z+TO+*%5Dstart=0rows=300sort=at+desc%2C+id+descfl=id,hive,timestamp,type,message_id,file_instance_id,scoreHTTP/1.1
200 OK
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
2BB
{
responseHeader:{
status:0,
QTime:3,
params:{
fl:id,hive,timestamp,type,message_id,file_instance_id,score,
sort:at desc, id desc,
indent:on,
start:0,
q:+(*),
wt:json,
fq:+hive:vdates AND +channel:(adam bethany dev notifications preet
share) AND at:[2014-07-10T21:27:56Z TO *],
rows:300}},
response:{numFound:1,start:0,maxScore:1.0,docs:[
{
hive:vdates,
timestamp:1405027721000,
type:MESSAGE,
id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,
message_id:2014-07-10-77a6614c-66e4-4ddb-8566-dff4bfb743d1,
score:1.0}]
}}
0
16.24 ⇒ 16.207
CAP 2
9415 14:30:00.310995
Update Channel
POST /solr/d-_v22/update?commit=true HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/json; charset=UTF-8
request_id: null 92fa6c11-78d8-44cc-a143-9ff3e4c132f4 2115
User-Agent: HivePoint (Factory JSON client:null:2114)
X-Forwarded-For: 10.220.16.229
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 102
Connection: keep-alive
[{id:4b2c4d09-31e2-4fe2-b767-3868efbdcda1,channel: {add:
preet},channel: {add: adam}}]HTTP/1.1 400 Bad Request
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
96
{responseHeader:{status:400,QTime:1},error:{msg:[doc=4b2c4d09-31e2-4fe2-b767-3868efbdcda1]
missing required field: timestamp,code:400}}
0
CAP 2
9602 14:30:08.082758
Subsequent search, after update
POST /solr/d-_v22/select?indent=onwt=json HTTP/1.1
host: d01-solr.srv.hivepoint.com
Accept-Encoding: gzip,deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
request_id: null 196bee69-e79c-455e-b0cb-6ad6ecdab4e0 1813
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[ https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062796#comment-14062796 ] Nathan Neulinger commented on SOLR-6251: this is the occurrence of the error on the server the update ran on 2014-07-10 21:30:00,313 ERROR qtp1599863753-30801 [org.apache.solr.core.SolrCore ] - org.apache.solr.common.SolrException: [doc=4b2c4d09-31e2-4fe2-b767-3868efbdcda1] missing required field: timestamp at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:189) at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:234) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557) at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:393) at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:118) at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:102) at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:66) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) incorrect
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062800#comment-14062800
]
Nathan Neulinger commented on SOLR-6251:
and here's an update in that same debug log from shortly before the error (the
distribute from the insert of the document on solr1):
2014-07-10 21:29:49,313 INFO qtp1599863753-30844
[solr.update.processor.LogUpdateProcessor] - [d-_v22_shard1_replica2]
webapp=/solr path=/update
params={distrib.from=http://10.220.16.204:8983/solr/d-_v22_shard1_replica1/update.distrib=TOLEADERwt=javabinversion=2}
{add=[4b2c4d09-31e2-4fe2-b767-3868efbdcda1 (1473278419196182528)]} 0 11
2014-07-10 21:29:49,416 INFO qtp1599863753-30844
[org.apache.solr.update.UpdateHandler] - start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
incorrect 'missing required field' during update - document definitely has it
-
Key: SOLR-6251
URL: https://issues.apache.org/jira/browse/SOLR-6251
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.8
Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All
on EC2. The two hosts are round-robin'd behind an ELB.
Reporter: Nathan Neulinger
Labels: replication
Document added on solr1. We can see the distribute take place from solr1 to
solr2 and returning a success. Subsequent searches returning document,
clearly showing the field as being there. Later on, an update is done to add
to an element of the document - and the update fails. The update was sent to
solr2 instance.
Schema marks the 'timestamp' field as required, so the initial insert should
not work if the field isn't present.
Symptom is intermittent - we're seeing this randomly, with no warning or
triggering that we can see, but in all cases, it's getting the error in
response to an update when the instance tries to distribute the change to the
other node.
Searches that were run AFTER the update also show the field as being present
in the document.
Will add full trace of operations in the comments shortly. pcap captures of
ALL traffic for the two nodes on 8983 is available if requested.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[
https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062802#comment-14062802
]
Hoss Man commented on SOLR-6251:
Please ask questions about things like this on the solr-user@lucene list prior
to filing a bug.
You have not provided details about your schema, but based on the details you
have provided, it appears that your timestamp field is not stored, therefore
you are probably hitting a documented limitation of using partial updates...
https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
{panel}
All original source fields must be stored for field modifiers to work
correctly, which is the Solr default.
{panel}
incorrect 'missing required field' during update - document definitely has it
-
Key: SOLR-6251
URL: https://issues.apache.org/jira/browse/SOLR-6251
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.8
Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All
on EC2. The two hosts are round-robin'd behind an ELB.
Reporter: Nathan Neulinger
Labels: replication
Document added on solr1. We can see the distribute take place from solr1 to
solr2 and returning a success. Subsequent searches returning document,
clearly showing the field as being there. Later on, an update is done to add
to an element of the document - and the update fails. The update was sent to
solr2 instance.
Schema marks the 'timestamp' field as required, so the initial insert should
not work if the field isn't present.
Symptom is intermittent - we're seeing this randomly, with no warning or
triggering that we can see, but in all cases, it's getting the error in
response to an update when the instance tries to distribute the change to the
other node.
Searches that were run AFTER the update also show the field as being present
in the document.
Will add full trace of operations in the comments shortly. pcap captures of
ALL traffic for the two nodes on 8983 is available if requested.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[ https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062806#comment-14062806 ] Nathan Neulinger commented on SOLR-6251: We are open to diagnostic suggestions on this, but are at a loss since this appears to be very intermittent and non-reproducible other than by waiting. Looking at solrconfig.xml compared to what is currently in 4.8.0 example - there are a variety of differences, mostly look like due to this config originally being based on 4.4 solrconfig.xml example. incorrect 'missing required field' during update - document definitely has it - Key: SOLR-6251 URL: https://issues.apache.org/jira/browse/SOLR-6251 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.8 Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All on EC2. The two hosts are round-robin'd behind an ELB. Reporter: Nathan Neulinger Labels: replication Document added on solr1. We can see the distribute take place from solr1 to solr2 and returning a success. Subsequent searches returning document, clearly showing the field as being there. Later on, an update is done to add to an element of the document - and the update fails. The update was sent to solr2 instance. Schema marks the 'timestamp' field as required, so the initial insert should not work if the field isn't present. Symptom is intermittent - we're seeing this randomly, with no warning or triggering that we can see, but in all cases, it's getting the error in response to an update when the instance tries to distribute the change to the other node. Searches that were run AFTER the update also show the field as being present in the document. Will add full trace of operations in the comments shortly. pcap captures of ALL traffic for the two nodes on 8983 is available if requested. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[ https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062907#comment-14062907 ] Nathan Neulinger commented on SOLR-6251: Leaving closed, but adding more information in case Hoss Man will comment additionally. 'timestamp' is: stored=true indexed=false That seems to meet all of the requirements stated for partial updates unless 'indexed=true' is also required and not documented. incorrect 'missing required field' during update - document definitely has it - Key: SOLR-6251 URL: https://issues.apache.org/jira/browse/SOLR-6251 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.8 Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All on EC2. The two hosts are round-robin'd behind an ELB. Reporter: Nathan Neulinger Labels: replication Attachments: schema.xml Document added on solr1. We can see the distribute take place from solr1 to solr2 and returning a success. Subsequent searches returning document, clearly showing the field as being there. Later on, an update is done to add to an element of the document - and the update fails. The update was sent to solr2 instance. Schema marks the 'timestamp' field as required, so the initial insert should not work if the field isn't present. Symptom is intermittent - we're seeing this randomly, with no warning or triggering that we can see, but in all cases, it's getting the error in response to an update when the instance tries to distribute the change to the other node. Searches that were run AFTER the update also show the field as being present in the document. Will add full trace of operations in the comments shortly. pcap captures of ALL traffic for the two nodes on 8983 is available if requested. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6251) incorrect 'missing required field' during update - document definitely has it
[ https://issues.apache.org/jira/browse/SOLR-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062909#comment-14062909 ] Nathan Neulinger commented on SOLR-6251: Additionally - since this works 99.9% of the time - I would surely think that a blatant problem as that would have been more visible. The incremental updates work normally without issue, and just randomly fail. incorrect 'missing required field' during update - document definitely has it - Key: SOLR-6251 URL: https://issues.apache.org/jira/browse/SOLR-6251 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.8 Environment: 4.8.0. Two nodes, SolrCloud, external ZK ensemble. All on EC2. The two hosts are round-robin'd behind an ELB. Reporter: Nathan Neulinger Labels: replication Attachments: schema.xml Document added on solr1. We can see the distribute take place from solr1 to solr2 and returning a success. Subsequent searches returning document, clearly showing the field as being there. Later on, an update is done to add to an element of the document - and the update fails. The update was sent to solr2 instance. Schema marks the 'timestamp' field as required, so the initial insert should not work if the field isn't present. Symptom is intermittent - we're seeing this randomly, with no warning or triggering that we can see, but in all cases, it's getting the error in response to an update when the instance tries to distribute the change to the other node. Searches that were run AFTER the update also show the field as being present in the document. Will add full trace of operations in the comments shortly. pcap captures of ALL traffic for the two nodes on 8983 is available if requested. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
