[jira] [Comment Edited] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230053#comment-14230053
]
yuanyun.cn edited comment on SOLR-6776 at 12/1/14 5:20 PM:
---
The finish of UpdateProcessoris is always called in
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
SolrQueryResponse).
When we add a doc without softcommit or commit, the
org.apache.solr.update.processor.RunUpdateProcessor.finish() will call
getUpdateLog().finish() to fsync the tlog.
But if add a doc with softcommit=true, RunUpdateProcessor.finish() will not
call getUpdateLog().finish(), and will not fsync the tlog.
This is kind of not right.
User enables transaction log for data durability, to make sure there is no data
lost.
So I think it should always fsync the tlog after add this doc to solr and
before the hard commit.
was (Author: yuanyun.cn):
The finish of UpdateProcessoris always in
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
SolrQueryResponse).
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230063#comment-14230063
]
Yonik Seeley edited comment on SOLR-6776 at 12/1/14 5:26 PM:
-
If this is reproducible by someone, it represents a bug.
{quote}
To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
if (!cmd.softCommit)
{ changesSinceCommit = false; }
{quote}
It's not clear what this change is trying to fix (or why it changes anything
for the reporter), but one should not have to softCommit (or commit) in order
to not lose data.
edit: I understand now (mis-read the patch the first time, missing the !).
You are correct, this is a bug and the fix looks correct.
was (Author: ysee...@gmail.com):
If this is reproducible by someone, it represents a bug.
{quote}
To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
if (!cmd.softCommit)
{ changesSinceCommit = false; }
{quote}
It's not clear what this change is trying to fix (or why it changes anything
for the reporter), but one should not have to softCommit (or commit) in order
to not lose data.
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For
