[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231728#comment-14231728
]
ASF subversion and git services commented on SOLR-6776:
---
Commit 1642946 from [~yo...@apache.org] in branch 'dev/trunk'
[ https://svn.apache.org/r1642946 ]
SOLR-6776: only clear changesSinceCommit on a hard commit so tlog will still be
flushed on a softCommit
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Assignee: Yonik Seeley
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231732#comment-14231732
]
ASF subversion and git services commented on SOLR-6776:
---
Commit 1642950 from [~yo...@apache.org] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1642950 ]
SOLR-6776: only clear changesSinceCommit on a hard commit so tlog will still be
flushed on a softCommit
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Assignee: Yonik Seeley
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231733#comment-14231733
]
ASF subversion and git services commented on SOLR-6776:
---
Commit 1642951 from [~yo...@apache.org] in branch
'dev/branches/lucene_solr_4_10'
[ https://svn.apache.org/r1642951 ]
SOLR-6776: only clear changesSinceCommit on a hard commit so tlog will still be
flushed on a softCommit
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Assignee: Yonik Seeley
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230053#comment-14230053
]
yuanyun.cn commented on SOLR-6776:
--
The finish of UpdateProcessoris always in
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
SolrQueryResponse).
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230063#comment-14230063
]
Yonik Seeley commented on SOLR-6776:
If this is reproducible by someone, it represents a bug.
{quote}
To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
if (!cmd.softCommit)
{ changesSinceCommit = false; }
{quote}
It's not clear what this change is trying to fix (or why it changes anything
for the reporter), but one should not have to softCommit (or commit) in order
to not lose data.
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230071#comment-14230071
]
yuanyun.cn commented on SOLR-6776:
--
Hi, Yonik:
The problem here is that if we add a doc with softcommit=true(user wants the
data to be immediately searchable), processCommit will set
changesSinceCommit=false,
then in finish method, it will not call
updateHandler.getUpdateLog().finish(null) to fsync the tlog.
If we kill the java after that, then data will be lost.
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227062#comment-14227062
]
Mark Miller commented on SOLR-6776:
---
By default, the tlog doesnt fsync, it just flushes and leans on replicas. You
can configure the sync level in solrconfig.xml.
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6776) Data lost when use SoftCommit and TLog
[
https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225445#comment-14225445
]
Xu Zhang commented on SOLR-6776:
Probably this is not a bug. Finish() is about flush tlog into hard disk and
soft-commit is just about visibility.
Data lost when use SoftCommit and TLog
--
Key: SOLR-6776
URL: https://issues.apache.org/jira/browse/SOLR-6776
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: yuanyun.cn
Labels: softCommit, updateLog
Fix For: 4.10.3
We enabled update log and change autoCommit to some bigger value 10 mins.
After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=adddocfield
name=idid1/field/doc/addsoftCommit=true
Then we kill the java process after a min.
After restart, Tlog failed to replay with following exception, and there is
no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û
Failure to open existing log file (non fatal)
E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.000:org.apache.solr.common.SolrException:
java.io.EOFException
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.init(UpdateHandler.java:94)
at
org.apache.solr.update.DirectUpdateHandler2.init(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.init(SolrCore.java:835)
at org.apache.solr.core.SolrCore.init(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at
org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at
org.apache.solr.update.TransactionLog.init(TransactionLog.java:159)
... 19 more
Check the code: seems this is related with:
org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it
sets changesSinceCommit=false(even we are using softCommit)
So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit updateHandler.getUpdateLog() != null) {
updateHandler.getUpdateLog().finish(null);
}
super.finish();
}
To fix this issue: I have to change RunUpdateProcessor.processCommit like
below:
if (!cmd.softCommit) {
changesSinceCommit = false;
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
