[jira] [Commented] (LUCENE-9981) CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with default maxDeterminizedStates limit
[
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353964#comment-17353964
]
Ori D commented on LUCENE-9981:
---
[~rcmuir],
We don't expose the entire elasticsearch DSL of course.
We do require authentication.
But the business needs also requires us to allow users for specifying regexp
and then we perform some kind of a regexp query in elastiscearch with that
regexp.
It's not exactly like any other slow query, because we control our queries
(through API layer) and our data. However, in this specific case, the user may
provide a "malicious" regexp that can cause high CPU.
It's true that the best solution is to simply avoid any regexp coming from
end-users. However, this is not aligned with our business needs at the moment.
We can find several ways to mitigate or minimize this possibility of High CPU
(for example, don't allow regexp with '{}' ). But I prefer having a solid
solution integrated into the elasticsearch engine.
I hope I was able to explain it better.
Please consider some fix to 8.x - even a simple one like you originally
suggested of a 1-liner patch - so we can enjoy the fix in the near future.
Thanks,
Ori.
> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with
> default maxDeterminizedStates limit
>
>
> Key: LUCENE-9981
> URL: https://issues.apache.org/jira/browse/LUCENE-9981
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9981.patch, LUCENE-9981.patch, LUCENE-9981.patch,
> LUCENE-9981_nfaprefix.patch, LUCENE-9981_test.patch,
> three-repeats-reverse-det.png, three-repeats.png
>
>
> We have a {{maxDeterminizedStates = 1}} limit designed to keep
> regexp-type queries from blowing up.
> But we have an adversary that will run for 268s on my laptop before hitting
> exception, first reported here:
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x7fff7006ca80 nid=0x231c8
> runnable [0x7fff8b7f]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
> at
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
> at
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
> at
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:247)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:104)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:82)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:138)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:114)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:72)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:62)
> at
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be
> an "up-front" optimization to make the actual subsequent terms-intensive part
> of the query faster. But it makes the whole query run for nearly 5 minutes
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be
> more "best-effort". For example, it can reduce the lower bound to {{1000}}
> and catch the exception like such:
> {code}
> try {
>// this is slow, and just an opto anyway, so don't burn cycles on it for
> some crazy worst-case.
>// if we don't set this common suffix, the query will just run a bit
> slower, that's all.
>int limit = Math.min(1000, maxDeterminizedStates);
>BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
> commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo".
> By computing that the common suffix is "foo" we can spend less CPU in the
> terms dictionary because we can first do a memcmp before having to run data
> thru any finite state machine. It's really a microopt and we shouldn't be
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the
> behavior today, maybe there is a
[jira] [Commented] (LUCENE-9981) CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with default maxDeterminizedStates limit
[
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353939#comment-17353939
]
Ori D commented on LUCENE-9981:
---
Hi Guys,
Looks great!
My only concern now is that the whole thing has started from OpenSearch, which
uses Lucene 8.x , and as I understand this patch will only be applied to
version 9.x.
Please see if you can do similar patch to 8.x as well, so we can get a quick
fix in OpenSearch (and probably other projects using Lucene).
Thanks
> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with
> default maxDeterminizedStates limit
>
>
> Key: LUCENE-9981
> URL: https://issues.apache.org/jira/browse/LUCENE-9981
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9981.patch, LUCENE-9981.patch, LUCENE-9981.patch,
> LUCENE-9981_nfaprefix.patch, LUCENE-9981_test.patch,
> three-repeats-reverse-det.png, three-repeats.png
>
>
> We have a {{maxDeterminizedStates = 1}} limit designed to keep
> regexp-type queries from blowing up.
> But we have an adversary that will run for 268s on my laptop before hitting
> exception, first reported here:
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x7fff7006ca80 nid=0x231c8
> runnable [0x7fff8b7f]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
> at
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
> at
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
> at
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:247)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:104)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:82)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:138)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:114)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:72)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:62)
> at
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be
> an "up-front" optimization to make the actual subsequent terms-intensive part
> of the query faster. But it makes the whole query run for nearly 5 minutes
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be
> more "best-effort". For example, it can reduce the lower bound to {{1000}}
> and catch the exception like such:
> {code}
> try {
>// this is slow, and just an opto anyway, so don't burn cycles on it for
> some crazy worst-case.
>// if we don't set this common suffix, the query will just run a bit
> slower, that's all.
>int limit = Math.min(1000, maxDeterminizedStates);
>BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
> commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo".
> By computing that the common suffix is "foo" we can spend less CPU in the
> terms dictionary because we can first do a memcmp before having to run data
> thru any finite state machine. It's really a microopt and we shouldn't be
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the
> behavior today, maybe there is a bigger issue and I don't want to shove
> something under the rug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9981) CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with default maxDeterminizedStates limit
[
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353750#comment-17353750
]
Ori D commented on LUCENE-9981:
---
[~rcmuir],
I'm not saying a timeout must be added as a solution, I'm only suggesting that
the final solution would not lead to an execution time of more than few
hundreds of ms.
The fact a user is authenticated doesn't necessarily protect you from DOS
attacks. It indeed reduces the chance, but does not completely eliminate the
risk.
For example, in our application an arbitrary user can download a trial. The
user is registered and authenticated, but can still perform legal applicative
operations that might lead to a long Elasticsearch/Lucene regexp processing on
our server.
Even an enterprise user can make non-intentional mistake and write a bad script
that calls many APIs on our server, and load it so much in a way the server
will not be available for other customers/tenants.
The point is that it will be much harder to load the server with 200 ms
operation than with 5 seconds one.
With 200ms, an application usually have other protective mechanisms to cope
with DOS attacks, since it is still considered a valid REST api time.
> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with
> default maxDeterminizedStates limit
>
>
> Key: LUCENE-9981
> URL: https://issues.apache.org/jira/browse/LUCENE-9981
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9981_test.patch
>
>
> We have a {{maxDeterminizedStates = 1}} limit designed to keep
> regexp-type queries from blowing up.
> But we have an adversary that will run for 268s on my laptop before hitting
> exception, first reported here:
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x7fff7006ca80 nid=0x231c8
> runnable [0x7fff8b7f]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
> at
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
> at
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
> at
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:247)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:104)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:82)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:138)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:114)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:72)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:62)
> at
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be
> an "up-front" optimization to make the actual subsequent terms-intensive part
> of the query faster. But it makes the whole query run for nearly 5 minutes
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be
> more "best-effort". For example, it can reduce the lower bound to {{1000}}
> and catch the exception like such:
> {code}
> try {
>// this is slow, and just an opto anyway, so don't burn cycles on it for
> some crazy worst-case.
>// if we don't set this common suffix, the query will just run a bit
> slower, that's all.
>int limit = Math.min(1000, maxDeterminizedStates);
>BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
> commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo".
> By computing that the common suffix is "foo" we can spend less CPU in the
> terms dictionary because we can first do a memcmp before having to run data
> thru any finite state machine. It's really a microopt and we shouldn't be
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the
> behavior today, maybe there is a bigger issue and I don't want to shove
> something under the rug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---
[jira] [Commented] (LUCENE-9981) CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with default maxDeterminizedStates limit
[
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17353741#comment-17353741
]
Ori D commented on LUCENE-9981:
---
Hi [~rcmuir],
Thanks for handling this (y)
Just wanted to add that I could not agree with you more on the fact that 5
seconds is still too high.
>From security perspective, even 5 seconds of loading the CPU can be considered
>as very high and can easily cause a DOS for any server if executed multiple
>times concurrently.
I think that we should aim to no more than few hundred of milliseconds.
Whatever we cannot achieve in 200-500 ms is probably not a good optimization
anyhow.
Not sure if the solution means just decrease the default limit from 1000 to 100
in your patch, or maybe add additional code that limits the whole calculation
by (configurable) time.
Cheers,
Ori.
> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with
> default maxDeterminizedStates limit
>
>
> Key: LUCENE-9981
> URL: https://issues.apache.org/jira/browse/LUCENE-9981
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9981_test.patch
>
>
> We have a {{maxDeterminizedStates = 1}} limit designed to keep
> regexp-type queries from blowing up.
> But we have an adversary that will run for 268s on my laptop before hitting
> exception, first reported here:
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x7fff7006ca80 nid=0x231c8
> runnable [0x7fff8b7f]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
> at
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
> at
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
> at
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:247)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:104)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:82)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:138)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:114)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:72)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:62)
> at
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be
> an "up-front" optimization to make the actual subsequent terms-intensive part
> of the query faster. But it makes the whole query run for nearly 5 minutes
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be
> more "best-effort". For example, it can reduce the lower bound to {{1000}}
> and catch the exception like such:
> {code}
> try {
>// this is slow, and just an opto anyway, so don't burn cycles on it for
> some crazy worst-case.
>// if we don't set this common suffix, the query will just run a bit
> slower, that's all.
>int limit = Math.min(1000, maxDeterminizedStates);
>BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
> commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo".
> By computing that the common suffix is "foo" we can spend less CPU in the
> terms dictionary because we can first do a memcmp before having to run data
> thru any finite state machine. It's really a microopt and we shouldn't be
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the
> behavior today, maybe there is a bigger issue and I don't want to shove
> something under the rug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14911) Logger : UpdateLog Error Message : java.io.EOFException
[
https://issues.apache.org/jira/browse/SOLR-14911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
D updated SOLR-14911:
-
Summary: Logger : UpdateLog Error Message : java.io.EOFException (was:
Looger : UpdateLog Error Message : java.io.EOFException)
> Logger : UpdateLog Error Message : java.io.EOFException
> ---
>
> Key: SOLR-14911
> URL: https://issues.apache.org/jira/browse/SOLR-14911
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCLI
>Affects Versions: 7.5
>Reporter: D
>Priority: Blocker
> Attachments: GC events.PNG
>
>
> *Events:*
> # GC logs showing continuos Full GC events. Log report attached.
> # Core filling failed , showing less data than expected.
> # following warnings showing on dashboard before error.
> |Level|Logger|Message|
> |WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable.
> PLEASE REMOVE THIS FILE.|
> |WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable.
> PLEASE REMOVE THIS FILE.|
> |WARN false|SolrResourceLoader|Solr loaded a deprecated plugin/analysis class
> [solr.TrieDateField]. Please consult documentation how to replace it
> accordingly.|
> |WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
> managed, but the non-managed schema schema.xml is still loadable.
> PLEASE REMOVE THIS FILE.|
> |WARN false|UpdateLog|Starting log replay
> tlog\{file=\data\tlog\tlog.0445482 refcount=2}
> active=false starting pos=0 inSortedOrder=false|
> * Total data in all cores around 8 GB
> * *Other Configurations:*
> ** -XX:+UseG1GC
> ** -XX:+UseStringDeduplication
> ** -XX:MaxGCPauseMillis=500
> ** -Xms15g
> ** -Xmx15g
> ** -Xss256k
> * *OS Environment :*
> ** Windows 10,
> ** Filling cores by calling SQL query using jtds-1.3.1 library.
> ** Solr Version 7.5
> ** Runtime: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.2 11.0.2+9
> ** Processors : 48
> ** System Physical Memory : 128 GB
> ** Swap Space : 256GB
> * solr-spec7.5.0
> ** solr-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
> 2018-09-18 13:07:55
> * lucene-spec7.5.0
> ** lucene-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
> 2018-09-18 13:01:1
> *Error Message :*
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:863)
> at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:857)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:266)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at
> org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:603)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:315)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:747)
> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:272)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
> at
> org.apache.solr.update.TransactionLog$LogReader.next(TransactionLog.java:673)
> at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1832)
> at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1747)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14911) Looger : UpdateLog Error Message : java.io.EOFException
D created SOLR-14911:
Summary: Looger : UpdateLog Error Message : java.io.EOFException
Key: SOLR-14911
URL: https://issues.apache.org/jira/browse/SOLR-14911
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCLI
Affects Versions: 7.5
Reporter: D
Attachments: GC events.PNG
*Events:*
# GC logs showing continuos Full GC events. Log report attached.
# Core filling failed , showing less data than expected.
# following warnings showing on dashboard before error.
|Level|Logger|Message|
|WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.|
|WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.|
|WARN false|SolrResourceLoader|Solr loaded a deprecated plugin/analysis class
[solr.TrieDateField]. Please consult documentation how to replace it
accordingly.|
|WARN false|ManagedIndexSchemaFactory|The schema has been upgraded to
managed, but the non-managed schema schema.xml is still loadable. PLEASE
REMOVE THIS FILE.|
|WARN false|UpdateLog|Starting log replay
tlog\{file=\data\tlog\tlog.0445482 refcount=2} active=false
starting pos=0 inSortedOrder=false|
* Total data in all cores around 8 GB
* *Other Configurations:*
** -XX:+UseG1GC
** -XX:+UseStringDeduplication
** -XX:MaxGCPauseMillis=500
** -Xms15g
** -Xmx15g
** -Xss256k
* *OS Environment :*
** Windows 10,
** Filling cores by calling SQL query using jtds-1.3.1 library.
** Solr Version 7.5
** Runtime: Oracle Corporation OpenJDK 64-Bit Server VM 11.0.2 11.0.2+9
** Processors : 48
** System Physical Memory : 128 GB
** Swap Space : 256GB
* solr-spec7.5.0
** solr-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:07:55
* lucene-spec7.5.0
** lucene-impl7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi -
2018-09-18 13:01:1
*Error Message :*
java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:863)
at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:857)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:266)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at
org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:603)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:315)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:747)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:272)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
at
org.apache.solr.update.TransactionLog$LogReader.next(TransactionLog.java:673)
at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1832)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1747)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
