[
https://issues.apache.org/jira/browse/SOLR-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Liang updated SOLR-18299:
-------------------------------
Description:
+Description:+
In Solr 9.8.1 (and likely Solr 10) cloud mode with TLOG/PULL, if a segment file
is exactly a multiple of PACKET_SZ (defaults to 1MB) IndexFetcher will always
fail to download it and hit an unexpected EOF. This will occur for some time
until the culprit file gets merged away, and then the cloud will continue to
operate as normal.
+Impact:+
The non-TLOG replicas will be unable to replicate any further index files and
will go "stale" until fixed.
+Stack Trace:+
{code:java}
2026-06-28 13:57:00.443 WARN IndexFetcher [solr9_collection shard36
core_node478 solr9_collection_shard36_replica_p477] ?
[indexFetcher-170-thread-1] - Error in fetching file: _7k9z8.cfs (downloaded
1048576 of 1048576 bytes)
java.io.EOFException: null
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:178)
~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:170)
~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1845)
~[solr-core-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
{code}
A main hallmark of this issue is the java.io.EOFException, however you will
always see a "downloaded X of X bytes" (which should ring alarm bells that
something is out of place).
+Steps to reproduce:+
It's hard to force in a live environment, but theoretically you could fill a
dummy segment file with a set number of bytes and add it to the index manifest.
It is 100% reproducible via integration test
was:
+Description:+
In Solr 9.8.1 (and likely Solr 10) cloud mode with TLOG/PULL, if a segment file
is exactly a multiple of PACKET_SZ (defaults to 1MB) IndexFetcher will always
fail to download it and hit an unexpected EOF. This will occur for some time
until the culprit file gets merged away, and then the cloud will continue to
operate as normal.
+Impact:+
The non-TLOG replicas will be unable to replicate any further index files and
will go "stale" until fixed.
+Stack Trace:+
{code:java}
2026-06-28 13:57:00.443 WARN IndexFetcher [solr9_collection shard36
core_node478 solr9_collection_shard36_replica_p477] ?
[indexFetcher-170-thread-1] - Error in fetching file: _7k9z8.cfs (downloaded
1048576 of 1048576 bytes)
java.io.EOFException: null
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:178)
~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:170)
~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1845)
~[solr-core-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
cloud-user - 2025-10-01 20:57:41]
{code}
A main hallmark of this issue is the java.io.EOFException, however you will
always see a "downloaded X of X bytes" (which should ring alarm bells that
something is out of place).
> IndexFetcher fails on files of exactly a multiple of PACKET_SZ
> --------------------------------------------------------------
>
> Key: SOLR-18299
> URL: https://issues.apache.org/jira/browse/SOLR-18299
> Project: Solr
> Issue Type: Bug
> Reporter: Kevin Liang
> Priority: Major
>
> +Description:+
> In Solr 9.8.1 (and likely Solr 10) cloud mode with TLOG/PULL, if a segment
> file is exactly a multiple of PACKET_SZ (defaults to 1MB) IndexFetcher will
> always fail to download it and hit an unexpected EOF. This will occur for
> some time until the culprit file gets merged away, and then the cloud will
> continue to operate as normal.
> +Impact:+
> The non-TLOG replicas will be unable to replicate any further index files and
> will go "stale" until fixed.
> +Stack Trace:+
>
> {code:java}
> 2026-06-28 13:57:00.443 WARN IndexFetcher [solr9_collection shard36
> core_node478 solr9_collection_shard36_replica_p477] ?
> [indexFetcher-170-thread-1] - Error in fetching file: _7k9z8.cfs (downloaded
> 1048576 of 1048576 bytes)
> java.io.EOFException: null
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:178)
> ~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
> cloud-user - 2025-10-01 20:57:41]
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:170)
> ~[solr-solrj-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
> cloud-user - 2025-10-01 20:57:41]
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1845)
> ~[solr-core-9.8.1.jar:9.8.1 2a113bf20593cbee7446d8130f2a68bbb6748916 -
> cloud-user - 2025-10-01 20:57:41]
> {code}
>
> A main hallmark of this issue is the java.io.EOFException, however you will
> always see a "downloaded X of X bytes" (which should ring alarm bells that
> something is out of place).
> +Steps to reproduce:+
> It's hard to force in a live environment, but theoretically you could fill a
> dummy segment file with a set number of bytes and add it to the index
> manifest. It is 100% reproducible via integration test
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
