[
https://issues.apache.org/jira/browse/KAFKA-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114714#comment-18114714
]
Mykyta Bozhenko edited comment on KAFKA-16304 at 9/13/26 3:48 AM:
------------------------------------------------------------------
Reproduced on the released 4.3.1 distribution, and the ticket's description
holds exactly.
*Setup.* Single-node KRaft broker, two log directories, one of them on a
{{dm-dust}} device. The tail of one segment is marked unreadable while its head
and the indexes stay in the page cache, so offset translation succeeds and the
failure can only land in the transfer. Then one large fetch spans into the
marked tail. This needs {{CONFIG_DM_DUST}}, which stock distribution kernels do
not enable, so the broker runs inside a User-Mode Linux guest built with the
device-mapper fault-injection targets.
*What happens.* From the guest's tracefs:
{code}
data-plane-kafk-1296 sys_sendfile64(out_fd: 0xb3, in_fd: 0xaf, offset:
0xc8ffdea8, count: 0xd737a)
data-plane-kafk-1296 sys_sendfile64 -> 0xfffffffffffffffb <- -EIO
{code}
162 of 699 {{sendfile64}} calls returned {{-EIO}}, across three
{{data-plane-kafka-network-thread}} threads. On the Java side every one of them
became a bare {{java.io.IOException: Input/output error}} at
{{FileDispatcherImpl.transferTo0}} -> {{PlaintextTransportLayer.transferFrom}}
-> {{FileRecords.writeTo}} -> ... -> {{Selector.pollSelectionKeys}}, and was
logged as {{DEBUG ... Connection with /127.0.0.1 ... disconnected}}.
{{KafkaStorageException}} count 0, offlined directories 0, {{kafka-log-dirs}}
reported {{error: null}} before and after, and 2000 records appended to the same
partition afterwards all succeeded. The broker stayed alive and healthy-looking
throughout.
*Why nothing below Kafka can help here.* A failed read of file data produces no
ext4 error and no kernel log line, and leaves the filesystem writable —
measured:
{{EXT4-fs error}} count 0, empty {{dmesg}}, and a write to the same filesystem
immediately after the failures succeeded. ({{Buffer I/O error}} is emitted on
the
{{buffer_head}} path used for metadata, which data reads do not take.) The errno
handed to the application is the only signal in the entire stack, which is what
makes this issue matter for latent sector errors — the common HDD failure, where
reads die while writes keep working.
*Disambiguation is possible without parsing error text.* On the error path,
re-reading the region that failed to transfer separates the two cases cleanly.
Measured on the same guest, same JDK, same call sequence, page cache dropped
before both runs:
|| case || transfer exception || message || verification re-read ||
| medium unreadable ({{dm-dust}}) | {{java.io.IOException}} | Input/output
error | fails |
| client aborts mid-transfer (RST) | {{java.io.IOException}} | Broken pipe |
succeeds |
The transfer stopped at exactly the first injected byte, so the failure point is
the injected one. Keying off the message text would not be portable: that text
is the platform's {{strerror}} output, and {{sendfile(2)}} error sets differ
across platforms.
*Patch.* [PR 23445|https://github.com/apache/kafka/pull/23445] does the
disambiguation half — {{FileRecords}}
re-reads the region and throws {{KafkaStorageException}} naming the file and the
position, leaving a socket failure exactly as it arrived — with three unit tests
and an end-to-end run of the released distribution with and without it:
|| || before || after ||
| {{sendfile64}} returning -EIO | 162 | 168 |
| {{KafkaStorageException}} in the log | 0 | 168 |
| {{DEBUG ... disconnected}} | 170 | 8 |
| {{WARN ... Unexpected error}} | 0 | 168 |
| log lines naming the unreadable file | 3 | 171 |
| log directory offlined | 0 | 0 |
162 of the 170 disconnects were disk failures wearing a network label; the 8
remaining are real client disconnects.
*The open question, before I write the second half.* Offlining needs the typed
exception to reach {{LogDirFailureChannel}}, and {{clients}} cannot depend on
{{storage}}. Two routes:
# let {{ChannelState}} carry the cause — it already carries
{{AuthenticationException}} for exactly this purpose — and have
{{SocketServer.Processor.processDisconnected}} route it;
# inject a listener into {{Selector}} from the server side: cleaner layering,
more code.
Related, and a policy call rather than a technical one: Kafka's failure unit is
the log directory, so a single unreadable 4 KiB block would offline a directory
of otherwise healthy replicas.
Disclosure per the ASF generative tooling guidance: the measurements and the
patch were produced with AI assistance; I have reviewed and am responsible for
all of it.
was (Author: JIRAUSER314651):
Reproduced on the released 4.3.1 distribution, and the ticket's description
holds exactly.
*Setup.* Single-node KRaft broker, two log directories, one of them on a
{{dm-dust}} device. The tail of one segment is marked unreadable while its head
and the indexes stay in the page cache, so offset translation succeeds and the
failure can only land in the transfer. Then one large fetch spans into the
marked tail. This needs {{CONFIG_DM_DUST}}, which stock distribution kernels do
not enable, so the broker runs inside a User-Mode Linux guest built with the
device-mapper fault-injection targets.
*What happens.* From the guest's tracefs:
{code}
data-plane-kafk-1296 sys_sendfile64(out_fd: 0xb3, in_fd: 0xaf, offset:
0xc8ffdea8, count: 0xd737a)
data-plane-kafk-1296 sys_sendfile64 -> 0xfffffffffffffffb <- -EIO
{code}
162 of 699 {{sendfile64}} calls returned {{-EIO}}, across three
{{data-plane-kafka-network-thread}} threads. On the Java side every one of them
became a bare {{java.io.IOException: Input/output error}} at
{{FileDispatcherImpl.transferTo0}} -> {{PlaintextTransportLayer.transferFrom}}
-> {{FileRecords.writeTo}} -> ... -> {{Selector.pollSelectionKeys}}, and was
logged as {{DEBUG ... Connection with /127.0.0.1 ... disconnected}}.
{{KafkaStorageException}} count 0, offlined directories 0, {{kafka-log-dirs}}
reported {{error: null}} before and after, and 2000 records appended to the same
partition afterwards all succeeded. The broker stayed alive and healthy-looking
throughout.
*Why nothing below Kafka can help here.* A failed read of file data produces no
ext4 error and no kernel log line, and leaves the filesystem writable —
measured:
{{EXT4-fs error}} count 0, empty {{dmesg}}, and a write to the same filesystem
immediately after the failures succeeded. ({{Buffer I/O error}} is emitted on
the
{{buffer_head}} path used for metadata, which data reads do not take.) The errno
handed to the application is the only signal in the entire stack, which is what
makes this issue matter for latent sector errors — the common HDD failure, where
reads die while writes keep working.
*Disambiguation is possible without parsing error text.* On the error path,
re-reading the region that failed to transfer separates the two cases cleanly.
Measured on the same guest, same JDK, same call sequence, page cache dropped
before both runs:
|| case || transfer exception || message || verification re-read ||
| medium unreadable ({{dm-dust}}) | {{java.io.IOException}} | Input/output
error | fails |
| client aborts mid-transfer (RST) | {{java.io.IOException}} | Broken pipe |
succeeds |
The transfer stopped at exactly the first injected byte, so the failure point is
the injected one. Keying off the message text would not be portable: that text
is the platform's {{strerror}} output, and {{sendfile(2)}} error sets differ
across platforms.
*Patch.* [PR 23445|https://github.com/apache/kafka/pull/23445] does the
disambiguation half — {{FileRecords}}
re-reads the region and throws {{KafkaStorageException}} naming the file and the
position, leaving a socket failure exactly as it arrived — with three unit tests
and an end-to-end run of the released distribution with and without it:
|| || before || after ||
| {{sendfile64}} returning -EIO | 162 | 168 |
| {{KafkaStorageException}} in the log | 0 | 168 |
| {{DEBUG ... disconnected}} | 170 | 8 |
| {{WARN ... Unexpected error}} | 0 | 168 |
| log lines naming the unreadable file | 3 | 171 |
| log directory offlined | 0 | 0 |
162 of the 170 disconnects were disk failures wearing a network label; the 8
remaining are real client disconnects.
*The open question, before I write the second half.* Offlining needs the typed
exception to reach {{LogDirFailureChannel}}, and {{clients}} cannot depend on
{{storage}}. Two routes:
# let {{ChannelState}} carry the cause — it already carries
{{AuthenticationException}} for exactly this purpose — and have
{{SocketServer.Processor.processDisconnected}} route it;
# inject a listener into {{Selector}} from the server side: cleaner layering,
more code.
Related, and a policy call rather than a technical one: Kafka's failure unit is
the log directory, so a single unreadable 4 KiB block would offline a directory
of otherwise healthy replicas.
Which route do you prefer, and does either need a KIP? I am happy to write it.
I do not have the contributor role in this project, so I cannot assign the issue
to myself. If this looks worth pursuing, could someone grant it and assign the
issue to me? The offlining half would follow once the route above is settled.
Disclosure per the ASF generative tooling guidance: the measurements and the
patch were produced with AI assistance; I have reviewed and am responsible for
all of it.
> Disk I/O error does not offline log directory while handling Fetch requests
> ---------------------------------------------------------------------------
>
> Key: KAFKA-16304
> URL: https://issues.apache.org/jira/browse/KAFKA-16304
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.6.1, 3.7.0
> Reporter: Gaurav Narula
> Priority: Critical
>
> The Fetch API handler delays reading of record batches from disk using
> {{FileRecords::writeTo}} until the bytes are ready to be sent to the wire.
> Therefore, a disk I/O error that happens while reading the log segment in
> either {{SslTransportLayer::transferFrom}} or
> {{PlaintextTransportLayer::transferFrom}} does not cause the log directory to
> be offlined. Instead, we assume that such an {{IOException}} can only occur
> because of network related causes and close the socket instead
> [[0]|https://github.com/apache/kafka/blob/7ac50a86113c9a489667451eecf61b6ab80870db/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L610]
>
> [[1]|https://github.com/apache/kafka/blob/7ac50a86113c9a489667451eecf61b6ab80870db/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L629]
> We should disambiguate between IOExceptions due to network and disk
> operations and offline the log directory in case of the latter as we do for
> other disk related failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
