[
https://issues.apache.org/jira/browse/CASSSIDECAR-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sampreeth Reddy B updated CASSSIDECAR-509:
------------------------------------------
Description:
*Problem*
When a live-migration client issues a {{GET
/api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against
the sidecar and the server-side path-validation code throws an unexpected
exception, the HTTP response is never closed on the server side. The client's
underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits
for a body it will never receive, blocking the live-migration task's
thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}}
fires (default: operation-wide timeout, often an hour).
The hang is caused by a missing {{.onFailure(...)}} handler on the
{{executorPools.service().executeBlocking(...)}} chain in
[{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
*Steps to Reproduce*
# Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any valid
Cassandra data directory.
# Issue a {{GET}} for an obviously-invalid path through either the Vert.x
client or {{{}curl{}}}:
{{curl -v
http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The
server correctly returns {{HTTP/1.1 404 Not Found}} .
# Repeat the request via the Vert.x client:
{{sidecarClient.liveMigrationStreamFileAsync(instance,
"this_path_does_not_exist.db", "/tmp/dest.db")
.get() this never completes}}
# Observed: the {{CompletableFuture}} returned by
{{liveMigrationStreamFileAsync}} never completes (success or failure). A thread
dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
{*}RCA{*}:
LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch
chained to the {{executeBlocking(...)}} future. Vert.x's documented contract is
that unhandled Future failures bubble up through {{UndeliverableException}} and
are logged, but the {{RoutingContext}} is not auto-failed by the framework the
handler is responsible for calling {{rc.fail(cause)}} or ending the response.
h4. Proposed Fix
chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or
{{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using the
project's existing error-wrapping convention
{{{}HttpExceptions.wrapHttpException(...){}}}):
was:
*Problem*
**
When a live-migration client issues a {{GET
/api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against
the sidecar and the server-side path-validation code throws an unexpected
exception, the HTTP response is never closed on the server side. The client's
underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits
for a body it will never receive, blocking the live-migration task's
thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}}
fires (default: operation-wide timeout, often an hour).
The hang is caused by a missing {{.onFailure(...)}} handler on the
{{executorPools.service().executeBlocking(...)}} chain in
[{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
*Steps to Reproduce*
# Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any valid
Cassandra data directory.
# Issue a {{GET}} for an obviously-invalid path through either the Vert.x
client or {{{}curl{}}}:
{{curl -v
http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The
server correctly returns {{HTTP/1.1 404 Not Found}} .
# Repeat the request via the Vert.x client:
{{sidecarClient.liveMigrationStreamFileAsync(instance,
"this_path_does_not_exist.db", "/tmp/dest.db")
.get() this never completes}}
# Observed: the {{CompletableFuture}} returned by
{{liveMigrationStreamFileAsync}} never completes (success or failure). A thread
dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
{*}RCA{*}:
LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch
chained to the {{executeBlocking(...)}} future. Vert.x's documented contract is
that unhandled Future failures bubble up through {{UndeliverableException}} and
are logged, but the {{RoutingContext}} is not auto-failed by the framework the
handler is responsible for calling {{rc.fail(cause)}} or ending the response.
h4. Proposed Fix
chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or
{{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using the
project's existing error-wrapping convention
{{{}HttpExceptions.wrapHttpException(...){}}}):
> Live-migration download hangs indefinitely when source returns a non-200 for
> an invalid file path
> -------------------------------------------------------------------------------------------------
>
> Key: CASSSIDECAR-509
> URL: https://issues.apache.org/jira/browse/CASSSIDECAR-509
> Project: Sidecar for Apache Cassandra
> Issue Type: Bug
> Components: Rest API
> Reporter: Sampreeth Reddy B
> Priority: Major
>
> *Problem*
> When a live-migration client issues a {{GET
> /api/v1/cassandra/live-migration/\{dir-type}/\{dir-index}/files/...}} against
> the sidecar and the server-side path-validation code throws an unexpected
> exception, the HTTP response is never closed on the server side. The client's
> underlying Vert.x {{HttpRequest.send()}} keeps the connection open and waits
> for a body it will never receive, blocking the live-migration task's
> thread-pool slot until the client-side {{HttpClientConfig.timeoutMillis()}}
> fires (default: operation-wide timeout, often an hour).
> The hang is caused by a missing {{.onFailure(...)}} handler on the
> {{executorPools.service().executeBlocking(...)}} chain in
> [{{LiveMigrationFileResolveHandler.handleInternal(...)}}|https://github.com/apache/cassandra-sidecar/blob/trunk/server/src/main/java/org/apache/cassandra/sidecar/handlers/livemigration/LiveMigrationFileResolveHandler.java].
> *Steps to Reproduce*
> # Start a sidecar (release 0.4.0 or main) and point {{data_dirs}} at any
> valid Cassandra data directory.
> # Issue a {{GET}} for an obviously-invalid path through either the Vert.x
> client or {{{}curl{}}}:
> {{curl -v
> http://<sidecar-host>:9043/api/v1/cassandra/live-migration/instance/1/files/this_path_does_not_exist.db}}The
> server correctly returns {{HTTP/1.1 404 Not Found}} .
> # Repeat the request via the Vert.x client:
> {{sidecarClient.liveMigrationStreamFileAsync(instance,
> "this_path_does_not_exist.db", "/tmp/dest.db")
> .get() this never completes}}
> # Observed: the {{CompletableFuture}} returned by
> {{liveMigrationStreamFileAsync}} never completes (success or failure). A
> thread dump shows the calling thread parked on {{{}CompletableFuture.get{}}}.
> {*}RCA{*}:
> LiveMigrationFileResolveHandler does not have an {{.onFailure(...)}} branch
> chained to the {{executeBlocking(...)}} future. Vert.x's documented contract
> is that unhandled Future failures bubble up through
> {{UndeliverableException}} and are logged, but the {{RoutingContext}} is not
> auto-failed by the framework the handler is responsible for calling
> {{rc.fail(cause)}} or ending the response.
> h4. Proposed Fix
> chain {{.onFailure(...)}} that calls {{rc.fail(500, cause)}} (or
> {{rc.response().setStatusCode(500).end(...)}} with a short JSON body, using
> the project's existing error-wrapping convention
> {{{}HttpExceptions.wrapHttpException(...){}}}):
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
