[
https://issues.apache.org/jira/browse/TIKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104897#comment-18104897
]
ASF GitHub Bot commented on TIKA-4815:
--------------------------------------
tballison merged PR #3026:
URL: https://github.com/apache/tika/pull/3026
> tika-grpc: all parse requests share one single-threaded PipesClient
> -------------------------------------------------------------------
>
> Key: TIKA-4815
> URL: https://issues.apache.org/jira/browse/TIKA-4815
> Project: Tika
> Issue Type: Bug
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Davide Polato
> Priority: Major
> Labels: grpc, pipes
> Attachments: ConcurrencyLocalDiagnostic.java
>
>
> The gRPC server creates exactly one PipesClient and every parse handler
> thread uses it. PipesClient's own javadoc says it is single-threaded, and
> nothing in TikaGrpcServerImpl synchronizes access to it. The management RPCs
> are not involved - this is about fetchAndParse and its streaming variants,
> which all funnel through the same client.
> Attached is a small repro: one sequential warmup call so the forked worker is
> up and serving, then 4 concurrent fetchAndParse calls at an in-process server
> built without directExecutor (the production server uses a thread pool too).
> Measured on current master:
> DIAG-CONC WARMUP status=PARSE_SUCCESS
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> DIAG-CONC OK ... status=FAILED_TO_INITIALIZE
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> SUMMARY concurrency=4 answered=4 transportFailed=0 elapsedMs=61358
> Every call got a normal gRPC OK answer - transport-level success masking
> application-level failure. The statuses say the worker died or never came up
> for the request, and four tiny HTML files took a minute. A crawler pointed at
> this records every page as unparseable while the server looks perfectly
> healthy.
> Why no test caught it: TikaGrpcServerTest sends its requests sequentially and
> builds servers with directExecutor(), so nothing ever overlaps.
> Effectively today's server has an undeclared limit of one in-flight parse.
> This is also what blocks cancellation (see TIKA-4795): you can't interrupt
> one request's parse without disturbing the worker everyone else is on.
> Measured on current master (4.0.0-SNAPSHOT). The 3.x server has the same
> shape but I haven't run the probe there.
> tika-pipes already has the right mechanism: PipesParser keeps a queue of
> clients sized by numClients, admission with a timeout, release in finally.
> tika-grpc bypasses it and builds a bare client. Moving the server onto a pool
> looks like most of the fix, with two open questions: what should pool
> exhaustion map to on the wire (I'd lean gRPC RESOURCE_EXHAUSTED -
> infrastructure outcomes as status, parse outcomes in the reply - but that's
> this project's call), and what marks a client as unreusable before pooling -
> PipesClient.process() rethrows InterruptedException without closing the
> connection, and PipesParser's finally re-offers the client unconditionally.
> I can work on this - the repro converts to a regression test directly. Drop
> the attachment into tika-grpc/src/test/java/org/apache/tika/pipes/grpc/ and
> run with -Dtest=ConcurrencyLocalDiagnostic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
