[
https://issues.apache.org/jira/browse/TIKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104854#comment-18104854
]
ASF GitHub Bot commented on TIKA-4815:
--------------------------------------
dpol1 commented on PR #3022:
URL: https://github.com/apache/tika/pull/3022#issuecomment-5296040860
thanks! RESOURCE_EXHAUSTED is the grpc native 429. we return the status in
the reply instead because that's how v1 already works, and onError would kill
the whole bidi stream - transport mapping can be its own follow-up if you want
it.
the abandoned-worker finding was real, fixed on both interrupt paths with
tests. dropped the dead catch too and tightened docs. for the backpressure one
I'd settle the rpc contract rather than paper over it with logs.
Addressed by 9316d8a354.
> tika-grpc: all parse requests share one single-threaded PipesClient
> -------------------------------------------------------------------
>
> Key: TIKA-4815
> URL: https://issues.apache.org/jira/browse/TIKA-4815
> Project: Tika
> Issue Type: Bug
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Davide Polato
> Priority: Major
> Labels: grpc, pipes
> Attachments: ConcurrencyLocalDiagnostic.java
>
>
> The gRPC server creates exactly one PipesClient and every parse handler
> thread uses it. PipesClient's own javadoc says it is single-threaded, and
> nothing in TikaGrpcServerImpl synchronizes access to it. The management RPCs
> are not involved - this is about fetchAndParse and its streaming variants,
> which all funnel through the same client.
> Attached is a small repro: one sequential warmup call so the forked worker is
> up and serving, then 4 concurrent fetchAndParse calls at an in-process server
> built without directExecutor (the production server uses a thread pool too).
> Measured on current master:
> DIAG-CONC WARMUP status=PARSE_SUCCESS
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> DIAG-CONC OK ... status=FAILED_TO_INITIALIZE
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> DIAG-CONC OK ... status=UNSPECIFIED_CRASH
> SUMMARY concurrency=4 answered=4 transportFailed=0 elapsedMs=61358
> Every call got a normal gRPC OK answer - transport-level success masking
> application-level failure. The statuses say the worker died or never came up
> for the request, and four tiny HTML files took a minute. A crawler pointed at
> this records every page as unparseable while the server looks perfectly
> healthy.
> Why no test caught it: TikaGrpcServerTest sends its requests sequentially and
> builds servers with directExecutor(), so nothing ever overlaps.
> Effectively today's server has an undeclared limit of one in-flight parse.
> This is also what blocks cancellation (see TIKA-4795): you can't interrupt
> one request's parse without disturbing the worker everyone else is on.
> Measured on current master (4.0.0-SNAPSHOT). The 3.x server has the same
> shape but I haven't run the probe there.
> tika-pipes already has the right mechanism: PipesParser keeps a queue of
> clients sized by numClients, admission with a timeout, release in finally.
> tika-grpc bypasses it and builds a bare client. Moving the server onto a pool
> looks like most of the fix, with two open questions: what should pool
> exhaustion map to on the wire (I'd lean gRPC RESOURCE_EXHAUSTED -
> infrastructure outcomes as status, parse outcomes in the reply - but that's
> this project's call), and what marks a client as unreusable before pooling -
> PipesClient.process() rethrows InterruptedException without closing the
> connection, and PipesParser's finally re-offers the client unconditionally.
> I can work on this - the repro converts to a regression test directly. Drop
> the attachment into tika-grpc/src/test/java/org/apache/tika/pipes/grpc/ and
> run with -Dtest=ConcurrencyLocalDiagnostic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
