This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new bea17ce11f TIKA-4978: per-request limits may tighten, not loosen, the
server's (#3367)
bea17ce11f is described below
commit bea17ce11f0655eebe48b56d53b2b65444a3636d
Author: Tim Allison <[email protected]>
AuthorDate: Mon Oct 19 17:39:07 2026 -0400
TIKA-4978: per-request limits may tighten, not loosen, the server's (#3367)
---
CHANGES.txt | 2 +
.../ROOT/pages/advanced/setting-limits.adoc | 19 +++
docs/modules/ROOT/pages/configuration/pages.adoc | 6 +-
.../ROOT/pages/developers/serialization.adoc | 15 +-
.../migration-to-4x/migrating-tika-server-4x.adoc | 6 +-
.../pages/migration-to-4x/serialization-4x.adoc | 10 +-
docs/modules/ROOT/pages/pipes/timeouts.adoc | 5 +-
docs/modules/ROOT/pages/security.adoc | 19 ++-
docs/modules/ROOT/pages/using-tika/grpc/index.adoc | 3 +-
.../ROOT/pages/using-tika/server/index.adoc | 40 +++++-
.../org/apache/tika/config/EmbeddedLimits.java | 26 ++++
.../java/org/apache/tika/config/OutputLimits.java | 24 ++++
.../StandardMetadataLimiterFactory.java | 27 ++++
.../org/apache/tika/parser/pages/PagesConfig.java | 56 ++++++++
.../org/apache/tika/renderer/RenderSettings.java | 38 +++++
.../apache/tika/config/LimitsTightenedToTest.java | 76 ++++++++++
.../apache/tika/parser/pages/PagesConfigTest.java | 54 +++++++
.../apache/tika/renderer/RenderSettingsTest.java | 23 +++
.../apache/tika/parser/image/RawTiffParser.java | 2 +
.../parser/microsoft/AbstractMetafileParser.java | 3 +-
.../java/org/apache/tika/parser/pdf/PDFParser.java | 2 +-
.../apache/tika/parser/pdf/PDFParserConfig.java | 1 +
.../tika/pipes/core/config/ConfigMerger.java | 19 +++
.../tika/pipes/core/config/ConfigOverrides.java | 38 +++++
.../tika/pipes/core/server/ConnectionHandler.java | 6 -
.../apache/tika/pipes/core/server/PipesServer.java | 34 +++--
.../tika/pipes/core/server/ServerProtocolIO.java | 107 ++++++++++++++
.../pipes/core/server/SharedServerResources.java | 6 +-
.../tika/pipes/core/config/ConfigMergerTest.java | 25 ++++
.../tika/pipes/core/server/PresetMergeTest.java | 39 ++++++
.../pipes/core/server/ServerProtocolIOTest.java | 148 ++++++++++++++++++-
.../apache/tika/pipes/fork/PipesForkParser.java | 6 +
.../tika/server/core/PipesRequestLimitsTest.java | 156 +++++++++++++++++++++
33 files changed, 995 insertions(+), 46 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index e750a38274..40f3a277e3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,7 @@
Release 4.2.0 - unreleased
+ * Per-request limits may tighten the server's but no longer loosen them
(TIKA-4978).
+
* Image checks (byte-size, min-image-size, blank-image-check; new
tika-image-processing
module) make it less likely that blank or tiny images reach text
recognizers. VLM
output gets tk:ocr-warnings (truncated, repetition, empty); VLM parsers
take
diff --git a/docs/modules/ROOT/pages/advanced/setting-limits.adoc
b/docs/modules/ROOT/pages/advanced/setting-limits.adoc
index 957fe2e9e1..7284ffef52 100644
--- a/docs/modules/ROOT/pages/advanced/setting-limits.adoc
+++ b/docs/modules/ROOT/pages/advanced/setting-limits.adoc
@@ -121,6 +121,25 @@ Tests: `AllLimitsTest`, `EmbeddedLimitsTest`,
`OutputLimitsTest`, `TimeoutLimits
`ExceptionReportingConfigLoadTest` under
`tika-serialization/src/test/java/org/apache/tika/config/`;
`ExceptionReportingParseTest` (tika-core) drives the policy through a real
parse.
+== Per-request limits
+
+A request to the pipes worker -- tika-server `/pipes`, `/async` and the
multipart `config`
+part, tika-grpc, `PipesParser` callers -- may carry its own `embedded-limits`,
+`output-limits`, `standard-metadata-limiter-factory` and content-handler
`writeLimit`. Each is
+clamped field by field to the operator's value (the server's `parse-context`
plus any preset,
+or the class default when neither sets one), so a request can tighten a limit
but not loosen
+it. A clamp is logged as a warning in the worker. A request may not replace a
configured
+metadata limiter with a different kind. The same holds for the render limits
in a `pages` block
+(`render` and `emit.render`: `dpi`, `maxImagePixels`, `maxWidth`, `maxHeight`,
including the
+deprecated `ocr` aliases), at the top level or under a parser; the ceiling is
what the
+operator's own `pages` layers resolve to for that parser. Timeouts follow
their own rule: a request may raise
+them up to `pipes.maxTotalTaskTimeoutMillis`
+(xref:pipes/timeouts.adoc#_per_request_overrides[details]).
+
+To let callers go further, raise the limit in the server's config or in a
preset.
+`PipesForkParser` writes the limits set on its `PipesForkParserConfig` into
its fork's config,
+so they are the ceiling rather than a request.
+
== Embedded document limits
`EmbeddedLimits` bounds how deep and how many embedded documents are parsed.
diff --git a/docs/modules/ROOT/pages/configuration/pages.adoc
b/docs/modules/ROOT/pages/configuration/pages.adoc
index 352aade554..d1b6c05c4e 100644
--- a/docs/modules/ROOT/pages/configuration/pages.adoc
+++ b/docs/modules/ROOT/pages/configuration/pages.adoc
@@ -73,7 +73,11 @@ So "PDFs render at 200 dpi, previews in colour", kept across
requests, is
and a request that sends `{"pages": {"text": "OCR"}}` OCRs every page at 200
dpi, since the PDF's
overlay still holds. Had the 200 dpi been under `parse-context.pages` instead,
that request would
-have replaced it and OCR'd at 300: a request's `pages` is the whole block. The
same overlay also
+have replaced the block (a request's `pages` is the whole block), but it would
still render at
+200: in tika-server, tika-grpc and other pipes workers a request may lower the
render limits
+(`dpi`, `maxImagePixels`, `maxWidth`, `maxHeight`, in `render` and
`emit.render`) but not raise
+them above what the operator's layers resolve to
+(xref:advanced/setting-limits.adoc#_per_request_limits[details]). The same
overlay also
means a request cannot undo a parser's own `pages` with the general block; it
sends
`{"pdf-parser": {"pages": {...}}}`. In code, `context.set(PagesConfig.class,
overlay)` is the
request layer and `PDFParserConfig.pages()` the parser's.
diff --git a/docs/modules/ROOT/pages/developers/serialization.adoc
b/docs/modules/ROOT/pages/developers/serialization.adoc
index 06322b87c1..d8bb1d796c 100644
--- a/docs/modules/ROOT/pages/developers/serialization.adoc
+++ b/docs/modules/ROOT/pages/developers/serialization.adoc
@@ -259,14 +259,17 @@ passed through to the component unexamined. So while a
request cannot bind a new
`{"parse-context": {"pages": {"text": "EXTRACT_AND_OCR"}}}`
will reach `PDFParser` and take effect.
-That is why per-request configuration is gated separately by
-`allowPerRequestConfig`, which is off by default. Treat "the caller may supply
+That is why per-request configuration is an operator decision: tika-server's
+`allowPerRequestConfig` for the `/config` routes, and `allowPipes` for `/pipes`
+and `/async`, whose tuples carry a `parse-context`. Treat "the caller may
supply
per-request config" as equivalent to "the caller may set any parser option,
including options that spawn external processes such as OCR" — not as something
-the allowlist constrains. The one exception is a property marked
`@OperatorOnly`
-(a path, executable, class name, URL, credential or operator-sized limit; see
-<<_operator_only_properties,Operator-only properties>>): a per-request block
-that names one is refused.
+the allowlist constrains. Two things still bound it. A property marked
+`@OperatorOnly` (a path, executable, class name, URL, credential or
+operator-sized limit; see <<_operator_only_properties,Operator-only
properties>>)
+is refused in a per-request block. And the pipes worker clamps request-supplied
+output, embedded-document, metadata and timeout limits so they can tighten the
+operator's but not loosen them (`ServerProtocolIO.clampRequestLimits`).
====
[source,json]
diff --git
a/docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc
b/docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc
index 6ef28fc200..a522a8f2c0 100644
--- a/docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc
+++ b/docs/modules/ROOT/pages/migration-to-4x/migrating-tika-server-4x.adoc
@@ -297,8 +297,10 @@ granting per-request parser configuration are separate
decisions:
The capabilities are two default-`false` flags in the `server` section:
-* `allowPipes` gates the `/pipes` and `/async` endpoints, which drive
process-isolated batch parsing through your fetchers and emitters. Selecting
either without `allowPipes` causes the server to refuse to start with a clear
error.
-* `allowPerRequestConfig` gates per-request parser configuration: the
`/config` family of endpoints and the multipart `config` part. When off, such
requests are rejected with 403.
+* `allowPipes` gates the `/pipes` and `/async` endpoints, which drive
process-isolated batch parsing through your fetchers and emitters. Selecting
either without `allowPipes` causes the server to refuse to start with a clear
error. It includes each tuple's `parse-context`, which is per-request
configuration in its own right; `allowPerRequestConfig` is not needed for it.
+* `allowPerRequestConfig` gates per-request parser configuration on the parse
endpoints: the `/config` family of endpoints and the multipart `config` part.
When off, such requests are rejected with 403.
+
+Per-request configuration that either flag admits is still bounded:
`@OperatorOnly` properties are refused, and limits may tighten the server's but
not loosen them
(xref:advanced/setting-limits.adoc#_per_request_limits[details]).
`/status` is no longer gated: it exposes only aggregate counters, so it is
enabled simply by listing `status` under `endpoints`.
diff --git a/docs/modules/ROOT/pages/migration-to-4x/serialization-4x.adoc
b/docs/modules/ROOT/pages/migration-to-4x/serialization-4x.adoc
index 6dfb09674d..4118934835 100644
--- a/docs/modules/ROOT/pages/migration-to-4x/serialization-4x.adoc
+++ b/docs/modules/ROOT/pages/migration-to-4x/serialization-4x.adoc
@@ -134,6 +134,10 @@ context-key types: only metadata/output-shaping components
(e.g. `MetadataFilter
Note the boundary: the allowlist blocks *binding a component* from the wire,
not *configuring
one that is already loaded*. Self-configuring components — every `Parser`
among them — have
their config subtree passed through unscanned, so a per-request config can
still set parser
-options (including ones that spawn external processes, such as OCR). This is
why
-`allowPerRequestConfig` is a separate gate and is off by default; the
allowlist alone does
-not make per-request configuration safe to expose.
+options (including ones that spawn external processes, such as OCR). This is
why accepting
+per-request configuration is an operator decision: in tika-server,
`allowPerRequestConfig`
+for the `config` part and `allowPipes` for `/pipes` and `/async`, whose tuples
carry a
+`parse-context`. Either way the configuration stays bounded: `@OperatorOnly`
properties are
+refused, and limits may tighten the operator's but not loosen them
+(xref:advanced/setting-limits.adoc#_per_request_limits[details]). The
allowlist alone does not
+make per-request configuration safe to expose.
diff --git a/docs/modules/ROOT/pages/pipes/timeouts.adoc
b/docs/modules/ROOT/pages/pipes/timeouts.adoc
index 2b983ee67d..6215800ac2 100644
--- a/docs/modules/ROOT/pages/pipes/timeouts.adoc
+++ b/docs/modules/ROOT/pages/pipes/timeouts.adoc
@@ -412,8 +412,9 @@ timeoutMillis)` directly instead of `checkCommand()`.
== Per-Request Overrides
-When using Tika Server with `allowPerRequestConfig: true`, timeouts can be
overridden
-per-request by including `TimeoutLimits` in the `ParseContext` of a
`FetchEmitTuple`:
+Timeouts can be overridden per request by including `TimeoutLimits` in the
`ParseContext`
+of a `FetchEmitTuple` (tika-server `/pipes` and `/async`, or the multipart
`config` part
+with `allowPerRequestConfig: true`):
[source,java]
----
diff --git a/docs/modules/ROOT/pages/security.adoc
b/docs/modules/ROOT/pages/security.adoc
index 10c6b8bce3..4a481120e8 100644
--- a/docs/modules/ROOT/pages/security.adoc
+++ b/docs/modules/ROOT/pages/security.adoc
@@ -74,16 +74,27 @@ callers — that requirement stands regardless of how the
flags are set. All def
* `allowPipes` gates the `/pipes` and `/async` endpoints (process-isolated
fetching and parsing
through your fetchers/emitters). Selecting either without it causes the
server to refuse to start.
+ A caller who reaches these endpoints can do whatever your configured
fetchers, emitters and
+ parsers can do, with your credentials and network position, including
configuring how each
+ document parses through the tuple's `parse-context`. Enable it only for
trusted callers; see
+
xref:using-tika/server/index.adoc#_allowpipes_the_pipes_and_async_endpoints[what
it grants]
+ (an illustration, not a complete list).
* `allowPerRequestConfig` gates per-request parser configuration — the
`/config` endpoints and the
multipart `config` part; when off, such requests are rejected with HTTP 403.
A caller who
reaches it can choose how documents parse, including options that spawn
external processes
- such as OCR, and can defeat memory and timeout limits.
+ such as OCR.
-*tika-grpc* is *more exposed by default* than tika-server: it ships with no
transport security and
+Either way, per-request configuration stays bounded: a request may not set an
`@OperatorOnly`
+property (a path, executable, class name, URL or credential), and it may
tighten the server's
+output, embedded-document, metadata and timeout limits but not loosen them
+(xref:advanced/setting-limits.adoc#_per_request_limits[details]).
+
+*tika-grpc* ships unlocked; securing it is the deployer's job. It has no
transport security and
no per-caller authorization, and its `FetchAndParse` surface is always on.
`allowPerRequestConfig`
and `allowComponentManagement` — the latter lets clients add, modify, and
delete fetchers/iterators
-and read back stored configs, which can contain secrets — are off by default.
Run it only behind
-network controls and, ideally, mutual TLS. See
xref:using-tika/grpc/index.adoc[Tika gRPC].
+and read back stored configs, which can contain secrets — are off by default,
but they are
+conveniences, not guarantees. Run it only behind network controls and,
ideally, mutual TLS. See
+xref:using-tika/grpc/index.adoc[Tika gRPC].
For the upgrade from the former `enableUnsecureFeatures` flag, which is now
split into
`allowPipes` and `allowPerRequestConfig`, see
diff --git a/docs/modules/ROOT/pages/using-tika/grpc/index.adoc
b/docs/modules/ROOT/pages/using-tika/grpc/index.adoc
index e088ab5bf2..ca0927e892 100644
--- a/docs/modules/ROOT/pages/using-tika/grpc/index.adoc
+++ b/docs/modules/ROOT/pages/using-tika/grpc/index.adoc
@@ -84,7 +84,8 @@ readable file, when `--client-auth-required` is set.
The primary rule is *trusted callers only*. tika-grpc is *even more exposed by
default than
tika-server*: no transport security, no per-caller authorization, and its core
`FetchAndParse`
surface always on. Anyone who can reach the gRPC port can fetch and parse
whatever the server's
-configured fetchers can reach. Run it only behind strict network controls.
+configured fetchers can reach. tika-grpc ships unlocked; securing it is the
deployer's job.
+Run it only behind strict network controls.
Its most dangerous capabilities — runtime mutation of the fetcher/iterator
store (for example
`SaveFetcher`, which can read files and load code on the server host) and
per-request parse
diff --git a/docs/modules/ROOT/pages/using-tika/server/index.adoc
b/docs/modules/ROOT/pages/using-tika/server/index.adoc
index bc27b79b17..220e61c97d 100644
--- a/docs/modules/ROOT/pages/using-tika/server/index.adoc
+++ b/docs/modules/ROOT/pages/using-tika/server/index.adoc
@@ -226,11 +226,37 @@ top-level `"preset": "name"` field, resolved the same way
beneath the tuple's ow
=== `allowPipes` — the `/pipes` and `/async` endpoints
`/pipes` and `/async` drive process-isolated batch parsing through your
configured fetchers and
-emitters, so whoever can reach them gains the read access of your fetchers and
the write access
-of your emitters (again
-https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3271[CVE-2015-3271]).
Listing them under
-`endpoints` is not enough — without `allowPipes` the server refuses to start,
deliberately making
-this an explicit choice:
+emitters. Enable them only for callers you trust as you trust your own
configuration: whoever can
+reach them can do whatever your fetchers, emitters and parsers can do, with
your configuration,
+your credentials and your server's network position (again
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3271[CVE-2015-3271]).
+
+What that includes depends on the plugins you configure. The following
illustrates it; it is
+*not* a complete list, and anything not listed here is not thereby safe:
+
+* *Reading* anything a configured fetcher can reach, at a key the caller
chooses: files under a
+ file-system fetcher's `basePath` (or anywhere, with `allowAbsolutePaths`),
and any object the
+ credentials of an S3, GCS, Azure or other cloud fetcher can read.
+* *Server-side request forgery* through the HTTP fetcher: the fetch key is the
URL, so a caller
+ can make the server request loopback, private-network and cloud metadata
addresses, and a
+ tuple's `parse-context` can add request headers
+ (xref:pipes/plugins/http.adoc#security-notes[details]).
+* *Writing* through any configured emitter, at a key the caller chooses:
overwriting anything
+ under a file-system emitter's `basePath` (or anywhere, with
`allowAbsolutePaths`), including
+ other callers' output, and writing to any store an emitter's credentials
allow. A fetcher and
+ an emitter together can copy data to wherever the caller can read it back.
+* *Configuring how each document parses*: a tuple's `parse-context` is
per-request
+ configuration, with the same bounds as `allowPerRequestConfig` (no
`@OperatorOnly` properties,
+ and limits that may tighten yours but not loosen them). Those bounds limit
what a request can
+ change; they do not make the endpoints safe for untrusted callers.
+* *Consuming resources*: every tuple runs a fetch, a parse and an emit in your
forked workers.
+
+`allowPipes` and the bounds above are defense in depth, not a security
boundary. Restrict
+`/pipes` and `/async` to trusted callers with your own authentication,
authorization and network
+controls, and configure only the fetchers and emitters you would hand to every
one of them.
+
+Listing them under `endpoints` is not enough — without `allowPipes` the server
refuses to start,
+deliberately making this an explicit choice:
[source,json]
----
@@ -629,8 +655,8 @@ Server behavior beyond host/port comes from the `server`
section of the JSON con
|`allowPipes`
|`false`
-|Opt-in for `/pipes` and `/async`. The server refuses to start if either is
selected without it.
-See <<_security,Security>>.
+|Opt-in for `/pipes` and `/async`, including each tuple's `parse-context`. The
server refuses to
+start if either is selected without it. See <<_security,Security>>.
|`allowPerRequestConfig`
|`false`
diff --git a/tika-core/src/main/java/org/apache/tika/config/EmbeddedLimits.java
b/tika-core/src/main/java/org/apache/tika/config/EmbeddedLimits.java
index 653e553421..359b76ffe7 100644
--- a/tika-core/src/main/java/org/apache/tika/config/EmbeddedLimits.java
+++ b/tika-core/src/main/java/org/apache/tika/config/EmbeddedLimits.java
@@ -188,6 +188,32 @@ public class EmbeddedLimits implements Serializable {
return limits != null ? limits : new EmbeddedLimits();
}
+ /**
+ * Returns this instance if neither limit is looser than {@code
ceiling}'s, otherwise a
+ * copy with each looser limit reduced to the ceiling's. Used at trust
boundaries so a
+ * request may tighten operator limits but not loosen them.
+ *
+ * @param ceiling the operator's limits
+ * @return this, or a tightened copy
+ * @since Apache Tika 4.2
+ */
+ public EmbeddedLimits tightenedTo(EmbeddedLimits ceiling) {
+ int depth = tighter(maxDepth, ceiling.maxDepth);
+ int count = tighter(maxCount, ceiling.maxCount);
+ if (depth == maxDepth && count == maxCount) {
+ return this;
+ }
+ return new EmbeddedLimits(depth, throwOnMaxDepth, count,
throwOnMaxCount);
+ }
+
+ /** The tighter of two limits where a negative value means unlimited. */
+ static int tighter(int requested, int ceiling) {
+ if (ceiling < 0) {
+ return requested;
+ }
+ return requested < 0 ? ceiling : Math.min(requested, ceiling);
+ }
+
@Override
public String toString() {
return "EmbeddedLimits{" +
diff --git a/tika-core/src/main/java/org/apache/tika/config/OutputLimits.java
b/tika-core/src/main/java/org/apache/tika/config/OutputLimits.java
index 0e7bd540b9..a2a1e6c4a5 100644
--- a/tika-core/src/main/java/org/apache/tika/config/OutputLimits.java
+++ b/tika-core/src/main/java/org/apache/tika/config/OutputLimits.java
@@ -229,6 +229,30 @@ public class OutputLimits implements Serializable {
this.zipBombRatio = zipBombRatio;
}
+ /**
+ * Returns this instance if no limit is looser than {@code ceiling}'s,
otherwise a copy
+ * with each looser limit reduced to the ceiling's. A negative write limit
is unlimited.
+ * Used at trust boundaries so a request may tighten operator limits but
not loosen them.
+ *
+ * @param ceiling the operator's limits
+ * @return this, or a tightened copy
+ * @since Apache Tika 4.2
+ */
+ public OutputLimits tightenedTo(OutputLimits ceiling) {
+ int wl = EmbeddedLimits.tighter(writeLimit, ceiling.writeLimit);
+ int xmlDepth = Math.min(maxXmlDepth, ceiling.maxXmlDepth);
+ int pkgDepth = Math.min(maxPackageEntryDepth,
ceiling.maxPackageEntryDepth);
+ long threshold = Math.min(zipBombThreshold, ceiling.zipBombThreshold);
+ long ratio = Math.min(zipBombRatio, ceiling.zipBombRatio);
+ if (wl == writeLimit && xmlDepth == maxXmlDepth && pkgDepth ==
maxPackageEntryDepth
+ && threshold == zipBombThreshold && ratio == zipBombRatio) {
+ return this;
+ }
+ OutputLimits tightened = new OutputLimits(wl, throwOnWriteLimit,
xmlDepth, threshold, ratio);
+ tightened.maxPackageEntryDepth = pkgDepth;
+ return tightened;
+ }
+
/**
* Helper method to get OutputLimits from ParseContext with defaults.
*
diff --git
a/tika-core/src/main/java/org/apache/tika/metadata/writelimiter/StandardMetadataLimiterFactory.java
b/tika-core/src/main/java/org/apache/tika/metadata/writelimiter/StandardMetadataLimiterFactory.java
index ed3d32a79b..0cfd13e851 100644
---
a/tika-core/src/main/java/org/apache/tika/metadata/writelimiter/StandardMetadataLimiterFactory.java
+++
b/tika-core/src/main/java/org/apache/tika/metadata/writelimiter/StandardMetadataLimiterFactory.java
@@ -81,6 +81,33 @@ public class StandardMetadataLimiterFactory implements
MetadataWriteLimiterFacto
excludeFields, includeEmpty);
}
+ /**
+ * Returns this instance if no size limit is looser than {@code
ceiling}'s, otherwise a
+ * copy with each looser size limit reduced to the ceiling's. Field
selection is the
+ * request's. Used at trust boundaries so a request may tighten operator
limits but not
+ * loosen them.
+ *
+ * @param ceiling the operator's limiter
+ * @return this, or a tightened copy
+ * @since Apache Tika 4.2
+ */
+ public StandardMetadataLimiterFactory
tightenedTo(StandardMetadataLimiterFactory ceiling) {
+ if (maxKeySize <= ceiling.maxKeySize && maxFieldSize <=
ceiling.maxFieldSize
+ && maxTotalBytes <= ceiling.maxTotalBytes
+ && maxValuesPerField <= ceiling.maxValuesPerField) {
+ return this;
+ }
+ StandardMetadataLimiterFactory tightened = new
StandardMetadataLimiterFactory();
+ tightened.setIncludeFields(includeFields);
+ tightened.setExcludeFields(excludeFields);
+ tightened.includeEmpty = includeEmpty;
+ tightened.maxKeySize = Math.min(maxKeySize, ceiling.maxKeySize);
+ tightened.maxFieldSize = Math.min(maxFieldSize, ceiling.maxFieldSize);
+ tightened.maxTotalBytes = Math.min(maxTotalBytes,
ceiling.maxTotalBytes);
+ tightened.maxValuesPerField = Math.min(maxValuesPerField,
ceiling.maxValuesPerField);
+ return tightened;
+ }
+
public void setIncludeFields(Set<String> includeFields) {
this.includeFields = includeFields != null ? new
HashSet<>(includeFields) : new HashSet<>();
}
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/pages/PagesConfig.java
b/tika-core/src/main/java/org/apache/tika/parser/pages/PagesConfig.java
index bf14f3ac42..602be8d2d2 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/pages/PagesConfig.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/pages/PagesConfig.java
@@ -26,6 +26,7 @@ import java.util.Locale;
import java.util.Set;
import org.apache.tika.annotation.TikaComponent;
+import org.apache.tika.config.JsonConfig;
import org.apache.tika.config.ParseContextConfig;
import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.metadata.Metadata;
@@ -89,6 +90,61 @@ public class PagesConfig implements Serializable {
return effective;
}
+ /**
+ * The operator's top-level {@code "pages"} (config plus preset; null when
neither sets one),
+ * which a request's own block replaces. The pipes worker records it so
that
+ * {@link #resolve(ParseContext, String, PagesConfig, PagesConfig)} can
bound a request by it.
+ *
+ * @since Apache Tika 4.2
+ */
+ public record OperatorPages(PagesConfig pages) implements Serializable {
+ }
+
+ /**
+ * The effective block for a rendering parser: as {@link
#resolve(ParseContext, PagesConfig...)}
+ * with the parser's {@code operatorOverlay} and its per-parse {@code
localOverlay}. In the
+ * pipes worker (where {@link OperatorPages} is set), when the context's
{@code "pages"} or
+ * the parser's {@code componentName} config is untrusted request JSON,
the render limits
+ * ({@link RenderSettings#tightenedTo}) may tighten the operator's but not
loosen them.
+ *
+ * @since Apache Tika 4.2
+ */
+ public static PagesConfig resolve(ParseContext context, String
componentName,
+ PagesConfig operatorOverlay, PagesConfig
localOverlay)
+ throws TikaConfigException, IOException {
+ PagesConfig effective = resolve(context, operatorOverlay,
localOverlay);
+ OperatorPages operator = context == null ? null :
context.get(OperatorPages.class);
+ if (operator == null || !(fromRequest(context, KEY) ||
fromRequest(context, componentName))) {
+ return effective;
+ }
+ return
effective.tightenedTo(defaults().over(operator.pages()).over(operatorOverlay));
+ }
+
+ private static boolean fromRequest(ParseContext context, String key) {
+ JsonConfig json = context.getJsonConfig(key);
+ return json != null && !json.trusted();
+ }
+
+ /**
+ * This effective block with its render and emitted-render limits reduced
to
+ * {@code ceiling}'s where they are looser; this instance if none is.
+ */
+ PagesConfig tightenedTo(PagesConfig ceiling) {
+ RenderSettings tightRender = render.tightenedTo(ceiling.render);
+ RenderSettings emitted = tightRender.over(emit == null ? null :
emit.render);
+ RenderSettings tightEmitted =
emitted.tightenedTo(ceiling.emittedRender());
+ if (tightRender == render && tightEmitted == emitted) {
+ return this;
+ }
+ PagesConfig result = over(null);
+ result.render = tightRender;
+ if (tightEmitted != emitted) {
+ result.emit = emit == null ? new Emit() : emit.copy();
+ result.emit.render = tightEmitted;
+ }
+ return result;
+ }
+
/** A copy of this with every set field of {@code overlay} applied; null
overlay is this. */
public PagesConfig over(PagesConfig overlay) {
PagesConfig result = new PagesConfig();
diff --git
a/tika-core/src/main/java/org/apache/tika/renderer/RenderSettings.java
b/tika-core/src/main/java/org/apache/tika/renderer/RenderSettings.java
index 53a047c112..03431601d2 100644
--- a/tika-core/src/main/java/org/apache/tika/renderer/RenderSettings.java
+++ b/tika-core/src/main/java/org/apache/tika/renderer/RenderSettings.java
@@ -101,6 +101,44 @@ public class RenderSettings implements Serializable {
return result;
}
+ /**
+ * This with {@code dpi}, {@code maxImagePixels}, {@code maxWidth} and
{@code maxHeight}
+ * reduced to {@code ceiling}'s where they are looser; this instance if
none is. A null
+ * field on either side is left alone. Used at trust boundaries so a
request may tighten
+ * the operator's render limits but not loosen them.
+ *
+ * @since Apache Tika 4.2
+ */
+ public RenderSettings tightenedTo(RenderSettings ceiling) {
+ Integer tDpi = dpi == null || ceiling.dpi == null ? dpi :
Math.min(dpi, ceiling.dpi);
+ Long tPixels = tighter(maxImagePixels, ceiling.maxImagePixels);
+ Long tWidth = tighter(toLong(maxWidth), toLong(ceiling.maxWidth));
+ Long tHeight = tighter(toLong(maxHeight), toLong(ceiling.maxHeight));
+ if (Objects.equals(tDpi, dpi) && Objects.equals(tPixels,
maxImagePixels)
+ && Objects.equals(tWidth, toLong(maxWidth))
+ && Objects.equals(tHeight, toLong(maxHeight))) {
+ return this;
+ }
+ RenderSettings result = copy();
+ result.dpi = tDpi;
+ result.maxImagePixels = tPixels;
+ result.maxWidth = tWidth == null ? null : tWidth.intValue();
+ result.maxHeight = tHeight == null ? null : tHeight.intValue();
+ return result;
+ }
+
+ /** The tighter of two limits where {@link #NO_LIMIT} means unlimited. */
+ private static Long tighter(Long requested, Long ceiling) {
+ if (requested == null || ceiling == null || ceiling == NO_LIMIT) {
+ return requested;
+ }
+ return requested == NO_LIMIT ? ceiling : Math.min(requested, ceiling);
+ }
+
+ private static Long toLong(Integer value) {
+ return value == null ? null : value.longValue();
+ }
+
public RenderSettings copy() {
RenderSettings result = new RenderSettings();
result.dpi = dpi;
diff --git
a/tika-core/src/test/java/org/apache/tika/config/LimitsTightenedToTest.java
b/tika-core/src/test/java/org/apache/tika/config/LimitsTightenedToTest.java
new file mode 100644
index 0000000000..bd3d60a5df
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/config/LimitsTightenedToTest.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.config;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+public class LimitsTightenedToTest {
+
+ @Test
+ public void testOutputLimitsTighterRequestPassesThrough() {
+ OutputLimits request = new OutputLimits(1000, false, 50, 500_000, 10);
+ assertSame(request, request.tightenedTo(new OutputLimits()));
+ }
+
+ @Test
+ public void testOutputLimitsLooserRequestIsClamped() {
+ OutputLimits ceiling = new OutputLimits(1000, false, 50, 500_000, 10);
+ OutputLimits request = new OutputLimits(OutputLimits.UNLIMITED, true,
10_000,
+ Long.MAX_VALUE, 5);
+
+ OutputLimits clamped = request.tightenedTo(ceiling);
+
+ assertEquals(1000, clamped.getWriteLimit());
+ assertEquals(50, clamped.getMaxXmlDepth());
+ assertEquals(500_000, clamped.getZipBombThreshold());
+ // already tighter than the ceiling
+ assertEquals(5, clamped.getZipBombRatio());
+ // not a ceiling: the request's choice stands
+ assertTrue(clamped.isThrowOnWriteLimit());
+ }
+
+ @Test
+ public void testUnlimitedCeilingAllowsAnyWriteLimit() {
+ OutputLimits request = new OutputLimits();
+ request.setWriteLimit(OutputLimits.UNLIMITED);
+ assertSame(request, request.tightenedTo(new OutputLimits()));
+ }
+
+ @Test
+ public void testEmbeddedLimits() {
+ EmbeddedLimits ceiling = new EmbeddedLimits(5, false, 100, false);
+
+ EmbeddedLimits looser = new EmbeddedLimits(EmbeddedLimits.UNLIMITED,
true, 1000, true);
+ EmbeddedLimits clamped = looser.tightenedTo(ceiling);
+ assertEquals(5, clamped.getMaxDepth());
+ assertEquals(100, clamped.getMaxCount());
+ assertTrue(clamped.isThrowOnMaxDepth());
+ assertTrue(clamped.isThrowOnMaxCount());
+
+ EmbeddedLimits tighter = new EmbeddedLimits(2, false, 10, false);
+ assertSame(tighter, tighter.tightenedTo(ceiling));
+
+ // default maxCount is unlimited, so any request count is allowed
+ EmbeddedLimits anyCount = new EmbeddedLimits(5, false, 1_000_000,
false);
+ assertSame(anyCount, anyCount.tightenedTo(new EmbeddedLimits(20, false,
+ EmbeddedLimits.UNLIMITED, false)));
+ }
+}
diff --git
a/tika-core/src/test/java/org/apache/tika/parser/pages/PagesConfigTest.java
b/tika-core/src/test/java/org/apache/tika/parser/pages/PagesConfigTest.java
index 66d92c3c90..8a2463749f 100644
--- a/tika-core/src/test/java/org/apache/tika/parser/pages/PagesConfigTest.java
+++ b/tika-core/src/test/java/org/apache/tika/parser/pages/PagesConfigTest.java
@@ -169,4 +169,58 @@ public class PagesConfigTest {
PagesConfig.Ocr ocr = new PagesConfig.Ocr();
assertThrows(IllegalArgumentException.class, () ->
ocr.setMaxPages(-3));
}
+
+ private static ParseContext requestFor(String componentName, PagesConfig
operatorTop) {
+ ParseContext context = new ParseContext();
+ // the parser's block came over the wire
+ context.setJsonConfig(componentName, "{}", false);
+ context.set(PagesConfig.OperatorPages.class, new
PagesConfig.OperatorPages(operatorTop));
+ return context;
+ }
+
+ @Test
+ public void testRequestMayNotRaiseRenderLimits() throws Exception {
+ PagesConfig request = new PagesConfig();
+ request.render().setDpi(1200);
+ request.render().setMaxImagePixels((long) RenderSettings.NO_LIMIT);
+ request.emit().render().setDpi(2400);
+
+ PagesConfig pages = PagesConfig.resolve(requestFor("x-parser", null),
"x-parser",
+ null, request);
+
+ assertEquals(300, pages.getRender().getDpi());
+ assertEquals(100_000_000L, pages.getRender().getMaxImagePixels());
+ assertEquals(300, pages.emittedRender().getDpi());
+ }
+
+ @Test
+ public void testOperatorLayersAreTheCeiling() throws Exception {
+ PagesConfig operatorTop = new PagesConfig();
+ operatorTop.render().setDpi(150);
+ // this parser's operator overlay allows more than the top-level block
+ PagesConfig operatorOverlay = new PagesConfig();
+ operatorOverlay.render().setDpi(600);
+ PagesConfig request = new PagesConfig();
+ request.render().setDpi(1200);
+
+ PagesConfig pages = PagesConfig.resolve(requestFor("x-parser",
operatorTop), "x-parser",
+ operatorOverlay, request);
+ assertEquals(600, pages.getRender().getDpi());
+
+ request.render().setDpi(96);
+ pages = PagesConfig.resolve(requestFor("x-parser", operatorTop),
"x-parser",
+ operatorOverlay, request);
+ assertEquals(96, pages.getRender().getDpi());
+ }
+
+ @Test
+ public void testNoClampOutsideTheWorker() throws Exception {
+ ParseContext context = new ParseContext();
+ context.setJsonConfig("x-parser", "{}", false);
+ PagesConfig request = new PagesConfig();
+ request.render().setDpi(1200);
+
+ assertEquals(1200, PagesConfig.resolve(context, "x-parser", null,
request)
+ .getRender().getDpi());
+ }
}
diff --git
a/tika-core/src/test/java/org/apache/tika/renderer/RenderSettingsTest.java
b/tika-core/src/test/java/org/apache/tika/renderer/RenderSettingsTest.java
index da8a3ba14a..8df913cb39 100644
--- a/tika-core/src/test/java/org/apache/tika/renderer/RenderSettingsTest.java
+++ b/tika-core/src/test/java/org/apache/tika/renderer/RenderSettingsTest.java
@@ -21,6 +21,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -131,4 +132,26 @@ public class RenderSettingsTest {
s.setMinWidth(0);
assertEquals(-1, s.getMaxWidth());
}
+
+ @Test
+ public void testTightenedTo() {
+ RenderSettings ceiling = RenderSettings.defaults();
+ ceiling.setDpi(150);
+ ceiling.setMaxWidth(2000);
+
+ RenderSettings looser = RenderSettings.defaults();
+ looser.setDpi(1200);
+ looser.setMaxImagePixels((long) RenderSettings.NO_LIMIT);
+ RenderSettings tight = looser.tightenedTo(ceiling);
+ assertEquals(150, tight.getDpi());
+ assertEquals(100_000_000L, tight.getMaxImagePixels());
+ // no box asked for, but the operator's box stands
+ assertEquals(2000, tight.getMaxWidth());
+ assertEquals(RenderSettings.NO_LIMIT, tight.getMaxHeight());
+
+ RenderSettings tighter = RenderSettings.defaults();
+ tighter.setDpi(72);
+ tighter.setMaxWidth(500);
+ assertSame(tighter, tighter.tightenedTo(ceiling));
+ }
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-image-module/src/main/java/org/apache/tika/parser/image/RawTiffParser.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-image-module/src/main/java/org/apache/tika/parser/image/RawTiffParser.java
index 8bacdfec00..49c382662f 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-image-module/src/main/java/org/apache/tika/parser/image/RawTiffParser.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-image-module/src/main/java/org/apache/tika/parser/image/RawTiffParser.java
@@ -39,6 +39,7 @@ import org.xml.sax.SAXException;
import org.apache.tika.annotation.TikaComponent;
import org.apache.tika.config.ConfigDeserializer;
import org.apache.tika.config.JsonConfig;
+import org.apache.tika.config.OperatorOnly;
import org.apache.tika.config.ParseContextConfig;
import org.apache.tika.exception.TikaException;
import org.apache.tika.extractor.EmbeddedDocumentExtractor;
@@ -566,6 +567,7 @@ public class RawTiffParser extends TiffParser {
return maxTotalPreviewBytes;
}
+ @OperatorOnly
public void setMaxTotalPreviewBytes(long maxTotalPreviewBytes) {
this.maxTotalPreviewBytes = maxTotalPreviewBytes;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/AbstractMetafileParser.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/AbstractMetafileParser.java
index d91e8d409f..5443a740fa 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/AbstractMetafileParser.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/AbstractMetafileParser.java
@@ -55,7 +55,8 @@ abstract class AbstractMetafileParser implements Parser,
RenderingParser {
/** The effective {@code "pages"} block for this parse; the default's
overlay first, as in PDFParser. */
PagesConfig pages(ParseContext context, MetafileParserConfig config)
throws TikaException, IOException {
- return PagesConfig.resolve(context, defaultConfig.getPages(),
config.getPages());
+ return PagesConfig.resolve(context, componentName(),
defaultConfig.getPages(),
+ config.getPages());
}
/**
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
index 4e947b62ba..121f489236 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParser.java
@@ -173,7 +173,7 @@ public class PDFParser implements Parser, RenderingParser,
EnrichingParser {
PDFParserConfig localConfig = getConfig(context);
// the default's overlay first: a class-keyed request config replaces
it rather than
// extending it, and the JSON path is a no-op second application
- PagesConfig pages = PagesConfig.resolve(context,
defaultConfig.getPages(),
+ PagesConfig pages = PagesConfig.resolve(context, "pdf-parser",
defaultConfig.getPages(),
localConfig.getPages());
if (localConfig.isSetKCMS()) {
System.setProperty("sun.java2d.cmm",
"sun.java2d.cmm.kcms.KcmsServiceProvider");
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
index ef5bf21fc7..c52b1da120 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
@@ -590,6 +590,7 @@ public class PDFParserConfig implements Serializable {
return maxMainMemoryBytes;
}
+ @OperatorOnly
public void setMaxMainMemoryBytes(long maxMainMemoryBytes) {
this.maxMainMemoryBytes = maxMainMemoryBytes;
}
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigMerger.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigMerger.java
index 458448956b..8baa532017 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigMerger.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigMerger.java
@@ -33,6 +33,7 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.config.loader.TikaJsonConfig;
import org.apache.tika.config.loader.TikaObjectMapperFactory;
@@ -251,6 +252,24 @@ public class ConfigMerger {
tl.getProgressTimeoutMillis());
}
+ if (overrides.getEmbeddedLimits() != null) {
+ EmbeddedLimits el = overrides.getEmbeddedLimits();
+ ObjectNode parseContextNode = getOrCreateObject(mapper, root,
"parse-context");
+ ObjectNode node = getOrCreateObject(mapper, parseContextNode,
"embedded-limits");
+ node.put("maxDepth", el.getMaxDepth());
+ node.put("throwOnMaxDepth", el.isThrowOnMaxDepth());
+ node.put("maxCount", el.getMaxCount());
+ node.put("throwOnMaxCount", el.isThrowOnMaxCount());
+ LOG.debug("Applied embedded limits: {}", el);
+ }
+
+ if (overrides.getWriteLimit() != null) {
+ ObjectNode parseContextNode = getOrCreateObject(mapper, root,
"parse-context");
+ getOrCreateObject(mapper, parseContextNode, "output-limits")
+ .put("writeLimit", overrides.getWriteLimit());
+ LOG.debug("Applied writeLimit: {}", overrides.getWriteLimit());
+ }
+
byte[] json =
mapper.writerWithDefaultPrettyPrinter().writeValueAsBytes(root);
// Return the first generated fetcher/emitter ID (or null if none)
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigOverrides.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigOverrides.java
index 40caad25f9..6411302aa5 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigOverrides.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/config/ConfigOverrides.java
@@ -22,6 +22,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.pipes.core.EmitStrategy;
@@ -51,6 +52,8 @@ public class ConfigOverrides {
private final String pluginRoots;
private final EmitStrategy emitStrategy;
private final TimeoutLimits timeoutLimits;
+ private final EmbeddedLimits embeddedLimits;
+ private final Integer writeLimit;
private final boolean classpathPlugins;
private ConfigOverrides(Builder builder) {
@@ -60,6 +63,8 @@ public class ConfigOverrides {
this.pluginRoots = builder.pluginRoots;
this.emitStrategy = builder.emitStrategy;
this.timeoutLimits = builder.timeoutLimits;
+ this.embeddedLimits = builder.embeddedLimits;
+ this.writeLimit = builder.writeLimit;
this.classpathPlugins = builder.classpathPlugins;
}
@@ -91,6 +96,14 @@ public class ConfigOverrides {
return timeoutLimits;
}
+ public EmbeddedLimits getEmbeddedLimits() {
+ return embeddedLimits;
+ }
+
+ public Integer getWriteLimit() {
+ return writeLimit;
+ }
+
/** Whether the fork is told to load plugins from its classpath; see
ConfigMerger. */
public boolean isClasspathPlugins() {
return classpathPlugins;
@@ -214,6 +227,8 @@ public class ConfigOverrides {
private String pluginRoots;
private EmitStrategy emitStrategy;
private TimeoutLimits timeoutLimits;
+ private EmbeddedLimits embeddedLimits;
+ private Integer writeLimit;
private boolean classpathPlugins;
private Builder() {
@@ -313,6 +328,29 @@ public class ConfigOverrides {
return this;
}
+ /**
+ * Set the embedded limits to write to the parse-context section.
Config-tier limits
+ * are the ceiling a request's own limits are clamped to.
+ *
+ * @param embeddedLimits the embedded limits (may be null to use
defaults)
+ * @return this builder
+ */
+ public Builder setEmbeddedLimits(EmbeddedLimits embeddedLimits) {
+ this.embeddedLimits = embeddedLimits;
+ return this;
+ }
+
+ /**
+ * Set {@code output-limits.writeLimit} in the parse-context section.
+ *
+ * @param writeLimit the write limit, negative for unlimited (may be
null to leave unset)
+ * @return this builder
+ */
+ public Builder setWriteLimit(Integer writeLimit) {
+ this.writeLimit = writeLimit;
+ return this;
+ }
+
/**
* Set the plugin roots path.
*
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ConnectionHandler.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ConnectionHandler.java
index c29e25d7ee..e3fcf569bd 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ConnectionHandler.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ConnectionHandler.java
@@ -53,7 +53,6 @@ import org.apache.tika.pipes.core.protocol.PipesMessage;
import org.apache.tika.pipes.core.protocol.PipesMessageType;
import org.apache.tika.pipes.core.serialization.JsonPipesIpc;
import org.apache.tika.pipes.core.serialization.PipesRequest;
-import org.apache.tika.serialization.ParseContextUtils;
/**
* Handles a single client connection in shared server mode.
@@ -218,11 +217,6 @@ public class ConnectionHandler implements Runnable,
Closeable {
long ctxStart = System.nanoTime();
mergedContext = resources.createMergedParseContext(
fetchEmitTuple.getParseContext(),
fetchEmitTuple.getPresetName());
- ParseContextUtils.resolveAll(mergedContext,
getClass().getClassLoader());
-
ServerProtocolIO.validateParseContext(mergedContext);
- ServerProtocolIO.clampRequestTimeoutLimits(
- fetchEmitTuple.getParseContext(),
mergedContext,
-
pipesConfig.getMaxTotalTaskTimeoutMillis());
// After resolveAll: the payload is typed runtime
state for
// BytesFetcher, never a resolvable config entry.
pipesRequest.applyTo(mergedContext);
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java
index 97c20dc91b..db1458095c 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/PipesServer.java
@@ -460,11 +460,6 @@ public class PipesServer implements AutoCloseable {
try {
mergedContext = createMergedParseContext(
fetchEmitTuple.getParseContext(),
fetchEmitTuple.getPresetName());
- ParseContextUtils.resolveAll(mergedContext,
getClass().getClassLoader());
-
ServerProtocolIO.validateParseContext(mergedContext);
- ServerProtocolIO.clampRequestTimeoutLimits(
- fetchEmitTuple.getParseContext(),
mergedContext,
-
pipesConfig.getMaxTotalTaskTimeoutMillis());
// After resolveAll: the payload is typed runtime
state for
// BytesFetcher, never a resolvable config entry.
pipesRequest.applyTo(mergedContext);
@@ -831,7 +826,7 @@ public class PipesServer implements AutoCloseable {
*
* @param requestContext the ParseContext from FetchEmitTuple
* @param presetName name of the preset to overlay at config-tier trust,
or null
- * @return a new ParseContext with defaults + preset + request overrides
+ * @return a new, resolved ParseContext with defaults + preset + clamped
request overrides
*/
private ParseContext createMergedParseContext(ParseContext requestContext,
String presetName)
throws TikaConfigException {
@@ -844,12 +839,35 @@ public class PipesServer implements AutoCloseable {
// extraction for every non-UNPACK parse mode. UNPACK mode sets its own
// EmbeddedDocumentExtractor + UnpackedByteCount in PipesWorker's
UNPACK-mode setup.
mergePreset(presetRegistry, presetName, mergedContext);
- // Request-level values override config defaults and the preset
- mergedContext.copyFrom(requestContext);
+ overlayRequest(requestContext, mergedContext, pipesConfig,
defaultContentHandlerFactory);
seedCacheMemoryBudget(mergedContext);
return mergedContext;
}
+ /**
+ * Overlays the request's values on the operator's (config plus preset),
resolves the
+ * result, and clamps what the request supplied: limits may be tightened,
not loosened,
+ * and timeouts are capped at {@code pipes.maxTotalTaskTimeoutMillis}.
+ * {@code defaultContentHandlerFactory} is the operator's when the context
sets none.
+ */
+ static void overlayRequest(ParseContext requestContext, ParseContext
mergedContext,
+ PipesConfig pipesConfig,
+ ContentHandlerFactory
defaultContentHandlerFactory)
+ throws TikaConfigException {
+ ParseContext operatorContext = new ParseContext();
+ operatorContext.copyFrom(mergedContext);
+ if (operatorContext.get(ContentHandlerFactory.class) == null
+ && defaultContentHandlerFactory != null) {
+ operatorContext.set(ContentHandlerFactory.class,
defaultContentHandlerFactory);
+ }
+ mergedContext.copyFrom(requestContext);
+ ParseContextUtils.resolveAll(mergedContext,
PipesServer.class.getClassLoader());
+ ServerProtocolIO.validateParseContext(mergedContext);
+ ServerProtocolIO.clampRequestTimeoutLimits(requestContext,
mergedContext,
+ pipesConfig.getMaxTotalTaskTimeoutMillis());
+ ServerProtocolIO.clampRequestLimits(operatorContext, mergedContext);
+ }
+
/**
* Overlays the named preset, resolved from this server's own config at
config-tier
* trust; only the name arrived on the wire. The caller's untrusted delta
is copied
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ServerProtocolIO.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ServerProtocolIO.java
index dc1a60b108..18185b8224 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ServerProtocolIO.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/ServerProtocolIO.java
@@ -27,11 +27,17 @@ import com.fasterxml.jackson.core.JsonProcessingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.ExceptionReporting;
+import org.apache.tika.config.OutputLimits;
+import org.apache.tika.config.ParseContextConfig;
import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.writelimiter.MetadataWriteLimiterFactory;
+import org.apache.tika.metadata.writelimiter.StandardMetadataLimiterFactory;
import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.pages.PagesConfig;
import org.apache.tika.pipes.api.ParseMode;
import org.apache.tika.pipes.api.PipesResult;
import org.apache.tika.pipes.core.extractor.UnpackConfig;
@@ -39,6 +45,9 @@ import org.apache.tika.pipes.core.protocol.PipesMessage;
import org.apache.tika.pipes.core.protocol.PipesMessageType;
import org.apache.tika.pipes.core.protocol.ShutDownReceivedException;
import org.apache.tika.pipes.core.serialization.JsonPipesIpc;
+import org.apache.tika.sax.BasicContentHandlerFactory;
+import org.apache.tika.sax.ContentHandlerFactory;
+import org.apache.tika.sax.WriteLimiter;
import org.apache.tika.utils.ExceptionUtils;
import org.apache.tika.utils.StringUtils;
@@ -405,6 +414,104 @@ public class ServerProtocolIO {
}
}
+ /**
+ * Trust boundary: a request may tighten the operator's {@link
OutputLimits},
+ * {@link EmbeddedLimits}, content-handler write limit, metadata limiter
and
+ * {@code maxUnpackBytes} but not loosen them. Render limits in {@code
"pages"} are bounded
+ * where the parser resolves them, by the {@link
PagesConfig.OperatorPages} recorded here.
+ * {@code operatorContext} is the server's config plus any preset, before
the request was
+ * overlaid; entries it already holds are the operator's and are left
alone. Must run after
+ * {@code ParseContextUtils.resolveAll}.
+ */
+ public static void clampRequestLimits(ParseContext operatorContext,
ParseContext merged)
+ throws TikaConfigException {
+ OutputLimits operatorOutput = OutputLimits.get(operatorContext);
+ OutputLimits output = merged.get(OutputLimits.class);
+ if (output != null && output !=
operatorContext.get(OutputLimits.class)) {
+ OutputLimits tightened = output.tightenedTo(operatorOutput);
+ if (tightened != output) {
+ LOG.warn("request-supplied {} is looser than the server's {};
clamping",
+ output, operatorOutput);
+ merged.set(OutputLimits.class, tightened);
+ }
+ }
+ EmbeddedLimits operatorEmbedded = EmbeddedLimits.get(operatorContext);
+ EmbeddedLimits embedded = merged.get(EmbeddedLimits.class);
+ if (embedded != null && embedded !=
operatorContext.get(EmbeddedLimits.class)) {
+ EmbeddedLimits tightened = embedded.tightenedTo(operatorEmbedded);
+ if (tightened != embedded) {
+ LOG.warn("request-supplied {} is looser than the server's {};
clamping",
+ embedded, operatorEmbedded);
+ merged.set(EmbeddedLimits.class, tightened);
+ }
+ }
+ MetadataWriteLimiterFactory operatorLimiter =
+ operatorContext.get(MetadataWriteLimiterFactory.class);
+ MetadataWriteLimiterFactory metadataLimiter =
merged.get(MetadataWriteLimiterFactory.class);
+ if (operatorLimiter != null && metadataLimiter != operatorLimiter) {
+ if (!(operatorLimiter instanceof StandardMetadataLimiterFactory
operatorStandard)
+ || !(metadataLimiter instanceof
StandardMetadataLimiterFactory standard)) {
+ throw new TikaConfigException("a request may not replace the
server's " +
+ "metadata limiter with " + metadataLimiter);
+ }
+ StandardMetadataLimiterFactory tightened =
standard.tightenedTo(operatorStandard);
+ if (tightened != standard) {
+ LOG.warn("request-supplied {} is looser than the server's {};
clamping",
+ standard, operatorStandard);
+ merged.set(MetadataWriteLimiterFactory.class, tightened);
+ }
+ }
+ try {
+ merged.set(PagesConfig.OperatorPages.class, new
PagesConfig.OperatorPages(
+ ParseContextConfig.getConfig(operatorContext,
PagesConfig.KEY,
+ PagesConfig.class, null)));
+ } catch (IOException e) {
+ throw new TikaConfigException("failed to read the server's pages
config", e);
+ }
+ UnpackConfig operatorUnpack = operatorContext.get(UnpackConfig.class);
+ UnpackConfig unpack = merged.get(UnpackConfig.class);
+ if (unpack != null && unpack != operatorUnpack) {
+ long ceiling = operatorUnpack != null ?
operatorUnpack.getMaxUnpackBytes()
+ : UnpackConfig.DEFAULT_MAX_UNPACK_BYTES;
+ long maxUnpackBytes = ceiling < 0 ? unpack.getMaxUnpackBytes()
+ : unpack.getMaxUnpackBytes() < 0 ? ceiling
+ : Math.min(unpack.getMaxUnpackBytes(), ceiling);
+ if (maxUnpackBytes != unpack.getMaxUnpackBytes()) {
+ LOG.warn("request-supplied maxUnpackBytes {} exceeds the
server's {}; clamping",
+ unpack.getMaxUnpackBytes(), ceiling);
+ // the request's own instance, deserialized for this task
+ unpack.setMaxUnpackBytes(maxUnpackBytes);
+ }
+ }
+ ContentHandlerFactory operatorFactory =
operatorContext.get(ContentHandlerFactory.class);
+ ContentHandlerFactory factory =
merged.get(ContentHandlerFactory.class);
+ if (factory instanceof WriteLimiter limiter && factory !=
operatorFactory) {
+ int ceiling = operatorOutput.getWriteLimit();
+ if (operatorFactory instanceof WriteLimiter operatorWriteLimiter) {
+ ceiling = tighter(operatorWriteLimiter.getWriteLimit(),
ceiling);
+ }
+ int writeLimit = tighter(limiter.getWriteLimit(), ceiling);
+ if (writeLimit != limiter.getWriteLimit()) {
+ if (!(factory instanceof BasicContentHandlerFactory basic)) {
+ throw new TikaConfigException("request-supplied write
limit " +
+ limiter.getWriteLimit() + " exceeds the server's "
+ ceiling);
+ }
+ LOG.warn("request-supplied write limit {} exceeds the server's
{}; clamping",
+ limiter.getWriteLimit(), ceiling);
+ // the request's own instance, deserialized for this task
+ basic.setWriteLimit(writeLimit);
+ }
+ }
+ }
+
+ /** The tighter of two limits where a negative value means unlimited. */
+ private static int tighter(int requested, int ceiling) {
+ if (ceiling < 0) {
+ return requested;
+ }
+ return requested < 0 ? ceiling : Math.min(requested, ceiling);
+ }
+
/**
* An {@link OutputStream} backed by a {@link ByteArrayOutputStream} that
aborts
* with an {@link IOException} the moment accumulated bytes would exceed
{@code limit}.
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/SharedServerResources.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/SharedServerResources.java
index 6f5b623694..314da6d960 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/SharedServerResources.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/server/SharedServerResources.java
@@ -175,7 +175,7 @@ public class SharedServerResources implements Closeable {
*
* @param requestContext the ParseContext from FetchEmitTuple
* @param presetName name of the preset to overlay at config-tier trust,
or null
- * @return a new ParseContext with defaults + preset + request overrides
+ * @return a new, resolved ParseContext with defaults + preset + clamped
request overrides
*/
public ParseContext createMergedParseContext(ParseContext requestContext,
String presetName)
throws TikaConfigException {
@@ -184,8 +184,8 @@ public class SharedServerResources implements Closeable {
// createMergedParseContext for why defaulting it would silently
disable embedded
// content extraction for every non-UNPACK parse mode.
PipesServer.mergePreset(presetRegistry, presetName, mergedContext);
- // Request-level values override config defaults and the preset
- mergedContext.copyFrom(requestContext);
+ PipesServer.overlayRequest(requestContext, mergedContext, pipesConfig,
+ defaultContentHandlerFactory);
PipesServer.seedCacheMemoryBudget(mergedContext);
return mergedContext;
}
diff --git
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/config/ConfigMergerTest.java
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/config/ConfigMergerTest.java
index 9d88fd17c3..98ed690781 100644
---
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/config/ConfigMergerTest.java
+++
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/config/ConfigMergerTest.java
@@ -34,6 +34,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.pipes.core.EmitStrategy;
public class ConfigMergerTest {
@@ -82,6 +83,30 @@ public class ConfigMergerTest {
Files.deleteIfExists(result.configPath());
}
+ /** Limits set in code land in the config, where they are the ceiling for
a request's. */
+ @Test
+ public void testEmbeddedAndWriteLimitsGoToParseContext() throws
IOException {
+ Path existing = tempDir.resolve("existing.json");
+ Files.writeString(existing,
+ "{\"parse-context\": {\"output-limits\": {\"maxXmlDepth\":
50}}}");
+ ConfigOverrides overrides = ConfigOverrides.builder()
+ .setEmbeddedLimits(new EmbeddedLimits(50, true, 1000, false))
+ .setWriteLimit(5000)
+ .build();
+
+ JsonNode parseContext = new ObjectMapper()
+ .readTree(ConfigMerger.merge(existing, overrides).json())
+ .get("parse-context");
+
+ JsonNode embedded = parseContext.get("embedded-limits");
+ assertEquals(50, embedded.get("maxDepth").asInt());
+ assertTrue(embedded.get("throwOnMaxDepth").asBoolean());
+ assertEquals(1000, embedded.get("maxCount").asInt());
+ assertEquals(5000,
parseContext.get("output-limits").get("writeLimit").asInt());
+ // merged into the user's block, not replacing it
+ assertEquals(50,
parseContext.get("output-limits").get("maxXmlDepth").asInt());
+ }
+
/** TIKA-4834: the config docs permit comments; the merge step must not
reject them. */
@Test
public void testMergeWithCommentedConfig() throws IOException {
diff --git
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/PresetMergeTest.java
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/PresetMergeTest.java
index 6be2dbf8fe..dcd451309e 100644
---
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/PresetMergeTest.java
+++
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/PresetMergeTest.java
@@ -25,10 +25,13 @@ import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.TimeoutLimits;
import org.apache.tika.config.loader.PresetRegistry;
import org.apache.tika.config.loader.TikaJsonConfig;
import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.pages.PagesConfig;
+import org.apache.tika.pipes.core.PipesConfig;
import org.apache.tika.serialization.ParseContextUtils;
/**
@@ -81,6 +84,42 @@ public class PresetMergeTest {
assertEquals(60_000,
TimeoutLimits.get(merged).getTotalTaskTimeoutMillis());
}
+ @Test
+ public void testPresetLimitsAreTheRequestCeiling() throws Exception {
+ PresetRegistry registry = registry("""
+ {"presets": {"deep": {"embedded-limits": {"maxDepth": 100}}}}
+ """);
+ ParseContext merged = new ParseContext();
+ PipesServer.mergePreset(registry, "deep", merged);
+ ParseContext requestContext = new ParseContext();
+ requestContext.setJsonConfig("embedded-limits", "{\"maxDepth\":
1000}");
+
+ PipesServer.overlayRequest(requestContext, merged, new PipesConfig(),
null);
+ // looser than the preset: clamped to it, not to the class default of
20
+ assertEquals(100, EmbeddedLimits.get(merged).getMaxDepth());
+ }
+
+ @Test
+ public void testRequestPagesMayNotLoosenOperatorRender() throws Exception {
+ ParseContext merged = new ParseContext();
+ merged.setJsonConfig("pages", "{\"render\": {\"dpi\": 150}}", true);
+ ParseContext requestContext = new ParseContext();
+ // replaces the operator's block: without the bound, dpi would fall
back to 300
+ requestContext.setJsonConfig("pages", "{\"text\": \"AUTO\"}", false);
+
+ PipesServer.overlayRequest(requestContext, merged, new PipesConfig(),
null);
+ assertEquals(150, PagesConfig.resolve(merged, "pdf-parser", null, null)
+ .getRender().getDpi());
+
+ ParseContext higher = new ParseContext();
+ higher.setJsonConfig("pages", "{\"render\": {\"dpi\": 1200}}", false);
+ ParseContext merged2 = new ParseContext();
+ merged2.setJsonConfig("pages", "{\"render\": {\"dpi\": 150}}", true);
+ PipesServer.overlayRequest(higher, merged2, new PipesConfig(), null);
+ assertEquals(150, PagesConfig.resolve(merged2, "pdf-parser", null,
null)
+ .getRender().getDpi());
+ }
+
@Test
public void testUnknownPresetIsTaskLevelError() throws Exception {
PresetRegistry registry = registry("{}");
diff --git
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/ServerProtocolIOTest.java
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/ServerProtocolIOTest.java
index 41458feffa..59cf56fb06 100644
---
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/ServerProtocolIOTest.java
+++
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/server/ServerProtocolIOTest.java
@@ -19,6 +19,7 @@ package org.apache.tika.pipes.core.server;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.DataInputStream;
@@ -30,20 +31,30 @@ import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.Test;
+import org.apache.tika.config.EmbeddedLimits;
import org.apache.tika.config.ExceptionReporting;
+import org.apache.tika.config.OutputLimits;
import org.apache.tika.config.TimeoutLimits;
+import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.metadata.writelimiter.MetadataWriteLimiterFactory;
+import org.apache.tika.metadata.writelimiter.StandardMetadataLimiterFactory;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.pipes.api.PipesResult;
import org.apache.tika.pipes.core.emitter.EmitDataImpl;
+import org.apache.tika.pipes.core.extractor.UnpackConfig;
import org.apache.tika.pipes.core.protocol.PipesMessage;
import org.apache.tika.pipes.core.protocol.PipesMessageType;
import org.apache.tika.pipes.core.serialization.JsonPipesIpc;
+import org.apache.tika.sax.BasicContentHandlerFactory;
+import org.apache.tika.sax.ContentHandlerFactory;
+import org.apache.tika.serialization.ParseContextUtils;
/**
* Unit tests for payload-size protection in {@link
ServerProtocolIO#writeFinished}
- * and timeout-limit clamping in {@link
ServerProtocolIO#clampRequestTimeoutLimits}.
+ * and request-limit clamping in {@link
ServerProtocolIO#clampRequestTimeoutLimits} and
+ * {@link ServerProtocolIO#clampRequestLimits}.
*/
class ServerProtocolIOTest {
@@ -88,6 +99,141 @@ class ServerProtocolIOTest {
assertEquals(7_200_000L,
merged.get(TimeoutLimits.class).getTotalTaskTimeoutMillis());
}
+ // ---- output/embedded/write limit clamping tests ----
+
+ private static ParseContext overlay(ParseContext operator, ParseContext
request) {
+ ParseContext merged = new ParseContext();
+ merged.copyFrom(operator);
+ merged.copyFrom(request);
+ return merged;
+ }
+
+ @Test
+ public void testRequestMayNotLoosenOperatorLimits() throws Exception {
+ ParseContext operator = new ParseContext();
+ operator.set(OutputLimits.class, new OutputLimits(1000, false, 50,
1_000_000, 100));
+ operator.set(EmbeddedLimits.class, new EmbeddedLimits(5, false, 100,
false));
+ ParseContext request = new ParseContext();
+ request.set(OutputLimits.class, new OutputLimits(-1, false, 10_000,
Long.MAX_VALUE, 100));
+ request.set(EmbeddedLimits.class, new EmbeddedLimits(-1, false, -1,
false));
+ ParseContext merged = overlay(operator, request);
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ OutputLimits output = merged.get(OutputLimits.class);
+ assertEquals(1000, output.getWriteLimit());
+ assertEquals(50, output.getMaxXmlDepth());
+ assertEquals(1_000_000, output.getZipBombThreshold());
+ assertEquals(5, merged.get(EmbeddedLimits.class).getMaxDepth());
+ assertEquals(100, merged.get(EmbeddedLimits.class).getMaxCount());
+ }
+
+ @Test
+ public void testRequestMayTightenOperatorLimits() throws Exception {
+ ParseContext request = new ParseContext();
+ OutputLimits tighter = new OutputLimits(10, true, 20, 1000, 10);
+ request.set(OutputLimits.class, tighter);
+ ParseContext operator = new ParseContext();
+ ParseContext merged = overlay(operator, request);
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ assertEquals(tighter, merged.get(OutputLimits.class));
+ }
+
+ @Test
+ public void testOperatorLimitsAreNotClamped() throws Exception {
+ ParseContext operator = new ParseContext();
+ // looser than the class defaults, which are the ceiling when nothing
is configured
+ OutputLimits loose = new OutputLimits(-1, false, 10_000,
Long.MAX_VALUE, Long.MAX_VALUE);
+ operator.set(OutputLimits.class, loose);
+ ParseContext merged = overlay(operator, new ParseContext());
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ assertEquals(10_000, merged.get(OutputLimits.class).getMaxXmlDepth());
+ }
+
+ @Test
+ public void testDefaultsAreTheCeilingWhenNothingIsConfigured() throws
Exception {
+ ParseContext request = new ParseContext();
+ request.setJsonConfig("output-limits", "{\"maxXmlDepth\": 100000}");
+ ParseContext operator = new ParseContext();
+ ParseContext merged = overlay(operator, request);
+ ParseContextUtils.resolveAll(merged,
+ getClass().getClassLoader());
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ assertEquals(new OutputLimits().getMaxXmlDepth(),
+ merged.get(OutputLimits.class).getMaxXmlDepth());
+ }
+
+ @Test
+ public void testRequestHandlerFactoryWriteLimitIsClamped() throws
Exception {
+ ParseContext operator = new ParseContext();
+ OutputLimits operatorLimits = new OutputLimits();
+ operatorLimits.setWriteLimit(1000);
+ operator.set(OutputLimits.class, operatorLimits);
+ ParseContext request = new ParseContext();
+ request.set(ContentHandlerFactory.class,
+ new
BasicContentHandlerFactory(BasicContentHandlerFactory.HANDLER_TYPE.TEXT, -1));
+ ParseContext merged = overlay(operator, request);
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ assertEquals(1000,
+ ((BasicContentHandlerFactory)
merged.get(ContentHandlerFactory.class)).getWriteLimit());
+ }
+
+ @Test
+ public void testRequestMetadataLimiterIsClamped() throws Exception {
+ ParseContext operator = new ParseContext();
+ StandardMetadataLimiterFactory operatorLimiter = new
StandardMetadataLimiterFactory();
+ operatorLimiter.setMaxTotalBytes(1000);
+ operator.set(MetadataWriteLimiterFactory.class, operatorLimiter);
+ ParseContext request = new ParseContext();
+ StandardMetadataLimiterFactory requestLimiter = new
StandardMetadataLimiterFactory();
+ requestLimiter.setMaxTotalBytes(Integer.MAX_VALUE);
+ requestLimiter.setMaxValuesPerField(2);
+ request.set(MetadataWriteLimiterFactory.class, requestLimiter);
+ ParseContext merged = overlay(operator, request);
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ StandardMetadataLimiterFactory clamped =
+ (StandardMetadataLimiterFactory)
merged.get(MetadataWriteLimiterFactory.class);
+ assertEquals(1000, clamped.getMaxTotalBytes());
+ assertEquals(2, clamped.getMaxValuesPerField());
+ }
+
+ @Test
+ public void testRequestMayNotReplaceMetadataLimiterWithAnotherKind() {
+ ParseContext operator = new ParseContext();
+ operator.set(MetadataWriteLimiterFactory.class, new
StandardMetadataLimiterFactory());
+ ParseContext request = new ParseContext();
+ request.set(MetadataWriteLimiterFactory.class, () -> null);
+ ParseContext merged = overlay(operator, request);
+
+ assertThrows(TikaConfigException.class,
+ () -> ServerProtocolIO.clampRequestLimits(operator, merged));
+ }
+
+ @Test
+ public void testRequestMaxUnpackBytesIsClamped() throws Exception {
+ ParseContext request = new ParseContext();
+ UnpackConfig unpack = new UnpackConfig();
+ unpack.setMaxUnpackBytes(-1);
+ request.set(UnpackConfig.class, unpack);
+ ParseContext operator = new ParseContext();
+ ParseContext merged = overlay(operator, request);
+
+ ServerProtocolIO.clampRequestLimits(operator, merged);
+
+ assertEquals(UnpackConfig.DEFAULT_MAX_UNPACK_BYTES,
+ merged.get(UnpackConfig.class).getMaxUnpackBytes());
+ }
+
// ---- payload guard tests ----
/**
diff --git
a/tika-pipes/tika-pipes-fork-parser/src/main/java/org/apache/tika/pipes/fork/PipesForkParser.java
b/tika-pipes/tika-pipes-fork-parser/src/main/java/org/apache/tika/pipes/fork/PipesForkParser.java
index 0454973652..26f94cf821 100644
---
a/tika-pipes/tika-pipes-fork-parser/src/main/java/org/apache/tika/pipes/fork/PipesForkParser.java
+++
b/tika-pipes/tika-pipes-fork-parser/src/main/java/org/apache/tika/pipes/fork/PipesForkParser.java
@@ -52,6 +52,7 @@ import org.apache.tika.pipes.core.fetcher.BytesFetcher;
import org.apache.tika.pipes.core.fetcher.InlineBytes;
import org.apache.tika.pipes.core.fetcher.PayloadRouter;
import org.apache.tika.sax.ContentHandlerFactory;
+import org.apache.tika.sax.WriteLimiter;
/**
* A ForkParser implementation backed by {@link PipesParser}.
@@ -463,6 +464,11 @@ public class PipesForkParser implements Closeable {
if (config.getTimeoutLimits() != null) {
builder.setTimeoutLimits(config.getTimeoutLimits());
}
+ // The fork clamps per-request limits to its config's; these are ours,
so seed them there.
+ builder.setEmbeddedLimits(config.getEmbeddedLimits());
+ if (config.getContentHandlerFactory() instanceof WriteLimiter limiter)
{
+ builder.setWriteLimit(limiter.getWriteLimit());
+ }
ConfigOverrides overrides = builder.build();
diff --git
a/tika-server/tika-server-core/src/test/java/org/apache/tika/server/core/PipesRequestLimitsTest.java
b/tika-server/tika-server-core/src/test/java/org/apache/tika/server/core/PipesRequestLimitsTest.java
new file mode 100644
index 0000000000..8a9690eac3
--- /dev/null
+++
b/tika-server/tika-server-core/src/test/java/org/apache/tika/server/core/PipesRequestLimitsTest.java
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.server.core;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Reader;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import jakarta.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
+import org.apache.cxf.jaxrs.client.WebClient;
+import org.apache.cxf.jaxrs.lifecycle.ResourceProvider;
+import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+
+import org.apache.tika.config.loader.TikaJsonConfig;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.pipes.core.EmitStrategy;
+import org.apache.tika.pipes.core.EmitStrategyConfig;
+import org.apache.tika.pipes.core.PipesConfig;
+import org.apache.tika.pipes.core.PipesParser;
+import org.apache.tika.serialization.JsonMetadataList;
+import org.apache.tika.server.core.resource.PipesResource;
+import org.apache.tika.server.core.writer.JSONObjWriter;
+
+/**
+ * A /pipes tuple's parse-context may tighten the server's limits but not
loosen them: with the
+ * server's writeLimit at 5, a tuple asking for an unlimited handler still
hits the limit.
+ */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class PipesRequestLimitsTest extends CXFTestBase {
+
+ private static final String HELLO_WORLD = "hello_world.xml";
+
+ private Path tmpDir;
+ private Path tmpOutputFile;
+ private Path tikaConfigPath;
+ private PipesParser pipesParser;
+
+ @Override
+ @BeforeAll
+ public void setUp() throws Exception {
+ tmpDir = Files.createTempDirectory("tika-pipes-request-limits-");
+ Path inputDir = tmpDir.resolve("input");
+ Path outputDir = tmpDir.resolve("output");
+ tmpOutputFile = outputDir.resolve(HELLO_WORLD + ".json");
+ Files.createDirectories(inputDir);
+ Files.createDirectories(outputDir);
+
Files.copy(PipesRequestLimitsTest.class.getResourceAsStream("/test-documents/mock/"
+ HELLO_WORLD),
+ inputDir.resolve(HELLO_WORLD));
+ tikaConfigPath = Files.createTempFile(tmpDir, "tika-pipes-config-",
".json");
+ CXFTestBase.createPluginsConfig(tikaConfigPath, inputDir, outputDir,
null, 10000L);
+
+ ObjectMapper mapper = new ObjectMapper();
+ ObjectNode root = (ObjectNode)
mapper.readTree(tikaConfigPath.toFile());
+ ((ObjectNode)
root.get("parse-context")).putObject("output-limits").put("writeLimit", 5);
+ mapper.writeValue(tikaConfigPath.toFile(), root);
+ super.setUp();
+ }
+
+ @Override
+ @AfterAll
+ public void tearDown() throws Exception {
+ if (pipesParser != null) {
+ pipesParser.close();
+ pipesParser = null;
+ }
+ super.tearDown();
+ if (tmpDir != null) {
+ FileUtils.deleteDirectory(tmpDir.toFile());
+ }
+ }
+
+ @Override
+ protected void setUpResources(JAXRSServerFactoryBean sf) {
+ List<ResourceProvider> providers = new ArrayList<>();
+ try {
+ TikaJsonConfig tikaJsonConfig =
TikaJsonConfig.load(tikaConfigPath);
+ PipesConfig pipesConfig = PipesConfig.load(tikaJsonConfig);
+ pipesConfig.setEmitStrategy(new
EmitStrategyConfig(EmitStrategy.EMIT_ALL));
+ pipesParser = PipesParser.load(tikaJsonConfig, pipesConfig);
+ providers.add(new SingletonResourceProvider(new
PipesResource(pipesParser, pipesConfig)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ sf.setResourceProviders(providers);
+ }
+
+ @Override
+ protected void setUpProviders(JAXRSServerFactoryBean sf) {
+ List<Object> providers = new ArrayList<>();
+ providers.add(new TikaServerParseExceptionMapper());
+ providers.add(new BadRequestExceptionMapper());
+ providers.add(new JSONObjWriter());
+ sf.setProviders(providers);
+ }
+
+ @Override
+ protected InputStream getTikaConfigInputStream() throws IOException {
+ return new ByteArrayInputStream(Files.readAllBytes(tikaConfigPath));
+ }
+
+ @Test
+ public void testRequestMayNotRaiseWriteLimit() throws Exception {
+ String tuple = "{\"id\":\"" + HELLO_WORLD + "\"," +
+ "\"fetcher\":\"" + FETCHER_ID + "\"," +
+ "\"fetchKey\":\"" + HELLO_WORLD + "\"," +
+ "\"emitter\":\"" + EMITTER_JSON_ID + "\"," +
+ "\"emitKey\":\"\"," +
+ "\"parse-context\":{\"basic-content-handler-factory\":" +
+
"{\"type\":\"TEXT\",\"writeLimit\":-1,\"throwOnWriteLimitReached\":false}}}";
+ Response response = WebClient
+ .create(endPoint + "/pipes")
+ .accept("application/json")
+ .post(tuple);
+ assertEquals(200, response.getStatus());
+
+ List<Metadata> metadataList;
+ try (Reader reader = Files.newBufferedReader(tmpOutputFile)) {
+ metadataList = JsonMetadataList.fromJson(reader);
+ }
+ Metadata metadata = metadataList.get(0);
+ assertEquals("true",
metadata.get(TikaCoreProperties.WRITE_LIMIT_REACHED), metadata.toString());
+ String content = metadata.get(TikaCoreProperties.TIKA_CONTENT);
+ assertTrue(content == null || !content.contains("hello world"),
content);
+ }
+}
