This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 18b948be33 TIKA-4889: a request may not supply the embedding filters;
they are deprecated (#3171)
18b948be33 is described below
commit 18b948be339940f305948b7ad52eb23e34edcb9c
Author: Tim Allison <[email protected]>
AuthorDate: Tue Sep 15 07:38:26 2026 -0400
TIKA-4889: a request may not supply the embedding filters; they are
deprecated (#3171)
---
CHANGES.txt | 8 ++++++++
.../ROOT/pages/configuration/metadata-filters.adoc | 11 +++++++----
.../pages/migration-to-4x/migrating-to-4x.adoc | 10 ++++++++++
.../tika/inference/AbstractEmbeddingFilter.java | 14 ++++++++++++++
.../apache/tika/inference/JinaEmbeddingFilter.java | 2 ++
.../tika/inference/OpenAIEmbeddingFilter.java | 2 ++
.../tika/serialization/ComponentNameResolver.java | 21 +++++++++++++++++++++
.../serdes/ParseContextDeserializer.java | 3 ++-
.../WireRestrictedParseContextTest.java | 10 ++++++++++
.../tika/server/standard/TikaResourceTest.java | 22 ++++++++++++++++++++++
10 files changed, 98 insertions(+), 5 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 5f0746ff9c..31bfb2e562 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,13 @@
Release 4.1.0 - unreleased
+ * A request can no longer supply openai-embedding-filter or
+ jina-embedding-filter in a per-request metadata-filters list: 4.0.x
+ built the filter from the request, endpoint and key included, and
+ posted the document's text there. The flat per-request skipEmbedding
+ override is unchanged. The two filters are deprecated; configure the
+ endpoint under "engines" and bind it with a TEXT inference binding.
+ Removal in 4.2.0 (TIKA-4889).
+
* PDF: "pdf-parser": {"text": ...} says where a page's text comes from:
EXTRACT, AUTO (default), EXTRACT_AND_OCR, OCR, or the new NONE, which
writes no text from any source while pages are still rendered for
diff --git a/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
b/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
index dc19f1fd8b..e1770b4c47 100644
--- a/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
+++ b/docs/modules/ROOT/pages/configuration/metadata-filters.adoc
@@ -115,10 +115,13 @@ Adding the module to the classpath is enough to make the
name resolvable.
|`openai-embedding-filter`, `jina-embedding-filter`
|`tika-inference`
-|Chunks `tk:content` and calls an embeddings endpoint, putting the vectors on
`tk:chunks`, one
-document per request. Since 4.1.0 a `TEXT`
xref:configuration/inference.adoc#text[inference
-binding] does the same for the whole tree in batched requests; it runs before
this chain and
-is not part of it. See xref:migration-to-4x/chunk-strategies.adoc[Chunk
Strategies].
+|*Deprecated in 4.1.0, removed in 4.2.0.* Chunks `tk:content` and calls an
embeddings endpoint,
+putting the vectors on `tk:chunks`, one document per request. Configure the
endpoint under
+`engines` and bind it with a `TEXT`
xref:configuration/inference.adoc#text[inference binding]
+instead: the whole tree in batched requests, before this chain. A request may
not supply these
+filters in a `metadata-filters` list (they would post text to a caller-chosen
endpoint); the
+flat `skipEmbedding` override below still works. See
+xref:migration-to-4x/chunk-strategies.adoc[Chunk Strategies].
|`charsoup-metadata-filter`, `optimaize-metadata-filter`,
`open-nlp-metadata-filter`
|`tika-langdetect-charsoup`, `-optimaize`, `-opennlp`
diff --git a/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
b/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
index 6799a2579b..f74222b56a 100644
--- a/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
+++ b/docs/modules/ROOT/pages/migration-to-4x/migrating-to-4x.adoc
@@ -256,6 +256,16 @@ What changes for you:
* VLM parsers gain `textRecognizer` (default `true`); set it to `false` for a
captioning or tagging
prompt so the PDF parser's `AUTO` OCR never replaces extracted text with the
VLM's output.
+=== Embedding filters deprecated (4.1.0+)
+
+`openai-embedding-filter` and `jina-embedding-filter` are deprecated and will
be removed in
+4.2.0. Configure the endpoint once under `engines` and bind it with a `TEXT`
+xref:configuration/inference.adoc#text[inference binding]; it embeds the whole
document tree in
+batched requests. Until then they keep working from the server-side config. A
request can no
+longer name them in a per-request `metadata-filters` list (4.0.x accepted a
caller-supplied
+`baseUrl` there); the flat per-request `{"openai-embedding-filter":
{"skipEmbedding": true}}`
+form is unchanged.
+
=== PDF text policy: `ocr.strategy` becomes `text` (4.1.0+)
The PDF parser's `"ocr": {"strategy": ...}` is now the parser's own `"text"`,
since it decides
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/AbstractEmbeddingFilter.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/AbstractEmbeddingFilter.java
index b134dd0da8..a40448189b 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/AbstractEmbeddingFilter.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/AbstractEmbeddingFilter.java
@@ -55,9 +55,15 @@ import org.apache.tika.parser.ParseContext;
* Thread safety: instances are safe for concurrent {@link #filter} calls once
* fully constructed. Setters must not be called concurrently with
* {@link #filter}.
+ *
+ * @deprecated since 4.1.0, removed in 4.2.0: configure the endpoint once under
+ * {@code "engines"} and bind it with a {@code TEXT} {@code "inference"}
binding, which embeds
+ * the whole document tree in batched requests.
*/
+@Deprecated
public abstract class AbstractEmbeddingFilter extends MetadataFilter
implements SelfConfiguring {
+
private static final long serialVersionUID = 1L;
private static final String REQUIRED_HANDLER_TYPE = "MARKDOWN";
@@ -68,10 +74,18 @@ public abstract class AbstractEmbeddingFilter extends
MetadataFilter implements
private InferenceConfig defaultConfig = new InferenceConfig();
protected AbstractEmbeddingFilter() {
+ warnDeprecated();
}
protected AbstractEmbeddingFilter(InferenceConfig config) {
this.defaultConfig = config;
+ warnDeprecated();
+ }
+
+ private void warnDeprecated() {
+ LOG.warn("{} is deprecated since 4.1.0 and will be removed in 4.2.0:
configure the "
+ + "endpoint under \"engines\" and bind it with a TEXT
\"inference\" binding",
+ getClass().getSimpleName());
}
/**
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/JinaEmbeddingFilter.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/JinaEmbeddingFilter.java
index 2985c77498..e684ed16fb 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/JinaEmbeddingFilter.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/JinaEmbeddingFilter.java
@@ -39,7 +39,9 @@ import org.apache.tika.utils.StringUtils;
* Configuration key: {@code "jina-embedding-filter"}
*
* @since Apache Tika 4.0
+ * @deprecated since 4.1.0, removed in 4.2.0; see {@link
AbstractEmbeddingFilter}.
*/
+@Deprecated
@TikaComponent(name = "jina-embedding-filter", spi = false)
public class JinaEmbeddingFilter extends OpenAIEmbeddingFilter {
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingFilter.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingFilter.java
index 54afc1712b..c03bb28ac5 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingFilter.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingFilter.java
@@ -45,7 +45,9 @@ import org.apache.tika.utils.StringUtils;
* Configuration key: {@code "openai-embedding-filter"}
*
* @since Apache Tika 4.0
+ * @deprecated since 4.1.0, removed in 4.2.0; see {@link
AbstractEmbeddingFilter}.
*/
+@Deprecated
@TikaComponent(name = "openai-embedding-filter", spi = false)
public class OpenAIEmbeddingFilter extends AbstractEmbeddingFilter {
diff --git
a/tika-serialization/src/main/java/org/apache/tika/serialization/ComponentNameResolver.java
b/tika-serialization/src/main/java/org/apache/tika/serialization/ComponentNameResolver.java
index 91a1872d7b..c73d67c293 100644
---
a/tika-serialization/src/main/java/org/apache/tika/serialization/ComponentNameResolver.java
+++
b/tika-serialization/src/main/java/org/apache/tika/serialization/ComponentNameResolver.java
@@ -96,6 +96,14 @@ public final class ComponentNameResolver {
* either: operator policy a caller must not be able to relax.
*/
private static final Set<Class<?>> WIRE_BLOCKED_CONFIG_KEYS = new
HashSet<>();
+ /**
+ * Implementations of a wire-allowed type that break its promise: a wire
request may name
+ * them nowhere it could instantiate one. The embedding filters POST
document text to a
+ * configured endpoint, so a request must not supply the endpoint. Their
flat per-request
+ * config ({@code skipEmbedding}) is unaffected: it tunes the loaded
instance and is
+ * guarded by the filter's RuntimeConfig.
+ */
+ private static final Set<String> WIRE_BLOCKED_COMPONENT_NAMES = new
HashSet<>();
static {
// Allowed: bounded to this request's metadata/output; no exec or IO.
@@ -115,6 +123,9 @@ public final class ComponentNameResolver {
WIRE_BLOCKED_CONTEXT_KEYS.add(EmbeddedDocumentExtractor.class);
WIRE_BLOCKED_CONFIG_KEYS.add(ExceptionReporting.class);
+
+ WIRE_BLOCKED_COMPONENT_NAMES.add("openai-embedding-filter");
+ WIRE_BLOCKED_COMPONENT_NAMES.add("jina-embedding-filter");
}
private static final Map<String, ComponentRegistry> REGISTRIES = new
ConcurrentHashMap<>();
@@ -311,6 +322,16 @@ public final class ComponentNameResolver {
return Collections.unmodifiableSet(WIRE_INSTANTIABLE_CONTEXT_KEYS);
}
+ /** Component names a wire ParseContext may not instantiate although their
type is allowed. */
+ public static Set<String> getWireBlockedComponentNames() {
+ return Collections.unmodifiableSet(WIRE_BLOCKED_COMPONENT_NAMES);
+ }
+
+ /** True if a wire ParseContext must not instantiate this component by
name. */
+ public static boolean isWireBlockedName(String friendlyName) {
+ return WIRE_BLOCKED_COMPONENT_NAMES.contains(friendlyName);
+ }
+
/** True if a wire ParseContext may bind this context-key type
(default-deny). */
public static boolean isWireInstantiable(Class<?> contextKey) {
return WIRE_INSTANTIABLE_CONTEXT_KEYS.contains(contextKey);
diff --git
a/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextDeserializer.java
b/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextDeserializer.java
index cd3da8d8c7..9b79b412ce 100644
---
a/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextDeserializer.java
+++
b/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextDeserializer.java
@@ -221,7 +221,8 @@ public class ParseContextDeserializer extends
JsonDeserializer<ParseContext> {
return;
}
Class<?> contextKey =
ComponentNameResolver.determineContextKey(infoOpt.get());
- if (ComponentNameResolver.isWireBlocked(contextKey)) {
+ if (ComponentNameResolver.isWireBlocked(contextKey)
+ || ComponentNameResolver.isWireBlockedName(name)) {
throw new IOException(wireBlockedMessage(name, contextKey));
}
}
diff --git
a/tika-serialization/src/test/java/org/apache/tika/serialization/WireRestrictedParseContextTest.java
b/tika-serialization/src/test/java/org/apache/tika/serialization/WireRestrictedParseContextTest.java
index 8282c82d4c..8d5a41864d 100644
---
a/tika-serialization/src/test/java/org/apache/tika/serialization/WireRestrictedParseContextTest.java
+++
b/tika-serialization/src/test/java/org/apache/tika/serialization/WireRestrictedParseContextTest.java
@@ -71,6 +71,16 @@ public class WireRestrictedParseContextTest {
}
}
+ /** Names blocked although their type is allowed: the embedding filters,
which do network IO. */
+ @Test
+ public void embeddingFiltersAreBlockedByName() {
+ Set<String> names =
ComponentNameResolver.getWireBlockedComponentNames();
+ assertTrue(names.contains("openai-embedding-filter"));
+ assertTrue(names.contains("jina-embedding-filter"));
+
assertTrue(ComponentNameResolver.isWireBlockedName("openai-embedding-filter"));
+
assertFalse(ComponentNameResolver.isWireBlockedName("include-field-metadata-filter"));
+ }
+
@Test
public void restrictedRejectsTypedParserInjection() {
// A blocked component nested under a wrapper key must be refused
regardless of the wrapper.
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaResourceTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaResourceTest.java
index 3b510f5635..3ad98d20e2 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaResourceTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaResourceTest.java
@@ -17,6 +17,7 @@
package org.apache.tika.server.standard;
import static org.apache.cxf.helpers.HttpHeaderHelper.CONTENT_ENCODING;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -448,6 +449,27 @@ public class TikaResourceTest extends CXFTestBase {
"expected wire-blocked rejection, got: " + rootMessage(e));
}
+ /**
+ * MetadataFilter is wire-instantiable on the promise that no
implementation does IO; the
+ * embedding filters POST document text to their baseUrl, so they are
blocked by name.
+ */
+ @Test
+ public void testConfigRejectsEmbeddingFilterFromTheWire() {
+ TikaObjectMapperFactory.getMapper();
+ for (String name : new String[]{"openai-embedding-filter",
"jina-embedding-filter"}) {
+ String configJson = "{\"metadata-filters\":[{\"" + name
+ + "\":{\"baseUrl\":\"http://127.0.0.1:1/\"}}]}";;
+ Exception e = assertThrows(Exception.class,
+ () -> TikaResource.mergeParseContextFromConfig(configJson,
new ParseContext()),
+ name);
+ assertTrue(rootMessage(e).contains("may not be supplied via a
request parseContext"),
+ name + ": expected wire-blocked rejection, got: " +
rootMessage(e));
+ }
+ // the flat per-request config tunes the loaded filter and stays
allowed
+ assertDoesNotThrow(() -> TikaResource.mergeParseContextFromConfig(
+ "{\"openai-embedding-filter\":{\"skipEmbedding\":true}}", new
ParseContext()));
+ }
+
private static String rootMessage(Throwable t) {
Throwable r = t;
while (r.getCause() != null && r.getCause() != r) {
