[
https://issues.apache.org/jira/browse/TIKA-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115714#comment-18115714
]
ASF GitHub Bot commented on TIKA-4889:
--------------------------------------
Copilot commented on code in PR #3174:
URL: https://github.com/apache/tika/pull/3174#discussion_r4018816598
##########
tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java:
##########
@@ -185,9 +191,10 @@ public void parse(TikaInputStream tis, ContentHandler
handler,
@Override
public void initialize() throws TikaConfigException {
- LOG.info("openai-image-embedding-parser runs one request per image;
the \"engines\" + "
- + "\"inference\" shape (openai-embedding-engine, input IMAGES,
task embed) batches "
- + "a document's images into one request");
+ LOG.warn("openai-image-embedding-parser is deprecated since 4.1.0 and
will be removed in "
+ + "4.2.0: configure the endpoint as an openai-embedding-engine
under \"engines\" "
+ + "and bind it with an IMAGES \"inference\" binding (task
embed), which sends a "
+ + "document's images in requests of the engine's batch size");
Review Comment:
`ParserLoader.finish()` now emits `warnEngineUnderParsers` for every
`ContentEnricher` under `parsers`, but this `initialize()` method also emits a
deprecation WARN for the same parser. Configuring
`openai-image-embedding-parser` under `parsers` therefore produces two startup
WARNs, despite the documented one-WARN-per-entry behavior; coordinate these
paths so direct `text-recognizers` use remains covered without duplicating the
legacy-shape warning.
##########
tika-serialization/src/main/java/org/apache/tika/config/loader/ParserLoader.java:
##########
@@ -205,66 +205,71 @@ private static String names(List<Parser> parsers) {
return names.toString();
}
- /**
- * Enrichers named directly under {@code "parsers"} that the composite
never dispatches
- * to: every type they advertise is claimed by another parser there, or
they advertise
- * none (engine unavailable, or told to skip). Both shapes look configured
and do
- * nothing as parsers.
- */
- static List<Parser> undispatchedEnrichers(Parser root) {
- List<Parser> inert = new ArrayList<>();
+ /** Engines (enrichers) named directly under {@code "parsers"}: the
deprecated 4.0 shape. */
+ static List<Parser> enginesUnderParsers(Parser root) {
+ List<Parser> engines = new ArrayList<>();
if (!(root instanceof CompositeParser composite) || root instanceof
DefaultParser) {
- return inert;
+ return engines;
+ }
+ for (Parser member : composite.getAllComponentParsers()) {
+ if (ContentEnrichers.isEnricher(member)) {
+ engines.add(member);
+ }
+ }
+ return engines;
+ }
+
+ /** The types the composite dispatches to this member as their parser. */
+ static Set<MediaType> parsedTypes(Parser root, Parser member) {
+ Set<MediaType> parsed = new TreeSet<>();
+ if (!(root instanceof CompositeParser composite)) {
+ return parsed;
}
ParseContext empty = new ParseContext();
Map<MediaType, Parser> dispatch = composite.getParsers(empty);
MediaTypeRegistry registry = composite.getMediaTypeRegistry();
- for (Parser member : composite.getAllComponentParsers()) {
- if (!ContentEnrichers.isEnricher(member)) {
- continue;
- }
- boolean dispatched = false;
- for (MediaType type : member.getSupportedTypes(empty)) {
- if (dispatch.get(registry.normalize(type)) == member) {
- dispatched = true;
- break;
- }
- }
- if (!dispatched) {
- inert.add(member);
+ for (MediaType type : member.getSupportedTypes(empty)) {
+ if (dispatch.get(registry.normalize(type)) == member) {
+ parsed.add(type);
}
}
- return inert;
+ return parsed;
}
- // the 4.0 shape still works, so it is INFO; an entry that never runs at
all is a WARN
- private static void logUndispatched(Parser inert, CompositeContentEnricher
enrichers,
- boolean listConfigured) {
- String name = ParserUtils.getParserClassname(inert);
- Set<MediaType> advertised = inert.getSupportedTypes(new
ParseContext());
+ /** One WARN per engine under "parsers": the deprecation, then what the
entry does today. */
+ private static void warnEngineUnderParsers(Parser engine, Parser root,
+ CompositeContentEnricher
enrichers,
+ boolean listConfigured) {
+ String name = ParserUtils.getParserClassname(engine);
+ String lead = name + " is named under \"parsers\", which is deprecated
for engines since "
+ + "4.1.0 and unsupported in 4.2.0: configure it under
\"engines\" and name it in "
+ + "\"text-recognizers\". ";
Review Comment:
This guidance is not correct for every `ContentEnricher` found under
`parsers`: `OpenAIImageEmbeddingParser` is an annotator, not a
`TextRecognizer`, and its replacement is an `openai-embedding-engine` with an
`IMAGES` `inference` binding. The current WARN tells users to put it in
`text-recognizers` (and the later branches call it a text recognizer), so make
the recommendation depend on the engine/binding or state both supported
migration paths.
##########
docs/modules/ROOT/pages/configuration/inference.adoc:
##########
@@ -231,9 +231,10 @@ request" (see
xref:configuration/inference-recipes.adoc[Recipes]). Names only, s
xref:using-tika/server/index.adoc[preset]. A name not in the configured list
fails the request.
Engines cannot be defined or changed per request.
-== From `openai-image-embedding-parser`
+== From `openai-image-embedding-parser` (deprecated)
-The 4.0 `openai-image-embedding-parser` still works as a text-recognizers
entry and embeds one
-image per request. The same endpoint as an engine plus a binding on `IMAGES`
embeds a
-document's images in one request and puts the vectors where they belong; move
the `baseUrl`,
-`model` and `apiKey` to the engine and drop the parser entry.
+The 4.0 `openai-image-embedding-parser` is deprecated since 4.1.0 and removed
in 4.2.0. It
+still works as a text-recognizers entry, with a WARN at startup, and embeds
one image per
+request. The same endpoint as an engine plus a binding on `IMAGES` embeds a
document's images
+in one request and puts the vectors where they belong; move the `baseUrl`,
`model` and
+`apiKey` to the engine and drop the parser entry.
Review Comment:
An `IMAGES` binding batches units up to the engine's `maxBatchSize`, so a
document with more images than that produces multiple requests. Saying the
replacement embeds all images "in one request" is inaccurate and conflicts with
the engine's batching behavior; describe batched requests instead.
> Simplify inference configuration
> --------------------------------
>
> Key: TIKA-4889
> URL: https://issues.apache.org/jira/browse/TIKA-4889
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
