This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new fa7ea09968 TIKA-4888: a picture's vector lands on the document it
appears in (#3152)
fa7ea09968 is described below
commit fa7ea0996857fd785d782d0cb887c04b996dddb9
Author: Tim Allison <[email protected]>
AuthorDate: Thu Sep 10 19:21:40 2026 -0400
TIKA-4888: a picture's vector lands on the document it appears in (#3152)
---
CHANGES.txt | 8 +
.../pages/migration-to-4x/chunk-strategies.adoc | 47 +++++-
tika-app/pom.xml | 7 +
.../org/apache/tika/cli/TikaCLIChunkLiftTest.java | 182 +++++++++++++++++++++
.../org/apache/tika/extractor/ParentMetadata.java | 39 +++++
.../apache/tika/parser/RecursiveParserWrapper.java | 15 +-
.../org/apache/tika/inference/ChunkSerializer.java | 22 ++-
.../org/apache/tika/inference/ChunkTarget.java | 85 ++++++++++
.../tika/inference/ImageEmbeddingConfig.java | 15 ++
.../tika/inference/OpenAIImageEmbeddingParser.java | 12 +-
.../tika/inference/locator/EmbeddedLocator.java | 45 +++++
.../apache/tika/inference/locator/Locators.java | 22 ++-
.../apache/tika/inference/ChunkSerializerTest.java | 19 +++
.../org/apache/tika/inference/ChunkTargetTest.java | 98 +++++++++++
.../inference/OpenAIImageEmbeddingParserTest.java | 36 ++++
15 files changed, 644 insertions(+), 8 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 8c6512827a..7d5a977ed0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,13 @@
Release 4.1.0 - unreleased
+ * The image embedder writes a picture's vector onto the document the
+ picture appears in (INLINE and RENDERING children) with an "embedded"
+ locator naming the picture; attachments keep their own, and
+ liftToParent: false keeps every picture's vector on the picture.
+ RecursiveParserWrapper exposes the parent's Metadata to a child's parse
+ as ParentMetadata, and ChunkTarget (tika-inference) resolves where a
+ child's chunks land (TIKA-4888).
+
* A PDF page rendered as an embedded document (imageStrategy RENDER_PAGES_*)
is enriched once, by the PDF parser: the text recognizer where the OCR
strategy says so, every other text-recognizers entry on every page,
results on
diff --git a/docs/modules/ROOT/pages/migration-to-4x/chunk-strategies.adoc
b/docs/modules/ROOT/pages/migration-to-4x/chunk-strategies.adoc
index 1acbc366db..c4e7698c19 100644
--- a/docs/modules/ROOT/pages/migration-to-4x/chunk-strategies.adoc
+++ b/docs/modules/ROOT/pages/migration-to-4x/chunk-strategies.adoc
@@ -48,7 +48,9 @@ CLIP-like image embeddings endpoint, e.g. Jina CLIP v2.
Carries a `vector` and a
|===
`SpatialLocator` (bounding box) and `TemporalLocator` (millisecond range) are
already defined
-and round-tripped by `ChunkSerializer`; no component produces them yet.
+and round-tripped by `ChunkSerializer`; no component produces them yet.
`EmbeddedLocator`
+(`id_path`, `name`) marks a chunk that was lifted from an embedded document
into its parent;
+see <<lifting>>.
A single chunk looks like:
@@ -109,6 +111,49 @@ Why this shape:
The trade-offs are the `nested` kNN caveats below, and large documents for
files with many
pages.
+[#lifting]
+== Where a Chunk Lands: the Parent
+
+An embedded document that is *part of* its parent, a picture in the body of a
docx or an
+email, should not be a search hit of its own: its vector belongs to the
document it appears in.
+The image embedder therefore writes a picture's chunk onto the parent, not
onto the picture,
+so a docx carries one vector per picture in its body and an email the vectors
of its inline
+pictures. Nothing to configure:
+
+[source,json]
+----
+{
+ "text-recognizers": [
+ { "openai-image-embedding-parser": { "baseUrl": "http://clip-server:8000";,
+ "model": "jinaai/jina-clip-v2" } }
+ ],
+ "metadata-filters": [
+ { "openai-embedding-filter": { "baseUrl": "http://embed-server:8000";,
+ "model": "text-embedding-3-small" } }
+ ]
+}
+----
+
+The rules:
+
+* `INLINE` and `RENDERING` children hand their chunks to the parent.
Attachments keep their
+ own: a PDF inside an email is still its own document with its own vectors,
as above.
+* The parent is the immediate container, so a picture inside an attached docx
reaches the
+ docx, not the email.
+* Every chunk written to a parent carries an `embedded` locator,
+ `{"id_path": "/1", "name": "image1.png"}`, naming the child it came from,
beside any
+ locator it already had.
+* `"liftToParent": false` on the embedder keeps every picture's vector on the
picture, for an
+ index where each picture is a hit of its own.
+* The parent is known only in the recursive (`/rmeta`, `-J`, pipes) parse,
where the
+ recursive wrapper exposes it to each child's parse as `ParentMetadata`; a
component that
+ writes chunks for a child resolves the target through `ChunkTarget`
(`tika-inference`).
+
+PDF page renders follow the same rule by a different route: the PDF parser
enriches its own
+renders and their vectors land on the PDF directly, one per page with a
`paginated` locator,
+whether the pages were rendered for OCR or emitted as embedded documents (see
+xref:configuration/parsers/pdf-parser.adoc[PDF Parser]).
+
== Elasticsearch/OpenSearch Mapping
With `nested` kNN support, a mapping like this works:
diff --git a/tika-app/pom.xml b/tika-app/pom.xml
index fea6bc9351..e161e0ba80 100644
--- a/tika-app/pom.xml
+++ b/tika-app/pom.xml
@@ -105,6 +105,13 @@
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</dependency>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>tika-http-jdk</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
<!-- The zip deps are what force reactor ordering; the zips are consumed
by dependency-plugin below. -->
<dependency>
<groupId>org.apache.tika</groupId>
diff --git
a/tika-app/src/test/java/org/apache/tika/cli/TikaCLIChunkLiftTest.java
b/tika-app/src/test/java/org/apache/tika/cli/TikaCLIChunkLiftTest.java
new file mode 100644
index 0000000000..a9a554d9dc
--- /dev/null
+++ b/tika-app/src/test/java/org/apache/tika/cli/TikaCLIChunkLiftTest.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.cli;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.awt.image.BufferedImage;
+import java.io.ByteArrayOutputStream;
+import java.io.PrintStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+import javax.imageio.ImageIO;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import org.apache.tika.http.TikaTestHttpServer;
+import org.apache.tika.inference.Chunk;
+import org.apache.tika.inference.ChunkSerializer;
+
+/**
+ * The consumer's path: a JSON config names the image embedder in
text-recognizers, and -J
+ * shows the vectors of the pictures in a docx body on the docx itself, each
naming the
+ * picture it came from. No other configuration: a picture's vector belongs to
its document.
+ */
+public class TikaCLIChunkLiftTest {
+
+ private static final String EMBEDDING_RESPONSE =
+ "{\"data\":[{\"embedding\":[0.1,0.2,0.3]}]}";
+
+ @TempDir
+ Path tmp;
+
+ @Test
+ public void testPictureVectorsLiftIntoTheDocx() throws Exception {
+ Path docx = tmp.resolve("two-pictures.docx");
+ writeDocx(docx);
+ try (TikaTestHttpServer server = new TikaTestHttpServer()) {
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
EMBEDDING_RESPONSE));
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
EMBEDDING_RESPONSE));
+ Path config = tmp.resolve("tika-config.json");
+ Files.writeString(config, """
+ {
+ "text-recognizers": [
+ { "openai-image-embedding-parser": { "baseUrl":
"BASE_URL", "model": "clip" } }
+ ]
+ }
+ """.replace("BASE_URL", server.url()));
+
+ JsonNode output = new ObjectMapper().readTree(
+ run("--config=" + config, "-J", docx.toUri().toString()));
+
+ assertEquals(2, server.getRequestCount(), "one embedding request
per picture");
+ JsonNode root = null;
+ List<JsonNode> pictures = new ArrayList<>();
+ for (JsonNode m : output) {
+ if (!m.has("tk:embedded-id-path")) {
+ root = m;
+ } else if ("INLINE".equals(text(m,
"tk:embedded-resource-type"))) {
+ pictures.add(m);
+ }
+ }
+ assertNotNull(root);
+ assertEquals(2, pictures.size());
+ Set<String> picturePaths = new HashSet<>();
+ for (JsonNode picture : pictures) {
+ assertFalse(picture.has("tk:chunks"), "chunks moved off the
picture");
+ picturePaths.add(text(picture, "tk:embedded-id-path"));
+ }
+
+ List<Chunk> chunks = ChunkSerializer.fromJson(text(root,
"tk:chunks"));
+ assertEquals(2, chunks.size());
+ Set<String> origins = new HashSet<>();
+ for (Chunk chunk : chunks) {
+ assertEquals(3, chunk.getVector().length);
+
origins.add(chunk.getLocators().getEmbedded().get(0).getIdPath());
+
assertTrue(chunk.getLocators().getEmbedded().get(0).getName().endsWith(".png"));
+ }
+ assertEquals(picturePaths, origins);
+ }
+ }
+
+ /** A minimal docx: one paragraph and two pictures related to the main
document part. */
+ private static void writeDocx(Path docx) throws Exception {
+ String rel =
"http://schemas.openxmlformats.org/officeDocument/2006/relationships/";;
+ try (ZipOutputStream zip = new
ZipOutputStream(Files.newOutputStream(docx))) {
+ entry(zip, "[Content_Types].xml", """
+ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+ <Types
xmlns="http://schemas.openxmlformats.org/package/2006/content-types";>
+ <Default Extension="rels"
ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
+ <Default Extension="xml" ContentType="application/xml"/>
+ <Default Extension="png" ContentType="image/png"/>
+ <Override PartName="/word/document.xml"
ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>
+ </Types>
+ """.getBytes(UTF_8));
+ entry(zip, "_rels/.rels", ("""
+ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+ <Relationships
xmlns="http://schemas.openxmlformats.org/package/2006/relationships";>
+ <Relationship Id="rId1" Type="RELofficeDocument"
Target="word/document.xml"/>
+ </Relationships>
+ """).replace("REL", rel).getBytes(UTF_8));
+ entry(zip, "word/document.xml", """
+ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+ <w:document
xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main";>
+ <w:body><w:p><w:r><w:t>two
pictures</w:t></w:r></w:p></w:body></w:document>
+ """.getBytes(UTF_8));
+ entry(zip, "word/_rels/document.xml.rels", ("""
+ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+ <Relationships
xmlns="http://schemas.openxmlformats.org/package/2006/relationships";>
+ <Relationship Id="rId1" Type="RELimage"
Target="media/image1.png"/>
+ <Relationship Id="rId2" Type="RELimage"
Target="media/image2.png"/>
+ </Relationships>
+ """).replace("REL", rel).getBytes(UTF_8));
+ entry(zip, "word/media/image1.png", png(1));
+ entry(zip, "word/media/image2.png", png(2));
+ }
+ }
+
+ private static void entry(ZipOutputStream zip, String name, byte[] bytes)
throws Exception {
+ zip.putNextEntry(new ZipEntry(name));
+ zip.write(bytes);
+ zip.closeEntry();
+ }
+
+ private static byte[] png(int shade) throws Exception {
+ BufferedImage image = new BufferedImage(20, 20,
BufferedImage.TYPE_INT_RGB);
+ for (int x = 0; x < 20; x++) {
+ for (int y = 0; y < 20; y++) {
+ image.setRGB(x, y, shade * 0x3F3F3F);
+ }
+ }
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ ImageIO.write(image, "png", out);
+ return out.toByteArray();
+ }
+
+ private static String text(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ assertNotNull(value, field);
+ return value.isArray() ? value.get(0).asText() : value.asText();
+ }
+
+ private static String run(String... args) throws Exception {
+ PrintStream stdout = System.out;
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ try {
+ System.setOut(new PrintStream(out, true, UTF_8.name()));
+ TikaCLI.main(args);
+ } finally {
+ System.setOut(stdout);
+ }
+ String json = out.toString(UTF_8);
+ assertTrue(json.startsWith("["), json);
+ return json;
+ }
+}
diff --git
a/tika-core/src/main/java/org/apache/tika/extractor/ParentMetadata.java
b/tika-core/src/main/java/org/apache/tika/extractor/ParentMetadata.java
new file mode 100644
index 0000000000..6505e70bf4
--- /dev/null
+++ b/tika-core/src/main/java/org/apache/tika/extractor/ParentMetadata.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.extractor;
+
+import org.apache.tika.metadata.Metadata;
+
+/**
+ * The metadata of the document an embedded document belongs to. The recursive
parser wrapper
+ * sets it for the duration of a child's parse, so a component enriching the
child can put a
+ * result where it belongs: a picture's vector on the page it appears in.
+ *
+ * @since Apache Tika 4.1
+ */
+public class ParentMetadata {
+
+ private final Metadata metadata;
+
+ public ParentMetadata(Metadata metadata) {
+ this.metadata = metadata;
+ }
+
+ public Metadata getMetadata() {
+ return metadata;
+ }
+}
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
b/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
index 7809deb96a..79470dce11 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
@@ -31,6 +31,7 @@ import org.apache.tika.exception.WriteLimitReachedException;
import org.apache.tika.exception.ZeroByteFileException;
import org.apache.tika.extractor.EmbeddedDocumentUtil;
import org.apache.tika.extractor.ParentContentHandler;
+import org.apache.tika.extractor.ParentMetadata;
import org.apache.tika.io.FilenameUtils;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
@@ -140,7 +141,7 @@ public class RecursiveParserWrapper extends ParserDecorator
{
"ContentHandler must implement
RecursiveParserWrapperHandler");
}
EmbeddedParserDecorator decorator =
- new EmbeddedParserDecorator(getWrappedParser(), "/", "/",
parserState);
+ new EmbeddedParserDecorator(getWrappedParser(), "/", "/",
parserState, metadata);
context.set(Parser.class, decorator);
ContentHandler localHandler =
parserState.recursiveParserWrapperHandler.createHandler();
@@ -214,10 +215,12 @@ public class RecursiveParserWrapper extends
ParserDecorator {
private String location = null;
private String embeddedIdPath = null;
-
+ // the document whose embedded documents this decorator parses
+ private final Metadata parentMetadata;
private EmbeddedParserDecorator(Parser parser, String location,
- String embeddedIdPath, ParserState
parseState) {
+ String embeddedIdPath, ParserState
parseState,
+ Metadata parentMetadata) {
super(parser);
this.location = location;
if (!this.location.endsWith("/")) {
@@ -225,6 +228,7 @@ public class RecursiveParserWrapper extends ParserDecorator
{
}
this.embeddedIdPath = embeddedIdPath;
this.parserState = parseState;
+ this.parentMetadata = parentMetadata;
}
@Override
@@ -251,7 +255,7 @@ public class RecursiveParserWrapper extends ParserDecorator
{
Parser preContextParser = context.get(Parser.class);
context.set(Parser.class,
new EmbeddedParserDecorator(getWrappedParser(),
objectLocation,
- idPath, parserState));
+ idPath, parserState, metadata));
long started = System.currentTimeMillis();
//store the handler that was used before this parse
//so that you can return it back to its state at the end of this
parse
@@ -259,6 +263,8 @@ public class RecursiveParserWrapper extends ParserDecorator
{
ParentContentHandler preParseParentHandler =
context.get(ParentContentHandler.class);
context.set(ParentContentHandler.class, new
ParentContentHandler(preParseHandler));
+ ParentMetadata preParseParentMetadata =
context.get(ParentMetadata.class);
+ context.set(ParentMetadata.class, new
ParentMetadata(parentMetadata));
ContentHandler secureContentHandler =
new RecursivelySecureContentHandler(localHandler, tis,
preParseHandler.handlerCounter,
preParseHandler.throwOnWriteLimitReached, context);
@@ -296,6 +302,7 @@ public class RecursiveParserWrapper extends ParserDecorator
{
context.set(Parser.class, preContextParser);
context.set(RecursivelySecureContentHandler.class,
preParseHandler);
context.set(ParentContentHandler.class, preParseParentHandler);
+ context.set(ParentMetadata.class, preParseParentMetadata);
long elapsedMillis = System.currentTimeMillis() - started;
metadata.set(TikaCoreProperties.PARSE_TIME_MILLIS,
Long.toString(elapsedMillis));
parserState.recursiveParserWrapperHandler
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkSerializer.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkSerializer.java
index 2119532c65..bcef910dd2 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkSerializer.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkSerializer.java
@@ -25,6 +25,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.tika.inference.locator.EmbeddedLocator;
import org.apache.tika.inference.locator.Locators;
import org.apache.tika.inference.locator.PaginatedLocator;
import org.apache.tika.inference.locator.SpatialLocator;
@@ -37,7 +38,7 @@ import org.apache.tika.metadata.TikaCoreProperties;
* Vectors are stored as base64-encoded big-endian float32 via
* {@link VectorSerializer}. Locators are nested under a {@code "locators"}
* object with optional {@code text}, {@code paginated}, {@code spatial},
- * and {@code temporal} arrays.
+ * {@code temporal} and {@code embedded} arrays.
*/
public final class ChunkSerializer {
@@ -172,6 +173,17 @@ public final class ChunkSerializer {
o.put("end_ms", t.getEndMs());
}
}
+
+ if (loc.getEmbedded() != null && !loc.getEmbedded().isEmpty()) {
+ ArrayNode arr = locNode.putArray("embedded");
+ for (EmbeddedLocator e : loc.getEmbedded()) {
+ ObjectNode o = arr.addObject();
+ o.put("id_path", e.getIdPath());
+ if (e.getName() != null) {
+ o.put("name", e.getName());
+ }
+ }
+ }
}
private static Locators deserializeLocators(JsonNode locNode) {
@@ -216,6 +228,14 @@ public final class ChunkSerializer {
}
}
+ JsonNode embArr = locNode.get("embedded");
+ if (embArr != null && embArr.isArray()) {
+ for (JsonNode n : embArr) {
+ String name = n.has("name") ? n.get("name").asText() : null;
+ locators.addEmbedded(new
EmbeddedLocator(n.get("id_path").asText(), name));
+ }
+ }
+
return locators;
}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkTarget.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkTarget.java
new file mode 100644
index 0000000000..e7b4088705
--- /dev/null
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ChunkTarget.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.inference;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.tika.extractor.ParentMetadata;
+import org.apache.tika.inference.locator.EmbeddedLocator;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+
+/**
+ * Where the chunks produced for a document land. A picture in the body of a
docx or an
+ * email, or a rendering of a page, is part of its parent, so its chunks go
onto the parent
+ * with an {@link EmbeddedLocator} naming the child. An attachment, a
top-level file, or a
+ * child parsed outside the recursive wrapper keeps its own.
+ */
+public final class ChunkTarget {
+
+ private static final Set<String> LIFTED = Set.of(
+ TikaCoreProperties.EmbeddedResourceType.INLINE.name(),
+ TikaCoreProperties.EmbeddedResourceType.RENDERING.name());
+
+ private final Metadata metadata;
+ private final EmbeddedLocator locator;
+
+ private ChunkTarget(Metadata metadata, EmbeddedLocator locator) {
+ this.metadata = metadata;
+ this.locator = locator;
+ }
+
+ public static ChunkTarget resolve(Metadata target, ParseContext context) {
+ ParentMetadata parent = context.get(ParentMetadata.class);
+ String idPath = target.get(TikaCoreProperties.EMBEDDED_ID_PATH);
+ String type = target.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE);
+ if (parent == null || idPath == null || type == null ||
!LIFTED.contains(type)) {
+ return new ChunkTarget(target, null);
+ }
+ return new ChunkTarget(parent.getMetadata(),
+ new EmbeddedLocator(idPath,
target.get(TikaCoreProperties.RESOURCE_NAME_KEY)));
+ }
+
+ /** The target keeps its own chunks. */
+ public static ChunkTarget self(Metadata target) {
+ return new ChunkTarget(target, null);
+ }
+
+ public Metadata getMetadata() {
+ return metadata;
+ }
+
+ /** Names the child the chunks came from; null when they stay on that
child. */
+ public EmbeddedLocator getLocator() {
+ return locator;
+ }
+
+ /** Tags the chunks with the locator, if any, and appends them to the
field. */
+ public void write(List<Chunk> chunks, String field) throws IOException {
+ if (locator != null) {
+ for (Chunk chunk : chunks) {
+ if (chunk.getLocators() != null) {
+ chunk.getLocators().addEmbedded(locator);
+ }
+ }
+ }
+ ChunkSerializer.mergeInto(metadata, chunks, field);
+ }
+}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ImageEmbeddingConfig.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ImageEmbeddingConfig.java
index f4de52a093..1d8da98f2f 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ImageEmbeddingConfig.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/ImageEmbeddingConfig.java
@@ -45,6 +45,13 @@ public class ImageEmbeddingConfig implements Serializable {
*/
private String outputField = TikaCoreProperties.TIKA_CHUNKS.getName();
+ /**
+ * Put a picture's vector on the document the picture appears in (a docx,
an email, a
+ * rendered page's PDF) rather than on the picture's own embedded
document; attachments
+ * always keep their own. See {@link ChunkTarget}.
+ */
+ private boolean liftToParent = true;
+
public String getBaseUrl() {
return baseUrl;
}
@@ -109,6 +116,14 @@ public class ImageEmbeddingConfig implements Serializable {
this.outputField = outputField;
}
+ public boolean isLiftToParent() {
+ return liftToParent;
+ }
+
+ public void setLiftToParent(boolean liftToParent) {
+ this.liftToParent = liftToParent;
+ }
+
/**
* Runtime-only config that prevents modification of security-sensitive
* and cost-sensitive fields ({@code baseUrl}, {@code apiKey},
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java
index 92091193e9..bc04bd4954 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java
@@ -171,7 +171,9 @@ public class OpenAIImageEmbeddingParser implements Parser,
Initializable, Closea
Chunk chunk = new Chunk(null, locators);
chunk.setVector(vector);
- ChunkSerializer.mergeInto(metadata, List.of(chunk),
config.getOutputField());
+ ChunkTarget target = config.isLiftToParent()
+ ? ChunkTarget.resolve(metadata, parseContext) :
ChunkTarget.self(metadata);
+ target.write(List.of(chunk), config.getOutputField());
XHTMLContentHandler xhtml = new XHTMLContentHandler(
handler, metadata, parseContext);
@@ -376,6 +378,14 @@ public class OpenAIImageEmbeddingParser implements Parser,
Initializable, Closea
defaultConfig.setOutputField(outputField);
}
+ public boolean isLiftToParent() {
+ return defaultConfig.isLiftToParent();
+ }
+
+ public void setLiftToParent(boolean liftToParent) {
+ defaultConfig.setLiftToParent(liftToParent);
+ }
+
// ---- Azure / endpoint config getters/setters
----------------------------
public String getEmbeddingsPath() {
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/EmbeddedLocator.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/EmbeddedLocator.java
new file mode 100644
index 0000000000..8f4aec8808
--- /dev/null
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/EmbeddedLocator.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.inference.locator;
+
+/**
+ * The embedded document a chunk was derived from, for a chunk that was lifted
into its
+ * parent's {@code tk:chunks}: a page render or an inline image whose vector
belongs to the
+ * document it is part of.
+ */
+public class EmbeddedLocator {
+
+ private final String idPath;
+ private final String name;
+
+ /**
+ * @param idPath the child's {@code tk:embedded-id-path}
+ * @param name the child's resource name, or null if it had none
+ */
+ public EmbeddedLocator(String idPath, String name) {
+ this.idPath = idPath;
+ this.name = name;
+ }
+
+ public String getIdPath() {
+ return idPath;
+ }
+
+ public String getName() {
+ return name;
+ }
+}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/Locators.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/Locators.java
index edd525c66a..b2047fb0a3 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/Locators.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/locator/Locators.java
@@ -33,6 +33,7 @@ public class Locators {
private List<PaginatedLocator> paginated;
private List<SpatialLocator> spatial;
private List<TemporalLocator> temporal;
+ private List<EmbeddedLocator> embedded;
public Locators() {
}
@@ -109,6 +110,24 @@ public class Locators {
return this;
}
+ // ---- embedded ---------------------------------------------------------
+
+ public List<EmbeddedLocator> getEmbedded() {
+ return embedded;
+ }
+
+ public void setEmbedded(List<EmbeddedLocator> embedded) {
+ this.embedded = embedded;
+ }
+
+ public Locators addEmbedded(EmbeddedLocator locator) {
+ if (this.embedded == null) {
+ this.embedded = new ArrayList<>();
+ }
+ this.embedded.add(locator);
+ return this;
+ }
+
/**
* @return true if no locators of any type are present
*/
@@ -116,6 +135,7 @@ public class Locators {
return (text == null || text.isEmpty())
&& (paginated == null || paginated.isEmpty())
&& (spatial == null || spatial.isEmpty())
- && (temporal == null || temporal.isEmpty());
+ && (temporal == null || temporal.isEmpty())
+ && (embedded == null || embedded.isEmpty());
}
}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkSerializerTest.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkSerializerTest.java
index 2f320731dd..a0357b00d3 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkSerializerTest.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkSerializerTest.java
@@ -25,6 +25,7 @@ import java.util.List;
import org.junit.jupiter.api.Test;
+import org.apache.tika.inference.locator.EmbeddedLocator;
import org.apache.tika.inference.locator.Locators;
import org.apache.tika.inference.locator.PaginatedLocator;
import org.apache.tika.inference.locator.SpatialLocator;
@@ -180,4 +181,22 @@ public class ChunkSerializerTest {
assertEquals("He said \"hello\" & <goodbye>",
restored.get(0).getText());
}
+
+ @Test
+ void testRoundTripEmbeddedLocator() throws Exception {
+ Chunk c = new Chunk(null, new Locators()
+ .addPaginated(new PaginatedLocator(3))
+ .addEmbedded(new EmbeddedLocator("/1/2", "page-3.png")));
+ c.setVector(new float[]{0.5f});
+ Chunk unnamed = new Chunk(null, new Locators().addEmbedded(new
EmbeddedLocator("/4", null)));
+
+ List<Chunk> restored =
ChunkSerializer.fromJson(ChunkSerializer.toJson(List.of(c, unnamed)));
+
+ EmbeddedLocator e = restored.get(0).getLocators().getEmbedded().get(0);
+ assertEquals("/1/2", e.getIdPath());
+ assertEquals("page-3.png", e.getName());
+ assertEquals(3,
restored.get(0).getLocators().getPaginated().get(0).getPage());
+ assertEquals("/4",
restored.get(1).getLocators().getEmbedded().get(0).getIdPath());
+
assertNull(restored.get(1).getLocators().getEmbedded().get(0).getName());
+ }
}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkTargetTest.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkTargetTest.java
new file mode 100644
index 0000000000..fbd7df4ca0
--- /dev/null
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/ChunkTargetTest.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.inference;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.extractor.ParentMetadata;
+import org.apache.tika.inference.locator.Locators;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+
+public class ChunkTargetTest {
+
+ private static Metadata child(String type, String idPath, String name) {
+ Metadata m = new Metadata();
+ if (type != null) {
+ m.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, type);
+ }
+ if (idPath != null) {
+ m.set(TikaCoreProperties.EMBEDDED_ID_PATH, idPath);
+ }
+ m.set(TikaCoreProperties.RESOURCE_NAME_KEY, name);
+ return m;
+ }
+
+ private static ParseContext withParent(Metadata parent) {
+ ParseContext context = new ParseContext();
+ context.set(ParentMetadata.class, new ParentMetadata(parent));
+ return context;
+ }
+
+ @Test
+ public void testInlineAndRenderingLiftToParent() throws Exception {
+ Metadata parent = new Metadata();
+ for (String type : List.of("INLINE", "RENDERING")) {
+ Metadata child = child(type, "/1", "image1.png");
+ ChunkTarget target = ChunkTarget.resolve(child,
withParent(parent));
+ assertSame(parent, target.getMetadata());
+ assertEquals("/1", target.getLocator().getIdPath());
+ assertEquals("image1.png", target.getLocator().getName());
+ }
+ }
+
+ @Test
+ public void testAttachmentTopLevelAndNoParentKeepTheirOwn() {
+ Metadata parent = new Metadata();
+ Metadata attachment = child("ATTACHMENT", "/2", "report.pdf");
+ assertSame(attachment, ChunkTarget.resolve(attachment,
withParent(parent)).getMetadata());
+ assertNull(ChunkTarget.resolve(attachment,
withParent(parent)).getLocator());
+
+ Metadata untyped = child(null, "/3", "mystery.bin");
+ assertSame(untyped, ChunkTarget.resolve(untyped,
withParent(parent)).getMetadata());
+
+ Metadata orphan = child("INLINE", "/4", "image4.png");
+ assertSame(orphan, ChunkTarget.resolve(orphan, new
ParseContext()).getMetadata());
+
+ Metadata topLevel = new Metadata();
+ assertSame(topLevel, ChunkTarget.resolve(topLevel,
withParent(parent)).getMetadata());
+ assertSame(topLevel, ChunkTarget.self(topLevel).getMetadata());
+ }
+
+ @Test
+ public void testWriteTagsAndAppends() throws Exception {
+ Metadata parent = new Metadata();
+ ParseContext context = withParent(parent);
+ for (int i = 1; i <= 2; i++) {
+ Chunk chunk = new Chunk(null, new Locators());
+ chunk.setVector(new float[]{i});
+ ChunkTarget.resolve(child("INLINE", "/" + i, "image" + i +
".png"), context)
+ .write(List.of(chunk),
TikaCoreProperties.TIKA_CHUNKS.getName());
+ }
+ List<Chunk> chunks =
ChunkSerializer.fromJson(parent.get(TikaCoreProperties.TIKA_CHUNKS));
+ assertEquals(2, chunks.size());
+ assertEquals("/1",
chunks.get(0).getLocators().getEmbedded().get(0).getIdPath());
+ assertEquals("image2.png",
chunks.get(1).getLocators().getEmbedded().get(0).getName());
+ }
+}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIImageEmbeddingParserTest.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIImageEmbeddingParserTest.java
index 7ef3b7189d..ffc0e2e15d 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIImageEmbeddingParserTest.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIImageEmbeddingParserTest.java
@@ -32,6 +32,7 @@ import org.junit.jupiter.api.Test;
import org.xml.sax.helpers.DefaultHandler;
import org.apache.tika.exception.TikaException;
+import org.apache.tika.extractor.ParentMetadata;
import org.apache.tika.http.TikaTestHttpServer;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.HttpHeaders;
@@ -242,6 +243,41 @@ public class OpenAIImageEmbeddingParserTest {
assertEquals(4, merged.get(1).getVector().length);
}
+ /** A picture in a docx body: its vector lands on the docx, naming the
picture. */
+ @Test
+ void testInlineChildVectorLandsOnParent() throws Exception {
+ Metadata parent = new Metadata();
+ Metadata picture = new Metadata();
+ picture.set(HttpHeaders.CONTENT_TYPE, "image/png");
+ picture.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, "INLINE");
+ picture.set(TikaCoreProperties.EMBEDDED_ID_PATH, "/1");
+ picture.set(TikaCoreProperties.RESOURCE_NAME_KEY, "image1.png");
+ ParseContext context = new ParseContext();
+ context.set(ParentMetadata.class, new ParentMetadata(parent));
+
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
buildEmbeddingResponse(3)));
+ try (TikaInputStream tis = TikaInputStream.get(new byte[]{(byte) 0x89,
'P', 'N', 'G'})) {
+ parser.parse(tis, new DefaultHandler(), picture, context);
+ }
+ assertNull(picture.get(TikaCoreProperties.TIKA_CHUNKS), "the picture
keeps nothing");
+ List<Chunk> chunks =
ChunkSerializer.fromJson(parent.get(TikaCoreProperties.TIKA_CHUNKS));
+ assertEquals(1, chunks.size());
+ assertEquals("/1",
chunks.get(0).getLocators().getEmbedded().get(0).getIdPath());
+ assertEquals("image1.png",
chunks.get(0).getLocators().getEmbedded().get(0).getName());
+
+ ImageEmbeddingConfig own = new ImageEmbeddingConfig();
+ own.setBaseUrl(server.url());
+ own.setModel("clip");
+ own.setLiftToParent(false);
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
buildEmbeddingResponse(3)));
+ try (OpenAIImageEmbeddingParser keeping = new
OpenAIImageEmbeddingParser(own);
+ TikaInputStream tis = TikaInputStream.get(new byte[]{(byte)
0x89, 'P', 'N', 'G'})) {
+ keeping.parse(tis, new DefaultHandler(), picture, context);
+ }
+ assertNotNull(picture.get(TikaCoreProperties.TIKA_CHUNKS),
"liftToParent: false");
+ assertEquals(1,
ChunkSerializer.fromJson(parent.get(TikaCoreProperties.TIKA_CHUNKS)).size());
+ }
+
@Test
void testSupportedTypes() {
assertTrue(parser.getSupportedTypes(new ParseContext())
