This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 8443ce4973 TIKA-4856: thumbnails catalog preset and
ThumbnailUnpackSelector (#3179)
8443ce4973 is described below
commit 8443ce49733c889601a12cb54fbfcde31c309910
Author: Tim Allison <[email protected]>
AuthorDate: Wed Sep 16 08:19:45 2026 -0400
TIKA-4856: thumbnails catalog preset and ThumbnailUnpackSelector (#3179)
---
CHANGES.txt | 2 +
.../ROOT/pages/using-tika/server/index.adoc | 48 +++++
.../org/apache/tika/extractor/UnpackSelector.java | 9 +
.../core/extractor/ThumbnailUnpackSelector.java | 95 ++++++++++
.../tika/pipes/core/extractor/UnpackExtractor.java | 2 +-
.../META-INF/tika/preset-catalog.properties | 19 ++
.../tika/pipes/core/presets/render-thumbnail.json | 12 ++
.../apache/tika/pipes/core/presets/thumbnail.json | 20 ++
.../extractor/ThumbnailUnpackSelectorTest.java | 82 +++++++++
.../apache/tika/config/loader/PresetRegistry.java | 15 ++
.../server/standard/RenderThumbnailPresetTest.java | 134 ++++++++++++++
.../tika/server/standard/ThumbnailPresetTest.java | 201 +++++++++++++++++++++
.../configs/tika-config-thumbnail-preset.json | 5 +
.../test-documents/testDOCX_Thumbnail.docx | Bin 0 -> 13810 bytes
.../resources/test-documents/testMP3_twoCovers.mp3 | Bin 0 -> 40065 bytes
.../resources/test-documents/testPDF_bookmarks.pdf | Bin 0 -> 9487 bytes
16 files changed, 643 insertions(+), 1 deletion(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 207eeee4f7..ff34ddd949 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,7 @@
Release 4.1.0 - unreleased
+ * New catalog presets, inert until the config names them. (TIKA-4856).
+
* The inference dispatcher no longer copies the top-level document's own
bytes into a buffer file of its own: the stream that owns them is open
until after the flush, so the file is referenced. Embedded documents'
diff --git a/docs/modules/ROOT/pages/using-tika/server/index.adoc
b/docs/modules/ROOT/pages/using-tika/server/index.adoc
index db0bac2f68..a7f3441ea4 100644
--- a/docs/modules/ROOT/pages/using-tika/server/index.adoc
+++ b/docs/modules/ROOT/pages/using-tika/server/index.adoc
@@ -155,6 +155,54 @@ not (detectors, embedded-document extraction, exception
reporting, ...) and rais
limits above the per-request clamp. Every active preset resolves at startup,
so a bad preset
fails the server, not its first request.
+WARNING: An active preset is a public route: anyone who can reach the server
can call it, and
+it runs at config trust — it can bind components `/config` cannot and raise
timeout limits
+above the per-request clamp. Vet a preset as you would
`allowPerRequestConfig`, scoped to what
+the preset turns on. Tika's catalog presets are kept bounded (page and render
caps, no OCR
+unless the preset is about OCR, no timeout raises); hold your own to the same
bar. Startup
+logs one line per active preset naming the components it configures.
+
+==== The thumbnail catalog presets
+
+Tika ships two catalog presets in `tika-pipes-core`. They are not active out
of the box: the
+operator names them in the server's `tika-config.json`, and a server without
that line answers
+`404` on their routes.
+
+[source,json]
+----
+{ "presets": { "thumbnail": true, "render-thumbnail": true } }
+----
+
+Both render the same two things the same way — the first page of a PDF at 96
dpi in colour
+(`rendering`, bounded by its `maxImagePixels`) and the EMF/WMF thumbnail of an
Office document
+(that metafile only, not the document's pictures) — so an image found through
one is the image
+the other returns, dimensions included. Both name the standard parsers, so
they need
+`tika-parsers-standard-package` on the classpath.
+
+`/unpack/preset/thumbnail` — "just give me the thumbnail". The zip holds at
most one raster
+image per request and nothing else: the stored thumbnail (Office docProps,
audio cover art,
+raw camera previews, epub, iWork), the rasterized EMF/WMF thumbnail, or the
PDF page; `204`
+when the document has none. An image file is its own thumbnail: it has no
embedded documents,
+so it answers `204` too. Text is read from the first PDF page only, no OCR or
inference runs,
+and embedded documents are parsed two levels deep at most, so the parse is
cheap.
+
+`/rmeta/preset/render-thumbnail` — "give me everything, thumbnail included". A
full parse
+plus the two renders above, so an index gets the thumbnail's type and
dimensions from the same
+request as the text. Nothing is selected for the client; it applies the rule
below. The preset
+names only rendering settings, and they merge with what the config's `parsers`
block sets for
+`pdf-parser`, `emf-parser` and `wmf-parser`; a same-named entry in the
config's `parse-context`
+block is replaced by the preset's, as with any parse-context layer, so keep
server-wide parser
+settings under `parsers`.
+
+The selection rule, for a client that reads `/rmeta` and wants the answer
`thumbnail` gives:
+take the first embedded document with an `image/*` `Content-Type` other than
EMF, WMF or SVG
+that is a `tk:embedded-resource-type` of `THUMBNAIL` at `tk:embedded-depth` 1,
a `THUMBNAIL` at
+depth 2 carrying `tk:rendering:rendered-by` (the rendering of a vector
thumbnail), or a
+`RENDERING` at depth 1 (a PDF page). The thumbnail of a document inside an
archive is that
+document's, not the archive's. Plain `/rmeta` lists stored thumbnails already;
the PDF page and
+the EMF/WMF rendering exist only when a request turns rendering on, which is
what
+`render-thumbnail` does.
+
Output format on the preset routes: an explicit format segment
(`/tika/preset/{name}/text`)
always wins; without one, a `ContentHandlerFactory` the preset binds decides,
then the
config's, then the endpoint default (Markdown).
diff --git
a/tika-core/src/main/java/org/apache/tika/extractor/UnpackSelector.java
b/tika-core/src/main/java/org/apache/tika/extractor/UnpackSelector.java
index b346759d1a..d93db3d5ba 100644
--- a/tika-core/src/main/java/org/apache/tika/extractor/UnpackSelector.java
+++ b/tika-core/src/main/java/org/apache/tika/extractor/UnpackSelector.java
@@ -17,6 +17,7 @@
package org.apache.tika.extractor;
import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.ParseContext;
public interface UnpackSelector {
@@ -29,4 +30,12 @@ public interface UnpackSelector {
UnpackSelector ACCEPT_ALL = new AcceptAll();
boolean select(Metadata metadata);
+
+ /**
+ * Per-parse variant; the default ignores the context. Override to keep
state per parse,
+ * since one selector instance may serve many parses.
+ */
+ default boolean select(Metadata metadata, ParseContext context) {
+ return select(metadata);
+ }
}
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelector.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelector.java
new file mode 100644
index 0000000000..bece7215b5
--- /dev/null
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelector.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.pipes.core.extractor;
+
+import java.util.Locale;
+import java.util.Set;
+
+import org.apache.tika.annotation.TikaComponent;
+import org.apache.tika.config.TransientParseState;
+import org.apache.tika.extractor.UnpackSelector;
+import org.apache.tika.metadata.HttpHeaders;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.Rendering;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.mime.MediaType;
+import org.apache.tika.parser.ParseContext;
+
+/**
+ * Unpacks the one embedded document a client would show as the document's
thumbnail and
+ * nothing else. A candidate is a raster image that is a {@code THUMBNAIL} at
depth 1 (stored
+ * thumbnail), a rendered {@code THUMBNAIL} at depth 2 (the rasterized vector
thumbnail of an
+ * Office document; {@code tk:rendering:rendered-by} marks it), or a {@code
RENDERING} at
+ * depth 1 (the first page of a PDF); the first candidate of a parse wins. The
thumbnail of a
+ * document inside an archive is that document's, not the archive's, and is
not a candidate.
+ * Pair it with parser config that renders only those, as the catalog preset
{@code thumbnail}
+ * does; clients reading {@code /rmeta} apply the same rule.
+ *
+ * @since Apache Tika 4.1.0
+ */
+@TikaComponent
+public class ThumbnailUnpackSelector implements UnpackSelector {
+
+ private static final Set<String> VECTOR_TYPES = Set.of("image/emf",
"image/x-emf",
+ "image/wmf", "image/x-wmf", "image/svg+xml");
+
+ /**
+ * Marks a parse whose thumbnail is taken; the selector itself may be
shared. Lives as
+ * long as the ParseContext, like ParseRecord's embedded counts: one
context per parse.
+ */
+ private static final class Taken implements TransientParseState {
+ }
+
+ @Override
+ public boolean select(Metadata metadata) {
+ return isCandidate(metadata);
+ }
+
+ /** The stateless rule; {@code /rmeta} clients pick the first embedded
document it accepts. */
+ public static boolean isCandidate(Metadata metadata) {
+ String type = metadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE);
+ Integer depth = metadata.getInt(TikaCoreProperties.EMBEDDED_DEPTH);
+ if (type == null || depth == null || !isRaster(metadata)) {
+ return false;
+ }
+ if
(TikaCoreProperties.EmbeddedResourceType.THUMBNAIL.name().equals(type)) {
+ return depth == 1
+ || (depth == 2 && metadata.get(Rendering.RENDERED_BY) !=
null);
+ }
+ return
TikaCoreProperties.EmbeddedResourceType.RENDERING.name().equals(type)
+ && depth == 1;
+ }
+
+ @Override
+ public boolean select(Metadata metadata, ParseContext context) {
+ if (context.get(Taken.class) != null || !isCandidate(metadata)) {
+ return false;
+ }
+ context.set(Taken.class, new Taken());
+ return true;
+ }
+
+ private static boolean isRaster(Metadata metadata) {
+ String contentType = metadata.get(HttpHeaders.CONTENT_TYPE);
+ MediaType mediaType = contentType == null ? null :
MediaType.parse(contentType);
+ if (mediaType == null) {
+ return false;
+ }
+ String base =
mediaType.getBaseType().toString().toLowerCase(Locale.ROOT);
+ return base.startsWith("image/") && !VECTOR_TYPES.contains(base);
+ }
+}
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
index 6bc5be097a..b469238b88 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackExtractor.java
@@ -181,7 +181,7 @@ public class UnpackExtractor extends
ParsingEmbeddedDocumentExtractor {
// Get UnpackSelector from ParseContext - if configured, use it to
filter
// If no selector configured, accept all embedded documents
UnpackSelector selector = context.get(UnpackSelector.class);
- if (selector != null && !selector.select(metadata)) {
+ if (selector != null && !selector.select(metadata, context)) {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("skipping embedded bytes {} <-> {}",
metadata.get(HttpHeaders.CONTENT_TYPE),
diff --git
a/tika-pipes/tika-pipes-core/src/main/resources/META-INF/tika/preset-catalog.properties
b/tika-pipes/tika-pipes-core/src/main/resources/META-INF/tika/preset-catalog.properties
new file mode 100644
index 0000000000..8f369767db
--- /dev/null
+++
b/tika-pipes/tika-pipes-core/src/main/resources/META-INF/tika/preset-catalog.properties
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Presets shipped with tika-pipes-core. Inert until a config activates one:
"presets": {"thumbnail": true}
+thumbnail=/org/apache/tika/pipes/core/presets/thumbnail.json
+render-thumbnail=/org/apache/tika/pipes/core/presets/render-thumbnail.json
diff --git
a/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/render-thumbnail.json
b/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/render-thumbnail.json
new file mode 100644
index 0000000000..98af0ebec9
--- /dev/null
+++
b/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/render-thumbnail.json
@@ -0,0 +1,12 @@
+{
+ // the same first-page and EMF/WMF renders as "thumbnail", added to a full
parse. Settings
+ // under the config's "parsers" block merge with these; a pdf-parser,
emf-parser or wmf-parser
+ // entry in the config's "parse-context" is replaced by the preset's, like
any parse-context layer
+ "pdf-parser": {
+ "imageStrategy": "RENDER_PAGES_BEFORE_PARSE",
+ "maxRenderedPages": 1,
+ "rendering": {"dpi": 96, "imageType": "RGB"}
+ },
+ "emf-parser": {"renderImage": true, "renderOnlyEmbeddedResourceTypes":
["THUMBNAIL"]},
+ "wmf-parser": {"renderImage": true, "renderOnlyEmbeddedResourceTypes":
["THUMBNAIL"]}
+}
diff --git
a/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/thumbnail.json
b/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/thumbnail.json
new file mode 100644
index 0000000000..a98b91b5b0
--- /dev/null
+++
b/tika-pipes/tika-pipes-core/src/main/resources/org/apache/tika/pipes/core/presets/thumbnail.json
@@ -0,0 +1,20 @@
+{
+ // at most one raster thumbnail per parse, nothing else; the rule is in
ThumbnailUnpackSelector
+ "thumbnail-unpack-selector": {},
+ // first page only, in colour; text comes from the content stream, never OCR
+ "pdf-parser": {
+ "imageStrategy": "RENDER_PAGES_BEFORE_PARSE",
+ "maxPages": 1,
+ "maxRenderedPages": 1,
+ "text": "EXTRACT",
+ "rendering": {"dpi": 96, "imageType": "RGB"}
+ },
+ // rasterize the EMF/WMF thumbnail of an Office document, not its embedded
pictures
+ "emf-parser": {"renderImage": true, "renderOnlyEmbeddedResourceTypes":
["THUMBNAIL"]},
+ "wmf-parser": {"renderImage": true, "renderOnlyEmbeddedResourceTypes":
["THUMBNAIL"]},
+ // no OCR or embeddings over the thumbnail itself
+ "text-recognizers": {"enabled": false},
+ "inference": {"enabled": false},
+ // the deepest candidate is the rendering of a vector thumbnail; nothing
below it is one
+ "embedded-limits": {"maxDepth": 2}
+}
diff --git
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelectorTest.java
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelectorTest.java
new file mode 100644
index 0000000000..b0fd0112bf
--- /dev/null
+++
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/ThumbnailUnpackSelectorTest.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.pipes.core.extractor;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.metadata.HttpHeaders;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.Rendering;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+
+public class ThumbnailUnpackSelectorTest {
+
+ @Test
+ public void testCandidates() {
+ ThumbnailUnpackSelector selector = new ThumbnailUnpackSelector();
+ assertTrue(selector.select(metadata("image/jpeg", "THUMBNAIL", 1)),
"stored thumbnail");
+ assertTrue(selector.select(rendered(metadata("image/png", "THUMBNAIL",
2))),
+ "rendering of a vector thumbnail");
+ assertFalse(selector.select(metadata("image/jpeg", "THUMBNAIL", 2)),
+ "stored thumbnail of a document inside an archive");
+ assertTrue(selector.select(metadata("image/png", "RENDERING", 1)),
"first page render");
+ assertTrue(selector.select(metadata("image/png; charset=binary",
"THUMBNAIL", 1)));
+ assertFalse(selector.select(metadata("image/emf", "THUMBNAIL", 1)),
"vector thumbnail");
+ assertFalse(selector.select(metadata("image/png", "RENDERING", 2)),
+ "render of an embedded document");
+ assertFalse(selector.select(rendered(metadata("image/png",
"THUMBNAIL", 3))),
+ "rendered thumbnail of a document inside an archive");
+ assertFalse(selector.select(metadata("image/png", "INLINE", 1)));
+ assertFalse(selector.select(metadata("application/pdf", "ATTACHMENT",
1)));
+ assertFalse(selector.select(metadata(null, "THUMBNAIL", 1)));
+ Metadata noDepth = metadata("image/png", "THUMBNAIL", 1);
+ noDepth.remove(TikaCoreProperties.EMBEDDED_DEPTH.getName());
+ assertFalse(selector.select(noDepth));
+ }
+
+ @Test
+ public void testFirstWinsPerParse() {
+ ThumbnailUnpackSelector selector = new ThumbnailUnpackSelector();
+ ParseContext parse = new ParseContext();
+ assertFalse(selector.select(metadata("image/png", "INLINE", 1), parse),
+ "a rejected candidate does not take the slot");
+ assertTrue(selector.select(metadata("image/jpeg", "THUMBNAIL", 1),
parse));
+ assertFalse(selector.select(metadata("image/jpeg", "THUMBNAIL", 1),
parse),
+ "second thumbnail of the same parse");
+ assertTrue(selector.select(metadata("image/png", "RENDERING", 1), new
ParseContext()),
+ "a shared selector starts over on the next parse");
+ }
+
+ private static Metadata rendered(Metadata metadata) {
+ metadata.add(Rendering.RENDERED_BY, "poi-metafile-renderer");
+ return metadata;
+ }
+
+ private static Metadata metadata(String contentType, String type, int
depth) {
+ Metadata metadata = new Metadata();
+ if (contentType != null) {
+ metadata.set(HttpHeaders.CONTENT_TYPE, contentType);
+ }
+ metadata.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, type);
+ metadata.set(TikaCoreProperties.EMBEDDED_DEPTH, depth);
+ return metadata;
+ }
+}
diff --git
a/tika-serialization/src/main/java/org/apache/tika/config/loader/PresetRegistry.java
b/tika-serialization/src/main/java/org/apache/tika/config/loader/PresetRegistry.java
index b2245a9424..1206d387da 100644
---
a/tika-serialization/src/main/java/org/apache/tika/config/loader/PresetRegistry.java
+++
b/tika-serialization/src/main/java/org/apache/tika/config/loader/PresetRegistry.java
@@ -20,17 +20,21 @@ import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashMap;
+import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.tika.exception.TikaConfigException;
import org.apache.tika.parser.ParseContext;
@@ -65,6 +69,8 @@ import
org.apache.tika.serialization.serdes.ParseContextDeserializer;
*/
public final class PresetRegistry {
+ private static final Logger LOG =
LoggerFactory.getLogger(PresetRegistry.class);
+
public static final String CONFIG_KEY = "presets";
private static final String INDEX_RESOURCE =
"META-INF/tika/preset-catalog.properties";
@@ -133,10 +139,19 @@ public final class PresetRegistry {
if (resolved.get(ContentHandlerFactory.class) != null) {
withContentHandlerFactory.add(e.getKey());
}
+ // an active preset is a public route at config trust; say what it
binds
+ LOG.info("preset '{}' active, callable by any client; configures
{}",
+ e.getKey(), componentNames(e.getValue()));
}
return new PresetRegistry(presets, withContentHandlerFactory, loader);
}
+ private static List<String> componentNames(JsonNode content) {
+ List<String> names = new ArrayList<>();
+ content.fieldNames().forEachRemaining(names::add);
+ return names;
+ }
+
// Trusted-tier resolution: operator config, so no wire-block screening.
private static ParseContext resolve(String name, JsonNode content,
ClassLoader loader)
throws TikaConfigException {
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/RenderThumbnailPresetTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/RenderThumbnailPresetTest.java
new file mode 100644
index 0000000000..7fb818386d
--- /dev/null
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/RenderThumbnailPresetTest.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.server.standard;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.ArrayList;
+import java.util.List;
+
+import jakarta.ws.rs.core.Response;
+import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
+import org.apache.cxf.jaxrs.client.WebClient;
+import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;
+import org.apache.pdfbox.Loader;
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.metadata.HttpHeaders;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.PagedText;
+import org.apache.tika.metadata.Rendering;
+import org.apache.tika.metadata.TIFF;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.pipes.core.extractor.ThumbnailUnpackSelector;
+import org.apache.tika.serialization.JsonMetadataList;
+import org.apache.tika.server.core.CXFTestBase;
+import org.apache.tika.server.core.resource.RecursiveMetadataResource;
+import org.apache.tika.server.core.writer.MetadataListMessageBodyWriter;
+
+/**
+ * The catalog preset {@code render-thumbnail} adds the thumbnail renders to a
full parse, and
+ * the documented client rule finds the image {@code thumbnail} would have
unpacked (TIKA-4856).
+ */
+public class RenderThumbnailPresetTest extends CXFTestBase {
+
+ private static final String PRESET_PATH = "/rmeta/preset/render-thumbnail";
+
+ @Override
+ protected void setUpResources(JAXRSServerFactoryBean sf) {
+ sf.setResourceClasses(RecursiveMetadataResource.class);
+ sf.setResourceProvider(RecursiveMetadataResource.class,
+ new SingletonResourceProvider(new
RecursiveMetadataResource(tikaResource)));
+ }
+
+ @Override
+ protected void setUpProviders(JAXRSServerFactoryBean sf) {
+ List<Object> providers = new ArrayList<>();
+ providers.add(new MetadataListMessageBodyWriter());
+ sf.setProviders(providers);
+ }
+
+ /** The default pipes config carries the "presets" block into the worker.
*/
+ @Override
+ protected InputStream getTikaConfigInputStream() {
+ return
getClass().getResourceAsStream("/configs/tika-config-thumbnail-preset.json");
+ }
+
+ @Test
+ public void testPdfKeepsItsTextAndRendersPageOne() throws Exception {
+ List<Metadata> metadataList = rmeta("testPDF_bookmarks.pdf");
+ Metadata pdf = metadataList.get(0);
+ assertEquals(2, (int) pdf.getInt(PagedText.N_PAGES));
+ assertContains("page about Denmark",
pdf.get(TikaCoreProperties.TIKA_CONTENT)); // page 2
+
+ Metadata thumbnail = thumbnail(metadataList);
+ assertEquals("RENDERING",
thumbnail.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ assertEquals(1, (int)
thumbnail.getInt(TikaCoreProperties.EMBEDDED_DEPTH));
+ assertEquals("image/png", thumbnail.get(HttpHeaders.CONTENT_TYPE));
+ assertEquals(pixelsAcross("testPDF_bookmarks.pdf", 96), (int)
thumbnail.getInt(TIFF.IMAGE_WIDTH),
+ "the index sees the 96 dpi image /unpack/preset/thumbnail
returns");
+ assertEquals(1, renderings(metadataList), "only the first page is
rendered");
+ }
+
+ @Test
+ public void testDocxThumbnailIsRasterized() throws Exception {
+ Metadata thumbnail = thumbnail(rmeta("testDOCX_Thumbnail.docx"));
+ assertEquals("THUMBNAIL",
thumbnail.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ assertEquals(2, (int)
thumbnail.getInt(TikaCoreProperties.EMBEDDED_DEPTH));
+ assertNotNull(thumbnail.get(Rendering.RENDERED_BY));
+ assertNotNull(thumbnail.getInt(TIFF.IMAGE_WIDTH));
+ }
+
+ private List<Metadata> rmeta(String file) throws Exception {
+ Response response = WebClient.create(endPoint + PRESET_PATH)
+ .accept("application/json")
+ .put(ClassLoader.getSystemResourceAsStream("test-documents/" +
file));
+ assertEquals(200, response.getStatus(), file);
+ return JsonMetadataList.fromJson(
+ new InputStreamReader((InputStream) response.getEntity(),
UTF_8));
+ }
+
+ /** The documented client rule, as the selector implements it. */
+ private static Metadata thumbnail(List<Metadata> metadataList) {
+ for (Metadata m : metadataList) {
+ if (ThumbnailUnpackSelector.isCandidate(m)) {
+ return m;
+ }
+ }
+ throw new AssertionError("no thumbnail candidate in " +
metadataList.size() + " documents");
+ }
+
+ private static long renderings(List<Metadata> metadataList) {
+ return metadataList.stream()
+ .filter(m ->
"RENDERING".equals(m.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE)))
+ .count();
+ }
+
+ private static int pixelsAcross(String pdf, int dpi) throws Exception {
+ try (PDDocument document = Loader.loadPDF(
+ ClassLoader.getSystemResourceAsStream("test-documents/" +
pdf).readAllBytes())) {
+ // PDFBox floors the scaled width
+ return (int) Math.max(1,
+ Math.floor(document.getPage(0).getMediaBox().getWidth() *
dpi / 72f));
+ }
+ }
+}
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
new file mode 100644
index 0000000000..89fef28133
--- /dev/null
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.server.standard;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.awt.image.BufferedImage;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+import javax.imageio.ImageIO;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import jakarta.ws.rs.core.Response;
+import org.apache.commons.io.FileUtils;
+import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
+import org.apache.cxf.jaxrs.client.WebClient;
+import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;
+import org.apache.pdfbox.Loader;
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import org.apache.tika.serialization.config.JsonConfigHelper;
+import org.apache.tika.server.core.CXFTestBase;
+import org.apache.tika.server.core.TikaServerParseExceptionMapper;
+import org.apache.tika.server.core.resource.UnpackerResource;
+import org.apache.tika.server.core.writer.ZipWriter;
+
+/** The catalog preset {@code thumbnail} unpacks exactly one raster thumbnail
(TIKA-4856). */
+public class ThumbnailPresetTest extends CXFTestBase {
+
+ private static final String PRESET_PATH = "/unpack/preset/thumbnail";
+ private static final String UNPACK_CONFIG_TEMPLATE =
"/configs/cxf-unpack-test-template.json";
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private Path unpackTempDir;
+
+ @Override
+ protected void setUpResources(JAXRSServerFactoryBean sf) {
+ sf.setResourceClasses(UnpackerResource.class);
+ sf.setResourceProvider(UnpackerResource.class,
+ new SingletonResourceProvider(new
UnpackerResource(tikaResource)));
+ }
+
+ @Override
+ protected void setUpProviders(JAXRSServerFactoryBean sf) {
+ List<Object> providers = new ArrayList<>();
+ providers.add(new ZipWriter());
+ providers.add(new TikaServerParseExceptionMapper());
+ sf.setProviders(providers);
+ }
+
+ @Override
+ protected InputStream getTikaConfigInputStream() {
+ return
getClass().getResourceAsStream("/configs/tika-config-thumbnail-preset.json");
+ }
+
+ @Override
+ protected InputStream getPipesConfigInputStream() throws IOException {
+ unpackTempDir = Files.createTempDirectory("tika-unpack-test-");
+ Map<String, Object> replacements = new HashMap<>();
+ replacements.put("UNPACK_EMITTER_BASE_PATH",
unpackTempDir.toAbsolutePath().toString());
+ replacements.put("PLUGINS_PATHS",
+
Paths.get("target/plugins").toAbsolutePath().toString().replace("\\", "/"));
+ replacements.put("TIMEOUT_MILLIS", 60000L);
+ ObjectNode config = (ObjectNode) JsonConfigHelper.loadFromResource(
+ UNPACK_CONFIG_TEMPLATE, CXFTestBase.class, replacements);
+ // the worker resolves the preset name from its own config
+ config.putObject("presets").put("thumbnail", true);
+ return new
ByteArrayInputStream(MAPPER.writeValueAsString(config).getBytes(UTF_8));
+ }
+
+ @Override
+ protected Path getUnpackEmitterBasePath() {
+ return unpackTempDir;
+ }
+
+ @Override
+ @AfterAll
+ public void tearDown() throws Exception {
+ super.tearDown();
+ if (unpackTempDir != null && Files.exists(unpackTempDir)) {
+ FileUtils.deleteDirectory(unpackTempDir.toFile());
+ }
+ }
+
+ @ParameterizedTest
+ @CsvSource({
+ "testDOCX_Thumbnail.docx, png", // EMF thumbnail, rasterized by
the metafile renderer
+ "testPDFTwoTextBoxes.pdf, png", // first page rendering
+ "testMP3_twoCovers.mp3, png" // two stored covers; the first
one wins
+ })
+ public void testExactlyOneThumbnail(String file, String format) throws
Exception {
+ Map<String, byte[]> entries = unpack(PRESET_PATH, file);
+ assertEquals(1, entries.size(), file + " -> " + entries.keySet());
+ Map.Entry<String, byte[]> entry = entries.entrySet().iterator().next();
+ assertTrue(entry.getKey().endsWith("." + format), entry.getKey());
+ assertEquals(format, imageFormat(entry.getValue()), entry.getKey());
+ }
+
+ @Test
+ public void testPdfPageIsRenderedAt96DpiInColour() throws Exception {
+ byte[] png = unpack(PRESET_PATH,
"testPDFTwoTextBoxes.pdf").values().iterator().next();
+ BufferedImage image = ImageIO.read(new ByteArrayInputStream(png));
+ float pageWidthPt;
+ try (PDDocument document =
Loader.loadPDF(fixture("testPDFTwoTextBoxes.pdf").readAllBytes())) {
+ pageWidthPt = document.getPage(0).getMediaBox().getWidth();
+ }
+ // the preset's "rendering" block, not ocr's 300 dpi grayscale; PDFBox
floors the width
+ assertEquals((int) Math.floor(pageWidthPt * 96 / 72f),
image.getWidth());
+ assertEquals(3, image.getColorModel().getNumColorComponents());
+ }
+
+ @Test
+ public void testPlainUnpackKeepsBothCovers() throws Exception {
+ // the preset's selector, not the parser, is what drops the second
cover
+ assertEquals(2, unpack("/unpack", "testMP3_twoCovers.mp3").size());
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "testHTML.html", // no embedded documents at all
+ "test_recursive_embedded.docx" // embedded documents, none of them
a thumbnail
+ })
+ public void testNoThumbnailIs204(String file) throws Exception {
+ assertEquals(204, put(PRESET_PATH, fixture(file)).getStatus(), file);
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "testMP3_twoCovers.mp3", // raster thumbnails at depth 2
+ "testDOCX_Thumbnail.docx" // vector thumbnail at depth 2, its
rendering at depth 3
+ })
+ public void testArchiveHasNoThumbnail(String inner) throws Exception {
+ // the document inside carries a thumbnail; it is that document's, not
the zip's
+ ByteArrayOutputStream zip = new ByteArrayOutputStream();
+ try (ZipOutputStream zos = new ZipOutputStream(zip)) {
+ zos.putNextEntry(new ZipEntry(inner));
+ zos.write(fixture(inner).readAllBytes());
+ zos.closeEntry();
+ }
+ assertEquals(204, put(PRESET_PATH, new
ByteArrayInputStream(zip.toByteArray()))
+ .getStatus(), inner);
+ }
+
+ private Map<String, byte[]> unpack(String path, String file) throws
IOException {
+ Response response = put(path, fixture(file));
+ assertEquals(200, response.getStatus(), file);
+ return readZipArchiveBytes((InputStream) response.getEntity());
+ }
+
+ private Response put(String path, InputStream body) {
+ return WebClient.create(endPoint +
path).accept("application/zip").put(body);
+ }
+
+ private static InputStream fixture(String file) {
+ return ClassLoader.getSystemResourceAsStream("test-documents/" + file);
+ }
+
+ private static String imageFormat(byte[] bytes) {
+ if (bytes.length > 4 && (bytes[0] & 0xFF) == 0x89 && bytes[1] == 'P'
&& bytes[2] == 'N'
+ && bytes[3] == 'G') {
+ return "png";
+ }
+ if (bytes.length > 3 && (bytes[0] & 0xFF) == 0xFF && (bytes[1] & 0xFF)
== 0xD8) {
+ return "jpg";
+ }
+ return "unknown";
+ }
+}
diff --git
a/tika-server/tika-server-standard/src/test/resources/configs/tika-config-thumbnail-preset.json
b/tika-server/tika-server-standard/src/test/resources/configs/tika-config-thumbnail-preset.json
new file mode 100644
index 0000000000..49d111c782
--- /dev/null
+++
b/tika-server/tika-server-standard/src/test/resources/configs/tika-config-thumbnail-preset.json
@@ -0,0 +1,5 @@
+{
+ "presets": {"thumbnail": true, "render-thumbnail": true},
+ "auto-detect-parser": {"throwOnZeroBytes": false},
+ "parse-context": {"timeout-limits": {"progressTimeoutMillis": 60000}}
+}
diff --git
a/tika-server/tika-server-standard/src/test/resources/test-documents/testDOCX_Thumbnail.docx
b/tika-server/tika-server-standard/src/test/resources/test-documents/testDOCX_Thumbnail.docx
new file mode 100644
index 0000000000..85f19c78d0
Binary files /dev/null and
b/tika-server/tika-server-standard/src/test/resources/test-documents/testDOCX_Thumbnail.docx
differ
diff --git
a/tika-server/tika-server-standard/src/test/resources/test-documents/testMP3_twoCovers.mp3
b/tika-server/tika-server-standard/src/test/resources/test-documents/testMP3_twoCovers.mp3
new file mode 100644
index 0000000000..db86376738
Binary files /dev/null and
b/tika-server/tika-server-standard/src/test/resources/test-documents/testMP3_twoCovers.mp3
differ
diff --git
a/tika-server/tika-server-standard/src/test/resources/test-documents/testPDF_bookmarks.pdf
b/tika-server/tika-server-standard/src/test/resources/test-documents/testPDF_bookmarks.pdf
new file mode 100755
index 0000000000..e65be90249
Binary files /dev/null and
b/tika-server/tika-server-standard/src/test/resources/test-documents/testPDF_bookmarks.pdf
differ
