[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094375#comment-18094375
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on code in PR #2921:
URL: https://github.com/apache/tika/pull/2921#discussion_r3538982722
##########
tika-grpc-mapper/src/main/java/org/apache/tika/grpc/mapper/transform/PdfDocumentTransformer.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.grpc.mapper.transform;
+
+import java.util.Locale;
+import java.util.Set;
+
+import org.apache.tika.grpc.v1.Document;
+import org.apache.tika.grpc.v1.DocumentMetadata;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.Office;
+import org.apache.tika.metadata.PagedText;
+import org.apache.tika.metadata.TikaCoreProperties;
+
+/**
+ * PDF. Maps the common, cross-format facts into typed DocumentMetadata.
+ *
+ * PDF-specific properties (encryption flags, XMP, permissions, incremental
updates, ...)
+ * are NOT given their own proto fields. They flow into the tagged tail, typed
where Tika
+ * declares the type and string otherwise. That is the whole point: format
richness lives
+ * in code plus the tail, not in the wire contract - so the schema stays small
and stable
+ * and clients never rebuild when we add or change a PDF property.
+ */
+public final class PdfDocumentTransformer implements DocumentTransformer {
+
+ @Override
+ public boolean appliesTo(Metadata tika) {
+ String contentType = tika.get(Metadata.CONTENT_TYPE);
+ return contentType != null &&
contentType.toLowerCase(Locale.ROOT).contains("pdf");
+ }
+
+ @Override
+ public void transform(Metadata tika, Document.Builder document,
Set<String> consumed) {
+ DocumentMetadata.Builder meta = document.getMetadataBuilder();
+
+ // Common, cross-format fields -> typed (a date is a Timestamp, a
count is an int).
+ TransformSupport.setString(tika, TikaCoreProperties.TITLE,
meta::setTitle, consumed);
Review Comment:
Done. Factored into `TransformSupport.mapCommonFields(tika, meta,
consumed)`, with an overload taking the keywords `Property` since Office/RTF
use `Office.KEYWORDS` instead of `TikaCoreProperties.SUBJECT`. Six transformers
now call it. `ImageDocumentTransformer` intentionally doesn't (its fields come
from IPTC/EXIF, not the core properties) and has a comment saying so.
##########
tika-grpc/src/main/java/org/apache/tika/pipes/grpc/TikaGrpcServerImpl.java:
##########
@@ -312,16 +318,21 @@ private void fetchAndParseImpl(FetchAndParseRequest
request,
if
(pipesResult.status().equals(PipesResult.RESULT_STATUS.FETCH_EXCEPTION)) {
fetchReplyBuilder.setErrorMessage(pipesResult.message());
}
+ long parseTimeMs = (System.nanoTime() - parseStart) / 1_000_000L;
Review Comment:
Went with the rename, since fetch and parse happen together inside the
forked pipes worker and aren't separable from the server's vantage point. The
proto field is now `fetch_parse_time_ms` with a comment spelling out what's
measured, and the server/mapper/tests are renamed to match. Safe to rename
rather than reserve the tag since nothing has shipped with this proto yet.
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
