krickert commented on PR #3451:
URL: https://github.com/apache/tika/pull/3451#issuecomment-6493009529
To be more direct: it's not one per Tika parser. A {{MessageParser}} is
keyed by the incoming protobuf message type, and it's only for users who
already have a typed message (a WARC record, a Kafka record, a reply from their
own service) and want it back as Tika documents. File parsing through
FetchAndParse is unchanged.
The jar ships its own generated stubs and decides how its message maps onto
Tika's document shape. Tika never has to know the message's schema, and nothing
goes through strings.
h4. Sidenote: CEL (not planned in this ticket - not slop!)
For automated mapping later, CEL (Common Expression Language) is the
protobuf-native option. Expressions are type-checked against the message
descriptors and run directly on typed messages, so there's no protobuf to JSON
to protobuf round trip (my LLMs tried to offer JSON hacks often - steer them
away from that it can get quadratic-level slow). The closest comparison is CSS
selectors, but for protobuf fields and faster. There's a Java implementation
(cel-java).
**That's probably too big for this iteration**. A hand-written mapper keeps
the strong typing and is a lot faster than JSON, and we can add mapping and
ETL-style behavior to the v2 shape later (note: with this shape, we can add it
for later w/o needing a new version).
Mappings and transforms aren't really part of tika (at this point) - the
other parsers tend to do their mappings in a similar way. I'm suggesting for
later is an API addition in the future where you can offer this and it'll be
fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
