[
https://issues.apache.org/jira/browse/TIKA-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677326#comment-17677326
]
Nick Burch commented on TIKA-3703:
----------------------------------
A zip file gives you compression, and most clients won't accidentally try to
buffer it in memory. JSON with base-64 encoded data is negative compression,
and a high risk of clients OOM-ing due to trying to fit all of the raw JSON and
parsed JSON in memory at once
(If it was just thumbnails then I could see some advantages of JSON, but it
also works on container formats with potentially huge contents)
In terms of recursion, I think it should be off on the default endpoint (as
now), but with another that supports it. Maybe eg {{/unpack}} and
{{/unpack/recursive}} ?
> Consider adding a frictionless data package output format
> ---------------------------------------------------------
>
> Key: TIKA-3703
> URL: https://issues.apache.org/jira/browse/TIKA-3703
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> For those who want more than just text and metadata, e.g. bytes for
> thumbnails, or embedded images or embedded files or rendered pages, it would
> be great to return that data in a standard format. Our current /unpack
> endpoint uses a zip file but with our own "standard".
> I was thinking about heading down the pure json option by including these
> byte streams as base64 encoded metadata values in our current metadata
> object. Not sure which is the better way to go.
> I'm opening this issue to discuss options.
>
> Reference: [https://frictionlessdata.io/standards/#standards-toolkit]
> We'd want to make this available as an endpoint on tika-server
> (\{{/v2/unpack}} or something else?) and as a commandline option in tika-app.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
