bhavya-sl commented on issue #55325:
URL: https://github.com/apache/arrow/issues/55325#issuecomment-6569992338
Thanks for picking this up, @nikhilred2619.
My preference is (a), accepting the metadata at construction:
```
fmt.make_fragment(path, filesystem, partition_expression, row_groups,
file_size=size, metadata=file_metadata)
```
Why (a) over (b):
- `EnsureCompleteMetadata()` returns early when the fragment already has
metadata. So (b) would either silently ignore the argument, or need new replace
semantics on a fragment that a running scan may be using. With (a) the fragment
is complete from the start.
- It exposes what `ParquetDatasetFactory` already does: `MakeFragment()`
followed by `SetMetadata()`, before the fragment is handed out.
`ParquetFileFormat` is already a friend of `ParquetFileFragment`, so a
`MakeFragment` overload that takes a `FileMetaData` can call `SetMetadata()`
directly. Nothing is opened and there is no I/O at construction.
- A fragment can't drop its metadata once attached. Attaching to long-lived
fragments therefore only grows memory. With (a), a caller that needs bounded
memory builds short-lived fragments for the files one query touches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
