JingsongLi commented on code in PR #10475:
URL: https://github.com/apache/paimon/pull/10475#discussion_r4244796797
##########
paimon-python/pypaimon/common/blob_index_cache.py:
##########
@@ -72,6 +74,36 @@ def put(self, file_path, blob_lengths, blob_offsets):
with self._lock:
self._cache[file_path] = entry
+ def get_or_load(self, file_path, loader):
+ """Share an in-flight load without holding the cache lock during
I/O."""
+ if self.max_size_bytes == 0:
+ return loader()
+ with self._lock:
+ entry = self._cache.get(file_path)
+ if entry is not None:
+ return entry.blob_lengths, entry.blob_offsets
+ pending = self._loading.get(file_path)
+ owner = pending is None
+ if owner:
+ pending = Future()
+ self._loading[file_path] = pending
+
+ if not owner:
+ return pending.result()
Review Comment:
[P1] Reset in-flight loads inherited by forked workers
If a process forks while another parent thread is loading this file, the
child inherits `_loading[file_path]` as an incomplete Future, but the owning
thread exists only in the parent. `pending.result()` then waits forever. I
reproduced with an actual FormatBlobReader cold load: the cache lock was
explicitly free at fork; the parent load completed and cleared its Future
before the child read; the child PR reader still timed out, while the baseline
reader completed against the same inherited cache and physical BLOB file.
This is reachable through fork-based Torch workers:
TableRead.to_torch_dataset retains the reader, TorchIterDataset.__iter__ reuses
it, and the dataset's PID handling updates only distributed rank, not the
catalog BLOB cache. `__reduce__` does not run on fork inheritance. Please reset
pending loads and synchronization state after a process change/after fork
before waiting, and add a bounded fork regression alongside the pickle test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
