[ https://issues.apache.org/jira/browse/ARROW-18275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633701#comment-17633701 ]
Miles Granger commented on ARROW-18275: --------------------------------------- Does this overlap with [ARROW-18163 | https://issues.apache.org/jira/browse/ARROW-18163]? To me it sounds like if that were implemented, this could be accomplished by the implementation of the custom dataset format? > [Python] Allow custom reader/writer implementation for arrow dataset > read/write path > ------------------------------------------------------------------------------------ > > Key: ARROW-18275 > URL: https://issues.apache.org/jira/browse/ARROW-18275 > Project: Apache Arrow > Issue Type: New Feature > Components: Python > Affects Versions: 10.0.0 > Reporter: Chang She > Priority: Major > Labels: Python > > We're implementing a "versionable" data format where the read/write path has > some metadata handling which we currently can't plug into the native pyarrow > write_dataset and pa.dataset.dataset mechanism. > What we've done currently is have our own `lance.write_dataset` and > `lance.dataset` interfaces which knows about the versioning. And if you use > the native arrow ones, it reads/writes an unversioned dataset. > It would be great if: > 1. the arrow interfaces provided a way for custom data formats to provide > their own Arrow compliant reader/writer implementations, so we can delete our > custom interface and stick with native pyarrow interface. > 2. the pyarrow interface can support custom kwargs like "version=5" or > "as_of=<timestamp>" or "version='latest'" > for reference, this is what our custom C++ dataset implementation looks like: > https://github.com/eto-ai/lance/blob/main/cpp/include/lance/arrow/dataset.h -- This message was sent by Atlassian Jira (v8.20.10#820010)