SEZ9 commented on issue #13262:
URL: https://github.com/apache/seatunnel/issues/13262#issuecomment-6870114711
@chovy-3012 thanks for laying out the v1 scope so clearly.
Quick reactions, point by point:
1. **Batch-only, one split per fragment, pending splits in checkpoint
state** — sounds good; that is the right granularity for a bounded source and
keeps failover simple.
2./3. **No column projection, version selection or filter push-down in v1**
— fine with me. Please make sure the source doc states these limitations
explicitly so users aren't surprised, and note them as follow-ups in the PR
description.
4. **lance-core 0.33.0 shared with the sink** — good, one client version for
both ends is what we want.
5. **Type mapping** — the scalar table looks correct. Mapping list to
`ARRAY` with the element type from the child field is right, and I agree with
fail-fast on `FixedSizeBinary` vectors rather than coercing to bytes. For blob,
yes, please make it an explicit decision rather than whatever falls through:
either map the plain-binary case to bytes and document it, or reject with the
same field-naming error. Either is acceptable for v1 as long as the behavior is
deterministic, documented, and the error names the offending column.
On config consistency: mirroring the sink's `lance.write.storage.options`
keys as `lance.read.storage.options` and passing them via `ReadOptions` on
every `Dataset.open` is exactly the approach I'd like, so `s3://` datasets work
from job config alone without `AWS_*` environment variables.
Your comment appears to have been cut off at the docs section ("a new source
page in EN"). Could you post the rest? In particular, please confirm the source
documentation will cover both EN and ZH and include the unsupported-types /
no-projection / latest-version-only notes above.
Please link the PR here once it's open so we can continue the review there.
Looking forward to it.
<!-- streview-comment:2330 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]