chovy-3012 commented on issue #13262: URL: https://github.com/apache/seatunnel/issues/13262#issuecomment-6857821831
Thanks @SEZ9 for the detailed scope questions! Here is the planned scope for the first version: 1. **Batch-only**: the first version is a bounded batch source. The dataset will be split into one split per fragment and assigned round-robin across readers; pending splits stay in checkpoint state, so a failover only reassigns the unread fragments. 2. **Column projection and dataset version selection**: neither is planned for v1 — each split scans all columns of its fragment, and the dataset is always opened at its latest version. lance-core 0.33.0 already exposes the knobs (a `ScanOptions` column subset, a `ReadOptions` version), so both are natural follow-ups. 3. **Filter push-down**: out of scope for v1, for the same reason. 4. **SDK version**: lance-core 0.33.0, the version the existing sink already ships — source and sink share one client and no dependency change is needed. 5. **Type mapping** (the reverse of the sink's `SchemaUtils`): scalar types map 1:1 — int(32)/int(64) → int/bigint, float → float, double → double, Utf8/LargeUtf8 → string, Bool → boolean, Date → date, Time → time, Decimal → decimal with the dataset's precision and scale, Binary/LargeBinary → bytes. Timestamp maps to timestamp, and timezone-aware columns are converted through the timezone declared in the dataset schema. On the types you specifically called out: list → `ARRAY` with the element type derived from the list's child field (a list without an element field is rejected with a clear error); vector (Arrow `FixedSizeBinary`) has no faithful SeaTunnel counterpart, so it is rejected up front with an error naming the offending field — fail fast rather than silently coercing to bytes; blob is not specially handled in v1 (a column that surfaces as plain binary reads as bytes, anything else falls into the same fail-fast rejection) — happy to make that an e xplicit map-or-reject decision if you prefer. The PR I'm preparing will follow the Connector-V2 source conventions and reuse the sink's storage/credential options so both ends share one configuration: - **Config consistency**: the source will add `lance.read.storage.options` with the same keys as the sink's `lance.write.storage.options` (`aws_endpoint_url`, `aws_access_key_id`, `aws_secret_access_key`, `allow_http`, ...). Every `Dataset.open` in the source will pass these via `ReadOptions`, so `s3://` datasets can authenticate purely from the job config, with no `AWS_*` environment variables. - **Docs**: a new source page in EN and ZH with the options table, type mapping and examples, and the sink page's "there is no Lance source connector" sentence will link to the new source page. One finding from verifying the existing sink against MinIO: `LanceSinkWriter` passes storage options only to `Dataset.create` — its `Dataset.open` calls (initial open, post-create and post-append reopen) and the `Fragment.create` inside `FragmentConverter` never receive them, so `s3://` targets fail with `CredentialsNotLoaded` even when `lance.write.storage.options` is set, and writing to object storage currently only works through `AWS_*` environment variables. Since the source shares the same credential design, I plan to fix these paths in the same PR — happy to split them into a separate fix PR if you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
