tokoko commented on code in PR #4317:
URL: https://github.com/apache/arrow-adbc/pull/4317#discussion_r4211485401
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1516,6 +1518,23 @@ struct ADBC_EXPORT AdbcDriver {
AdbcStatusCode (*StatementExecuteMulti)(struct AdbcStatement*,
struct AdbcMultiResultSet*, struct
AdbcError*);
+ AdbcStatusCode (*ConnectionBeginIngestPartitions)(struct AdbcConnection*,
+ struct ArrowSchema*,
+ struct
AdbcSerializableHandle*,
+ struct AdbcError*);
+ AdbcStatusCode (*ConnectionWriteIngestPartition)(struct AdbcConnection*,
const uint8_t*,
+ size_t, struct
ArrowArrayStream*,
+ struct
AdbcSerializableHandle*,
+ struct AdbcError*);
+ AdbcStatusCode (*ConnectionCompleteIngestPartitions)(struct AdbcConnection*,
Review Comment:
@lidavidm I went with an out parameter as discussed here, but while mapping
this to table formats I found that two states (retryable / terminal) aren't
enough, so I ended up making a new outcome code rather than a boolean: `int*
outcome` on `AdbcStatementCompleteIngestPartitions` with the possible values of:
- `ADBC_INGEST_COMPLETE_SUCCEEDED` – set iff the call returns
`ADBC_STATUS_OK` (kinda redundant but a little cleaner)
- `ADBC_INGEST_COMPLETE_FAILED` – nothing was promoted and the ingest can't
be completed; abort and start over
- `ADBC_INGEST_COMPLETE_RETRYABLE` – nothing was promoted; call Complete
again with the same handle and receipts
- `ADBC_INGEST_COMPLETE_UNKNOWN` – the driver doesn't know whether the
commit landed; caller must *not* abort (a handle-scoped cleanup could delete
committed data) and should call Complete again. A driver that reports this has
to be able to resolve it on the repeated call without promoting the writes
twice.
The third state comes straight from Iceberg. A rough mapping of iceberg
exceptions to adbc complete states:
- `CommitFailedException` maps to `RETRYABLE`. This is often internally
retried by clients, but can surface if retries are exhausted.
- `ValidationException`, `NoSuchTableException`, `ForbiddenException` and
others map to `FAILED`. Most of these are more likely to show up during Begin
rather than Complete, but still a possibility.
- `CommitStateUnknownException` maps to `UNKNOWN` - basically driver has no
way of knowing whether promote worked and calling abort is dangerous.
The key difference between `RETRYABLE` and `UNKNOWN` is that abort is not
safe to be called for `UNKNOWN`. For example when `CommitStateUnknownException`
is surfaced, Iceberg's Spark writer fails the job and skips file cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
