zhangshenghang commented on issue #10196:
URL: https://github.com/apache/seatunnel/issues/10196#issuecomment-3665181553
> A much-needed feature,
>
> 1. May I ask if you have considered the possible errors when reading the
Source? For example, when reading data from a certain shard, if the shard is
damaged, the reading of the entire table may get stuck halfway;
> 2. How to implement Error Classification? This seems to be related to
different sinks.
> 3. I feel that the logic of the Queue Overflow Policy may not be
applicable in many scenarios. Generally, when we configure the ROUTE, we will
select a specific sink based on the original sink situation. For the second
type of overflow situation, I think it is also relatively difficult to judge.
1. **Source-side errors (e.g., corrupted shards)**
In the current version, **“row-level error handling” only applies to the
Transform / Sink stages**.
If the Source fails to read a shard (e.g., shard corruption, connection
failure), it is still treated as a **system-level error**: the job fails
immediately, and there is **no row-level bypass**.
2. **How errors are classified**
This is determined jointly by the **engine framework** and the **connector
implementation**:
* **Transform**: uses a wrapper class + (optional) `SupportRowLevelError` to
determine whether an error is row-level.
* **Sink**: only connectors that implement `SupportRowLevelError` will be
handled as row-level errors; otherwise, everything is treated as a system-level
error and the job fails.
* **JDBC**: classification is based on the `SQLState` of `SQLException`:
* `22xxx` / `23xxx` → row-level data / constraint errors
* others → system-level errors
3. **Positioning of Queue Overflow Policy**
It only applies to the **internal buffering queue of the error sink**, not
the main sink:
* `FAIL`: fail the job when the queue is full
* `DROP`: drop new errors when the queue is full
* `BLOCK`: block the error-producing thread when the queue is full
The engine does not distinguish between “too many errors” and “the error
sink itself is slow / malfunctioning”—both are treated as “the error side can’t
keep up.” You can choose a strategy that fits your business by combining:
* `max_error_records` / `max_error_ratio` +
* `queue_capacity` / `queue_overflow_policy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
