milton4code opened a new issue, #373:
URL: https://github.com/apache/doris-spark-connector/issues/373
### Component
Spark Doris Connector / Stream Load
### Environment
- Doris Spark Connector: `26.0.0`
- Spark: `3.2.3`
- Scala: `2.12`
- Java: `8`
- Deployment: Spark on YARN, client mode
- Stream Load body: `PipedInputStream` / `InputStreamEntity`
- Transfer mode: `Transfer-Encoding: chunked`
- `doris.sink.auto-redirect`: `false`
### Description
`AbstractStreamLoadProcessor.addCommonHeaders` unconditionally adds:
```java
req.setHeader(HttpHeaders.EXPECT, "100-continue");
```
The request body uses `PipedInputStream` and `InputStreamEntity`. Since its
length is unknown, HttpClient sends it with chunked transfer encoding. In our
Doris environment, the resulting request fails before the Stream Load body is
processed.
With the connector default header, the write fails with:
```text
StreamLoadException: stream load failed, status: 413, reason: Request Entity
Too Large
```
This is reproducible with only 5 records, so it is unrelated to the actual
payload size or `sink.batch.size`.
Trying to disable the header through Stream Load properties does not work:
```scala
.option("doris.sink.properties.expect", "")
```
Empty, blank, or placeholder values still leave an `Expect` header in the
request because `handleStreamLoadProperties` calls `HttpRequestBase.setHeader`.
The server then returns:
```text
StreamLoadException: stream load failed, status: 417, reason: Expectation
Failed
```
Calling `setHeader("Expect", "")` is not equivalent to calling
`removeHeaders(HttpHeaders.EXPECT)`. There is currently no connector option
that completely removes this hardcoded header.
### Steps to reproduce
```scala
df.write
.format("org.apache.doris.spark.sql.sources.DorisDataSource")
.option("doris.table.identifier", "database.table")
.option("doris.fenodes", "fe-host:http-port")
.option("user", "user")
.option("password", "password")
.option("doris.sink.auto-redirect", "false")
.option("sink.properties.format", "csv")
.mode("append")
.save()
```
### Verified workaround
We patched Connector 26.0.0 locally and changed:
```java
req.setHeader(HttpHeaders.EXPECT, "100-continue");
```
to:
```java
req.removeHeaders(HttpHeaders.EXPECT);
```
We removed all `doris.sink.properties.expect` options and retained
`doris.sink.auto-redirect=false`. The same Spark job, server, table, and data
then completed successfully. This was verified end to end in the server
environment.
### Expected behavior
The connector should provide an official, typed option for enabling or
disabling `Expect: 100-continue`, for example:
```text
doris.sink.http.expect-continue=true|false
```
For backward compatibility, the default may remain `true`. When disabled,
the connector should completely remove the header rather than set an empty
value:
```java
if (config.getValue(DorisOptions.DORIS_SINK_HTTP_EXPECT_CONTINUE)) {
req.setHeader(HttpHeaders.EXPECT, "100-continue");
} else {
req.removeHeaders(HttpHeaders.EXPECT);
}
```
Tests should verify both the default header and the complete absence of the
header when disabled.
### Additional context
A streaming `InputStreamEntity` is non-repeatable, so relying on an FE
redirect is also unsafe. We use `doris.sink.auto-redirect=false` to connect
directly to a BE.
I can submit a PR with the proposed configuration option and tests if the
maintainers agree with this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
