seokyun-ha-toss opened a new issue, #15030:
URL: https://github.com/apache/iceberg/issues/15030
### Apache Iceberg version
1.9.2
### Query engine
Kafka Connect
### Please describe the bug 🐞
Hello, guys.
When using the Iceberg Sink Connector on Kafka Connect, I occasionally
observe the following error: `CRLF expected at end of chunk`
The connector task fails with an unrecoverable exception when loading table
metadata from the Iceberg REST Catalog.
Restarting the connector resolves the issue and it continues to run normally.
Below is the full stack trace from the failed Kafka Sink Connector:
```
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due
to unrecoverable exception.
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:636)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:345)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:247)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:216)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:226)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)
at
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.iceberg.exceptions.RESTException: Failed to convert
HTTP response body to string
at
org.apache.iceberg.rest.HTTPClient.extractResponseBodyAsString(HTTPClient.java:150)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:295)
at org.apache.iceberg.rest.BaseHTTPClient.get(BaseHTTPClient.java:77)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:373)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:397)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99)
at org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:106)
at
org.apache.iceberg.connect.data.IcebergWriterFactory.createWriter(IcebergWriterFactory.java:59)
at
org.apache.iceberg.connect.data.SinkWriter.lambda$writerForTable$3(SinkWriter.java:139)
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1220)
at
org.apache.iceberg.connect.data.SinkWriter.writerForTable(SinkWriter.java:138)
at
org.apache.iceberg.connect.data.SinkWriter.lambda$routeRecordStatically$1(SinkWriter.java:98)
at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4204)
at
org.apache.iceberg.connect.data.SinkWriter.routeRecordStatically(SinkWriter.java:96)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:85)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.apache.iceberg.connect.data.SinkWriter.save(SinkWriter.java:68)
at org.apache.iceberg.connect.channel.Worker.save(Worker.java:124)
at
org.apache.iceberg.connect.channel.CommitterImpl.save(CommitterImpl.java:151)
at
org.apache.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:83)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:606)
... 11 more
Caused by: org.apache.hc.core5.http.MalformedChunkCodingException: CRLF
expected at end of chunk
at
org.apache.hc.core5.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251)
at
org.apache.hc.core5.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:223)
at
org.apache.hc.core5.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
at
org.apache.hc.core5.http.io.EofSensorInputStream.read(EofSensorInputStream.java:138)
at
java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:242)
at
java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at
org.apache.hc.client5.http.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:71)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.Reader.read(Reader.java:250)
at
org.apache.hc.core5.http.io.entity.EntityUtils.toCharArrayBuffer(EntityUtils.java:188)
at
org.apache.hc.core5.http.io.entity.EntityUtils.toString(EntityUtils.java:231)
at
org.apache.hc.core5.http.io.entity.EntityUtils.toString(EntityUtils.java:292)
at
org.apache.hc.core5.http.io.entity.EntityUtils.toString(EntityUtils.java:253)
at
org.apache.iceberg.rest.HTTPClient.extractResponseBodyAsString(HTTPClient.java:148)
```
My environment is
- Kafka Connect: `3.9.0`
- Kafka Broker `4.0`
- Iceberg Sink Connector `1.9.2`
- Iceberg REST API Catalog
Is this a known issue with the Iceberg REST Catalog client or Apache
HttpClient (chunked + gzip responses)?
Are there recommended configurations or workarounds to avoid this issue?
Thanks, good day
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
