fenil25 opened a new issue, #11818:
URL: https://github.com/apache/iceberg/issues/11818
### Apache Iceberg version
1.7.1 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
I have iceberg kafka-connect sink and when the volume of records in topic is
low, iceberg connector doesn't consume any records.
One would see a constant lag and iceberg will never produce any records and
commit to the table. Once, we have enough records, all the records will be
pushed through and committed to the table.
example -
```
/bin/kafka-consumer-groups.sh --bootstrap-server
cdc-kafka-prod.internal.ec2.tinyspeck.com:9092 --describe --group
cg-control-payments_group
Consumer group 'cg-control-payments_group' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
prod.aux2.payments_group 0 45 46 1
- - -
prod.aux2.payments_group 1 41 42 1
- - -
prod.aux2.payments_group 2 35 36 1
- - -
prod.aux2.payments_group 3 32 33 1
- - -
./bin/kafka-consumer-groups.sh --bootstrap-server
cdc-kafka-prod.internal.ec2.tinyspeck.com:9092 --describe --group
cg-control-payments_group
Consumer group 'cg-control-payments_group' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
prod.aux2.payments_group 0 45 46 1
- - -
prod.aux2.payments_group 1 41 42 1
- - -
prod.aux2.payments_group 2 35 36 1
- - -
prod.aux2.payments_group 3 32 33 1
- - -
```
I am observing this constant lag of 1 for all partition since a day now. No
commit has been made to the append-only table. I confirmed that the record
exists in the topic and is not expired. `fetch.min.bytes` is also set to
default value of 1.
When new records flow through the topic, all the records are committed to
the iceberg table. But, this comes with a lot of unnecessary delay.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
