arunkumarucet opened a new pull request, #17633:
URL: https://github.com/apache/pinot/pull/17633
## Summary
- Adds support for Kafka 4.x clients with a new `pinot-kafka-4.0` module
- **Deprecates Kafka 2.0** - all Kafka 2.0 consumer factory references now
redirect to Kafka 3.0
- Introduces Kafka 4.1.1 client dependency
## Changes
### Kafka 2.0 Deprecation
- Redirect handled through PluginManager.java
- Users specifying Kafka 2.0 consumer factory classes will automatically use
Kafka 3.0:
- `org.apache.pinot.core.realtime.impl.kafka2.KafkaConsumerFactory` →
`kafka30.KafkaConsumerFactory`
- `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` →
`kafka30.KafkaConsumerFactory`
- No configuration changes required - existing tables using Kafka 2.0 will
seamlessly use Kafka 3.0
### New Module: `pinot-kafka-4.0`
- Created new stream ingestion module for Kafka 4.x compatibility
- Implements `KafkaConsumerFactory`, `KafkaPartitionLevelConsumer`,
`KafkaStreamMetadataProvider`, and related classes
- Uses Testcontainers for unit testing (gracefully skips when Docker
unavailable)
### Backward Compatibility Mappings (PluginManager)
| User Config | Actual Class Used |
|-------------|-------------------|
| `kafka20.KafkaConsumerFactory` | `kafka30.KafkaConsumerFactory` |
| `kafka2.KafkaConsumerFactory` (old) | `kafka30.KafkaConsumerFactory` |
| `kafka3.KafkaConsumerFactory` (old) | `kafka30.KafkaConsumerFactory` |
| `kafka4.KafkaConsumerFactory` (old) | `kafka40.KafkaConsumerFactory` |
### Kafka 4.x Key Changes
- Kafka 4.x removed ZooKeeper dependency entirely (KRaft-only)
- Kafka 4.x removed Scala dependency (pure Java clients)
- Updated `ConsumerRecord` API usage for Kafka 4.x compatibility
### Testing
- Added `LLCRealtimeKafka4ClusterIntegrationTest` for end-to-end validation
- Unit tests use Testcontainers with Docker skip logic for environments
without Docker
- All existing Kafka 3.0 tests continue to pass
## Test plan
- [x] Unit tests pass (non-Docker tests)
- [x] `LLCRealtimeKafka4ClusterIntegrationTest` passes (16 tests)
- [x] Existing Kafka 3.0 integration tests unaffected
- [x] Build compiles successfully
## Configuration
To use Kafka 4.x, specify in your table config:
`"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory"
`
## Migration Notes
- Kafka 2.0 users: No action required. Your existing configuration will
automatically use Kafka 3.0.
- Kafka 3.0 users: No changes needed.
- Kafka 4.x users: Update your consumer factory class to
kafka40.KafkaConsumerFactory.
## Notes
- Kafka 4.x clients are backward compatible with Kafka 3.x brokers
- The integration tests use the existing Kafka 2.x embedded server (client
compatibility verified)
- Unit tests requiring Docker will skip gracefully on machines without Docker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
