pradeep-jangid opened a new issue, #16226:
URL: https://github.com/apache/pinot/issues/16226
Hi team, I’m facing an issue with real-time ingestion in Pinot where
messages published to a specific Kafka partition are not being ingested, even
though:
The same message works fine when sent to another partition
I'm using a single Pinot real-time server, so there's no distribution across
multiple instances
Pinot shows `CONSUMING` status for all partitions, including the problematic
one
Prometheus metrics `pinot_server_realtimeRowsConsumed_Count` increases,
indicating the server is reading Kafka messages
But the message does not show up in the Pinot table
No errors are visible in Pinot server logs (decode, index, null, etc.)
Environment & Setup:
Kafka topic: 24 partitions
Pinot ingestion config:
{
"streamType": "kafka",
"stream.kafka.decoder.prop.format": "JSON",
"stream.kafka.decoder.class.name":
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.type": "LOWLEVEL"
}
Segment status is IN_PROGRESS for the problematic partition
Schema is validated (field types are correct)
What I’ve Tried:
Restarted Pinot server
Verified schema consistency
Verified ingestion metrics are increasing
This happens only when the message lands in specific partition. Rest other
partitions ingest successfully.
Please let me know what else I can check.
Thanks a lot in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
