pauldheinrichs opened a new issue, #15800:
URL: https://github.com/apache/druid/issues/15800
### Description
Feel free to close this if there is a better place to put this or if I'm
incorrect. But I feel like there are improvements that could be made to the
kinesis ingestion to resolve the main "known issue" under the kinesis druid
docs.
> Before you deploy the Kinesis extension to production, consider the
following known issues:
Avoid implementing more than one Kinesis supervisor that reads from the same
Kinesis stream for ingestion. Kinesis has a per-shard read throughput limit and
having multiple supervisors on the same stream can reduce available read
throughput for an individual supervisor's tasks. Multiple supervisors ingesting
to the same Druid datasource can also cause increased contention for locks on
the datasource.
What does Enhanced fan out do?
> A consumer that uses enhanced fan-out doesn't have to contend with other
consumers who are receiving data from the stream
Source Docs:
- https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
- V1 docs:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesis.html#putRecords-com.amazonaws.services.kinesis.model.PutRecordsRequest-
Dependencies:
- It would appear likely the AWS SDK would need to updated to roughly
around ~1.11.400
https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
---
I have not dug into the kinesis consuming code enough to say what the lift
would be to support this, but thought posting this for visibility might be
worth while? :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
