AmatyaAvadhanula commented on PR #12792:
URL: https://github.com/apache/druid/pull/12792#issuecomment-1200761411
> Can you also add some Integration test (See integration test group
"kinesis-index")?
Added a few tests where empty shards are present
> Are we still handling the case when shardIterator returned by
kinesis.getShardIterator is null?
Yes, an end of shard marker is still being returned. If this doesn't happen
immediately, the task will handle it appropriately after assignment.
> Is the timeoutMillis still needed? Is the timeoutMillis use for handling
when shards are temporary closed due to SplitShard or MergeShards operation?
Could you please clarify what timeoutMillis is?
If you mean `fetchSequenceNumberTimeout`, it is the time that we poll
kinesis for records before determining that it is empty. This was needed as we
had to get at least one record before beginning ingestion and kinesis may not
return records at TRIM_HORIZON or LATEST in the first few iterations depending
on when data was added.
It isn't needed now as the new marker is returned and the task will handle
it appropriately.
This prevents multiple retries to fetch records and makes it faster as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
