As part of the proposal <https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion> to bypass deep store for segment completion, I plan to add a new optional string field *peerSegmentDownloadScheme* to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be *http* or *https*.
1. SplitSegmentCommitter <https://github.com/apache/incubator-pinot/blob/31c55afdb6a40f98189308ce6292587ead9d0dec/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SplitSegmentCommitter.java> will check this value. If it exists, the segment committer will be able to finish segment commit successfully even if the upload to the segment store fails. The committer will report a special marker to the controller about the segment is available in peer servers. 2. When Pinot servers fail to download segments from the segment store, they can also check this field's value. If it exists, it can download segments from peer servers using either HTTP and HTTPS segment fetchers as configured. (related PR <https://github.com/apache/incubator-pinot/pull/5336> in review for how to discover such servers.) Note this is a table level config. We will test the new download behavior in realtime tables in incremental fashion. Once fully proven, this config can be upgraded to server level config. Please let me know if you have any questions on this. Thanks @mcvsubbu for coming up with the idea and offline discussions. Ting Chen
