No, I mean the follwing: (1) Segment download URI in metadata will have either a valid URI (from deepstore) OR be empty. (2) In case it is empty, we construct the URIs of all selected peers and pass it to the segment fetcher (3) We add a new method to the SegmentFetcher interface that takes a list of URIs instead of a single URI (4) We modify the retry logic in the base class to pick a random one from the list (even if the list size is 1). (5) default implementation for the list could be to take a random URI (or the first one, or whatever) from the list and call the existing method of one URI
=Subbu On 2020/06/11 22:09:05, TING CHEN <tingc...@uber.com.INVALID> wrote: > You mean multiple URIs in a segment's download url. No for this project. > > On Thu, Jun 11, 2020 at 2:59 PM kishore g <g.kish...@gmail.com> wrote: > > > +1 peer. > > > > unrelated to this - do we support multiple URI's? > > > > On Thu, Jun 11, 2020 at 2:51 PM Subbu Subramaniam <mcvsu...@apache.org> > > wrote: > > > > > Hey Ting, > > > > > > I like the URI in metadata as "peer:///segmentName". This way, the URI > > > remains parsable and we can use the scheme to check for a segment > > fetcher, > > > > > > thanks > > > > > > -Subbu > > > > > > On 2020/06/10 01:09:25, TING CHEN <tingc...@uber.com.INVALID> wrote: > > > > As part of the deep store by-passing > > > > < > > > > > https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion > > > > > > > > project, a server is allowed to download segments from peer Pinot > > servers > > > > during Low Level Consumer (LLC) instead of deep segment stores. To > > enable > > > > this feature, we plan to add a new URI format for the field > > > > *segment.realtime.download.url > > > > *in LLCRealtimeSegmentZKMetadata. > > > > > > > > The new URI format serves the purpose of instructing a Pinot server to > > > find > > > > and download the segment from a peer server. Controller writes it to > > > Helix > > > > in case of segment upload failure or no deep store configured at all. > > We > > > > proposed the following format options and want to hear your feedback: > > > > > > > > 1. peer:///segmentName; (my preference) > > > > 2. simply an empty string *''* > > > > > > > > Both are in essence specially markers to indicate that the segment is > > not > > > > found in deep store and servers have to download them from peer > > servers. > > > > (1) has the benefit of better readability than (2) for debugging > > > purposes. > > > > > > > > Please let me know what you think. > > > > > > > > Ting Chen > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org > > > For additional commands, e-mail: dev-h...@pinot.apache.org > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
