Our current design <https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion> does not add a new pinotFS for *peer://*. This is a very interesting question though. The only operations our design uses today for peer "FS" is essentially *copyToLocal()* in the pinotFS interface. Our design basically has a class and a few supporting methods <https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#BypassingdeepstorerequirementforRealtimesegmentcompletion-EnablebesteffortsegmentuploadinSplitSegmentCommiteranddownloadsegmentfrompeerservers.> implementing the above method. That is why we do not add a full-fledged pinotFS subclass for this design.
On Thu, Jun 11, 2020 at 3:00 PM kishore g <[email protected]> wrote: > Also, will peer:// have an implementation of pinotFS ? > > On Thu, Jun 11, 2020 at 2:58 PM kishore g <[email protected]> wrote: > > > +1 peer. > > > > unrelated to this - do we support multiple URI's? > > > > On Thu, Jun 11, 2020 at 2:51 PM Subbu Subramaniam <[email protected]> > > wrote: > > > >> Hey Ting, > >> > >> I like the URI in metadata as "peer:///segmentName". This way, the URI > >> remains parsable and we can use the scheme to check for a segment > fetcher, > >> > >> thanks > >> > >> -Subbu > >> > >> On 2020/06/10 01:09:25, TING CHEN <[email protected]> wrote: > >> > As part of the deep store by-passing > >> > < > >> > https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion > >> > > >> > project, a server is allowed to download segments from peer Pinot > >> servers > >> > during Low Level Consumer (LLC) instead of deep segment stores. To > >> enable > >> > this feature, we plan to add a new URI format for the field > >> > *segment.realtime.download.url > >> > *in LLCRealtimeSegmentZKMetadata. > >> > > >> > The new URI format serves the purpose of instructing a Pinot server to > >> find > >> > and download the segment from a peer server. Controller writes it to > >> Helix > >> > in case of segment upload failure or no deep store configured at all. > We > >> > proposed the following format options and want to hear your feedback: > >> > > >> > 1. peer:///segmentName; (my preference) > >> > 2. simply an empty string *''* > >> > > >> > Both are in essence specially markers to indicate that the segment is > >> not > >> > found in deep store and servers have to download them from peer > servers. > >> > (1) has the benefit of better readability than (2) for debugging > >> purposes. > >> > > >> > Please let me know what you think. > >> > > >> > Ting Chen > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> >
