> There is a word doc with no implementation or PR. I think there could be an implementation / PR.
In the word doc there is a link to a POC implementation[1] showing this protocol working with a flight service, ucx and libcudf. The key piece here is that we're voting on adopting this protocol spec (i.e. I'll add it to the documentation website) rather than us explicitly providing full implementations or abstractions around it. We can provide reference implementations like the POC, but I don't think they should be in the Arrow monorepo or else we run the risk of a lot of the same issues that Flight has: i.e. Adding anything to Flight in C++ requires fully wrapping the grpc/flight primitives with Arrow equivalents to export which increases the maintenance burden on us and makes it more difficult for users to leverage the underlying knobs and dials. > For example, does any ADBC client respect this protocol today? If a flight server responds with an S3/HTTP URI will the ADBC client download the files from the correct place? Will it at least notice that the URI is not a GRPC URI and give a "I don't have a connector for downloading from HTTP/S3" error? I've split the S3/HTTP URI flight pieces out into a separate document and separate thing to vote on at the request of several people who wanted to view these as two separate proposals to vote on. So this vote *only* covers adopting the protocol spec as an "Experimental Protocol" so we can start seeing real world usage to help refine and improve it. That said, I believe all clients currently would reject any non-grpc URI. > I was speaking with someone yesterday and they explained that they ended up not choosing Flight for an internal project because Flight didn't support something called "cloud fetch" which I have now learned is I was reading through that link, and it seems like it's pretty much *identical* to Flight as it currently exists, except that it is using cloud storage (S3, GCS, etc.) URIs containing Arrow IPC *files*, rather than a service sitting in front of those serving up Arrow IPC *streams*. Which has been requested by others in the community, hence the second proposal that was split out [2]. > So a big +1 for the idea of disassociated transports but I'm not sure why we need a vote to start working on it (but I'm not opposed if a vote helps) Mostly I found that the google doc was easier for iterating on the protocol specification than a markdown PR for the Arrow documentation as I could more visually express things without a preview of the rendered markdown. If it would get people to be more likely to vote on this, I can write up the documentation markdown now and create a PR rather than waiting until we decide we're even going to adopt this protocol as an "official" arrow protocol. Lemme know if there's any other unanswered questions! --Matt [1]: https://github.com/zeroshade/cudf-flight-ucx [2]: https://docs.google.com/document/d/1-x7tHWDzpbgmsjtTUnVXeEO4b7vMWDHTu-lzxlK9_hE/edit#heading=h.ub6lgn7s75tq On Thu, Mar 28, 2024 at 4:53 PM Weston Pace <weston.p...@gmail.com> wrote: > I'm sorry for the very late reply. Until yesterday I had no real concept > of what this was talking about and so I had stayed out. > > I'm +0 only because it isn't clear what we are voting on. There is a word > doc with no implementation or PR. I think there could be an implementation > / PR. For example, does any ADBC client respect this protocol today? If a > flight server responds with an S3/HTTP URI will the ADBC client download > the files from the correct place? Will it at least notice that the URI is > not a GRPC URI and give a "I don't have a connector for downloading from > HTTP/S3" error? In general, I think we do want this in Flight (see > comments below) and I am very supportive of the idea. However, if adopting > this as an experimental proposal helps move this forward then I think > that's fine. > > That being said, I do want to express support for the proposal as a > concept, at least the "disassociated transports" portion (I can't speak to > UCX/etc.). I was speaking with someone yesterday and they explained that > they ended up not choosing Flight for an internal project because Flight > didn't support something called "cloud fetch" which I have now learned is > [1]. I had recalled looking at this proposal before and this person seemed > interested and optimistic to know this was being considered for Flight. > This proposal, as I understand it, should make it possible for cloud > servers to support a cloud fetch style API. From the discussion I got the > impression that this cloud fetch approach is useful and generally > applicable. > > So a big +1 for the idea of disassociated transports but I'm not sure why > we need a vote to start working on it (but I'm not opposed if a vote helps) > > [1] > > https://www.databricks.com/blog/2021/08/11/how-we-achieved-high-bandwidth-connectivity-with-bi-tools.html > > On Thu, Mar 28, 2024 at 1:04 PM Matt Topol <zotthewiz...@gmail.com> wrote: > > > I'll keep this new vote open for at least the next 72 hours. As before > > please reply with: > > > > [ ] +1 Accept this Proposal > > [ ] +0 > > [ ] -1 Do not accept this proposal because... > > > > Thanks everyone! > > > > On Wed, Mar 27, 2024 at 7:51 PM Benjamin Kietzman <bengil...@gmail.com> > > wrote: > > > > > +1 > > > > > > On Tue, Mar 26, 2024, 18:36 Matt Topol <zotthewiz...@gmail.com> wrote: > > > > > > > Should I start a new thread for a new vote? Or repeat the original > vote > > > > email here? > > > > > > > > Just asking since there hasn't been any responses so far. > > > > > > > > --Matt > > > > > > > > On Thu, Mar 21, 2024 at 11:46 AM Matt Topol <zotthewiz...@gmail.com> > > > > wrote: > > > > > > > > > Absolutely, it will be marked experimental until we see some people > > > using > > > > > it and can get more real-world feedback. > > > > > > > > > > There's also already a couple things that will be followed-up on > > after > > > > the > > > > > initial adoption for expansion which were discussed in the > comments. > > > > > > > > > > On Thu, Mar 21, 2024, 11:42 AM David Li <lidav...@apache.org> > wrote: > > > > > > > > > >> I think let's try again. Would it be reasonable to declare this > > > > >> 'experimental' for the time being, just as we did with > Flight/Flight > > > > >> SQL/etc? > > > > >> > > > > >> On Tue, Mar 19, 2024, at 15:24, Matt Topol wrote: > > > > >> > Hey All, It's been another month and we've gotten a whole bunch > of > > > > >> feedback > > > > >> > and engagement on the document from a variety of individuals. > > Myself > > > > >> and a > > > > >> > few others have proactively attempted to reach out to as many > > third > > > > >> parties > > > > >> > as we could, hoping to pull more engagement also. While it would > > be > > > > >> great > > > > >> > to get even more feedback, the comments have slowed down and we > > > > haven't > > > > >> > gotten anything in a few days at this point. > > > > >> > > > > > >> > If there's no objections, I'd like to try to open up for voting > > > again > > > > to > > > > >> > officially adopt this as a protocol to add to our docs. > > > > >> > > > > > >> > Thanks all! > > > > >> > > > > > >> > --Matt > > > > >> > > > > > >> > On Sat, Mar 2, 2024 at 6:43 PM Paul Whalen <pgwha...@gmail.com> > > > > wrote: > > > > >> > > > > > >> >> Agreed that it makes sense not to focus on in-place updating > for > > > this > > > > >> >> proposal. I’m not even sure it’s a great fit as a “general > > > purpose” > > > > >> Arrow > > > > >> >> protocol, because of all the assumptions and restrictions > > required > > > as > > > > >> you > > > > >> >> noted. > > > > >> >> > > > > >> >> I took another look at the proposal and don’t think there’s > > > anything > > > > >> >> preventing in-place updating in the future - ultimately the > data > > > body > > > > >> could > > > > >> >> just be in the same location for subsequent messages. > > > > >> >> > > > > >> >> Thanks! > > > > >> >> Paul > > > > >> >> > > > > >> >> On Fri, Mar 1, 2024 at 5:28 PM Matt Topol < > > zotthewiz...@gmail.com> > > > > >> wrote: > > > > >> >> > > > > >> >> > > @pgwhalen: As a potential "end user developer," (and > aspiring > > > > >> >> > contributor) this > > > > >> >> > immediately excited me when I first saw it. > > > > >> >> > > > > > >> >> > Yay! Good to hear that! > > > > >> >> > > > > > >> >> > > @pgwhalen: And it wasn't clear to me whether updating > batches > > > in > > > > >> >> > place (and the producer/consumer coordination that comes with > > > that) > > > > >> was > > > > >> >> > supported or encouraged as part of the proposal. > > > > >> >> > > > > > >> >> > So, updating batches in place was not a particular use-case > we > > > were > > > > >> >> > targeting with this approach. Instead using shared memory to > > > > produce > > > > >> and > > > > >> >> > consume the buffers/batches without having to physically copy > > the > > > > >> data. > > > > >> >> > Trying to update a batch in place is a dangerous prospect > for a > > > > >> number of > > > > >> >> > reasons, but as you've mentioned it can technically be made > > safe > > > if > > > > >> the > > > > >> >> > shape is staying the same and you're only modifying > fixed-width > > > > data > > > > >> >> types > > > > >> >> > (i.e. not only is the *shape* unchanged but the sizes of the > > > > >> underlying > > > > >> >> > data buffers are also remaining unchanged). The > > producer/consumer > > > > >> >> > coordination that would be needed for updating batches in > place > > > is > > > > >> not > > > > >> >> part > > > > >> >> > of this proposal but is definitely something we can look into > > as > > > a > > > > >> >> > follow-up to this for extending it. There's a number of > > > discussions > > > > >> that > > > > >> >> > would need to be had around that so I don't want to add on > > > another > > > > >> >> > complexity to this already complex proposal. > > > > >> >> > > > > > >> >> > That said, if you or anyone see something in this proposal > that > > > > would > > > > >> >> > hinder or prevent being able to use it for your use case > please > > > let > > > > >> me > > > > >> >> know > > > > >> >> > so we can address it. Even though the proposal as it > currently > > > > exists > > > > >> >> > doesn't fully support the in-place updating of batches, I > don't > > > > want > > > > >> to > > > > >> >> > make things harder for us in such a follow-up where we'd end > up > > > > >> requiring > > > > >> >> > an entirely new protocol to support that. > > > > >> >> > > > > > >> >> > > @octalene.dev: I know of a third party that is interested > in > > > > >> Arrow for > > > > >> >> > HPC environments that could be interested in the proposal > and I > > > can > > > > >> see > > > > >> >> if > > > > >> >> > they're interested in providing feedback. > > > > >> >> > > > > > >> >> > Awesome! Thanks much! > > > > >> >> > > > > > >> >> > > > > > >> >> > For reference to anyone who hasn't looked at the document in > a > > > > while, > > > > >> >> since > > > > >> >> > the original discussion thread on this I have added a full > > > > >> "Background > > > > >> >> > Context" page to the beginning of the proposal to help anyone > > who > > > > >> isn't > > > > >> >> > already familiar with the issues this protocol is trying to > > solve > > > > or > > > > >> >> isn't > > > > >> >> > already familiar with ucx or libfabric transports to better > > > > >> understand > > > > >> >> > *why* I'm > > > > >> >> > proposing this and what it is trying to solve. The point of > > this > > > > >> >> background > > > > >> >> > information is to help ensure that anyone who might have > > thoughts > > > > on > > > > >> >> > protocols in general or APIs should still be able to > understand > > > the > > > > >> base > > > > >> >> > reasons and goals that we're trying to achieve with this > > protocol > > > > >> >> proposal. > > > > >> >> > You don't need to already understand managing GPU/device > memory > > > or > > > > >> ucx to > > > > >> >> > be able to have meaningful input on the document. > > > > >> >> > > > > > >> >> > Thanks again to all who have contributed so far and please > > spread > > > > to > > > > >> any > > > > >> >> > contacts that you think might be interested in this for their > > > > >> particular > > > > >> >> > use cases. > > > > >> >> > > > > > >> >> > --Matt > > > > >> >> > > > > > >> >> > On Wed, Feb 28, 2024 at 1:39 AM Aldrin > > > <octalene....@pm.me.invalid > > > > > > > > > >> >> wrote: > > > > >> >> > > > > > >> >> > > I am interested in this as well, but I haven't gotten to a > > > point > > > > >> where > > > > >> >> I > > > > >> >> > > can have valuable input (I haven't tried other > transports). I > > > > know > > > > >> of a > > > > >> >> > > third party that is interested in Arrow for HPC > environments > > > that > > > > >> could > > > > >> >> > be > > > > >> >> > > interested in the proposal and I can see if they're > > interested > > > in > > > > >> >> > providing > > > > >> >> > > feedback. > > > > >> >> > > > > > > >> >> > > I glanced at the document before but I'll go through again > to > > > see > > > > >> if > > > > >> >> > there > > > > >> >> > > is anything I can comment on. > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > # ------------------------------ > > > > >> >> > > # Aldrin > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > https://github.com/drin/ > > > > >> >> > > https://gitlab.com/octalene > > > > >> >> > > https://keybase.io/octalene > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > On Tuesday, February 27th, 2024 at 17:43, Paul Whalen < > > > > >> >> > pgwha...@gmail.com> > > > > >> >> > > wrote: > > > > >> >> > > > > > > >> >> > > > As a potential "end user developer," (and aspiring > > > contributor) > > > > >> this > > > > >> >> > > > immediately excited me when I first saw it. > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > I work at a trading firm, and my team has developed an > IPC > > > > >> mechanism > > > > >> >> > for > > > > >> >> > > > efficiently transmitting pandas dataframes both remotely > > via > > > > TCP > > > > >> and > > > > >> >> > > > locally via shared memory, where the interface for the > > > > >> application > > > > >> >> > > > developer is the same for both. The data in the > dataframes > > > may > > > > >> change > > > > >> >> > > > rapidly, so when communicating locally via shared memory, > > if > > > > the > > > > >> >> shape > > > > >> >> > of > > > > >> >> > > > the dataframe doesn't change, we update the memory in > > place, > > > > >> >> > coordinating > > > > >> >> > > > between the producer and consumer via TCP. > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > We intend to move away from our remote TCP mechanism > > towards > > > > >> Arrow > > > > >> >> > > Flight, > > > > >> >> > > > or a lighter-weight version of Arrow IPC. For the local > > > shared > > > > >> memory > > > > >> >> > > > mechanism which we previously did not have a good answer > > for, > > > > it > > > > >> >> seems > > > > >> >> > > like > > > > >> >> > > > Disassociated Arrow IPC maps quite well to our problem. > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > So some features that enable our use case are: > > > > >> >> > > > - Updating existing batches in place is supported > > > > >> >> > > > - The interface is pretty similar to Flight > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > I'd imagine we're not the only financial firm to > implement > > > > >> something > > > > >> >> > like > > > > >> >> > > > this, given how widespread pandas usage is, so that could > > be > > > a > > > > >> place > > > > >> >> to > > > > >> >> > > > seek feedback. > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > As I was reading the proposal initially, I gleaned that > the > > > > most > > > > >> >> > > important > > > > >> >> > > > audience was those writing interfaces to GPUs/remote > > > > >> >> > memory/non-standard > > > > >> >> > > > transports/etc. And it wasn't clear to me whether > updating > > > > >> batches in > > > > >> >> > > > place (and the producer/consumer coordination that comes > > with > > > > >> that) > > > > >> >> was > > > > >> >> > > > supported or encouraged as part of the proposal. But > > > > regardless, > > > > >> as > > > > >> >> an > > > > >> >> > > end > > > > >> >> > > > user, this seems like an easier and more efficient way to > > > glue > > > > >> pieces > > > > >> >> > in > > > > >> >> > > > the Arrow ecosystem together if it was adopted broadly. > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > Paul > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > On Tue, Feb 27, 2024 at 6:05 PM Matt Topol > > > > >> zotthewiz...@gmail.com > > > > >> >> > wrote: > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > I'll continue my efforts of trying to reach out to > other > > > > >> interested > > > > >> >> > > > > parties, but if anyone else here has any contacts or > > > > >> connections > > > > >> >> that > > > > >> >> > > they > > > > >> >> > > > > think might be interested please forward them the link > to > > > the > > > > >> >> Google > > > > >> >> > > doc. > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > I really do want to get as much engagement and feedback > > as > > > > >> possible > > > > >> >> > on > > > > >> >> > > > > this. > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > Thanks! > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > On Tue, Feb 27, 2024, 6:38 PM Wes McKinney > > > > wesmck...@gmail.com > > > > >> >> > wrote: > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > > Have there been efforts to proactively reach out to > > other > > > > >> third > > > > >> >> > > parties > > > > >> >> > > > > > that might have an interest in this or be a potential > > > user > > > > at > > > > >> >> some > > > > >> >> > > point? > > > > >> >> > > > > > There are a lot of interested parties in Arrow that > may > > > not > > > > >> >> > actively > > > > >> >> > > > > > follow > > > > >> >> > > > > > the mailing list. > > > > >> >> > > > > > > > > > >> >> > > > > > > >> >> > > > > > Seems like folks from the Dask, Ray, RAPIDS > (especially > > > > >> folks at > > > > >> >> > > NVIDIA > > > > >> >> > > > > > or > > > > >> >> > > > > > working on UCX), or other communities like that might > > > have > > > > >> >> > > constructive > > > > >> >> > > > > > thoughts about this. DLPack ( > > > > >> >> https://dmlc.github.io/dlpack/latest/ > > > > >> >> > ) > > > > >> >> > > also > > > > >> >> > > > > > seems adjacent and worth reaching out to. Other ideas > > for > > > > >> >> projects > > > > >> >> > or > > > > >> >> > > > > > companies that could be reached out to for feedback. > > > > >> >> > > > > > > > > > >> >> > > > > > > >> >> > > > > > On Tue, Feb 27, 2024 at 5:23 PM Antoine Pitrou > > > > >> >> anto...@python.org > > > > >> >> > > > > > wrote: > > > > >> >> > > > > > > > > > >> >> > > > > > > >> >> > > > > > > If there's no engagement, then I'm afraid it might > > mean > > > > >> that > > > > >> >> > third > > > > >> >> > > > > > > parties have no interest in this. I don't really > have > > > any > > > > >> >> > solution > > > > >> >> > > for > > > > >> >> > > > > > > generating engagement except nagging and pinging > > people > > > > >> >> > explicitly > > > > >> >> > > :-) > > > > >> >> > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > Le 27/02/2024 à 19:09, Matt Topol a écrit : > > > > >> >> > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > I would like to see the same Antoine, currently > > given > > > > the > > > > >> >> lack > > > > >> >> > of > > > > >> >> > > > > > > > engagement (both for OR against) I was going to > > take > > > > the > > > > >> >> > silence > > > > >> >> > > as > > > > >> >> > > > > > > > assent > > > > >> >> > > > > > > > and hope for non-Voltron Data PMC members to vote > > in > > > > >> this. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > If anyone has any suggestions on how we could > > > > potentially > > > > >> >> > > generate > > > > >> >> > > > > > > > more > > > > >> >> > > > > > > > engagement and discussion on this, please let me > > know > > > > as > > > > >> I > > > > >> >> want > > > > >> >> > > as > > > > >> >> > > > > > > > many > > > > >> >> > > > > > > > parties in the community as possible to be part > of > > > > this. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > Thanks everyone. > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > --Matt > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > On Tue, Feb 27, 2024 at 12:48 PM Antoine Pitrou > > > > >> >> > > anto...@python.org > > > > >> >> > > > > > > > wrote: > > > > >> >> > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > Hello, > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > I'd really like to see more engagement and > > > criticism > > > > >> from > > > > >> >> > > > > > > > > non-Voltron > > > > >> >> > > > > > > > > Data parties before this is formally adopted as > > an > > > > >> Arrow > > > > >> >> > spec. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > Regards > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > Antoine. > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > Le 27/02/2024 à 18:35, Matt Topol a écrit : > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > Hey all, > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > I'd like to propose a vote for us to > officially > > > > >> adopt the > > > > >> >> > > protocol > > > > >> >> > > > > > > > > > described in the google doc[1] for > Dissociated > > > > Arrow > > > > >> IPC > > > > >> >> > > > > > > > > > Transports. > > > > >> >> > > > > > > > > > This > > > > >> >> > > > > > > > > > proposal was originally discussed at 2. Once > > this > > > > >> >> proposal > > > > >> >> > is > > > > >> >> > > > > > > > > > adopted, > > > > >> >> > > > > > > > > > I > > > > >> >> > > > > > > > > > will work on adding the necessary > documentation > > > to > > > > >> the > > > > >> >> > Arrow > > > > >> >> > > > > > > > > > website > > > > >> >> > > > > > > > > > along > > > > >> >> > > > > > > > > > with examples etc. > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > The vote will be open for at least 72 hours. > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > [ ] +1 Accept this Proposal > > > > >> >> > > > > > > > > > [ ] +0 > > > > >> >> > > > > > > > > > [ ] -1 Do not accept this proposal because... > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > Thank you everyone! > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > --Matt > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > > [1]: > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1zHbnyK1r6KHpMOtEdIg1EZKNzHx-MVgUMOzB87GuXyk/edit#heading=h.38515dnp2bdb > > > > >> >> > > > > > >> >> > > > > >> > > > > > > > > > > > > > > >
