Hi Bechir, GraphBlas is mainly an interface, a list of functions.
This is similar to how BLAS is implemented by OpenBlas and MKL. Gabor Szárnyas has a great collection on how they are implanted and how they perform: https://github.com/GraphBLAS/GraphBLAS-Pointers Best regards, Adam Lippai On Fri, Jun 30, 2023 at 08:04 Bechir Ben Daadouch <bechirche...@gmail.com> wrote: > Hey Adam, > > Thank you very much for taking the time to respond and for the suggestions > :) > > I am aiming to assess the viability of Apache Arrow for graph algorithms > and data structures, rather than addressing a particular issue. I am > interested to determine the extent to which I can utilize Apache Arrow for > this purpose and what potential challenges may arise. > > and since I have already encountered some roadblocks early on in my > evaluation of Apache Arrow for graph algorithms and data structures, I am > very interested in hearing your point of view on this topic. > > Thanks, > Bechir > > On Fri, Jun 30, 2023, 13:38 Adam Lippai <a...@rigo.sk> wrote: > > > Hi, > > > > I’d recommend integrating with GraphBlas/suitesparse which are sparse > > matrix multiplication based algorithms for common graph problems. There > > might be an overlap in the data structures used, eg creating and a new > > arrow type like the recent tensor type addition, or simply creating zero > > copy or memcopy based efficient converters would be something worth > > exploring. > > > > Best regards, > > Adam alippai > > > > On Fri, Jun 30, 2023 at 06:57 Bechir Ben Daadouch < > bechirche...@gmail.com> > > wrote: > > > > > Thank you for taking the time to answer :) > > > > > > I don't have a fix Use-Case, but I am trying yo build a POC and > evaluate > > > whether Apache Arrow could be adequate in the context of graphs. But I > > > found out very quickly that I won't be able to do all the necessary > > > algorithm steps using only Apache Arrow without resorting to other > > > libraries. > > > > > > On Fri, Jun 30, 2023, 07:36 Benson Muite <benson_mu...@emailplus.org> > > > wrote: > > > > > > > On 6/30/23 04:21, Bechir Ben Daadouch wrote: > > > > > Dear Apache Arrow Dev Community, > > > > > > > > > > My name is Bechir, I am currently working on a project that > involves > > > > > implementing graph algorithms in Apache Arrow. > > > > > > > > > > The initial plan was to construct a node structure and a subsequent > > > graph > > > > > that would encompass all the nodes. However, I quickly realized > that > > > due > > > > to > > > > > Apache Arrow's columnar format, this approach was not feasible. > > > > > > > > > > I tried a couple of things, including the implementation of the > > > > > shortest-path algorithm. However, I rapidly discovered that > > > manipulating > > > > > arrow objects, particularly when applying graph algorithms, proved > > more > > > > > complex than anticipated and it became very clear that I would need > > to > > > > > resort to some data structures outside of what arrow offers (i.e.: > > > Heapq > > > > > wouldn't be possible using arrow). > > > > > > > > > > I also gave a shot at doing it similar to a certain SQL method > (see: > > > > > https://ibb.co/0rPGB42 ), but ran into some roadblocks there too > > and I > > > > > ended up having to resort to using Pandas for some transformations. > > > > > > > > > > My next course of action is to experiment with compressed sparse > > rows, > > > > > hoping to execute Matrix Multiplication using this method. But > > > honestly, > > > > > with what I know right now, I remain skeptical about the > feasibility > > > > > of it. However, > > > > > before committing to this approach, I would greatly appreciate your > > > > opinion > > > > > based on your experience with Apache Arrow. > > > > > > > > > > Thank you very much for your time. > > > > > > > > > > Looking forward to potentially discussing this further. > > > > > > > > > > Many thanks, > > > > > Bechir > > > > > > > > > Arrow may not be the best choice for most graph algorithms as they > > > > typically require random memory accesses that will be difficult to > > > > coalesce into forms that allow for vectorization. If your data will > fit > > > > in memory of a single node, you might consider: > > > > https://github.com/DrTimothyAldenDavis/GraphBLAS > > > > https://pypi.org/project/python-graphblas/ > > > > https://github.com/JuliaSparse/SuiteSparseGraphBLAS.jl > > > > > > > > > >
