Regarding no universal property graph format, yes I agree this is a problem. At Aerospike we support the same csv format as Amazon Neptune, so this csv format will work for both those use cases. It's is fairly trivial to write something to convert this to something Neo4J can consume as well if we wanted to. I am not sure what other vendors use but the format we generate, while not a written standard, can be easily adapted to other formats if needed.
Regarding other drivers, right now that would probably be hard because I built it on JMH which for those who have not used it before, is a Java benchmarking tool. I do wonder if there would be a way to standardize this across drivers. I would be hesitant to embed other drivers in Java, but maybe we could over time convert it to something that generates the different languages. Regarding embedded use cases, we could easily make it support taking in a config.properties for a graph and then instead of making a Cluster -> GraphTraversalSource, it could go Graph -> GraphTraversalSource, so I think that use case would be much simpler. Regarding LDBC, yes this was something I actually looked at. It seemed like quite a bit of work to get it all working and then we would be married to their datasets and query patterns, instead I thought it was simpler to use whatever dataset we seem fit and generate queries around that. Also the couple LDBC results I looked at had non-real production systems behind them, a lot were very specifically made for executing these queries and used GPU's, not really made to solve general queries so it didn't seem very relevant for actually comparing results. That's my take on it, I could be wrong on that. -Lyndon On Wed, Nov 20, 2024 at 8:26 AM Josh Perryman <joshperry...@gmail.com> wrote: > I'd like to point out that in Europe there's a Linked Data Benchmark > Council (https://ldbcouncil.org/) which has some benchmarks and data > already in place. May not be exactly what the community has in mind, but at > the least it shows how domain description, queries and data can be packaged > together for implementation-agnostic benchmarking. > > -Josh > > On Wed, Nov 20, 2024 at 6:37 AM Stephen Mallette <spmalle...@gmail.com> > wrote: > > > Hey Lyndon - thanks for bringing this here. There are always questions > > about "how to benchmark" and the tendency is to point folks to long > > unmaintained projects. It would be nice to be able to point folks to > > something that TinkerPop maintains. Going back to the start of this > thread, > > it reads as though you intend it to mostly test remote graph systems > using > > the Java driver. Do you think it is easily adapted in some way to cover > > other drivers or even embedded use cases? > > > > On Tue, Nov 19, 2024 at 8:55 PM Cole Greer <cole.gr...@improving.com > > .invalid> > > wrote: > > > > > Hi Lyndon, > > > > > > I think it would be great to have a standardized large-scale > benchmarking > > > Framework in TinkerPop. Ideally such a framework should be as > accessible > > > as possible for providers and users. As such, I think it’s valuable to > > have > > > standardized datasets hosted by Apache. One limitation I see is that > the > > > CSV format used by Aerospike is not a standardized format recognized by > > > TinkerPop. The format does however appear to be emerging as a de facto > > > standard, as I have seen the same format in use from other TinkerPop > > > providers > > > and various other property graph related projects. > > > > > > Unfortunately, there is currently no such thing as a universal property > > > graph file > > > format and it would be unreasonable to pre-emptively transform and host > > the > > > datasets in many potentially useful formats. As such, I think it is > best > > > for the > > > if these datasets are hosted by Apache in the CSV format, and we agree > to > > > additionally host any future transformations from the community into > > other > > > popular competing formats. > > > > > > Thanks, > > > Cole > > > > > > From: Lyndon Bauto <lba...@aerospike.com.INVALID> > > > Date: Tuesday, November 19, 2024 at 2:02 PM > > > To: dev@tinkerpop.apache.org <dev@tinkerpop.apache.org> > > > Subject: Re: Benchmarking Framework > > > I am fine with ASF hosting the files if they are okay with it, I didn't > > > want to force the onus on apache in case the size of the files would be > > > problematic if we wanted file sets into the TB. > > > > > > On Tue, Nov 19, 2024 at 2:01 PM Lyndon Bauto <lba...@aerospike.com> > > wrote: > > > > > > > Interesting, I had tried sweeping min/max connection pool sizes and > > still > > > > had the problem but maybe this will have addressed the problem. I can > > > look > > > > into this when I have some time. > > > > > > > > The benchmark framework just expects a loaded graph, how the provider > > > > loads their graph and in what format is up to them. We chose csv > > because > > > we > > > > support it and others support csv inputs as well. > > > > > > > > So this loading is done outside the framework and anything can be > > loaded > > > > in any fashion as long as the queries being run in the framework line > > up > > > > with the dataset. It's very easy to add new queries. > > > > > > > > On Tue, Nov 19, 2024 at 1:56 PM Yang Xia <xia...@apache.org> wrote: > > > > > > > >> Thanks Lyndon. For the benchmarking framework itself, can it take in > > any > > > >> dataset in csv file format, or only the specific dataset you've > > > generated? > > > >> > > > >> Also I think if we want to standardize on a set of sample data for > > > users, > > > >> the best way may be to host the data on ASF, and It does look like > ASF > > > has > > > >> a server to host files: https://nightlies.apache.org/. Thoughts? > > > >> > > > >> Cheers, > > > >> > > > >> Yang > > > >> > > > >> On 2024/11/19 17:46:18 Lyndon Bauto wrote: > > > >> > I was planning on contributing the entire framework which is > > > basically a > > > >> > framework that's set up in a way where you can add additional > > > >> benchmarks to > > > >> > it similar to adding a test in junit, so if people want to make > new > > > >> > benchmarks on different things, it's easy. Not sure on the exact > > place > > > >> it'd > > > >> > like, gremlin-tools seems reasonable though. > > > >> > > > > >> > The dataset we used was basically an identity graph dataset (csv > > > format) > > > >> > that is generated from 350 GB, 3.5 TB, and 15TB. Aerospike will > host > > > >> this > > > >> > data in a public gcp bucket and others are welcome to use it. > > > >> > > > > >> > We will likely generate some other datasets and add to this over > > time, > > > >> this > > > >> > is not something that TinkerPop would be on the hook for hosting > or > > > >> > providing in any way. > > > >> > > > > >> > -Lyndon > > > >> > > > > >> > On Tue, Nov 19, 2024 at 8:33 AM Yang Xia <xia...@apache.org> > wrote: > > > >> > > > > >> > > Hi Lyndon, > > > >> > > > > > >> > > I also think this is something that can benefit users. I just > have > > > >> some > > > >> > > quick questions. > > > >> > > > > > >> > > Could you clarify what you plan to contribute into TinkerPop? Is > > it > > > >> the > > > >> > > benchmarking framework, the dataset, or both? For the > benchmarking > > > >> > > framework, are you looking to PR something into the > gremlin-tools > > > >> module? > > > >> > > For the dataset, what type of data does it have, how is it > > > generated? > > > >> Since > > > >> > > there are quite a few benchmarking dataset that exists out there > > > >> already, I > > > >> > > feel like we've usually kept datasets external. > > > >> > > > > > >> > > Cheers, > > > >> > > > > > >> > > Yang > > > >> > > > > > >> > > On 2024/11/15 20:25:03 Lyndon Bauto wrote: > > > >> > > > Reviving this thread. > > > >> > > > > > > >> > > > I think I have exposed a bottleneck in the Java driver. Not > sure > > > >> what it > > > >> > > > is, but if I scale the client machine up to 128 cores and 2:1 > > > >> thread:core > > > >> > > > ratio, I get no additional performance over a say, 16 core > > > machine. > > > >> > > However > > > >> > > > if I create additional JVM's running the benchmark I get > > > additional > > > >> > > > performance. It's still unclear whether this is the driver or > > > maybe > > > >> the > > > >> > > JVM > > > >> > > > running the driver. > > > >> > > > > > > >> > > > Anyway I would like to move forward with getting this into > > > >> TinkerPop. The > > > >> > > > plan I have is to make the datasets we have generated for it > so > > > far > > > >> > > public, > > > >> > > > then anyway can load the dataset into any graph provider's > graph > > > >> and run > > > >> > > > the benchmark. Additionally we will look into other datasets > > > >> separately. > > > >> > > > > > > >> > > > let me know if anyone has any concerns > > > >> > > > -Lyndon > > > >> > > > > > > >> > > > On Thu, Jul 11, 2024 at 8:54 AM Ken Hu <kenhu...@gmail.com> > > > wrote: > > > >> > > > > > > >> > > > > I think this would be very useful for the 3.x line that uses > > > >> > > WebSockets. > > > >> > > > > There's difficulty in recommending what the best connection > > > >> settings > > > >> > > are to > > > >> > > > > increase performance for different workloads and an > automated > > > >> tool to > > > >> > > > > discover that would be helpful to users. On a side note, a > > goal > > > >> during > > > >> > > the > > > >> > > > > transition to HTTP should be to make the connection settings > > > >> simpler so > > > >> > > > > that it is easier to figure out what the settings should be > > for > > > a > > > >> > > specific > > > >> > > > > workload. > > > >> > > > > > > > >> > > > > I feel like there are definitely some users that will > benefit > > > >> from a > > > >> > > > > benchmarking tool like this. > > > >> > > > > > > > >> > > > > On Wed, Jul 10, 2024 at 12:42 PM Lyndon Bauto > > > >> > > <lba...@aerospike.com.invalid > > > >> > > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Right now the dataset and benchmarking setup is really > > simple. > > > >> > > > > > > > > >> > > > > > It does some mergeV's, edge insertion, get vertex by id > > > >> > > > > (g.V(<id>).next()), > > > >> > > > > > and then some g.V(<id>).out().out().out().out(). > > > >> > > > > > > > > >> > > > > > The idea being to get results for queries that require a > > > decent > > > >> bit > > > >> > > of > > > >> > > > > > processing, as well as quick lookup and return queries > that > > > will > > > >> > > allow us > > > >> > > > > > to test the driver when it's under high throughput load > that > > > is > > > >> > > highly > > > >> > > > > > concurrent. We could also add a query that returns a lot > of > > > data > > > >> > > without > > > >> > > > > a > > > >> > > > > > lot of processing, so we could test the driver under a > > > scenario > > > >> > > where a > > > >> > > > > lot > > > >> > > > > > of data is coming back. > > > >> > > > > > > > > >> > > > > > This would help users identify what would be most > beneficial > > > for > > > >> > > their > > > >> > > > > use > > > >> > > > > > case, for example, maybe few connections and many in > process > > > per > > > >> > > > > connection > > > >> > > > > > gets better use of resources when the data returned is > > minimal > > > >> but > > > >> > > the > > > >> > > > > > number of queries running is very high, meanwhile more > > > >> connections > > > >> > > with > > > >> > > > > > less in process per connection might achieve better > results > > > when > > > >> > > queries > > > >> > > > > > are returning more data. > > > >> > > > > > > > > >> > > > > > Down the road adding things like identity graph use cases, > > > fraud > > > >> > > > > detection > > > >> > > > > > use cases, and others with datasets included and queries > to > > > >> > > benchmark in > > > >> > > > > > there would be a great way for providers to opt into > > providing > > > >> > > benchmarks > > > >> > > > > > that are relevant to their target customers but that is a > > > later > > > >> > > thing. > > > >> > > > > > > > > >> > > > > > - Lyndon > > > >> > > > > > > > > >> > > > > > On Tue, Jul 9, 2024 at 4:14 PM Ken Hu <kenhu...@gmail.com > > > > > >> wrote: > > > >> > > > > > > > > >> > > > > > > Hey Lyndon, > > > >> > > > > > > > > > >> > > > > > > This is a very interesting idea. You mentioned > throughput > > > >> testing > > > >> > > but > > > >> > > > > how > > > >> > > > > > > does this compare to other graph testing that use > specific > > > >> > > generated > > > >> > > > > > > datasets and specific queries? Asked another way, what > > kind > > > of > > > >> > > queries > > > >> > > > > > are > > > >> > > > > > > you using to test in this system? > > > >> > > > > > > > > > >> > > > > > > Regards, > > > >> > > > > > > Ken > > > >> > > > > > > > > > >> > > > > > > On Tue, Jul 9, 2024 at 2:00 PM Lyndon Bauto > > > >> > > > > <lba...@aerospike.com.invalid > > > >> > > > > > > > > > >> > > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > Hi devs, > > > >> > > > > > > > > > > >> > > > > > > > I've been working on a benchmarking framework for > > > tinkerpop, > > > >> > > > > > specifically > > > >> > > > > > > > the Java driver. > > > >> > > > > > > > > > > >> > > > > > > > The idea is to have a benchmarking framework that a > > > >> TinkerPop > > > >> > > user > > > >> > > > > can > > > >> > > > > > > > target their instance of gremlin-server with (can be > any > > > >> > > provider) > > > >> > > > > and > > > >> > > > > > > what > > > >> > > > > > > > this will allow them to do is fix some of their > configs > > of > > > >> the > > > >> > > driver > > > >> > > > > > > while > > > >> > > > > > > > having others as variables. The framework will then > run > > > >> through a > > > >> > > > > bunch > > > >> > > > > > > of > > > >> > > > > > > > different settings, recording latency and throughput. > > > >> > > > > > > > > > > >> > > > > > > > The output of the benchmarking framework would be > > guidance > > > >> for > > > >> > > the > > > >> > > > > user > > > >> > > > > > > of > > > >> > > > > > > > the Java driver for optimal configuration for both > > latency > > > >> and > > > >> > > > > > > throughput, > > > >> > > > > > > > that they can then use to optimize their workload > > outside > > > >> the > > > >> > > > > > framework. > > > >> > > > > > > > > > > >> > > > > > > > A provider could also use this to manually adjust > > > >> > > > > > > > gremlinPool/threadPoolWorkers/etc and run the > framework > > > >> under > > > >> > > > > different > > > >> > > > > > > > settings to optimize throughput and latency there as > > well. > > > >> > > > > > > > > > > >> > > > > > > > The benchmark is built on JMH and is build into a > docker > > > >> > > container so > > > >> > > > > > it > > > >> > > > > > > is > > > >> > > > > > > > very easy to use. The configs are passed at runtime, > so > > a > > > >> user > > > >> > > would > > > >> > > > > > just > > > >> > > > > > > > call a docker build then docker run script, with the > > > configs > > > >> > > setup in > > > >> > > > > > the > > > >> > > > > > > > docker config. > > > >> > > > > > > > > > > >> > > > > > > > We could also add other benchmarks at any scale to the > > > >> framework > > > >> > > that > > > >> > > > > > > allow > > > >> > > > > > > > benchmark publishing from providers who wish to > > > participate. > > > >> > > > > > > > > > > >> > > > > > > > Anyone have any thoughts on this? > > > >> > > > > > > > > > > >> > > > > > > > Cheers, > > > >> > > > > > > > Lyndon > > > >> > > > > > > > -- > > > >> > > > > > > > > > > >> > > > > > > > *Lyndon Bauto* > > > >> > > > > > > > *Senior Software Engineer* > > > >> > > > > > > > *Aerospike, Inc.* > > > >> > > > > > > > www.aerospike.com<http://www.aerospike.com> > > > >> > > > > > > > lba...@aerospike.com > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > -- > > > >> > > > > > > > > >> > > > > > *Lyndon Bauto* > > > >> > > > > > *Senior Software Engineer* > > > >> > > > > > *Aerospike, Inc.* > > > >> > > > > > www.aerospike.com<http://www.aerospike.com> > > > >> > > > > > lba...@aerospike.com > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > -- > > > >> > > > > > > >> > > > *Lyndon Bauto* > > > >> > > > *Senior Software Engineer* > > > >> > > > *Aerospike, Inc.* > > > >> > > > www.aerospike.com<http://www.aerospike.com> > > > >> > > > lba...@aerospike.com > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > -- > > > >> > > > > >> > *Lyndon Bauto* > > > >> > *Senior Software Engineer* > > > >> > *Aerospike, Inc.* > > > >> > www.aerospike.com<http://www.aerospike.com> > > > >> > lba...@aerospike.com > > > >> > > > > >> > > > > > > > > > > > > -- > > > > > > > > *Lyndon Bauto* > > > > *Senior Software Engineer* > > > > *Aerospike, Inc.* > > > > www.aerospike.com<http://www.aerospike.com> > > > > lba...@aerospike.com > > > > > > > > > > > > > -- > > > > > > *Lyndon Bauto* > > > *Senior Software Engineer* > > > *Aerospike, Inc.* > > > www.aerospike.com<http://www.aerospike.com> > > > lba...@aerospike.com > > > > > > -- *Lyndon Bauto* *Senior Software Engineer* *Aerospike, Inc.* www.aerospike.com lba...@aerospike.com
