Interesting, I had tried sweeping min/max connection pool sizes and still had the problem but maybe this will have addressed the problem. I can look into this when I have some time.
The benchmark framework just expects a loaded graph, how the provider loads their graph and in what format is up to them. We chose csv because we support it and others support csv inputs as well. So this loading is done outside the framework and anything can be loaded in any fashion as long as the queries being run in the framework line up with the dataset. It's very easy to add new queries. On Tue, Nov 19, 2024 at 1:56 PM Yang Xia <xia...@apache.org> wrote: > Thanks Lyndon. For the benchmarking framework itself, can it take in any > dataset in csv file format, or only the specific dataset you've generated? > > Also I think if we want to standardize on a set of sample data for users, > the best way may be to host the data on ASF, and It does look like ASF has > a server to host files: https://nightlies.apache.org/. Thoughts? > > Cheers, > > Yang > > On 2024/11/19 17:46:18 Lyndon Bauto wrote: > > I was planning on contributing the entire framework which is basically a > > framework that's set up in a way where you can add additional benchmarks > to > > it similar to adding a test in junit, so if people want to make new > > benchmarks on different things, it's easy. Not sure on the exact place > it'd > > like, gremlin-tools seems reasonable though. > > > > The dataset we used was basically an identity graph dataset (csv format) > > that is generated from 350 GB, 3.5 TB, and 15TB. Aerospike will host this > > data in a public gcp bucket and others are welcome to use it. > > > > We will likely generate some other datasets and add to this over time, > this > > is not something that TinkerPop would be on the hook for hosting or > > providing in any way. > > > > -Lyndon > > > > On Tue, Nov 19, 2024 at 8:33 AM Yang Xia <xia...@apache.org> wrote: > > > > > Hi Lyndon, > > > > > > I also think this is something that can benefit users. I just have some > > > quick questions. > > > > > > Could you clarify what you plan to contribute into TinkerPop? Is it the > > > benchmarking framework, the dataset, or both? For the benchmarking > > > framework, are you looking to PR something into the gremlin-tools > module? > > > For the dataset, what type of data does it have, how is it generated? > Since > > > there are quite a few benchmarking dataset that exists out there > already, I > > > feel like we've usually kept datasets external. > > > > > > Cheers, > > > > > > Yang > > > > > > On 2024/11/15 20:25:03 Lyndon Bauto wrote: > > > > Reviving this thread. > > > > > > > > I think I have exposed a bottleneck in the Java driver. Not sure > what it > > > > is, but if I scale the client machine up to 128 cores and 2:1 > thread:core > > > > ratio, I get no additional performance over a say, 16 core machine. > > > However > > > > if I create additional JVM's running the benchmark I get additional > > > > performance. It's still unclear whether this is the driver or maybe > the > > > JVM > > > > running the driver. > > > > > > > > Anyway I would like to move forward with getting this into > TinkerPop. The > > > > plan I have is to make the datasets we have generated for it so far > > > public, > > > > then anyway can load the dataset into any graph provider's graph and > run > > > > the benchmark. Additionally we will look into other datasets > separately. > > > > > > > > let me know if anyone has any concerns > > > > -Lyndon > > > > > > > > On Thu, Jul 11, 2024 at 8:54 AM Ken Hu <kenhu...@gmail.com> wrote: > > > > > > > > > I think this would be very useful for the 3.x line that uses > > > WebSockets. > > > > > There's difficulty in recommending what the best connection > settings > > > are to > > > > > increase performance for different workloads and an automated tool > to > > > > > discover that would be helpful to users. On a side note, a goal > during > > > the > > > > > transition to HTTP should be to make the connection settings > simpler so > > > > > that it is easier to figure out what the settings should be for a > > > specific > > > > > workload. > > > > > > > > > > I feel like there are definitely some users that will benefit from > a > > > > > benchmarking tool like this. > > > > > > > > > > On Wed, Jul 10, 2024 at 12:42 PM Lyndon Bauto > > > <lba...@aerospike.com.invalid > > > > > > > > > > > wrote: > > > > > > > > > > > Right now the dataset and benchmarking setup is really simple. > > > > > > > > > > > > It does some mergeV's, edge insertion, get vertex by id > > > > > (g.V(<id>).next()), > > > > > > and then some g.V(<id>).out().out().out().out(). > > > > > > > > > > > > The idea being to get results for queries that require a decent > bit > > > of > > > > > > processing, as well as quick lookup and return queries that will > > > allow us > > > > > > to test the driver when it's under high throughput load that is > > > highly > > > > > > concurrent. We could also add a query that returns a lot of data > > > without > > > > > a > > > > > > lot of processing, so we could test the driver under a scenario > > > where a > > > > > lot > > > > > > of data is coming back. > > > > > > > > > > > > This would help users identify what would be most beneficial for > > > their > > > > > use > > > > > > case, for example, maybe few connections and many in process per > > > > > connection > > > > > > gets better use of resources when the data returned is minimal > but > > > the > > > > > > number of queries running is very high, meanwhile more > connections > > > with > > > > > > less in process per connection might achieve better results when > > > queries > > > > > > are returning more data. > > > > > > > > > > > > Down the road adding things like identity graph use cases, fraud > > > > > detection > > > > > > use cases, and others with datasets included and queries to > > > benchmark in > > > > > > there would be a great way for providers to opt into providing > > > benchmarks > > > > > > that are relevant to their target customers but that is a later > > > thing. > > > > > > > > > > > > - Lyndon > > > > > > > > > > > > On Tue, Jul 9, 2024 at 4:14 PM Ken Hu <kenhu...@gmail.com> > wrote: > > > > > > > > > > > > > Hey Lyndon, > > > > > > > > > > > > > > This is a very interesting idea. You mentioned throughput > testing > > > but > > > > > how > > > > > > > does this compare to other graph testing that use specific > > > generated > > > > > > > datasets and specific queries? Asked another way, what kind of > > > queries > > > > > > are > > > > > > > you using to test in this system? > > > > > > > > > > > > > > Regards, > > > > > > > Ken > > > > > > > > > > > > > > On Tue, Jul 9, 2024 at 2:00 PM Lyndon Bauto > > > > > <lba...@aerospike.com.invalid > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > I've been working on a benchmarking framework for tinkerpop, > > > > > > specifically > > > > > > > > the Java driver. > > > > > > > > > > > > > > > > The idea is to have a benchmarking framework that a TinkerPop > > > user > > > > > can > > > > > > > > target their instance of gremlin-server with (can be any > > > provider) > > > > > and > > > > > > > what > > > > > > > > this will allow them to do is fix some of their configs of > the > > > driver > > > > > > > while > > > > > > > > having others as variables. The framework will then run > through a > > > > > bunch > > > > > > > of > > > > > > > > different settings, recording latency and throughput. > > > > > > > > > > > > > > > > The output of the benchmarking framework would be guidance > for > > > the > > > > > user > > > > > > > of > > > > > > > > the Java driver for optimal configuration for both latency > and > > > > > > > throughput, > > > > > > > > that they can then use to optimize their workload outside the > > > > > > framework. > > > > > > > > > > > > > > > > A provider could also use this to manually adjust > > > > > > > > gremlinPool/threadPoolWorkers/etc and run the framework under > > > > > different > > > > > > > > settings to optimize throughput and latency there as well. > > > > > > > > > > > > > > > > The benchmark is built on JMH and is build into a docker > > > container so > > > > > > it > > > > > > > is > > > > > > > > very easy to use. The configs are passed at runtime, so a > user > > > would > > > > > > just > > > > > > > > call a docker build then docker run script, with the configs > > > setup in > > > > > > the > > > > > > > > docker config. > > > > > > > > > > > > > > > > We could also add other benchmarks at any scale to the > framework > > > that > > > > > > > allow > > > > > > > > benchmark publishing from providers who wish to participate. > > > > > > > > > > > > > > > > Anyone have any thoughts on this? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Lyndon > > > > > > > > -- > > > > > > > > > > > > > > > > *Lyndon Bauto* > > > > > > > > *Senior Software Engineer* > > > > > > > > *Aerospike, Inc.* > > > > > > > > www.aerospike.com > > > > > > > > lba...@aerospike.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Lyndon Bauto* > > > > > > *Senior Software Engineer* > > > > > > *Aerospike, Inc.* > > > > > > www.aerospike.com > > > > > > lba...@aerospike.com > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Lyndon Bauto* > > > > *Senior Software Engineer* > > > > *Aerospike, Inc.* > > > > www.aerospike.com > > > > lba...@aerospike.com > > > > > > > > > > > > > -- > > > > *Lyndon Bauto* > > *Senior Software Engineer* > > *Aerospike, Inc.* > > www.aerospike.com > > lba...@aerospike.com > > > -- *Lyndon Bauto* *Senior Software Engineer* *Aerospike, Inc.* www.aerospike.com lba...@aerospike.com
