> If geode-benchmarks is included, that implies that an RC cannot be approved until reviewers can successfully run the benchmark suite from the geode-benchmarks source distribution. Is that what we want?
I think it would be sufficient to run the tests of the benchmarks, eg ./gradlew test > Deploying CI pipelines and running Benchmarks seems like a prime example of things we’d be happy to help others in the community with on the dev list — but not something we would expect questions about on the user list. I think it would be valuable to share our benchmarks with the geode user community. The benchmark framework itself (the harness module) is a fairly generic benchmarking framework than can be used to benchmark anything that can be spun up using java. The geode-benchmark module has geode benchmarks that could be used for testing specific hardware, for example. -Dan On Thu, Jan 16, 2020 at 12:37 PM Owen Nichols <onich...@pivotal.io> wrote: > When voting on RC candidates, PMC members "are required to download the > signed source code package, compile it as provided, and test the resulting > executable on their own platform”. > > If geode-benchmarks is included, that implies that an RC cannot be > approved until reviewers can successfully run the benchmark suite from the > geode-benchmarks source distribution. Is that what we want? > > Similarly, if CI is included, that seems to imply that an RC cannot be > approved until reviewers can stand up their own pipeline from the geode/ci > source distribution. Is that what we want? > > So far there doesn’t seem to be consensus on what to include in a Geode > source release, but let’s keep in mind that anything we add to the release > becomes an Act Of The Foundation and is held to a higher standard. Apache > makes a clear distinction between between development activity and official > releases to the public. Development activity is anything that should stay > within the dev list. Deploying CI pipelines and running Benchmarks seems > like a prime example of things we’d be happy to help others in the > community with on the dev list — but not something we would expect > questions about on the user list. > > > On Jan 16, 2020, at 10:23 AM, Dan Smith <dsm...@pivotal.io> wrote: > > > > We are supposed to be including all of the source necessary to test Geode > > in the source release [1] - I think that would include benchmarks as > well. > > > > I don't really see any compelling reason *not* to include the benchmarks, > > let's go ahead and get them into our source release! > > > > [1] > > > http://www.apache.org/legal/release-policy.html#what-must-every-release-contain > > > > On Wed, Jan 15, 2020 at 10:26 PM Owen Nichols <onich...@pivotal.io> > wrote: > > > >> +1 for no changes > >> > >> On Wed, Jan 15, 2020 at 5:57 PM Jacob Barrett <jbarr...@pivotal.io> > wrote: > >> > >>> We can live in areas of gray that don’t require any changes. Nobody is > >>> asking for benchmarks so let’s not do work to add them. Nobody is > >>> complaining they CI is included so let’s not do work to remove them. Is > >> it > >>> ideal, meh... > >>> > >>>> On Jan 15, 2020, at 5:50 PM, Mark Hanson <mhan...@pivotal.io> wrote: > >>>> > >>>> Just my two cents. > >>>> > >>>> I think that we should probably strip CI into a separate repo. The key > >>> indicator is that if something were wrong in the CI yaml, would I hold > a > >>> release for that? I think no. So that suggests to me it is a separate > >>> thing. Same goes for benchmarks. If we were failing a benchmark I would > >> be > >>> concerned, but if the script were broken, would I hold the release? I > >> think > >>> no as well. > >>>> > >>>> I think that says that the CI code should also be a separate repo. > >>>> > >>>> Thanks, > >>>> Mark > >>>> > >>>>> On Jan 14, 2020, at 10:21 PM, Jacob Barrett <jbarr...@pivotal.io> > >>> wrote: > >>>>> > >>>>> Until someone outside of the geode ci community is asking for it I > >> just > >>> don’t see utility in going through the motions of making a release for > >> it. > >>>>> > >>>>>>> On Jan 14, 2020, at 10:13 PM, Owen Nichols <onich...@pivotal.io> > >>> wrote: > >>>>>> > >>>>>> The source is already public, so on some level a source release is > >> no > >>> different from a git tag. Benchmarks has matured enough that I think > it > >>> makes sense to at least start branching and tagging the > geode-benchmarks > >>> repo to capture exactly what was used in that Geode release. > >>>>>> > >>>>>> Others in the dev and user community may find the benchmarks useful > >> in > >>> other ways than we use them. While our focus for CI is on tuning for > >>> repeatability, someone else might just want a load generator to break > in > >> a > >>> new cluster or get some rough numbers. Some might want to get under > the > >>> hood and tinker and tune, or contribute their own benchmarks, with the > >>> understanding that it’s not a turnkey or standalone product, but a tool > >>> that requires getting your hands dirty. > >>>>>> > >>>>>> Would a “1 page” readme with a few tips on “how to run on a laptop” > >> be > >>> enough to let other interested contributors help get geode-benchmarks > to > >> a > >>> “better state”? > >>>>>> > >>>>>>> On Jan 14, 2020, at 9:38 PM, Jacob Barrett <jbarr...@pivotal.io> > >>> wrote: > >>>>>>> > >>>>>>> I don’t think the benchmarks provide any material benefit to a user > >>> in their current state. They are heavily tuned for our CI process which > >>> relies on very beefy machines. Usage on other hardware will require > more > >>> tuning. I don’t think it’s worth putting the source in the release > until > >>> they are in a better state. > >>>>>>> > >>>>>>> -Jake > >>>>>>> > >>>>>>> > >>>>>>>>> On Jan 14, 2020, at 4:14 PM, Dan Smith <dsm...@pivotal.io> > wrote: > >>>>>>>> > >>>>>>>> On Tue, Jan 14, 2020 at 4:11 PM Owen Nichols < > onich...@pivotal.io > >>> > >>> wrote: > >>>>>>>> > >>>>>>>>> I believe the desire is to include the source code for > >>> geode-benchmarks as > >>>>>>>>> part of the official geode release, much like how we include > >>> geode-examples. > >>>>>>>>> > >>>>>>>> > >>>>>>>> Oh! I thought you meant running the benchmarks in the release > >>> pipeline - I > >>>>>>>> think last release we were running them but decided they were too > >>> flaky to > >>>>>>>> use. > >>>>>>>> > >>>>>>>> +1 to including the benchmark source in the source release. > >>>>>>>> > >>>>>>>> -Dan > >>>>>> > >>>> > >>> > >> > >
