I did some pretty heavy refactoring to the python test logic (altered a bit by some revision of the gherkin feature file language) and the result is a much more simplified test logic file:
https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-python/src/main/jython/radish/feature_steps.py About 120 lines of code (down from about 170). If you include the test logic "setup" file: https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/gremlin-python/src/main/jython/radish/terrain.py we end up with about 250 lines of test logic total (take out the comment/license and we're probably well under 200 total - not too bad). I think I'm closing in on the end to infrastructure building here so the basic framework is getting close to final at this point I believe. I'll keep scanning the tests looking for other types of assertions that I've not yet covered, but it's getting pretty solid I think. Hopefully, there won't need to be too many more lines of code needed to express the test logic as I like how things are looking right now. On Thu, Sep 28, 2017 at 8:02 AM, Jorge Bay Gondra <jorgebaygon...@gmail.com> wrote: > Great progress! I like how you avoided using ids, even if it adds some > complexity to the transformation required. > > The Python step definitions are still quite simple, I think it would be > mostly the same on the rest of the languages. > > On Thu, Sep 28, 2017 at 1:45 PM, Stephen Mallette <spmalle...@gmail.com> > wrote: > > > Just another update on progress with the test suite. The language of the > > .feature files is getting slightly more complex as I try to translate > more > > and more of our Java process suite tests into the language of the gherkin > > files. You can see here where I've added the ability to include > parameters: > > > > https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > > gremlin-test/features/filter/Has.feature#L33 > > > > I also came up with a method of asserting edges (didn't want to rely on > ids > > as it makes the gherkin harder to read, plus i didn't want to assume > > TinkerGraph identifiers in case these tests were every used with some > other > > graph database that didn't use longs): > > > > https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > > gremlin-test/features/map/Vertex.feature#L105-L111 > > > > for all those additions (and others) the logic required by the GLV to > > process these tests has stayed surprisingly simple: > > > > https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > > gremlin-python/src/main/jython/radish/feature_steps.py > > > > There's a fair bit of regex/string manipulation involved there, but it's > > processing strings from the feature file so that's the nature of it I > > suppose. I think I'm of the mind that I want to port all of the tests to > > feature files, so I wrote this unit test to help validate that none were > > missed: > > > > https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > > gremlin-test/src/test/java/org/apache/tinkerpop/gremlin/ > > structure/FeatureCoverageTest.java > > > > I don't know that we have to get to 100% porting right away, but I think > > that once we have these gherkin files written they not only become the > > basis for our current GLV testing work, but we might be able to simply > use > > them for testing the Java stuff as well - that would rid us of having the > > test code duplication. It also sets us up with a portable body of tests > > that can be re-used in TinkerPop 4.x. > > > > I'm open to suggestions if anyone has any. > > > > On Mon, Sep 25, 2017 at 11:23 AM, Jorge Bay Gondra < > > jorgebaygon...@gmail.com > > > wrote: > > > > > I was able to build a proof of concept for a Gherkin-based test runner > in > > > C#, that takes the proposed count and select features > > > <https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/ > > > gremlin-test/features/map> > > > and runs them using C# step definitions. > > > > > > It uses the Gherkin parser <https://github.com/cucumber/gherkin-dotnet > > > > > from > > > cucumber, there isn't a release of the parser with .NET Core support so > > > I've > > > asked them to release one > > > <https://github.com/cucumber/gherkin-dotnet/issues/11> (there is no > > > limitation from their source files). If they are not able to release it > > in > > > the next few days, we can implement our own as it should be pretty > > straight > > > forward. > > > > > > On Fri, Sep 22, 2017 at 2:23 PM, Stephen Mallette < > spmalle...@gmail.com> > > > wrote: > > > > > > > Thanks for the update. I'm trying to keep the test language as simple > > as > > > > possible so that we don't need an overly complicated test > > implementation. > > > > Hopefully that will help make the .NET approach as easy as possible. > > > > > > > > On Fri, Sep 22, 2017 at 8:20 AM, Jorge Bay Gondra < > > > > jorgebaygon...@gmail.com> > > > > wrote: > > > > > > > > > I've been looking into Gherkin support for .NET: SpecFlow, the > > cucumber > > > > > implementation for .NET <https://cucumber.io/docs# > > > > cucumber-implementations > > > > > >, > > > > > does not support .NET Core platform (we use .NET Core build tools > for > > > the > > > > > .NET GLV) and only supports .NET Framework. > > > > > > > > > > From what I can see <https://github.com/techtalk/ > SpecFlow/projects/2 > > >, > > > > > .NET > > > > > Core support on SpecFlow is coming at a very slow pace and we > > shouldn't > > > > > expect to land any time soon (there were some design decisions in > > > > SpecFlow > > > > > library that makes supporting other platforms non-trivial, like > > > requiring > > > > > code gen). > > > > > > > > > > The alternative would be to implement our own harness to support > it: > > > > from a > > > > > xunit test, look for certain types and parse the annotations, and > > > execute > > > > > them using the Gherkin features. > > > > > There is a .NET cross-platform Gherkin parser: > > > > > https://github.com/cucumber/gherkin-dotnet > > > > > I'll continue looking into this option and try to understand the > > effort > > > > > required... > > > > > > > > > > On Tue, Sep 19, 2017 at 6:21 PM, Jorge Bay Gondra < > > > > > jorgebaygon...@gmail.com> > > > > > wrote: > > > > > > > > > > > Nice! Gherkin will make our lives easier with a growing number of > > > GLVs. > > > > > > > > > > > > We should find a way to define the different features supported > by > > > each > > > > > > GLV, as it's reasonable to have different maturity levels per GLV > > > (ie: > > > > > > lambdas support, traversal strategy, ...). I don't know if it > will > > be > > > > > > beneficial to do it in the Gherkin files or within each GLV > > > > > implementation. > > > > > > Also, we should consider the process of rolling out a new method > / > > > > class > > > > > > in the java implementation, how that could affect each GLV. > > > > > > > > > > > > On Thu, Sep 14, 2017 at 11:12 PM, Stephen Mallette < > > > > spmalle...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > >> that's what i meant by "reflection" or as you suggest eval(). I > > > guess > > > > > the > > > > > >> point is that if the language can support some way of taking the > > > > string > > > > > >> value and turning it automatically into a traversal in that GLVs > > > style > > > > > >> then > > > > > >> we should do that. > > > > > >> > > > > > >> On Thu, Sep 14, 2017 at 4:45 PM, Daniel Kuppitz <m...@gremlin.guru > > > > > > > wrote: > > > > > >> > > > > > >> > For unparameterized queries it can probably be as easy as: > > > > > >> > > > > > > >> > @given("the traversal of") > > > > > >> > def translate_traversal(step): > > > > > >> > g = step.context.g > > > > > >> > step.context.traversal = eval(step.text) > > > > > >> > > > > > > >> > > > > > > >> > Cheers, > > > > > >> > Daniel > > > > > >> > > > > > > >> > > > > > > >> > On Thu, Sep 14, 2017 at 1:39 PM, Daniel Kuppitz > <m...@gremlin.guru > > > > > > > > >> wrote: > > > > > >> > > > > > > >> > > That's great stuff. I haven't used Cucumber / Gherkin for > > years, > > > > > but I > > > > > >> > > really like the BDD approach. > > > > > >> > > > > > > > >> > > and then you can look at the GLV Gremlin translations > > > specifically > > > > > >> here: > > > > > >> > >> https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > grem > > > > > >> > >> lin-python/src/main/jython/radish/count_features_step.py# > > > L34-L46 > > > > > >> > > > > > > > >> > > > > > > > >> > > This part is the only thing that looks weird to me. You're > > > > basically > > > > > >> > > writing every query twice; is there really no easier way to > do > > > > that? > > > > > >> > > > > > > > >> > > Cheers, > > > > > >> > > Daniel > > > > > >> > > > > > > > >> > > > > > > > >> > > On Thu, Sep 14, 2017 at 1:17 PM, Stephen Mallette < > > > > > >> spmalle...@gmail.com> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > >> I've brought this issue up in the past and had suggested > some > > > > > >> options I > > > > > >> > >> had > > > > > >> > >> in mind but now I've finally put the basics of those ideas > in > > > > place > > > > > >> so I > > > > > >> > >> figured I'd start a fresh thread. Recall that the issue at > > hand > > > > is > > > > > >> that > > > > > >> > we > > > > > >> > >> don't have a test suite for GLVs as gremlin-test is bound > to > > > the > > > > > >> JVM. We > > > > > >> > >> have some tricks that let us test gremlin-python with it > but > > > > those > > > > > >> > tricks > > > > > >> > >> won't work for every language and we now have the first > > > language > > > > in > > > > > >> > >> gremlin-dotnet and upcoming gremlin-javascript which won't > > > > support > > > > > it > > > > > >> > >> (yes, > > > > > >> > >> i know that gremlin-javascript can run on the jvm but there > > are > > > > > >> issues > > > > > >> > >> with > > > > > >> > >> getting it all to work with the test framework that make it > > > > unduly > > > > > >> > >> complicated). > > > > > >> > >> > > > > > >> > >> On other threads I offered the idea that we look to use > > Gherkin > > > > to > > > > > >> write > > > > > >> > >> general Gremlin test specifications, which then could be > read > > > and > > > > > >> > >> processed > > > > > >> > >> by the wide variety of test frameworks that can read that > > > format > > > > - > > > > > >> there > > > > > >> > >> tend to be Gherkin processors in just about every language > - > > > for > > > > > >> > example, > > > > > >> > >> see: > > > > > >> > >> > > > > > >> > >> https://cucumber.io/ > > > > > >> > >> > > > > > >> > >> I just created this issue: > > > > > >> > >> > > > > > >> > >> https://issues.apache.org/jira/browse/TINKERPOP-1784 > > > > > >> > >> > > > > > >> > >> and pushed this branch: > > > > > >> > >> > > > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784 > > > > > >> > >> > > > > > >> > >> which demonstrates how this works with gremlin-python. The > > > basic > > > > > >> anatomy > > > > > >> > >> of > > > > > >> > >> this setup involves this new directory in gremlin-test: > > > > > >> > >> > > > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/ > grem > > > > > >> > >> lin-test/features > > > > > >> > >> > > > > > >> > >> It contains the Gherkin .features files. These are the test > > > > > >> > >> specifications. > > > > > >> > >> They are written using gremlin-java as the "model" > language. > > > GLVs > > > > > >> will > > > > > >> > >> then > > > > > >> > >> need to write some infrastructure to process these Gherkin > > > files. > > > > > The > > > > > >> > key > > > > > >> > >> to making this "easy" to implement will lie in our abiilty > to > > > > keep > > > > > >> the > > > > > >> > >> assertions we want to have relatively simple. The more > > > simplistic > > > > > the > > > > > >> > >> language in the Gherkin .feature files the easier the job > it > > > will > > > > > be > > > > > >> for > > > > > >> > >> GLVs to build their infrastructure. Of course, once that > > > > > >> infrastructure > > > > > >> > is > > > > > >> > >> in place, the GLV developer just has to write the GLV > version > > > of > > > > > the > > > > > >> > >> Gremlin specified in the .feature file. So you can look at > > all > > > > the > > > > > >> > >> "infrastructure" code here in this pair of files: > > > > > >> > >> > > > > > >> > >> https://github.com/apache/tinkerpop/tree/TINKERPOP-1784/ > grem > > > > > >> > >> lin-python/src/main/jython/radish > > > > > >> > >> > > > > > >> > >> and then you can look at the GLV Gremlin translations > > > > specifically > > > > > >> here: > > > > > >> > >> > > > > > >> > >> https://github.com/apache/tinkerpop/blob/TINKERPOP-1784/ > grem > > > > > >> > >> lin-python/src/main/jython/radish/count_features_step.py# > > > L34-L46 > > > > > >> > >> > > > > > >> > >> I think this approach works pretty well and solves our > > general > > > > > >> problems > > > > > >> > >> for > > > > > >> > >> GLV testing. There is some pain up front in implementing > the > > > > > >> > >> "infrastructure" but after that new Gremlin tests added to > > > > .feature > > > > > >> > files > > > > > >> > >> just need to translated in the GLV. I suppose we could > > > > "automate" a > > > > > >> good > > > > > >> > >> portion of the translation with reflection of some sort. > > > Anything > > > > > >> else > > > > > >> > >> could just be handled manually. > > > > > >> > >> > > > > > >> > >> Not sure if we need to use this new model to wholly replace > > the > > > > old > > > > > >> one. > > > > > >> > >> The process test suite has its place in helping graph > > database > > > > > >> providers > > > > > >> > >> test their stuff. I also imagine that introducing this > > approach > > > > in > > > > > >> that > > > > > >> > >> context would create a breaking change which we would then > > need > > > > to > > > > > >> push > > > > > >> > >> off > > > > > >> > >> to 3.4.0. I suppose that gives us time to think, but for > now > > > it > > > > > >> might > > > > > >> > not > > > > > >> > >> be best to conflate the two and just treat them as separate > > > > aspects > > > > > >> of > > > > > >> > the > > > > > >> > >> test suite. > > > > > >> > >> > > > > > >> > >> Anyway - it's important we settle on an approach to testing > > as > > > we > > > > > >> really > > > > > >> > >> shouldn't do a GA release of the Gremlin .NET GLV without > > > getting > > > > > the > > > > > >> > test > > > > > >> > >> suite solid. please yell if you have any ideas or feedback > on > > > > this > > > > > >> > >> approach. > > > > > >> > >> > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
