I just pushed a change that allows python tests to pass against the docker image.
This will install the gremlin-python script support on the server so that's an extra download to wait for. I had to update the localhost references to point to the docker IP in gremlin-python: grep -Rl localhost gremlin-python/src/main/jython/ | xargs sed -i 's/localhost/172.17.0.2/g' Then I could `mvn clean install -pl :gremlin-python` Robert Dale On Wed, Mar 6, 2019 at 9:22 AM Robert Dale <robd...@gmail.com> wrote: > Florian makes a great point about using the image from gremlin-server. > It's much lighter weight. It also makes it easier for non-java, > client-only devs to use since it doesn't require maven if a published > version is used. I've pushed my changes. > > Thus, > # Will use the published gremlin-server image with the 'latest' tag. > ./docker/gremlin-server.sh > > # Will use the published gremlin-server image with the specified version > ./docker/gremlin-server.sh 3.4.0 > > # To use the working copy, build gremlin-server and the image. e.g.: > mvn clean install -pl :gremlin-server -am && mvn -DskipTests install > -Pdocker-images -pl :gremlin-server > > # You still have to know what the working copy version is. Docker will > check local images. > ./docker/gremlin-server.sh 3.4.1-SNAPSHOT > > In all cases, the test scripts are still copied from the working copy so > be aware of any version conflicts between gremlin-server version used and > working copy version test scripts. > > I've also retained the original image's ability to override the yaml file. > However, the version is required even when using 'latest' tag. > > # Will start latest published gremlin-server with the default config on > port 8182 > ./docker/gremlin-server.sh latest conf/gremlin-server.yaml > > > Robert Dale > > > On Fri, Mar 1, 2019 at 3:13 PM Florian Hockmann <f...@florian-hockmann.de> > wrote: > >> Wow, for some reason I didn't see your earlier mail and also the linked >> PR. So, I didn't know that there is already some automation to try out. >> >> That made my attempts to help of course a bit pointless :-/ Sorry >> Stephen if I waisted your time with that. >> >> Regarding our initial discussion here: I think that Testcontainers would >> be the best solution for the GLVs (gremlin-driver was probably a bad >> example), at least in the long term, as it would also enable us to test >> modification steps without breaking other tests. Nevertheless, I agree >> that a simpler approach might be the better solution for now and it is >> definitely an improvement over our current situation. >> >> I will probably wait for testcontainers-dotnet to support using images >> that are only available locally (so we can test with a locally built >> Gremlin Server) and then test out how good it works with our build >> process. >> >> Am 01.03.2019 um 20:50 schrieb Robert Dale: >> > I force pushed a fix. Wasn't listening on all nets. Also added a banner >> > for IP address that will appear like so: >> > ... >> > Successfully built b00dc1f2db35 >> > ####################### >> > IP is 172.17.0.2 >> > ####################### >> > [INFO] GremlinServer - 3.4.1-SNAPSHOT >> > ... >> > >> > Robert Dale >> > >> > >> > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <f...@florian-hockmann.de >> > >> > wrote: >> > >> >> I don't know what the problem could be, but two things I can think of >> >> that could be helpful to find the problem: >> >> >> >> * Connect to the container (via docker exec -it [container_id] bash) >> >> and then confirm with netstat (which you have to install, `apk >> >> update; apk add ospd-netstat` should work) that port 45940 is >> >> actually open and that the foreign address is 0.0.0.0 (`netstat >> -tlpn`) >> >> * Check whether you can connect to that port in general without >> adding >> >> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v >> >> localhost 45940`. >> >> >> >> Simply restarting Docker could also help. (At least on Windows some >> >> strange Docker networks problems are often gone after a restart.) >> >> >> >> You're probably using Gremlin Console from your host system and not >> with >> >> our Docker image, right? >> >> >> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette: >> >>> i tried adding that -p thing to the run command in the script, but i >> >> still >> >>> couldn't connect to it with Gremlin Console >> >>> >> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann < >> f...@florian-hockmann.de> >> >>> wrote: >> >>> >> >>>> Did you start the container with port mapping (e.g., docker run -p >> >>>> 45940:45940 [image-name]) or how are you trying to connect to the >> >>>> container? >> >>>> >> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette: >> >>>>> The server starts but something seems messed up with the ports. It's >> >>>> like I >> >>>>> can't connect to the 45940 port from the host. I tried a couple >> things >> >> I >> >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's >> >> probably >> >>>>> something simple...any ideas? >> >>>>> >> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <robd...@gmail.com> >> wrote: >> >>>>> >> >>>>>> I made a first attempt. Let me know what you think. >> >>>>>> https://github.com/apache/tinkerpop/pull/1075 >> >>>>>> >> >>>>>> Robert Dale >> >>>>>> >> >>>>>> >> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette < >> >> spmalle...@gmail.com >> >>>>>> wrote: >> >>>>>> >> >>>>>>>> If we agree on Testcontainers in general, then I can give this a >> >> try >> >>>>>> and >> >>>>>>> use Testcontainers in one of our Java projects that need Gremlin >> >> Server >> >>>>>> for >> >>>>>>> testing (gremlin-driver maybe?). >> >>>>>>> >> >>>>>>> The java integration tests for Gremlin Server flip a lot of >> >>>> configuration >> >>>>>>> bits per test. I don't know how that works with testcontainers >> and if >> >>>> it >> >>>>>>> will make our testing there any better. There are also a number of >> >>>> tests >> >>>>>>> that stop/start the server under different conditions - not sure >> how >> >>>> that >> >>>>>>> works with testcontainers either. Finally, wouldn't we expect >> adding >> >>>>>> docker >> >>>>>>> to the mix to slow down our integration test runs for java? >> >>>>>>> >> >>>>>>> I don't want to seem like i'm completely against testcontainers >> btw. >> >> I >> >>>>>> just >> >>>>>>> wonder if a script doesn't give us a 100% better experience than >> what >> >>>> we >> >>>>>>> have now (which is nothing). If so, and the script is easy and >> >>>>>> zero-impact >> >>>>>>> without new dependencies, then I think I like that approach. >> Also, It >> >>>>>> would >> >>>>>>> be nice to have the script for other sorts of manual testing, >> such as >> >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do >> >>>>>>> testcontainers for GLVs I suppose that can be done too, but for my >> >>>>>>> purposes, since I was the one who asked for help with this >> initially, >> >>>>>>> testcontainers doesn't sound like something that is an absolute >> >>>>>> requirement >> >>>>>>> for my needs. >> >>>>>>> >> >>>>>>> thanks robert/florian for all the discussion and help - >> appreciated. >> >>>>>>> >> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann < >> >>>>>> f...@florian-hockmann.de> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Yes, except for the detail that Testcontainers could make the >> script >> >>>>>>>> obsolete if it is also used to build the image by itself. That >> would >> >>>>>> have >> >>>>>>>> the advantage that developers wouldn't have to execute the >> script to >> >>>>>>> build >> >>>>>>>> the image. (Executing the script with Testcontainers could also >> be a >> >>>>>>>> possibility, but I'm not sure how good that works.) >> >>>>>>>> >> >>>>>>>> If we agree on Testcontainers in general, then I can give this a >> try >> >>>>>> and >> >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin >> >>>> Server >> >>>>>>> for >> >>>>>>>> testing (gremlin-driver maybe?). >> >>>>>>>> With such a PR it would probably also be easier for others to see >> >> what >> >>>>>>>> advantages / disadvantages using Testcontainers would have. >> >>>>>>>> >> >>>>>>>> I think using a Java project first makes most sense as we can use >> >> all >> >>>>>>>> functionality Testcontainers has to offer there. The downside is >> of >> >>>>>>> course >> >>>>>>>> that we might not have all the functionality afterwards for other >> >>>>>>> languages >> >>>>>>>> and then we might need to extend the Testcontainers libraries for >> >> the >> >>>>>>>> different languages. (My assumption here is that we won't need >> that >> >>>>>> much >> >>>>>>>> functionality so it should be limited to 1 or 2 missing >> features.) >> >>>>>>>> Alternatively, we could also start with a GLV where the >> >> Testcontainers >> >>>>>>>> library doesn't have the same functionality which probably means >> >> that >> >>>>>> we >> >>>>>>>> need a workaround to get an image of our current Gremlin Server >> >>>> version >> >>>>>>> for >> >>>>>>>> testing. >> >>>>>>>> >> >>>>>>>> -----Ursprüngliche Nachricht----- >> >>>>>>>> Von: Robert Dale <robd...@gmail.com> >> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49 >> >>>>>>>> An: dev@tinkerpop.apache.org >> >>>>>>>> Betreff: Re: docker and GLV development >> >>>>>>>> >> >>>>>>>> I think there are two components here. >> >>>>>>>> 1) Script to build the dockerized test server. Includes ability >> to >> >>>>>> start >> >>>>>>>> container manually, externally from tests. >> >>>>>>>> 2) Testcontainers integration which uses the docker image from >> #1. >> >>>>>>>> Testcontainers could also include executing the script in #1 to >> >> build >> >>>>>> the >> >>>>>>>> container. >> >>>>>>>> >> >>>>>>>> Does this sound right to you? >> >>>>>>>> >> >>>>>>>> Robert Dale >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann < >> >>>>>> f...@florian-hockmann.de >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>>> I do like the idea of being able to test against older versions >> >>>>>>> though. >> >>>>>>>>> That shouldn't be a problem in general as we could use multiple >> >>>>>>>>> different images. See for example this PR that adds >> Testcontainers >> >> to >> >>>>>>>>> JanusGraph to test different Elasticsearch versions: >> >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409 >> >>>>>>>>> >> >>>>>>>>>> Yes, an on-the-fly container would work >> >>>>>>>>> We could however also use an on-the-fly container with >> >>>>>> Testcontainers. >> >>>>>>>>> We would just need some automation to build the image in the >> >>>>>>> background. >> >>>>>>>>> What if we would build an image during the usual Maven build for >> >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we >> >> could >> >>>>>>>>> use that tag with Testcontainers in addition to tags for already >> >>>>>>>> released versions. >> >>>>>>>>> Testcontainers-Java can apparently even build images by itself. >> So, >> >>>>>>>>> that could also be a possibility to get an image with the latest >> >>>>>>>>> Gremlin Server with the current (unreleased) changes. >> >>>>>>>>> >> >>>>>>>>> In general, I would really like to get away from a situation >> where >> >> we >> >>>>>>>>> have to start some script in the background before tests can be >> >>>>>>> executed. >> >>>>>>>>> @Robert: >> >>>>>>>>>> What is the test configuration/data for GLVs? Could you list >> the >> >>>>>>>>> paths/files? >> >>>>>>>>> >> >>>>>>>>> For Gremlin.Net for example, you can find the automation >> happening >> >> in >> >>>>>>>>> the pom.xml of the testing part: >> >>>>>>>>> >> >>>>>>>>> >> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d >> >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179 >> >>>>>>>>> >> >>>>>>>>> It calls the test-server-start.groovy script before the tests >> are >> >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop >> the >> >>>>>>>> server again. >> >>>>>>>>> This has the obvious downside that the tests can only be >> executed >> >>>>>> from >> >>>>>>>>> the Maven build which isn't the build tool .NET developers would >> >> use >> >>>>>>>>> and also not the tool the IDE will use (the situation is >> basically >> >>>>>> the >> >>>>>>>>> same for other GLVs). >> >>>>>>>>> >> >>>>>>>>> -----Ursprüngliche Nachricht----- >> >>>>>>>>> Von: Robert Dale <robd...@gmail.com> >> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52 >> >>>>>>>>> An: dev@tinkerpop.apache.org >> >>>>>>>>> Betreff: Re: docker and GLV development >> >>>>>>>>> >> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-) >> >>>>>>>>> >> >>>>>>>>> What is the test configuration/data for GLVs? Could you list >> the >> >>>>>>>>> paths/files? >> >>>>>>>>> >> >>>>>>>>> Robert Dale >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette >> >>>>>>>>> <spmalle...@gmail.com> >> >>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the >> test >> >>>>>>>>>> configuration/data for GLVs. That's why i suggested the >> command as >> >>>>>> I >> >>>>>>>>>> did because it was similar to docker/build.sh. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <robd...@gmail.com >> > >> >>>>>>> wrote: >> >>>>>>>>>>> Do you want to create a docker container on-the-fly using the >> >>>>>>>>>>> current working copy kind of like we do for docker/build.sh? >> >>>>>>>>>>> >> >>>>>>>>>>> Robert Dale >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette >> >>>>>>>>>>> <spmalle...@gmail.com> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration >> and >> >>>>>>>>>>> automation, >> >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great. >> >>>>>>>>>>>> There are times when changes are occurring on both client and >> >>>>>>>>>>>> server so a local up-to-the-moment image would be necessary. >> I >> >>>>>>>>>>>> also think we might want >> >>>>>>>>>> to >> >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out >> >>>>>>>>>>>> more >> >>>>>>>>>>> clearly >> >>>>>>>>>>>> how those play into projects (there is a bit of discussion >> >>>>>> going >> >>>>>>>>>>>> on >> >>>>>>>>>> with >> >>>>>>>>>>>> that now). I do like the idea of being able to test against >> >>>>>>>>>>>> older >> >>>>>>>>>>> versions >> >>>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann < >> >>>>>>>>>> f...@florian-hockmann.de >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start >> >>>>>> tests >> >>>>>>>>>>>>> for Gremlin.Net from my IDE. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at >> >>>>>>>>>>>>> problems >> >>>>>>>>>>> like >> >>>>>>>>>>>>> this: >> >>>>>>>>>>>>> https://www.testcontainers.org/ >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker >> >>>>>>>>>> container >> >>>>>>>>>>> in >> >>>>>>>>>>>>> your tests for whatever external dependencies you need. >> >>>>>>>>>>>>> Databases >> >>>>>>>>>> are a >> >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us, >> >>>>>>>>>>>>> Gremlin >> >>>>>>>>>>> Server >> >>>>>>>>>>>>> would be such a container. >> >>>>>>>>>>>>> Testcontainers then uses the ability of the testing >> framework >> >>>>>>>> (e.g. >> >>>>>>>>>>>> JUnit) >> >>>>>>>>>>>>> to start the Docker container for the tests and waits until >> >>>>>>>>>>>>> they are >> >>>>>>>>>>>> fully >> >>>>>>>>>>>>> started (how they are started can be configured for each >> >>>>>>>>>>>>> container differently). After the tests have completed, it >> >>>>>>>>>>>>> also stops the >> >>>>>>>>>>>> containers >> >>>>>>>>>>>>> again. >> >>>>>>>>>>>>> That way, you don't have to manage the containers manually >> >>>>>>>> anymore. >> >>>>>>>>>>>>> Everything is done in the background by the IDE / the build >> >>>>>>> tool. >> >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now >> >>>>>>>>>>>>> also have libraries for different languages, including the >> >>>>>>>>>>>>> ones for which we >> >>>>>>>>>> have >> >>>>>>>>>>>>> GLVs: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> testcontainers-python: >> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python >> >>>>>>>>>>>>> testcontainers-dotnet >> >>>>>>>>>>>>> < >> >>>>>> >> https://github.com/testcontainers/testcontainers-pythontestcontainer >> >>>>>>>>>> s- >> >>>>>>>>>> dotnet >> >>>>>>>>>>>>> : >> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet >> >>>>>>>>>>>>> testcontainers-node >> >>>>>>>>>>>>> < >> >>>>>> >> https://github.com/testcontainers/testcontainers-dotnettestcontainer >> >>>>>>>>>> s- >> >>>>>>>>>> node >> >>>>>>>>>>>>> : >> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of >> >>>>>>>>> course. >> >>>>>>>>>>> The >> >>>>>>>>>>>>> others still lack some functionality, but I guess that we >> >>>>>>>>>>>>> could also >> >>>>>>>>>>> just >> >>>>>>>>>>>>> contribute to those libraries if an important feature is >> >>>>>>>>>>>>> missing for >> >>>>>>>>>>> us. >> >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but >> >>>>>>>>>>> testcontainers-dotnet >> >>>>>>>>>>>>> unfortunately can only use images that are available in some >> >>>>>>>>>> registry. >> >>>>>>>>>>>> So, >> >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to >> >>>>>>>>>>>>> DockerHub >> >>>>>>>>>> to >> >>>>>>>>>>>> use >> >>>>>>>>>>>>> it in our tests. That means that we could either only test >> >>>>>> the >> >>>>>>>>>>>>> GLVs >> >>>>>>>>>>>> against >> >>>>>>>>>>>>> already released versions or we could publish something like >> >>>>>>>>>>>>> nightly >> >>>>>>>>>>>> build >> >>>>>>>>>>>>> images of Gremlin Server. >> >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is >> >>>>>>>>>>>>> able to >> >>>>>>>>>> use >> >>>>>>>>>>>>> images that are only available locally (testcontainers-java >> >>>>>>>>>>>>> already >> >>>>>>>>>> has >> >>>>>>>>>>>>> that capability), but now that I think more about this, I >> >>>>>>>>>>>>> think that >> >>>>>>>>>> it >> >>>>>>>>>>>>> makes actually more sense to use images that are available >> in >> >>>>>>>>>>>>> the >> >>>>>>>>>>>> registry >> >>>>>>>>>>>>> as every developer would otherwise have to build test images >> >>>>>>>>>>>>> first >> >>>>>>>>>>>> locally >> >>>>>>>>>>>>> in order to use them for tests. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to >> >>>>>>>>>>>>> see >> >>>>>>>>>>> whether >> >>>>>>>>>>>>> you agree that it's a good solution to our problem and then >> >>>>>> we >> >>>>>>>>>>>>> can >> >>>>>>>>>> see >> >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests >> >>>>>> or >> >>>>>>>>>>>>> how we >> >>>>>>>>>>>> want >> >>>>>>>>>>>>> to handle that. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -----Ursprüngliche Nachricht----- >> >>>>>>>>>>>>> Von: Stephen Mallette <spmalle...@gmail.com> >> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24 >> >>>>>>>>>>>>> An: dev@tinkerpop.apache.org >> >>>>>>>>>>>>> Betreff: docker and GLV development >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be >> >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love >> to >> >>>>>>>>>>>>> be able to do >> >>>>>>>>>>>> something >> >>>>>>>>>>>>> like: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> docker/gremlin-server.sh -test >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> which would start Gremlin Server with our standard test >> >>>>>>>>>>>>> configuration (currently built into the maven tool chain, >> but >> >>>>>>>>>>>>> I think that could be extracted without too much >> difficulty). >> >>>>>>>>>>>>> That way, we could easily >> >>>>>>>>>> open >> >>>>>>>>>>> up >> >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly >> in >> >>>>>>>>>>>>> there, >> >>>>>>>>>>> use >> >>>>>>>>>>>>> the debugger, etc. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Sound useful? and if so, is that something that is >> >>>>>>>>>>>>> easy/possible? and >> >>>>>>>>>>>> then >> >>>>>>>>>>>>> further....any volunteers with the know-how to get that >> setup >> >>>>>>>>>>>>> (i'm >> >>>>>>>>>>> happy >> >>>>>>>>>>>> to >> >>>>>>>>>>>>> assist however i can)? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for >> GLV >> >>>>>>>>>>> development >> >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't >> >>>>>> mind >> >>>>>>>>>>>>> using >> >>>>>>>>>>> the >> >>>>>>>>>>>>> maven integration but i feel like i could be more efficient >> >>>>>>>>>>>>> with a >> >>>>>>>>>> more >> >>>>>>>>>>>>> streamlined setup. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>
