Wow, for some reason I didn't see your earlier mail and also the linked PR. So, I didn't know that there is already some automation to try out.
That made my attempts to help of course a bit pointless :-/ Sorry Stephen if I waisted your time with that. Regarding our initial discussion here: I think that Testcontainers would be the best solution for the GLVs (gremlin-driver was probably a bad example), at least in the long term, as it would also enable us to test modification steps without breaking other tests. Nevertheless, I agree that a simpler approach might be the better solution for now and it is definitely an improvement over our current situation. I will probably wait for testcontainers-dotnet to support using images that are only available locally (so we can test with a locally built Gremlin Server) and then test out how good it works with our build process. Am 01.03.2019 um 20:50 schrieb Robert Dale: > I force pushed a fix. Wasn't listening on all nets. Also added a banner > for IP address that will appear like so: > ... > Successfully built b00dc1f2db35 > ####################### > IP is 172.17.0.2 > ####################### > [INFO] GremlinServer - 3.4.1-SNAPSHOT > ... > > Robert Dale > > > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <[email protected]> > wrote: > >> I don't know what the problem could be, but two things I can think of >> that could be helpful to find the problem: >> >> * Connect to the container (via docker exec -it [container_id] bash) >> and then confirm with netstat (which you have to install, `apk >> update; apk add ospd-netstat` should work) that port 45940 is >> actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`) >> * Check whether you can connect to that port in general without adding >> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v >> localhost 45940`. >> >> Simply restarting Docker could also help. (At least on Windows some >> strange Docker networks problems are often gone after a restart.) >> >> You're probably using Gremlin Console from your host system and not with >> our Docker image, right? >> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette: >>> i tried adding that -p thing to the run command in the script, but i >> still >>> couldn't connect to it with Gremlin Console >>> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <[email protected]> >>> wrote: >>> >>>> Did you start the container with port mapping (e.g., docker run -p >>>> 45940:45940 [image-name]) or how are you trying to connect to the >>>> container? >>>> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette: >>>>> The server starts but something seems messed up with the ports. It's >>>> like I >>>>> can't connect to the 45940 port from the host. I tried a couple things >> I >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's >> probably >>>>> something simple...any ideas? >>>>> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <[email protected]> wrote: >>>>> >>>>>> I made a first attempt. Let me know what you think. >>>>>> https://github.com/apache/tinkerpop/pull/1075 >>>>>> >>>>>> Robert Dale >>>>>> >>>>>> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette < >> [email protected] >>>>>> wrote: >>>>>> >>>>>>>> If we agree on Testcontainers in general, then I can give this a >> try >>>>>> and >>>>>>> use Testcontainers in one of our Java projects that need Gremlin >> Server >>>>>> for >>>>>>> testing (gremlin-driver maybe?). >>>>>>> >>>>>>> The java integration tests for Gremlin Server flip a lot of >>>> configuration >>>>>>> bits per test. I don't know how that works with testcontainers and if >>>> it >>>>>>> will make our testing there any better. There are also a number of >>>> tests >>>>>>> that stop/start the server under different conditions - not sure how >>>> that >>>>>>> works with testcontainers either. Finally, wouldn't we expect adding >>>>>> docker >>>>>>> to the mix to slow down our integration test runs for java? >>>>>>> >>>>>>> I don't want to seem like i'm completely against testcontainers btw. >> I >>>>>> just >>>>>>> wonder if a script doesn't give us a 100% better experience than what >>>> we >>>>>>> have now (which is nothing). If so, and the script is easy and >>>>>> zero-impact >>>>>>> without new dependencies, then I think I like that approach. Also, It >>>>>> would >>>>>>> be nice to have the script for other sorts of manual testing, such as >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do >>>>>>> testcontainers for GLVs I suppose that can be done too, but for my >>>>>>> purposes, since I was the one who asked for help with this initially, >>>>>>> testcontainers doesn't sound like something that is an absolute >>>>>> requirement >>>>>>> for my needs. >>>>>>> >>>>>>> thanks robert/florian for all the discussion and help - appreciated. >>>>>>> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann < >>>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, except for the detail that Testcontainers could make the script >>>>>>>> obsolete if it is also used to build the image by itself. That would >>>>>> have >>>>>>>> the advantage that developers wouldn't have to execute the script to >>>>>>> build >>>>>>>> the image. (Executing the script with Testcontainers could also be a >>>>>>>> possibility, but I'm not sure how good that works.) >>>>>>>> >>>>>>>> If we agree on Testcontainers in general, then I can give this a try >>>>>> and >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin >>>> Server >>>>>>> for >>>>>>>> testing (gremlin-driver maybe?). >>>>>>>> With such a PR it would probably also be easier for others to see >> what >>>>>>>> advantages / disadvantages using Testcontainers would have. >>>>>>>> >>>>>>>> I think using a Java project first makes most sense as we can use >> all >>>>>>>> functionality Testcontainers has to offer there. The downside is of >>>>>>> course >>>>>>>> that we might not have all the functionality afterwards for other >>>>>>> languages >>>>>>>> and then we might need to extend the Testcontainers libraries for >> the >>>>>>>> different languages. (My assumption here is that we won't need that >>>>>> much >>>>>>>> functionality so it should be limited to 1 or 2 missing features.) >>>>>>>> Alternatively, we could also start with a GLV where the >> Testcontainers >>>>>>>> library doesn't have the same functionality which probably means >> that >>>>>> we >>>>>>>> need a workaround to get an image of our current Gremlin Server >>>> version >>>>>>> for >>>>>>>> testing. >>>>>>>> >>>>>>>> -----Ursprüngliche Nachricht----- >>>>>>>> Von: Robert Dale <[email protected]> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49 >>>>>>>> An: [email protected] >>>>>>>> Betreff: Re: docker and GLV development >>>>>>>> >>>>>>>> I think there are two components here. >>>>>>>> 1) Script to build the dockerized test server. Includes ability to >>>>>> start >>>>>>>> container manually, externally from tests. >>>>>>>> 2) Testcontainers integration which uses the docker image from #1. >>>>>>>> Testcontainers could also include executing the script in #1 to >> build >>>>>> the >>>>>>>> container. >>>>>>>> >>>>>>>> Does this sound right to you? >>>>>>>> >>>>>>>> Robert Dale >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann < >>>>>> [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>>> I do like the idea of being able to test against older versions >>>>>>> though. >>>>>>>>> That shouldn't be a problem in general as we could use multiple >>>>>>>>> different images. See for example this PR that adds Testcontainers >> to >>>>>>>>> JanusGraph to test different Elasticsearch versions: >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409 >>>>>>>>> >>>>>>>>>> Yes, an on-the-fly container would work >>>>>>>>> We could however also use an on-the-fly container with >>>>>> Testcontainers. >>>>>>>>> We would just need some automation to build the image in the >>>>>>> background. >>>>>>>>> What if we would build an image during the usual Maven build for >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we >> could >>>>>>>>> use that tag with Testcontainers in addition to tags for already >>>>>>>> released versions. >>>>>>>>> Testcontainers-Java can apparently even build images by itself. So, >>>>>>>>> that could also be a possibility to get an image with the latest >>>>>>>>> Gremlin Server with the current (unreleased) changes. >>>>>>>>> >>>>>>>>> In general, I would really like to get away from a situation where >> we >>>>>>>>> have to start some script in the background before tests can be >>>>>>> executed. >>>>>>>>> @Robert: >>>>>>>>>> What is the test configuration/data for GLVs? Could you list the >>>>>>>>> paths/files? >>>>>>>>> >>>>>>>>> For Gremlin.Net for example, you can find the automation happening >> in >>>>>>>>> the pom.xml of the testing part: >>>>>>>>> >>>>>>>>> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179 >>>>>>>>> >>>>>>>>> It calls the test-server-start.groovy script before the tests are >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop the >>>>>>>> server again. >>>>>>>>> This has the obvious downside that the tests can only be executed >>>>>> from >>>>>>>>> the Maven build which isn't the build tool .NET developers would >> use >>>>>>>>> and also not the tool the IDE will use (the situation is basically >>>>>> the >>>>>>>>> same for other GLVs). >>>>>>>>> >>>>>>>>> -----Ursprüngliche Nachricht----- >>>>>>>>> Von: Robert Dale <[email protected]> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52 >>>>>>>>> An: [email protected] >>>>>>>>> Betreff: Re: docker and GLV development >>>>>>>>> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-) >>>>>>>>> >>>>>>>>> What is the test configuration/data for GLVs? Could you list the >>>>>>>>> paths/files? >>>>>>>>> >>>>>>>>> Robert Dale >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette >>>>>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the test >>>>>>>>>> configuration/data for GLVs. That's why i suggested the command as >>>>>> I >>>>>>>>>> did because it was similar to docker/build.sh. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <[email protected]> >>>>>>> wrote: >>>>>>>>>>> Do you want to create a docker container on-the-fly using the >>>>>>>>>>> current working copy kind of like we do for docker/build.sh? >>>>>>>>>>> >>>>>>>>>>> Robert Dale >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette >>>>>>>>>>> <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration and >>>>>>>>>>> automation, >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great. >>>>>>>>>>>> There are times when changes are occurring on both client and >>>>>>>>>>>> server so a local up-to-the-moment image would be necessary. I >>>>>>>>>>>> also think we might want >>>>>>>>>> to >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out >>>>>>>>>>>> more >>>>>>>>>>> clearly >>>>>>>>>>>> how those play into projects (there is a bit of discussion >>>>>> going >>>>>>>>>>>> on >>>>>>>>>> with >>>>>>>>>>>> that now). I do like the idea of being able to test against >>>>>>>>>>>> older >>>>>>>>>>> versions >>>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann < >>>>>>>>>> [email protected] >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start >>>>>> tests >>>>>>>>>>>>> for Gremlin.Net from my IDE. >>>>>>>>>>>>> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at >>>>>>>>>>>>> problems >>>>>>>>>>> like >>>>>>>>>>>>> this: >>>>>>>>>>>>> https://www.testcontainers.org/ >>>>>>>>>>>>> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker >>>>>>>>>> container >>>>>>>>>>> in >>>>>>>>>>>>> your tests for whatever external dependencies you need. >>>>>>>>>>>>> Databases >>>>>>>>>> are a >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us, >>>>>>>>>>>>> Gremlin >>>>>>>>>>> Server >>>>>>>>>>>>> would be such a container. >>>>>>>>>>>>> Testcontainers then uses the ability of the testing framework >>>>>>>> (e.g. >>>>>>>>>>>> JUnit) >>>>>>>>>>>>> to start the Docker container for the tests and waits until >>>>>>>>>>>>> they are >>>>>>>>>>>> fully >>>>>>>>>>>>> started (how they are started can be configured for each >>>>>>>>>>>>> container differently). After the tests have completed, it >>>>>>>>>>>>> also stops the >>>>>>>>>>>> containers >>>>>>>>>>>>> again. >>>>>>>>>>>>> That way, you don't have to manage the containers manually >>>>>>>> anymore. >>>>>>>>>>>>> Everything is done in the background by the IDE / the build >>>>>>> tool. >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now >>>>>>>>>>>>> also have libraries for different languages, including the >>>>>>>>>>>>> ones for which we >>>>>>>>>> have >>>>>>>>>>>>> GLVs: >>>>>>>>>>>>> >>>>>>>>>>>>> testcontainers-python: >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python >>>>>>>>>>>>> testcontainers-dotnet >>>>>>>>>>>>> < >>>>>> https://github.com/testcontainers/testcontainers-pythontestcontainer >>>>>>>>>> s- >>>>>>>>>> dotnet >>>>>>>>>>>>> : >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet >>>>>>>>>>>>> testcontainers-node >>>>>>>>>>>>> < >>>>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer >>>>>>>>>> s- >>>>>>>>>> node >>>>>>>>>>>>> : >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node >>>>>>>>>>>>> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of >>>>>>>>> course. >>>>>>>>>>> The >>>>>>>>>>>>> others still lack some functionality, but I guess that we >>>>>>>>>>>>> could also >>>>>>>>>>> just >>>>>>>>>>>>> contribute to those libraries if an important feature is >>>>>>>>>>>>> missing for >>>>>>>>>>> us. >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but >>>>>>>>>>> testcontainers-dotnet >>>>>>>>>>>>> unfortunately can only use images that are available in some >>>>>>>>>> registry. >>>>>>>>>>>> So, >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to >>>>>>>>>>>>> DockerHub >>>>>>>>>> to >>>>>>>>>>>> use >>>>>>>>>>>>> it in our tests. That means that we could either only test >>>>>> the >>>>>>>>>>>>> GLVs >>>>>>>>>>>> against >>>>>>>>>>>>> already released versions or we could publish something like >>>>>>>>>>>>> nightly >>>>>>>>>>>> build >>>>>>>>>>>>> images of Gremlin Server. >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is >>>>>>>>>>>>> able to >>>>>>>>>> use >>>>>>>>>>>>> images that are only available locally (testcontainers-java >>>>>>>>>>>>> already >>>>>>>>>> has >>>>>>>>>>>>> that capability), but now that I think more about this, I >>>>>>>>>>>>> think that >>>>>>>>>> it >>>>>>>>>>>>> makes actually more sense to use images that are available in >>>>>>>>>>>>> the >>>>>>>>>>>> registry >>>>>>>>>>>>> as every developer would otherwise have to build test images >>>>>>>>>>>>> first >>>>>>>>>>>> locally >>>>>>>>>>>>> in order to use them for tests. >>>>>>>>>>>>> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to >>>>>>>>>>>>> see >>>>>>>>>>> whether >>>>>>>>>>>>> you agree that it's a good solution to our problem and then >>>>>> we >>>>>>>>>>>>> can >>>>>>>>>> see >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests >>>>>> or >>>>>>>>>>>>> how we >>>>>>>>>>>> want >>>>>>>>>>>>> to handle that. >>>>>>>>>>>>> >>>>>>>>>>>>> -----Ursprüngliche Nachricht----- >>>>>>>>>>>>> Von: Stephen Mallette <[email protected]> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24 >>>>>>>>>>>>> An: [email protected] >>>>>>>>>>>>> Betreff: docker and GLV development >>>>>>>>>>>>> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love to >>>>>>>>>>>>> be able to do >>>>>>>>>>>> something >>>>>>>>>>>>> like: >>>>>>>>>>>>> >>>>>>>>>>>>> docker/gremlin-server.sh -test >>>>>>>>>>>>> >>>>>>>>>>>>> which would start Gremlin Server with our standard test >>>>>>>>>>>>> configuration (currently built into the maven tool chain, but >>>>>>>>>>>>> I think that could be extracted without too much difficulty). >>>>>>>>>>>>> That way, we could easily >>>>>>>>>> open >>>>>>>>>>> up >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in >>>>>>>>>>>>> there, >>>>>>>>>>> use >>>>>>>>>>>>> the debugger, etc. >>>>>>>>>>>>> >>>>>>>>>>>>> Sound useful? and if so, is that something that is >>>>>>>>>>>>> easy/possible? and >>>>>>>>>>>> then >>>>>>>>>>>>> further....any volunteers with the know-how to get that setup >>>>>>>>>>>>> (i'm >>>>>>>>>>> happy >>>>>>>>>>>> to >>>>>>>>>>>>> assist however i can)? >>>>>>>>>>>>> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV >>>>>>>>>>> development >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't >>>>>> mind >>>>>>>>>>>>> using >>>>>>>>>>> the >>>>>>>>>>>>> maven integration but i feel like i could be more efficient >>>>>>>>>>>>> with a >>>>>>>>>> more >>>>>>>>>>>>> streamlined setup. >>>>>>>>>>>>> >>>>>>>>>>>>>
