hey folks! I wanted to gather any last thoughts that people might have. I'd
like to get started setting this up - anyone else have input?

S

On Thu, Jan 19, 2017 at 11:41 AM Stephen Sisk <s...@google.com> wrote:

> Glad to hear you support kubernetes (although to be clear, I'm rooting for
> the right solution for us in the long run - if anyone has a strong reason
> for dcos, I'm excited to hear it.)
>
> I agree with you that testing IO in failure scenarios seems like a
> fruitful area for future work, but that I don't want to tackle it just yet
> (and I'm not hearing that we think it affects our current decision - if
> someone does, I'd like to hear about it.) I am going to split off a thread
> for that discussion because I think the discussion informs how we write our
> unit tests currently, and want to clarify it.
>
> On Wed, Jan 18, 2017 at 1:42 PM Ismaël Mejía <ieme...@gmail.com> wrote:
>
> ​Hello again,
>
> Stephen, I agree with you the real question is what is the scope of the
> tests, maybe the discussion so far has been more about testing a ‘real’
> data store and finding infra/performance issues (and future regressions),
> but having a modern cluster manager opens the door to create more
> interesting integration tests like the ones I mentioned, in particular my
> idea is more oriented towards the validation of the ‘correct’expected
> behavior of the IOs and runners. But this is quite ambitious for a first
> goal, maybe we should first get things working and let this for later (if
> there is still interest).
>
> I am not sure that unit tests are enough to test distribution issues
> because they are harder to simulate in particular if we add the fact that
> we can have too many moving pieces. For example, imagine that we run a Beam
> pipeline deployed via Spark on a YARN cluster (where some nodes can fail)
> that reads from Kafka (with some slow partition) and writes to Cassandra
> (with a partition that goes down). You see, this is a quite complex
> combination of pieces (and possible issues), but it is not a totally
> artificial scenario, in fact this is a common architecture, and this can
> (at least in theory) be simulated with a cluster manager, but I don’t see
> how can I easily reproduce this with a unit test.
>
> Anyway, this scenario makes me think that the boundaries of what we want to
> test are really important. Complexity can be huge.
>
> About the Mesos package question, effectively I referred to Mesos Universe
> (the repo you linked), and what you said is sadly true, it is not easy to
> find multi-node instance packages that are the most interesting ones for
> our tests (in both k8s or mesos). I agree with your decision of using
> Kubernetes, I just wanted to mention that in some cases we will need to
> produce these multi-node packages to have interesting tests.
>
> Ismaël​
>
>
> On Wed, Jan 18, 2017 at 10:09 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
> wrote:
>
> > Yes, for both DCOS (Mesos+Marathon) and Kubernetes, I think we may find
> > single node config but not sure for multi-node setup. Anyway, I'm not
> sure
> > if we find a multi-node configuration, it would cover our needs.
> >
> > Regards
> > JB
> >
> > On 01/18/2017 12:52 PM, Stephen Sisk wrote:
> >
> >> ah! I looked around a bit more and found the dcos package repo -
> >> https://github.com/mesosphere/universe/tree/version-3.x/repo/packages
> >>
> >> poking around a bit, I can find a lot of packages for single node
> >> instances, but not many packages for multi-node instances. Single node
> >> instance packages are kind of useful, but I don't think it's *too*
> >> helpful.
> >> The multi-node instance packages that run the data store's high
> >> availability mode are where the real work is, and it seems like both
> >> kubernetes helm and dcos' package universe don't have a lot of those.
> >>
> >> S
> >>
> >> On Wed, Jan 18, 2017 at 9:56 AM Stephen Sisk <s...@google.com> wrote:
> >>
> >> Hi Ishmael,
> >>>
> >>> these are good questions, thanks for raising them.
> >>>
> >>> Ability to modify network/compute resources to simulate failures
> >>> =================================================
> >>> I see two real questions here:
> >>> 1. Is this something we want to do?
> >>> 2. Is it possible with both/either?
> >>>
> >>> So far, the test strategy I've been advocating is that we test problems
> >>> like this in unit tests rather than do this in ITs/Perf tests.
> Otherwise,
> >>> it's hard to re-create the same conditions.
> >>>
> >>> I can investigate whether it's possible, but I want to clarify whether
> >>> this is something that we care about. I know both support killing
> >>> individual nodes. I haven't seen a lot of network control in either,
> but
> >>> haven't tried to look for it.
> >>>
> >>> Availability of ready to play packages
> >>> ============================
> >>> I did look at this, and as far as I could tell, mesos didn't have any
> >>> pre-built packages for multi-node clusters of data stores. If there's a
> >>> good repository of them that we trust, that would definitely save us
> >>> time.
> >>> Can you point me at the mesos repository?
> >>>
> >>> S
> >>>
> >>>
> >>>
> >>> On Wed, Jan 18, 2017 at 8:37 AM Jean-Baptiste Onofré <j...@nanthrax.net>
> >>> wrote:
> >>>
> >>> ⁣​Hi Ismael
> >>>
> >>> Stephen will reply with details but I know he did a comparison and
> >>> evaluate different options.
> >>>
> >>> He tested with the jdbc Io itests.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On Jan 18, 2017, 08:26, at 08:26, "Ismaël Mejía" <ieme...@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks for your analysis Stephen, good arguments / references.
> >>>>
> >>>> One quick question. Have you checked the APIs of both
> >>>> (Mesos/Kubernetes) to
> >>>> see
> >>>> if we can do programmatically do more complex tests (I suppose so, but
> >>>> you
> >>>> don't mention how easy or if those are possible), for example to
> >>>> simulate a
> >>>> slow networking slave (to test stragglers), or to arbitrarily kill one
> >>>> slave (e.g. if I want to test the correct behavior of a runner/IO that
> >>>> is
> >>>> reading from it) ?
> >>>>
> >>>> Other missing point in the review is the availability of ready to play
> >>>> packages,
> >>>> I think in this area mesos/dcos seems more advanced no? I haven't
> >>>> looked
> >>>> recently but at least 6 months ago there were not many helm packages
> >>>> ready
> >>>> for
> >>>> example to test kafka or the hadoop echosystem stuff (hdfs, hbase,
> >>>> etc). Has
> >>>> this been improved ? because preparing this also is a considerable
> >>>> amount of
> >>>> work on the other hand this could be also a chance to contribute to
> >>>> kubernetes.
> >>>>
> >>>> Regards,
> >>>> Ismaël
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Jan 18, 2017 at 2:36 AM, Stephen Sisk <s...@google.com.invalid
> >
> >>>> wrote:
> >>>>
> >>>> hi!
> >>>>>
> >>>>> I've been continuing this investigation, and have some more info to
> >>>>>
> >>>> report,
> >>>>
> >>>>> and hopefully we can start making some decisions.
> >>>>>
> >>>>> To support performance testing, I've been investigating
> >>>>>
> >>>> mesos+marathon and
> >>>>
> >>>>> kubernetes for running data stores in their high availability mode. I
> >>>>>
> >>>> have
> >>>>
> >>>>> been examining features that kubernetes/mesos+marathon use to support
> >>>>>
> >>>> this.
> >>>>
> >>>>>
> >>>>> Setting up a multi-node cluster in a high availability mode tends to
> >>>>>
> >>>> be
> >>>>
> >>>>> more expensive time-wise than the single node instances I've played
> >>>>>
> >>>> around
> >>>>
> >>>>> with in the past. Rather than do a full build out with both
> >>>>>
> >>>> kubernetes and
> >>>>
> >>>>> mesos, I'd like to pick one of the two options to build the prototype
> >>>>> cluster with. If the prototype doesn't go well, we could still go
> >>>>>
> >>>> back to
> >>>>
> >>>>> the other option, but I'd like to change us from a mode of "let's
> >>>>>
> >>>> look at
> >>>>
> >>>>> all the options" to one of "here's the favorite, let's prove that
> >>>>>
> >>>> works for
> >>>>
> >>>>> us".
> >>>>>
> >>>>> Below are the features that I've seen are important to multi-node
> >>>>>
> >>>> instances
> >>>>
> >>>>> of data stores. I'm sure other folks on the list have done this
> >>>>>
> >>>> before, so
> >>>>
> >>>>> feel free to pipe up if I'm missing a good solution to a problem.
> >>>>>
> >>>>> DNS/Discovery
> >>>>>
> >>>>> --------------------
> >>>>>
> >>>>> Necessary for talking between nodes (eg, cassandra nodes all need to
> >>>>>
> >>>> be
> >>>>
> >>>>> able to talk to a set of seed nodes.)
> >>>>>
> >>>>> * Kubernetes has built-in DNS/discovery between nodes.
> >>>>>
> >>>>> * Mesos has supports this via mesos-dns, which isn't a part of core
> >>>>>
> >>>> mesos,
> >>>>
> >>>>> but is in dcos, which is the mesos distribution I've been using and
> >>>>>
> >>>> that I
> >>>>
> >>>>> would expect us to use.
> >>>>>
> >>>>> Instances properly distributed across nodes
> >>>>>
> >>>>> ------------------------------------------------------------
> >>>>>
> >>>>> If multiple instances of a data source end up on the same underlying
> >>>>>
> >>>> VM, we
> >>>>
> >>>>> may not get good performance out of those instances since the
> >>>>>
> >>>> underlying VM
> >>>>
> >>>>> may be more taxed than other VMs.
> >>>>>
> >>>>> * Kubernetes has a beta feature StatefulSets[1] which allow for
> >>>>>
> >>>> containers
> >>>>
> >>>>> distributed so that there's one container per underlying machine (as
> >>>>>
> >>>> well
> >>>>
> >>>>> as a lot of other useful features like easy stable dns names.)
> >>>>>
> >>>>> * Mesos can support this via the built in UNIQUE constraint [2]
> >>>>>
> >>>>> Load balancing
> >>>>>
> >>>>> --------------------
> >>>>>
> >>>>> Incoming requests from users need to be distributed to the various
> >>>>>
> >>>> machines
> >>>>
> >>>>> - this is important for many data stores' high availability modes.
> >>>>>
> >>>>> * Kubernetes supports easily hooking up to an external load balancer
> >>>>>
> >>>> when
> >>>>
> >>>>> on a cloud (and can be configured to work with a built-in load
> >>>>>
> >>>> balancer if
> >>>>
> >>>>> not)
> >>>>>
> >>>>> * Mesos supports this via marathon-lb [3], which is an install-able
> >>>>>
> >>>> package
> >>>>
> >>>>> in DC/OS
> >>>>>
> >>>>> Persistent Volumes tied to specific instances
> >>>>>
> >>>>> ------------------------------------------------------------
> >>>>>
> >>>>> Databases often need persistent state (for example to store the data
> >>>>>
> >>>> :), so
> >>>>
> >>>>> it's an important part of running our service.
> >>>>>
> >>>>> * Kubernetes StatefulSets supports this
> >>>>>
> >>>>> * Mesos+marathon apps with persistent volumes supports this [4] [5]
> >>>>>
> >>>>> As I mentioned above, I'd like to focus on either kubernetes or mesos
> >>>>>
> >>>> for
> >>>>
> >>>>> my investigation, and as I go further along, I'm seeing kubernetes as
> >>>>> better suited to our needs.
> >>>>>
> >>>>> (1) It supports more of the features we want out of the box and with
> >>>>> StatefulSets, Kubernetes handles them all together neatly - eg. DC/OS
> >>>>> requires marathon-lb to be installed and mesos-dns to be configured.
> >>>>>
> >>>>> (2) I'm also finding that there seem to be more examples of using
> >>>>> kubernetes to solve the types of problems we're working on. This is
> >>>>> somewhat subjective, but in my experience as I've tried to learn both
> >>>>> kubernetes and mesos, I personally found it generally easier to get
> >>>>> kubernetes running than mesos due to the tutorials/examples available
> >>>>>
> >>>> for
> >>>>
> >>>>> kubernetes.
> >>>>>
> >>>>> (3) Lower cost of initial setup - as I discussed in a previous
> >>>>>
> >>>> mail[6],
> >>>>
> >>>>> kubernetes was far easier to get set up even when I knew the exact
> >>>>>
> >>>> steps.
> >>>>
> >>>>> Mesos took me around 27 steps [7], which involved a lot of config
> >>>>>
> >>>> that was
> >>>>
> >>>>> easy to get wrong (it took me about 5 tries to get all the steps
> >>>>>
> >>>> correct in
> >>>>
> >>>>> one go.) Kubernetes took me around 8 steps and very little config.
> >>>>>
> >>>>> Given that, I'd like to focus my investigation/prototyping on
> >>>>>
> >>>> Kubernetes.
> >>>>
> >>>>> To
> >>>>> be clear, it's fairly close and I think both Mesos and Kubernetes
> >>>>>
> >>>> could
> >>>>
> >>>>> support what we need, so if we run into issues with kubernetes, Mesos
> >>>>>
> >>>> still
> >>>>
> >>>>> seems like a viable option that we could fall back to.
> >>>>>
> >>>>> Thanks,
> >>>>> Stephen
> >>>>>
> >>>>>
> >>>>> [1] Kubernetes StatefulSets
> >>>>>
> >>>>>
> >>>> https://kubernetes.io/docs/concepts/abstractions/controllers
> >>> /statefulsets/
> >>>
> >>>>
> >>>>> [2] mesos unique constraint -
> >>>>> https://mesosphere.github.io/marathon/docs/constraints.html
> >>>>>
> >>>>> [3]
> >>>>> https://mesosphere.github.io/marathon/docs/service-
> >>>>> discovery-load-balancing.html
> >>>>>  and https://mesosphere.com/blog/2015/12/04/dcos-marathon-lb/
> >>>>>
> >>>>> [4]
> >>>>>
> >>>> https://mesosphere.github.io/marathon/docs/persistent-volumes.html
> >>>>
> >>>>>
> >>>>> [5]
> >>>>>
> >>>> https://dcos.io/docs/1.7/usage/tutorials/marathon/stateful-services/
> >>>>
> >>>>>
> >>>>> [6] Container Orchestration software for hosting data stores
> >>>>> https://lists.apache.org/thread.html/5825b35b895839d0b33b6c726c1de0
> >>>>> e76bdb9653d1e913b1207c6c4d@%3Cdev.beam.apache.org%3E
> >>>>>
> >>>>> [7]
> https://github.com/ssisk/beam/blob/support/support/mesos/setup.md
> >>>>>
> >>>>>
> >>>>> On Thu, Dec 29, 2016 at 5:44 PM Davor Bonaci <da...@apache.org>
> >>>>>
> >>>> wrote:
> >>>>
> >>>>>
> >>>>> Just a quick drive-by comment: how tests are laid out has
> >>>>>>
> >>>>> non-trivial
> >>>>
> >>>>> tradeoffs on how/where continuous integration runs, and how results
> >>>>>>
> >>>>> are
> >>>>
> >>>>> integrated into the tooling. The current state is certainly not
> >>>>>>
> >>>>> ideal
> >>>>
> >>>>> (e.g., due to multiple test executions some links in Jenkins point
> >>>>>>
> >>>>> where
> >>>>
> >>>>> they shouldn't), but most other alternatives had even bigger
> >>>>>>
> >>>>> drawbacks at
> >>>>
> >>>>> the time. If someone has great ideas that don't explode the number
> >>>>>>
> >>>>> of
> >>>>
> >>>>> modules, please share ;-)
> >>>>>>
> >>>>>> On Mon, Dec 26, 2016 at 6:30 AM, Etienne Chauchot
> >>>>>>
> >>>>> <echauc...@gmail.com>
> >>>>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Stephen,
> >>>>>>>
> >>>>>>> Thanks for taking the time to comment.
> >>>>>>>
> >>>>>>> My comments are bellow in the email:
> >>>>>>>
> >>>>>>>
> >>>>>>> Le 24/12/2016 à 00:07, Stephen Sisk a écrit :
> >>>>>>>
> >>>>>>> hey Etienne -
> >>>>>>>>
> >>>>>>>> thanks for your thoughts and thanks for sharing your
> >>>>>>>>
> >>>>>>> experiences. I
> >>>>
> >>>>> generally agree with what you're saying. Quick comments below:
> >>>>>>>>
> >>>>>>>> IT are stored alongside with UT in src/test directory of the IO
> >>>>>>>>
> >>>>>>> but
> >>>>
> >>>>> they
> >>>>>
> >>>>>>
> >>>>>>>>> might go to dedicated module, waiting for a consensus
> >>>>>>>> I don't have a strong opinion or feel that I've worked enough
> >>>>>>>>
> >>>>>>> with
> >>>>
> >>>>> maven
> >>>>>
> >>>>>> to
> >>>>>>>> understand all the consequences - I'd love for someone with more
> >>>>>>>>
> >>>>>>> maven
> >>>>
> >>>>> experience to weigh in. If this becomes blocking, I'd say check
> >>>>>>>>
> >>>>>>> it in,
> >>>>
> >>>>> and
> >>>>>>
> >>>>>>> we can refactor later if it proves problematic.
> >>>>>>>>
> >>>>>>>> Sure, not a blocking point, it could be refactored afterwards.
> >>>>>>>
> >>>>>> Just as
> >>>>
> >>>>> a
> >>>>>
> >>>>>> reminder, JB mentioned that storing IT in separate module allows
> >>>>>>>
> >>>>>> to
> >>>>
> >>>>> have
> >>>>>
> >>>>>> more coherence between all IT (same behavior) and to do cross IO
> >>>>>>> integration tests. JB, have you experienced some long term
> >>>>>>>
> >>>>>> drawbacks of
> >>>>
> >>>>> storing IT in a separate module, like, for example, more
> >>>>>>>
> >>>>>> difficult
> >>>>
> >>>>> maintenance due to "distance" with production code?
> >>>>>>>
> >>>>>>>
> >>>>>>>   Also IMHO, it is better that tests load/clean data than doing
> >>>>>>>>
> >>>>>>> some
> >>>>
> >>>>>
> >>>>>>>>> assumptions about the running order of the tests.
> >>>>>>>> I definitely agree that we don't want to make assumptions about
> >>>>>>>>
> >>>>>>> the
> >>>>
> >>>>> running
> >>>>>>>> order of the tests - that way lies pain. :) It will be
> >>>>>>>>
> >>>>>>> interesting to
> >>>>
> >>>>> see
> >>>>>>
> >>>>>>> how the performance tests work out since they will need more
> >>>>>>>>
> >>>>>>> data (and
> >>>>
> >>>>> thus
> >>>>>>>> loading data can take much longer.)
> >>>>>>>>
> >>>>>>>> Yes, performance testing might push in the direction of data
> >>>>>>>
> >>>>>> loading
> >>>>
> >>>>> from
> >>>>>
> >>>>>> outside the tests due to loading time.
> >>>>>>>
> >>>>>>>   This should also be an easier problem
> >>>>>>>> for read tests than for write tests - if we have long running
> >>>>>>>>
> >>>>>>> instances,
> >>>>>
> >>>>>> read tests don't really need cleanup. And if write tests only
> >>>>>>>>
> >>>>>>> write a
> >>>>
> >>>>> small
> >>>>>>>> amount of data, as long as we are sure we're writing to uniquely
> >>>>>>>> identifiable locations (ie, new table per test or something
> >>>>>>>>
> >>>>>>> similar),
> >>>>
> >>>>> we
> >>>>>
> >>>>>> can clean up the write test data on a slower schedule.
> >>>>>>>>
> >>>>>>>> I agree
> >>>>>>>
> >>>>>>>
> >>>>>>>> this will tend to go to the direction of long running data store
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> instances rather than data store instances started (and
> >>>>>>>>
> >>>>>>> optionally
> >>>>
> >>>>> loaded)
> >>>>>>
> >>>>>>> before tests.
> >>>>>>>> It may be easiest to start with a "data stores stay running"
> >>>>>>>> implementation, and then if we see issues with that move towards
> >>>>>>>>
> >>>>>>> tests
> >>>>
> >>>>> that
> >>>>>>>> start/stop the data stores on each run. One thing I'd like to
> >>>>>>>>
> >>>>>>> make
> >>>>
> >>>>> sure
> >>>>>
> >>>>>> is
> >>>>>>
> >>>>>>> that we're not manually tweaking the configurations for data
> >>>>>>>>
> >>>>>>> stores.
> >>>>
> >>>>> One
> >>>>>
> >>>>>> way we could do that is to destroy/recreate the data stores on a
> >>>>>>>>
> >>>>>>> slower
> >>>>>
> >>>>>> schedule - maybe once per week. That way if the script is
> >>>>>>>>
> >>>>>>> changed or
> >>>>
> >>>>> the
> >>>>>
> >>>>>> data store instances are changed, we'd be able to detect it
> >>>>>>>>
> >>>>>>> relatively
> >>>>
> >>>>> soon
> >>>>>>>> while still removing the need for the tests to manage the data
> >>>>>>>>
> >>>>>>> stores.
> >>>>
> >>>>>
> >>>>>>>> I agree. In addition to configuration manual tweaking, there
> >>>>>>>
> >>>>>> might be
> >>>>
> >>>>> cases in which a data store re-partition data during a test or
> >>>>>>>
> >>>>>> after
> >>>>
> >>>>> some
> >>>>>
> >>>>>> tests while the dataset changes. The IO must be tolerant to that
> >>>>>>>
> >>>>>> but
> >>>>
> >>>>> the
> >>>>>
> >>>>>> asserts (number of bundles for example) in test must not fail in
> >>>>>>>
> >>>>>> that
> >>>>
> >>>>> case.
> >>>>>>
> >>>>>>> I would also prefer if possible that the tests do not manage data
> >>>>>>>
> >>>>>> stores
> >>>>>
> >>>>>> (not setup them, not start them, not stop them)
> >>>>>>>
> >>>>>>>
> >>>>>>> as a general note, I suspect many of the folks in the states
> >>>>>>>>
> >>>>>>> will be
> >>>>
> >>>>> on
> >>>>>
> >>>>>> holiday until Jan 2nd/3rd.
> >>>>>>>>
> >>>>>>>> S
> >>>>>>>>
> >>>>>>>> On Fri, Dec 23, 2016 at 7:48 AM Etienne Chauchot
> >>>>>>>>
> >>>>>>> <echauc...@gmail.com
> >>>>
> >>>>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Recently we had a discussion about integration tests of IOs.
> >>>>>>>>>
> >>>>>>>> I'm
> >>>>
> >>>>> preparing a PR for integration tests of the elasticSearch IO
> >>>>>>>>> (
> >>>>>>>>> https://github.com/echauchot/incubator-beam/tree/BEAM-1184-E
> >>>>>>>>> LASTICSEARCH-IO
> >>>>>>>>> as a first shot) which are very important IMHO because they
> >>>>>>>>>
> >>>>>>>> helped
> >>>>
> >>>>> catch
> >>>>>>
> >>>>>>> some bugs that UT could not (volume, data store instance
> >>>>>>>>>
> >>>>>>>> sharing,
> >>>>
> >>>>> real
> >>>>>
> >>>>>> data store instance ...)
> >>>>>>>>>
> >>>>>>>>> I would like to have your thoughts/remarks about points bellow.
> >>>>>>>>>
> >>>>>>>> Some
> >>>>
> >>>>> of
> >>>>>
> >>>>>> these points are also discussed here
> >>>>>>>>>
> >>>>>>>>> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-Np
> >>>>>>>>> rQ7vbf1jNVRgdqeEE8I/edit#heading=h.7ly6e7beup8a
> >>>>>>>>> :
> >>>>>>>>>
> >>>>>>>>> - UT and IT have a similar architecture, but while UT focus on
> >>>>>>>>>
> >>>>>>>> testing
> >>>>>
> >>>>>> the correct behavior of the code including corner cases and use
> >>>>>>>>>
> >>>>>>>> embedded
> >>>>>>
> >>>>>>> in memory data store, IT assume that the behavior is correct
> >>>>>>>>>
> >>>>>>>> (strong
> >>>>
> >>>>> UT)
> >>>>>>
> >>>>>>> and focus on higher volume testing and testing against real
> >>>>>>>>>
> >>>>>>>> data
> >>>>
> >>>>> store
> >>>>>
> >>>>>> instance(s)
> >>>>>>>>>
> >>>>>>>>> - For now, IT are stored alongside with UT in src/test
> >>>>>>>>>
> >>>>>>>> directory of
> >>>>
> >>>>> the
> >>>>>
> >>>>>> IO but they might go to dedicated module, waiting for a
> >>>>>>>>>
> >>>>>>>> consensus.
> >>>>
> >>>>> Maven
> >>>>>>
> >>>>>>> is not configured to run them automatically because data store
> >>>>>>>>>
> >>>>>>>> is not
> >>>>
> >>>>> available on jenkins server yet
> >>>>>>>>>
> >>>>>>>>> - For now, they only use DirectRunner, but they will  be run
> >>>>>>>>>
> >>>>>>>> against
> >>>>
> >>>>> each runner.
> >>>>>>>>>
> >>>>>>>>> - IT do not setup data store instance (like stated in the above
> >>>>>>>>> document) they assume that one is already running (hardcoded
> >>>>>>>>> configuration in test for now, waiting for a common solution to
> >>>>>>>>>
> >>>>>>>> pass
> >>>>
> >>>>> configuration to IT). A docker container script is provided in
> >>>>>>>>>
> >>>>>>>> the
> >>>>
> >>>>> contrib directory as a starting point to whatever orchestration
> >>>>>>>>>
> >>>>>>>> software
> >>>>>>
> >>>>>>> will be chosen.
> >>>>>>>>>
> >>>>>>>>> - IT load and clean test data before and after each test if
> >>>>>>>>>
> >>>>>>>> needed.
> >>>>
> >>>>> It
> >>>>>
> >>>>>> is simpler to do so because some tests need empty data store
> >>>>>>>>>
> >>>>>>>> (write
> >>>>
> >>>>> test) and because, as discussed in the document, tests might
> >>>>>>>>>
> >>>>>>>> not be
> >>>>
> >>>>> the
> >>>>>
> >>>>>> only users of the data store. Also IMHO, it is better that
> >>>>>>>>>
> >>>>>>>> tests
> >>>>
> >>>>> load/clean data than doing some assumptions about the running
> >>>>>>>>>
> >>>>>>>> order
> >>>>
> >>>>> of
> >>>>>
> >>>>>> the tests.
> >>>>>>>>>
> >>>>>>>>> If we generalize this pattern to all IT tests, this will tend
> >>>>>>>>>
> >>>>>>>> to go
> >>>>
> >>>>> to
> >>>>>
> >>>>>> the direction of long running data store instances rather than
> >>>>>>>>>
> >>>>>>>> data
> >>>>
> >>>>> store instances started (and optionally loaded) before tests.
> >>>>>>>>>
> >>>>>>>>> Besides if we where to change our minds and load data from
> >>>>>>>>>
> >>>>>>>> outside
> >>>>
> >>>>> the
> >>>>>
> >>>>>> tests, a logstash script is provided.
> >>>>>>>>>
> >>>>>>>>> If you have any thoughts or remarks I'm all ears :)
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>> Etienne
> >>>>>>>>>
> >>>>>>>>> Le 14/12/2016 à 17:07, Jean-Baptiste Onofré a écrit :
> >>>>>>>>>
> >>>>>>>>> Hi Stephen,
> >>>>>>>>>>
> >>>>>>>>>> the purpose of having in a specific module is to share
> >>>>>>>>>>
> >>>>>>>>> resources and
> >>>>
> >>>>> apply the same behavior from IT perspective and be able to
> >>>>>>>>>>
> >>>>>>>>> have IT
> >>>>
> >>>>> "cross" IO (for instance, reading from JMS and sending to
> >>>>>>>>>>
> >>>>>>>>> Kafka, I
> >>>>
> >>>>> think that's the key idea for integration tests).
> >>>>>>>>>>
> >>>>>>>>>> For instance, in Karaf, we have:
> >>>>>>>>>> - utest in each module
> >>>>>>>>>> - itest module containing itests for all modules all together
> >>>>>>>>>>
> >>>>>>>>>> Regards
> >>>>>>>>>> JB
> >>>>>>>>>>
> >>>>>>>>>> On 12/14/2016 04:59 PM, Stephen Sisk wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Etienne,
> >>>>>>>>>>>
> >>>>>>>>>>> thanks for following up and answering my questions.
> >>>>>>>>>>>
> >>>>>>>>>>> re: where to store integration tests - having them all in a
> >>>>>>>>>>>
> >>>>>>>>>> separate
> >>>>>
> >>>>>> module
> >>>>>>>>>>> is an interesting idea. I couldn't find JB's comments about
> >>>>>>>>>>>
> >>>>>>>>>> moving
> >>>>
> >>>>> them
> >>>>>>
> >>>>>>> into a separate module in the PR - can you share the reasons
> >>>>>>>>>>>
> >>>>>>>>>> for
> >>>>
> >>>>> doing so?
> >>>>>>>>>>> The IO integration/perf tests so it does seem like they'll
> >>>>>>>>>>>
> >>>>>>>>>> need to
> >>>>
> >>>>> be
> >>>>>
> >>>>>> treated in a special manner, but given that there is already
> >>>>>>>>>>>
> >>>>>>>>>> an IO
> >>>>
> >>>>> specific
> >>>>>>>>>>> module, it may just be that we need to treat all the ITs in
> >>>>>>>>>>>
> >>>>>>>>>> the IO
> >>>>
> >>>>> module
> >>>>>>>>>>> the same way. I don't have strong opinions either way right
> >>>>>>>>>>>
> >>>>>>>>>> now.
> >>>>
> >>>>>
> >>>>>>>>>>> S
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Dec 14, 2016 at 2:39 AM Etienne Chauchot <
> >>>>>>>>>>>
> >>>>>>>>>> echauc...@gmail.com>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi guys,
> >>>>>>>>>>>
> >>>>>>>>>>> @Stephen: I addressed all your comments directly in the PR,
> >>>>>>>>>>>
> >>>>>>>>>> thanks!
> >>>>
> >>>>> I just wanted to comment here about the docker image I used:
> >>>>>>>>>>>
> >>>>>>>>>> the
> >>>>
> >>>>> only
> >>>>>
> >>>>>> official Elastic image contains only ElasticSearch. But for
> >>>>>>>>>>>
> >>>>>>>>>> testing I
> >>>>>
> >>>>>> needed logstash (for ingestion) and kibana (not for
> >>>>>>>>>>>
> >>>>>>>>>> integration
> >>>>
> >>>>> tests,
> >>>>>>
> >>>>>>> but to easily test REST requests to ES using sense). This is
> >>>>>>>>>>>
> >>>>>>>>>> why I
> >>>>
> >>>>> use
> >>>>>>
> >>>>>>> an ELK (Elasticsearch+Logstash+Kibana) image. This one
> >>>>>>>>>>>
> >>>>>>>>>> isreleased
> >>>>
> >>>>> under
> >>>>>>>>>>> theapache 2 license.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Besides, there is also a point about where to store
> >>>>>>>>>>>
> >>>>>>>>>> integration
> >>>>
> >>>>> tests:
> >>>>>>
> >>>>>>> JB proposed in the PR to store integration tests to dedicated
> >>>>>>>>>>>
> >>>>>>>>>> module
> >>>>>
> >>>>>> rather than directly in the IO module (like I did).
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Etienne
> >>>>>>>>>>>
> >>>>>>>>>>> Le 01/12/2016 à 20:14, Stephen Sisk a écrit :
> >>>>>>>>>>>
> >>>>>>>>>>> hey!
> >>>>>>>>>>>>
> >>>>>>>>>>>> thanks for sending this. I'm very excited to see this
> >>>>>>>>>>>>
> >>>>>>>>>>> change. I
> >>>>
> >>>>> added some
> >>>>>>>>>>>> detail-oriented code review comments in addition to what
> >>>>>>>>>>>>
> >>>>>>>>>>> I've
> >>>>
> >>>>> discussed
> >>>>>>>>>>>> here.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The general goal is to allow for re-usable instantiation of
> >>>>>>>>>>>>
> >>>>>>>>>>> particular
> >>>>>>
> >>>>>>>
> >>>>>>>>>>>> data
> >>>>>>>>>>>
> >>>>>>>>>>> store instances and this seems like a good start. Looks like
> >>>>>>>>>>>>
> >>>>>>>>>>> you
> >>>>
> >>>>> also have
> >>>>>>>>>>>> a script to generate test data for your tests - that's
> >>>>>>>>>>>>
> >>>>>>>>>>> great.
> >>>>
> >>>>>
> >>>>>>>>>>>> The next steps (definitely not blocking your work) will be
> >>>>>>>>>>>>
> >>>>>>>>>>> to have
> >>>>
> >>>>> ways to
> >>>>>>>>>>>> create instances from the docker images you have here, and
> >>>>>>>>>>>>
> >>>>>>>>>>> use
> >>>>
> >>>>> them
> >>>>>
> >>>>>> in the
> >>>>>>>>>>>> tests. We'll need support in the test framework for that
> >>>>>>>>>>>>
> >>>>>>>>>>> since
> >>>>
> >>>>> it'll
> >>>>>
> >>>>>> be
> >>>>>>>>>>>> different on developer machines and in the beam jenkins
> >>>>>>>>>>>>
> >>>>>>>>>>> cluster,
> >>>>
> >>>>> but
> >>>>>
> >>>>>> your
> >>>>>>>>>>>> scripts here allow someone running these tests locally to
> >>>>>>>>>>>>
> >>>>>>>>>>> not have
> >>>>
> >>>>> to
> >>>>>>
> >>>>>>>
> >>>>>>>>>>>> worry
> >>>>>>>>>>>
> >>>>>>>>>>> about getting the instance set up and can manually adjust,
> >>>>>>>>>>>>
> >>>>>>>>>>> so this
> >>>>
> >>>>> is
> >>>>>>
> >>>>>>> a
> >>>>>>>>>>>> good incremental step.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have some thoughts now that I'm reviewing your scripts
> >>>>>>>>>>>>
> >>>>>>>>>>> (that I
> >>>>
> >>>>> didn't
> >>>>>>>>>>>> have previously, so we are learning this together):
> >>>>>>>>>>>> * It may be useful to try and document why we chose a
> >>>>>>>>>>>>
> >>>>>>>>>>> particular
> >>>>
> >>>>> docker
> >>>>>>>>>>>> image as the base (ie, "this is the official supported
> >>>>>>>>>>>>
> >>>>>>>>>>> elastic
> >>>>
> >>>>> search
> >>>>>>
> >>>>>>> docker image" or "this image has several data stores
> >>>>>>>>>>>>
> >>>>>>>>>>> together that
> >>>>
> >>>>> can be
> >>>>>>>>>>>> used for a couple different tests")  - I'm curious as to
> >>>>>>>>>>>>
> >>>>>>>>>>> whether
> >>>>
> >>>>> the
> >>>>>
> >>>>>> community thinks that is important
> >>>>>>>>>>>>
> >>>>>>>>>>>> One thing that I called out in the comment that's worth
> >>>>>>>>>>>>
> >>>>>>>>>>> mentioning
> >>>>
> >>>>> on the
> >>>>>>>>>>>> larger list - if you want to specify which specific runners
> >>>>>>>>>>>>
> >>>>>>>>>>> a test
> >>>>
> >>>>> uses,
> >>>>>>>>>>>> that can be controlled in the pom for the module. I updated
> >>>>>>>>>>>>
> >>>>>>>>>>> the
> >>>>
> >>>>> testing
> >>>>>>>>>>>>
> >>>>>>>>>>>> doc
> >>>>>>>>>>>
> >>>>>>>>>>> mentioned previously in this thread with a TODO to talk
> >>>>>>>>>>>>
> >>>>>>>>>>> about this
> >>>>
> >>>>> more. I
> >>>>>>>>>>>> think we should also make it so that IO modules have that
> >>>>>>>>>>>> automatically,
> >>>>>>>>>>>>
> >>>>>>>>>>>> so
> >>>>>>>>>>>
> >>>>>>>>>>> developers don't have to worry about it.
> >>>>>>>>>>>>
> >>>>>>>>>>>> S
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Dec 1, 2016 at 9:00 AM Etienne Chauchot <
> >>>>>>>>>>>>
> >>>>>>>>>>> echauc...@gmail.com>
> >>>>>>
> >>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Stephen,
> >>>>>>>>>>>>
> >>>>>>>>>>>> As discussed, I added injection script, docker containers
> >>>>>>>>>>>>
> >>>>>>>>>>> scripts
> >>>>
> >>>>> and
> >>>>>>
> >>>>>>> integration tests to the sdks/java/io/elasticsearch/contrib
> >>>>>>>>>>>> <
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://github.com/apache/incubator-beam/pull/1439/files/1e7
> >>>>>>>>>>>>
> >>>>>>>>>>> e2f0a6e1a1777d31ae2c886c920efccd708b5#diff-e243536428d06ade7
> >>>>>>>>> d824cefcb3ed0b9
> >>>>>>>>>
> >>>>>>>>> directory in that PR:
> >>>>>>>>>>
> >>>>>>>>>>> https://github.com/apache/incubator-beam/pull/1439.
> >>>>>>>>>>>>
> >>>>>>>>>>>> These work well but they are first shot. Do you have any
> >>>>>>>>>>>>
> >>>>>>>>>>> comments
> >>>>
> >>>>> about
> >>>>>>>>>>>> those?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Besides I am not very sure that these files should be in the
> >>>>>>>>>>>>
> >>>>>>>>>>> IO
> >>>>
> >>>>> itself
> >>>>>>
> >>>>>>> (even in contrib directory, out of maven source
> >>>>>>>>>>>>
> >>>>>>>>>>> directories). Any
> >>>>
> >>>>>
> >>>>>>>>>>>> thoughts?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Etienne
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Le 23/11/2016 à 19:03, Stephen Sisk a écrit :
> >>>>>>>>>>>>
> >>>>>>>>>>>> It's great to hear more experiences.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm also glad to hear that people see real value in the
> >>>>>>>>>>>>>
> >>>>>>>>>>>> high
> >>>>
> >>>>> volume/performance benchmark tests. I tried to capture that
> >>>>>>>>>>>>>
> >>>>>>>>>>>> in
> >>>>
> >>>>> the
> >>>>>
> >>>>>>
> >>>>>>>>>>>>> Testing
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> doc I shared, under "Reasons for Beam Test Strategy". [1]
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It does generally sound like we're in agreement here. Areas
> >>>>>>>>>>>>>
> >>>>>>>>>>>> of
> >>>>
> >>>>> discussion
> >>>>>>>>>>>>
> >>>>>>>>>>>>
>
>

Reply via email to