Re: Hosting data stores for IO Transform testing

2017-01-20 Thread Stephen Sisk
hey folks! I wanted to gather any last thoughts that people might have. I'd like to get started setting this up - anyone else have input? S On Thu, Jan 19, 2017 at 11:41 AM Stephen Sisk wrote: > Glad to hear you support kubernetes (although to be clear, I'm rooting for > the right solution for

Re: Hosting data stores for IO Transform testing

2017-01-19 Thread Stephen Sisk
Glad to hear you support kubernetes (although to be clear, I'm rooting for the right solution for us in the long run - if anyone has a strong reason for dcos, I'm excited to hear it.) I agree with you that testing IO in failure scenarios seems like a fruitful area for future work, but that I don't

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Ismaël Mejía
​Hello again, Stephen, I agree with you the real question is what is the scope of the tests, maybe the discussion so far has been more about testing a ‘real’ data store and finding infra/performance issues (and future regressions), but having a modern cluster manager opens the door to create more

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Jean-Baptiste Onofré
Yes, for both DCOS (Mesos+Marathon) and Kubernetes, I think we may find single node config but not sure for multi-node setup. Anyway, I'm not sure if we find a multi-node configuration, it would cover our needs. Regards JB On 01/18/2017 12:52 PM, Stephen Sisk wrote: ah! I looked around a bit

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Stephen Sisk
ah! I looked around a bit more and found the dcos package repo - https://github.com/mesosphere/universe/tree/version-3.x/repo/packages poking around a bit, I can find a lot of packages for single node instances, but not many packages for multi-node instances. Single node instance packages are kind

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Stephen Sisk
Hi Ishmael, these are good questions, thanks for raising them. Ability to modify network/compute resources to simulate failures = I see two real questions here: 1. Is this something we want to do? 2. Is it possible with both/either? So far, the tes

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Jean-Baptiste Onofré
⁣​Hi Ismael Stephen will reply with details but I know he did a comparison and evaluate different options. He tested with the jdbc Io itests. Regards JB On Jan 18, 2017, 08:26, at 08:26, "Ismaël Mejía" wrote: >Thanks for your analysis Stephen, good arguments / references. > >One quick questio

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Ismaël Mejía
Thanks for your analysis Stephen, good arguments / references. One quick question. Have you checked the APIs of both (Mesos/Kubernetes) to see if we can do programmatically do more complex tests (I suppose so, but you don't mention how easy or if those are possible), for example to simulate a slow

Re: Hosting data stores for IO Transform testing

2017-01-17 Thread Stephen Sisk
hi! I've been continuing this investigation, and have some more info to report, and hopefully we can start making some decisions. To support performance testing, I've been investigating mesos+marathon and kubernetes for running data stores in their high availability mode. I have been examining fe

Re: Hosting data stores for IO Transform testing

2016-12-29 Thread Davor Bonaci
Just a quick drive-by comment: how tests are laid out has non-trivial tradeoffs on how/where continuous integration runs, and how results are integrated into the tooling. The current state is certainly not ideal (e.g., due to multiple test executions some links in Jenkins point where they shouldn't

Re: Hosting data stores for IO Transform testing

2016-12-26 Thread Etienne Chauchot
Hi Stephen, Thanks for taking the time to comment. My comments are bellow in the email: Le 24/12/2016 à 00:07, Stephen Sisk a écrit : hey Etienne - thanks for your thoughts and thanks for sharing your experiences. I generally agree with what you're saying. Quick comments below: IT are stor

Re: Hosting data stores for IO Transform testing

2016-12-26 Thread Jean-Baptiste Onofré
We have two levels: - the "itest" for runners: basically, we execute samples on the different runners - the IO itest: same as the previous tests but we run on different IOs (Kafka, JMS/ActiveMQ, ...). In that case, we need to test again on different runners, but using different pipelines using

Re: Hosting data stores for IO Transform testing

2016-12-26 Thread Etienne Chauchot
JB, Thanks for the link! My comments are bellow in the email Le 24/12/2016 à 08:14, Jean-Baptiste Onofré a écrit : Hi Etienne, Thanks for sharing ! For the itest module, I'm in favor of a dedicated module too. It's what I did in Karaf: https://github.com/apache/karaf/tree/master/itests I

Re: Hosting data stores for IO Transform testing

2016-12-23 Thread Jean-Baptiste Onofré
Hi Etienne, Thanks for sharing ! For the itest module, I'm in favor of a dedicated module too. It's what I did in Karaf: https://github.com/apache/karaf/tree/master/itests Itests contains all the integration tests for Karaf, covering all modules/features. What we should add: - multi runner

Re: Hosting data stores for IO Transform testing

2016-12-23 Thread Stephen Sisk
hey Etienne - thanks for your thoughts and thanks for sharing your experiences. I generally agree with what you're saying. Quick comments below: > IT are stored alongside with UT in src/test directory of the IO but they might go to dedicated module, waiting for a consensus I don't have a strong o

Re: Hosting data stores for IO Transform testing

2016-12-23 Thread Etienne Chauchot
Hi, Recently we had a discussion about integration tests of IOs. I'm preparing a PR for integration tests of the elasticSearch IO (https://github.com/echauchot/incubator-beam/tree/BEAM-1184-ELASTICSEARCH-IO as a first shot) which are very important IMHO because they helped catch some bugs tha