Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Ismaël Mejía
​Hello again, Stephen, I agree with you the real question is what is the scope of the tests, maybe the discussion so far has been more about testing a ‘real’ data store and finding infra/performance issues (and future regressions), but having a modern cluster manager opens the door to create more

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Jean-Baptiste Onofré
Yes, for both DCOS (Mesos+Marathon) and Kubernetes, I think we may find single node config but not sure for multi-node setup. Anyway, I'm not sure if we find a multi-node configuration, it would cover our needs. Regards JB On 01/18/2017 12:52 PM, Stephen Sisk wrote: ah! I looked around a bit

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Stephen Sisk
ah! I looked around a bit more and found the dcos package repo - https://github.com/mesosphere/universe/tree/version-3.x/repo/packages poking around a bit, I can find a lot of packages for single node instances, but not many packages for multi-node instances. Single node instance packages are kind

Precommit Build Fixed

2017-01-18 Thread Jason Kuster
Hi all, I reenabled Coveralls last night on precommit and postcommit, but screwed up the invocation command on the precommit, which broke a bunch of builds today, my apologies. The change should be fixed (waiting on https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/6620/ to be sure),

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Stephen Sisk
Hi Ishmael, these are good questions, thanks for raising them. Ability to modify network/compute resources to simulate failures = I see two real questions here: 1. Is this something we want to do? 2. Is it possible with both/either? So far, the tes

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Jean-Baptiste Onofré
⁣​Hi Ismael Stephen will reply with details but I know he did a comparison and evaluate different options. He tested with the jdbc Io itests. Regards JB On Jan 18, 2017, 08:26, at 08:26, "Ismaël Mejía" wrote: >Thanks for your analysis Stephen, good arguments / references. > >One quick questio

Re: Hosting data stores for IO Transform testing

2017-01-18 Thread Ismaël Mejía
Thanks for your analysis Stephen, good arguments / references. One quick question. Have you checked the APIs of both (Mesos/Kubernetes) to see if we can do programmatically do more complex tests (I suppose so, but you don't mention how easy or if those are possible), for example to simulate a slow

Re: IO Integration tests - concrete proposal

2017-01-18 Thread Etienne Chauchot
Hi, Yes, thanks all for these clarifications about testing architecture. I agree that point 1 and 2 should be shared between tests as much as possible. Especially sharing data loading between tests is more time-effective and resource-effective: tests that need data (testRead, testSplit, ...)

Re: IO Integration tests - concrete proposal

2017-01-18 Thread Jean-Baptiste Onofré
In addition of Stephen's e-mail, I would like to add some points for IO contributors about unit tests in addition of integration tests (it's likely indirectly related ;)). Depending of the backend, it could be difficult to write unit tests using a running service for some IO. For instance, I s

Re: IO Integration tests - concrete proposal

2017-01-18 Thread Jean-Baptiste Onofré
Hi guys, Firs, great e-mail Stephen: complete and detailed proposal. Lukasz raised a good point: it makes sense to be able to leverage the same "bootstrap" script. We discussed about providing the following in each IO: 1. code to load data (java, script, whatever) 2. script to bootstrap the b

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

2017-01-18 Thread Etienne Chauchot
Thanks guys for your comments! Etienne Le 17/01/2017 à 18:41, Ben Chambers a écrit : We should start by understanding the goals. If elements are in different windows can they be out in the same batch? If they have different timestamps what timestamp should the batch have? As a composite trans

Re: [DISCUSS] Python SDK status and next steps

2017-01-18 Thread Ahmet Altay
Thank you all for the comments so far. I would follow the process as suggested by Davor and others in this thread. Ahmet On Tue, Jan 17, 2017 at 11:47 PM, Sergio Fernández wrote: > Hi > > On Tue, Jan 17, 2017 at 5:22 PM, Ahmet Altay > wrote: > > > > tl;dr: I would like to start a discussion ab